Image analysis apparatus

Abstract

To accurately analyze the state of an analysis object reflected in a two-dimensional image.SOLUTION: An acquisition unit 141 acquires a two-dimensional image PC in which a real space SR and an analysis object H are imaged by an image apparatus C. An object model generation unit 142 generates a three-dimensional object model M imitating the analysis object H. A depth estimation unit 143 estimates the depth of the analysis object H relative to the image apparatus C. An arrangement unit 144 arranges a three-dimensional object model M on a three-dimensional space model SV simulating the real space SR based on the depth estimated by the depth estimation unit 143 and the position of the analysis object H on the two-dimensional image PC. A correction unit 145 corrects the orientation of the three-dimensional object model M on the three-dimensional space model SV based on the positional relationship between the optical axis of the image apparatus C and the analysis object H.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image analysis device. [Background technology]

[0002] Conventionally, a technology is known that analyzes two-dimensional images captured by a surveillance camera or the like and estimates the behavior of an analysis target. For example, a specific behavior detection device described in Patent Document 1 below judges whether a patient has gotten out of bed, including the presence or absence of a behavior of getting up from the bed and sitting on the bed. The specific behavior detection device includes a camera, a bed detection unit, a person posture detection unit, a relative position calculation unit, a relative position information storage unit, and a behavior estimation unit. The camera captures an image of the entire monitoring area. The bed detection unit extracts the bed from the captured image of the camera and calculates the area of ​​the bed from the extracted information. The person posture detection unit obtains coordinate information of a specific part of the patient on the bed from the captured image of the camera, and detects the posture of the patient from the information. The relative position calculation unit calculates the relative position between the bed and the patient based on the information detected by the bed detection unit and the person posture detection unit. The relative position information storage unit stores the time-series relative position information calculated by the relative position calculation unit. The behavior estimation unit estimates the behavior of the patient on the bed based on multiple pieces of time-series information stored in the relative position information storage unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-108129 A Summary of the Invention [Problem to be solved by the invention]

[0004] A two-dimensional image captured by an imaging device is a projection of a point in three-dimensional space onto a two-dimensional plane. When a general lens is used, this projection is described by perspective projection. In perspective projection, even if the object to be analyzed is in the same state (for example, facing the same direction), the way it appears in the two-dimensional image changes depending on its positional relationship with the imaging device. For this reason, there are cases where the state of the object to be analyzed cannot be analyzed accurately.

[0005] An object of the present invention is to accurately analyze the state of an object to be analyzed that is captured in a two-dimensional image. [Means for solving the problem]

[0006] An image analysis device according to one embodiment of the present invention includes an acquisition unit that acquires a two-dimensional image of a real space and an analysis object located in the real space captured by an imaging device; an object model generation unit that generates a three-dimensional object model that imitates the analysis object based on the two-dimensional image; a depth estimation unit that estimates a depth of the analysis object relative to the imaging device based on the two-dimensional image; a placement unit that places the three-dimensional object model on a three-dimensional space model that imitates the real space based on the depth estimated by the depth estimation unit and the position of the analysis object on the two-dimensional image; and a correction unit that corrects the orientation of the three-dimensional object model on the three-dimensional space model based on the positional relationship between the optical axis of the imaging device and the analysis object. Effect of the Invention

[0007] According to one aspect of the present invention, the state of an analysis target captured in a two-dimensional image can be analyzed with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an image analyzing device 10 according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a two-dimensional image PC. [Diagram 3] FIG. 2 is a schematic diagram showing an example of a three-dimensional space model SV. [Figure 4A]13 is a diagram illustrating a schematic diagram of a result of motion estimation by a motion estimation unit 146. FIG. [Figure 4B] 13 is a diagram illustrating a schematic diagram of a result of motion estimation by a motion estimation unit 146. FIG. [Diagram 5] 1 is a diagram showing a schematic diagram of a position of an analysis object H in a real space SR. [Figure 6A] 1 is a diagram showing a schematic diagram of the positional relationship between an imaging device C and an analysis object H. FIG. [Figure 6B] 1 is a diagram showing a schematic diagram of the positional relationship between an imaging device C and an analysis object H. FIG. [Figure 7] FIG. 13 is a diagram showing a two-dimensional image PC2. [Figure 8A] FIG. 13 is a diagram showing a schematic view of the direction of the line of sight of a person H5. [Figure 8B] FIG. 13 is a diagram showing a schematic view of the direction of the line of sight of a person H5. [Figure 9A] FIG. 13 is a diagram showing a two-dimensional image PC3. [Figure 9B] FIG. 13 is a diagram showing a two-dimensional image PC3. [Figure 10A] FIG. 11 is a diagram illustrating a method for calculating a yaw correction angle. [Figure 10B] FIG. 11 is a diagram illustrating a method for calculating a pitch correction angle. [Figure 11] 4 is a flowchart showing the operation of the image analyzing device 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment 1 is a block diagram showing the configuration of an image analyzing device 10 according to a first embodiment. Based on a two-dimensional image PC captured in a real space SR, the image analyzing device 10 estimates the state of an analysis object H (e.g., people H1 to H6) appearing in the two-dimensional image PC. The state of the analysis object H is, for example, the posture of the analysis object H or the movement of the analysis object H.

[0010] Specifically, the image analyzing device 10 is an information processing terminal such as a personal computer, a tablet, or a smartphone. The image analyzing device 10 includes, for example, an input device 100, a display device 110, a communication device 120, a storage device 130, a processing device 140, and a bus 150. The input device 100, the display device 110, the communication device 120, and the storage device 130 are each connected to the processing device 140 by the bus 150 that mediates data transmission and reception. The bus 150 may be configured using a single bus, or may be configured using different buses between each element.

[0011] The input device 100 is an input device (e.g., a keyboard, a mouse, a switch, a button, or a sensor) that accepts input from the outside. The display device 110 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The input device 100 and the display device 110 may be integrated, for example, as a touch panel.

[0012] The communication device 120 includes a communication interface connectable to a network (not shown) and communicates with other devices via the network. The other devices are, for example, an imaging device C (see FIG. 5, etc.) that captures a two-dimensional image PC (see FIG. 2, etc.). The imaging device C is, for example, a camera. The communication interface of the communication device 120 may be a wireless communication interface or a wired communication interface.

[0013] The storage device 130 is a recording medium readable by the processing device 140. The storage device 130 may be configured with at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), and the like. The storage device 130 stores a program PG. The program PG is a program for operating the image analyzing device 10.

[0014] The processing device 140 includes one or more central processing units (CPUs). The processing device 140 reads the program PG from the storage device 130 when an instruction to execute the program PG is received through an input operation on the input device 100. The processing device 140 executes the read program PG. The processing device 140 executes the program PG to function as an acquisition unit 141, an object model generation unit 142, a depth estimation unit 143, an arrangement unit 144, a correction unit 145, and a motion estimation unit 146. At least some of the functions of the acquisition unit 141, the object model generation unit 142, the depth estimation unit 143, the arrangement unit 144, the correction unit 145, and the motion estimation unit 146 may be configured by circuits such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA).

[0015] The acquisition unit 141 acquires a two-dimensional image PC (PC1, PC2, etc.) in which the real space SR and an analysis target H located in the real space SR are captured by an imaging device C. The real space SR may be, for example, an indoor space such as a store, a station, an office, or a hospital, or an outdoor space such as a road, a theme park, or an event venue. The analysis target H is an entity that performs an action, such as a person, an animal, a vehicle, or a robot. The analysis target H may be, for example, an object B such as a thrown ball. In this embodiment, the real space SR is an indoor space, and the analysis target H is people H1 to H6 (see FIG. 2, etc.).

[0016] The acquiring unit 141 acquires the two-dimensional image PC from the imaging device C, for example, by using the communication device 120. When the imaging device C is located within a range where it can be connected to the image analyzing device 10 by wire, the acquiring unit 141 may acquire the two-dimensional image PC from the imaging device C via a connection interface (not shown) provided in the image analyzing device 10. The connection interface is an interface for connecting the image analyzing device 10 to another device. The connection interface may be, for example, a connector to which a USB (Universal Serial Bus) cable or the like is connected, or a communication module that uses short-range wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

[0017] The acquiring unit 141 may acquire different two-dimensional images PC from a plurality of imaging devices C provided at different locations in the real space SR. In addition, the acquiring unit 141 is not limited to acquiring the two-dimensional image PC directly from the imaging device C, and may acquire the two-dimensional image PC from another information processing device that has acquired the two-dimensional image PC from the imaging device C.

[0018] In this embodiment, the two-dimensional image PC is a still image constituting a moving image. The imaging device C captures the moving image by generating a still image (two-dimensional image PC) at a predetermined frame rate. The imaging device C transmits each still image (two-dimensional image PC) to the image analysis device 10 every time it generates it. By capturing the moving image with the imaging device C, the state of the analysis object H is continuously grasped, and the accuracy of estimating the movement is improved.

[0019] Fig. 2 is a schematic diagram showing an example of a two-dimensional image PC. In the two-dimensional image PC1 shown in Fig. 2, an indoor space is shown as a real space SR. In the real space SR, objects B (B1 to B6) exist. The objects B include, for example, a wall B1, a floor B2, a bench B3, a desk B4, a round stool B5, and a monitor B6. In addition, the two-dimensional image PC shows people H1 and H2 who are the analysis target H.

[0020] For the two-dimensional image PC1, the Xp axis is taken in the horizontal direction and the Yp axis is taken in the vertical direction. Each point (for example, each pixel) of the two-dimensional image PC1 can be specified by two-dimensional coordinate information (Xp, Yp) using the Xp axis and the Yp axis.

[0021] In addition, which of the objects B (in this embodiment, the object B includes inanimate objects such as vehicles, as well as living objects such as humans or animals) in the two-dimensional image PC1 is to be the analysis target H may be designated in advance before the two-dimensional image PC1 is acquired, or may be designated after the two-dimensional image PC1 is acquired. When the analysis target H is designated in advance, the analysis target H in the two-dimensional image PC1 is detected using, for example, a machine learning model using a neural network. For example, when it is desired to monitor the movement of people in a store, people in the store are detected using a first learning model that detects people (more specifically, an area in which a person appears) from the two-dimensional image PC1. When the analysis target H is not designated in advance, for example, an operator visually checks the two-dimensional image PC1 displayed on the display device 110, and specifies the part in which the object B to be the analysis target H appears using the input device 100. For example, when only a specific person is to be the analysis target H from the two-dimensional image PC1 in which many people appear, when the operator specifies the part in which the person to be the analysis target H appears, the area in which the person appears is detected by edge detection.

[0022] The object model generating unit 142 generates a three-dimensional object model M that imitates the analysis object H based on the two-dimensional image PC. The object model generating unit 142 generates the three-dimensional object model M using, for example, a machine learning model (second learning model) using a neural network. The three-dimensional object model M includes, for example, information on the positions and orientations of parts of the analysis object H. For example, when the analysis object H is people H1 and H2, the parts of the analysis object H may be some parts that constitute the analysis object H, such as the faces or hands of the people H1 and H2. The three-dimensional object model M may imitate, for example, the shape of the analysis object H. For example, when the analysis object H is people H1 and H2, the shape of the analysis object H is the posture of the people H1 and H2.

[0023] In this embodiment, the three-dimensional object model M is a skeletal model showing the skeletons of the people H1 and H2. The object model generating unit 142 detects feature points such as the joints and face parts of the person H1 from a partial image of an area in which the person H1 appears in the two-dimensional image PC1, for example, and generates a skeletal model by connecting the feature points. Similarly, for the person H2, a skeletal model is generated from a partial image of an area in which the person H2 appears. By using the three-dimensional object model M as a skeletal model, it is possible to prevent the individual people H1 and H2 from being identified and to protect privacy. In addition, by using the three-dimensional object model M as a skeletal model, information that is not directly related to the movements of the people H1 and H2, such as the clothes of the people H1 and H2, is removed, so that the processing load of the processing device 140 is reduced. Note that the three-dimensional object model M is not limited to a skeletal model, and may be, for example, an avatar imitating the people H1 and H2.

[0024] The depth estimation unit 143 estimates the depth of the analysis object H relative to the imaging device C that captures the two-dimensional image PC based on the two-dimensional image PC. The depth of the analysis object H relative to the imaging device C is, for example, the distance between a representative point of the imaging device C (for example, the center point O of the lens of the imaging device C shown in FIG. 5) and each point (or representative point) on the surface of the analysis object H. The depth estimation unit 143 estimates the depth of each pixel of the two-dimensional image PC using, for example, a machine learning model (third learning model) using a neural network. The depth of the pixel of the area of ​​the two-dimensional image PC in which the analysis object H appears is the depth of the analysis object H. In addition, the coordinates of each pixel of the two-dimensional image PC are associated with the depth of each pixel, thereby generating point cloud data of the real space SR including the analysis object H.

[0025] The placement unit 144 places the three-dimensional object model M in the three-dimensional space model SV simulating the real space SR, based on the depth estimated by the depth estimation unit 143 and the position of the analysis object H on the two-dimensional image PC. In the first embodiment, the placement unit 144 estimates three-dimensional coordinates (hereinafter referred to as "real coordinates") indicating the position of the analysis object H in the real space SR, based on the depth estimated by the depth estimation unit 143, the position of the analysis object H on the two-dimensional image PC, and the position of the imaging device C in the real space SR. Then, the placement unit 144 places the three-dimensional object model M at a position on the three-dimensional space model SV that corresponds to the real coordinates.

[0026] Fig. 5 is a diagram showing a schematic diagram of the position of an analysis target H (a person H1 as an example) in the real space SR. Any point in the real space SR can be specified by three-dimensional coordinate information (Xr, Yr, Zr) using the orthogonal Xr, Yr, and Zr axes. In Fig. 5, the center point O of the lens of the imaging device C is set as the origin, the optical axis direction of the imaging device C is set as the Zr axis, the upward direction on the plane perpendicular to the Zr axis is set as the Yr axis, and the rightward direction is set as the Xr axis.

[0027] Generally, since the analysis target H has a three-dimensional spread, the area that the analysis target H occupies in the real space SR is indicated by a plurality of pieces of coordinate information. The position of the analysis target H may be a plurality of pieces of coordinate information indicating the area that the analysis target H occupies in the real space SR, or may be coordinate information indicating the position of the representative point of the analysis target H in the real space SR. In the following, the center point D of the head of the person H1 is assumed to be the representative point of the person H1. That is, the position (real coordinates) of the person H1, who is the analysis target H, is assumed to be indicated by three-dimensional coordinates indicating the center point D. The three-dimensional coordinates indicating the center point D are assumed to be (Xrh, Yrh, Zrh). Note that the representative point of the person H1 is not limited to the center point D of the head of the person H1, and may be any other part as long as it is a feature point of the human body (for example, a waist or a leg) that can be detected from the two-dimensional image PC.

[0028] The mounting position of the imaging device C in the real space SR is known. For example, in FIG. 5, the imaging device C is mounted at a specific location on the ceiling of the room in which the person H1 is present. Furthermore, if the imaging direction and imaging magnification of the imaging device C are changeable (i.e., pan / tilt / zoom are possible), the acquisition unit 141 also acquires the imaging direction and imaging magnification of the imaging device C when acquiring the two-dimensional image PC. Furthermore, it is assumed that the camera parameters of the imaging device C are also known. Thus, the arrangement unit 144 can identify the range PA of the real space SR captured in the two-dimensional image PC.

[0029] Therefore, the position of the analysis object H on the two-dimensional image PC is projected onto the position in the real space SR, and the position in the depth direction relative to the imaging device C is identified based on the depth estimated by the depth estimation unit 143, thereby estimating three-dimensional coordinates indicating the position of the analysis object H in the real space SR. In the example of Fig. 5, three-dimensional coordinates (Xrh, Yrh, Zrh) indicating the position (center point D) of the person H1 are estimated.

[0030] The placement unit 144 places the three-dimensional object model M at a position on the three-dimensional space model SV that corresponds to the three-dimensional coordinates (Xrh, Yrh, Zrh). The three-dimensional space model SV is generated so as to reproduce the real space SR (indoor space in this embodiment) in which the two-dimensional image PC is captured. The three-dimensional space model SV may be generated automatically, for example, by three-dimensionally scanning the real space SR. Alternatively, an operator may manually generate the three-dimensional space model SV while referring to the dimensions of the real space SR, etc.

[0031] The degree of accuracy with which the three-dimensional space model SV is created varies depending on, for example, the accuracy with which the contents of the movement are estimated. For example, when the movement estimated by the movement estimation unit 146 is the viewing direction of the people H1 and H2, if it is sufficient to simply estimate in which direction the people H1 and H2 are facing, only the object B that defines the rough structure of the real space SR (if it is an indoor space, the object B that constitutes the layout of the room, such as the floor, wall, ceiling, and shelf) may be included in the three-dimensional space model SV. On the other hand, if it is desired to specifically specify what the people H1 and H2 are looking at, it is preferable to include the object B placed in the real space SR (if it is an indoor space, a poster on the wall or a cup placed on the table, etc.) in the three-dimensional space model SV in addition to the object B that defines the rough structure of the real space SR.

[0032] FIG. 3 is a schematic diagram showing an example of a three-dimensional space model SV. The three-dimensional space model SV shown in FIG. 3 reproduces the real space SR shown in the two-dimensional image PC shown in FIG. 2. The object models Bm (Bm1 to Bm6) included in the three-dimensional space model SV correspond to the object B (B1 to B6) in FIG. 2. Furthermore, three-dimensional target models (skeletal models) M1 and M2 imitating people H1 and H2 are arranged in the three-dimensional space model SV. In FIG. 3, the bodies (parts below the head) of the people H1 and H2 who are the analysis objects H are represented by straight lines, and the heads are represented by circles (spheres). Although the arrangement is omitted in FIG. 3 from the viewpoint of visibility, facial parts such as eyes may be arranged on the heads of the skeleton models.

[0033] Any point in the three-dimensional space model SV can be specified by three-dimensional coordinate information (Xv, Yv, Zv) using the orthogonal Xv, Yv, and Zv axes. The correspondence between the coordinate information (Xv, Yv, Zv) of the three-dimensional space model SV and the coordinate information (Xr, Yr, Zr) of the real space SR is known and can be converted into each other. The position of the object model Bm (Bm1 to Bm6) in the three-dimensional space model SV corresponds to the position of the object B (B1 to B6) in the real space SR. The three-dimensional coordinates indicating the position of the three-dimensional target model M1 in the three-dimensional space model SV (more specifically, the center point Dm of the head of the three-dimensional target model M1) are (Xvh, Yvh, Zvh). The placement unit 144 performs coordinate conversion processing on the three-dimensional coordinates (Xrh, Yrh, Zrh) of the person H1 in the real space SR to calculate three-dimensional coordinates (Xvh, Yvh, Zvh), and places the three-dimensional object model M1.

[0034] The correction unit 145 corrects the posture of the three-dimensional object model M based on the positional relationship between the optical axis K of the imaging device C and the analysis object H. In this embodiment, the correction unit 145 corrects the orientation of the three-dimensional object model M by rotating the three-dimensional object model M based on the position of the analysis object H with respect to the optical axis K.

[0035] Hereinafter, the principle of correction by the correction unit 145 will be described with reference to Figs. 6A to 10B. Figs. 6A and 6B are diagrams that typically show the positional relationship between the imaging device C and the analysis target H. Fig. 6A is a diagram of a room F in which the imaging device C and the analysis target H are arranged, as viewed from above, and Fig. 6B is a diagram of the room F as viewed from the side. The room F has walls Fw1 to Fw4, a ceiling Fc, and a floor Ff. The imaging device C is installed near the wall Fw3 with the optical axis K directed toward the wall Fw4. For convenience of explanation, it is assumed that the Zr axis (optical axis K) is parallel to the ceiling Fc and the floor Ff, the Yr axis is parallel to the up-down direction of the wall Fw3, and the Xr axis is parallel to the left-right direction of the wall Fw3. In Fig. 6A, the imaging range of the imaging device C is indicated by a dashed line.

[0036] When viewed from above the room F, position P1 is set on an extension of the optical axis K of the imaging device C, position P2 is set in an area on the right side of the optical axis K, and position P3 is set in an area on the left side of the optical axis K. Positions P1 to P3 are arranged on a straight line. Persons H3 to H5, who are the analysis subject H, are placed at positions P1 to P3 and imaged, respectively, to obtain a two-dimensional image PC2 shown in FIG. 7. As shown in FIG. 7 and other figures, the persons H3 to H5 are in a pose with both arms spread out along the left-right direction of the wall Fw1, and are facing their faces and gazes toward the wall Fw1. Note that persons H3 to H5 are shown diagrammatically in FIGS. 6A and 6B.

[0037] FIG. 7 is a diagram showing a two-dimensional image PC2. Walls Fw1, Fw2, Fw4, a floor Ff, and people H3 to H5 are shown in the two-dimensional image PC2. Here, the people H3 to H5 are in the same posture (with their arms spread along the left-right direction of the wall Fw1, and their faces and gazes are directed toward the wall Fw1), but they are shown differently in the two-dimensional image PC2. Specifically, the person H3, who is located at a position P1 on the extension line of the optical axis K, appears to be facing the direction toward the wall Fw1 (the direction along the Xr axis) as in the real space SR. In contrast, the person H4, who is located at a position P2 in the area to the right of the optical axis K, appears to be rotating counterclockwise relative to the wall Fw1 (or compared to the person H3). Moreover, the person H5, who is located at a position P3 in the area to the left of the optical axis K, appears to be rotating clockwise relative to the wall Fw1 (or compared to the person H3).

[0038] Such differences in appearance are due to the nature of perspective projection. The two-dimensional image PC2 is a projection of a point in three-dimensional space onto a two-dimensional plane using perspective projection. In perspective projection, object B is rendered so that its depthwise extent vanishes at a vanishing point. Therefore, even if people H3 to H5 are in the same pose, they are rendered in different shapes depending on their positions relative to the imaging device C. In the case of parallel projection, people H3 to H5 in the same pose are rendered in the same shape.

[0039] Here, the above-mentioned placement unit 144 places the three-dimensional object model M in the three-dimensional space model SV as it appears in the two-dimensional image PC2. For this reason, an error may occur between the orientation of the people H3 to H5 in the real space SR and the orientation of the three-dimensional object model M placed in the three-dimensional space model SV.

[0040] 8A and 8B are diagrams showing the direction of the gaze of person H5. As shown in FIG. 8A, when the direction of the gaze of person H5 in the real space SR is perpendicular to the wall Fw1, the direction of the gaze is indicated by an arrow Aw1. On the other hand, when the direction of the gaze of person H5 is estimated based on the two-dimensional image PC2 shown in FIG. 7, the direction of the gaze of person H5 is shifted clockwise by an angle Φ with respect to the arrow Ar2, as shown by the arrow Ar2 in FIG. 8B. Such an estimation result does not correctly reflect the direction of the gaze of person H5.

[0041] For this reason, the correction unit 145 corrects the posture of the three-dimensional object model M based on the positional relationship between the optical axis K of the imaging device C and the analysis object H. As described above, in this embodiment, the correction unit 145 corrects the orientation of the three-dimensional object model M by rotating the three-dimensional object model M based on the position of the analysis object H with respect to the optical axis K.

[0042] 9A and 9B are diagrams showing a two-dimensional image PC3. In FIG. 9A and FIG. 9B, a person H6, who is an analysis target H, is shown typically. FIG. 9A shows a two-dimensional image PC3 captured by an imaging device C installed near the ceiling Fc of a wall Fw3 of a room F. The person H6 appears in the two-dimensional image PC3. FIG. 9B is an enlarged view of a portion of FIG. 9A in which the face of the person H6 appears. The arrangement unit 144 estimates that the line of sight of the person H6 is directed in the direction of the arrow Ar3, based on the face of the person H6 captured in the two-dimensional image PC3.

[0043] The arrow Ar3 can be decomposed into Yaw, which is a rotation component around the Yr axis, Roll, which is a rotation component around the Zr axis, and Pitch, which is a rotation component around the Xr axis. The yaw of the arrow Ar3 is a°, the roll is b°, and the pitch is c°. The correction unit 145 corrects the yaw and pitch of the arrow Ar.

[0044] Fig. 10A is a diagram showing a schematic diagram of a method for calculating the yaw correction angle. The position of person H5 on the Xr-Zr axis plane is defined as coordinates (x1, z1). The angle y° between the optical axis K (Zr axis) of the imaging device C and the line segment Lc1 connecting the center point O and the position of person H5 can be calculated by the following formula (1). The angle y° is positive in the area Fr to the right of the optical axis K of the imaging device C, and negative in the area Fl to the left.

[0045]

number

[0046] The correction unit 145 calculates the corrected yaw a'° using the following equation (2).

[0047] a'°=a°+y° (2)

[0048] The correction unit 145 rotates the three-dimensional object model M on the three-dimensional space model by y° so that the yaw of the line of sight of the three-dimensional object model M becomes a'° after correction. If the angle y° is positive, the three-dimensional object model M is rotated clockwise (right-handed). If the angle y° is negative, the three-dimensional object model M is rotated counterclockwise (left-handed). In the example of FIG. 10A, the angle y° is positive. Therefore, the correction unit 145 corrects the three-dimensional object model M by y° clockwise so that the yaw coincides with the arrow Ar3y.

[0049] Fig. 10B is a diagram showing a schematic diagram of a method for calculating the pitch correction angle. The position of person H5 on the Zr-Yr axis plane is defined as coordinates (z1, y1). The angle p° between the optical axis K (Zr axis) of the imaging device C and the line segment Lc2 connecting the center point O and the position of person H5 can be calculated by the following formula (3). The angle p° is positive in the area Fu above the optical axis K of the imaging device C and negative in the area Fd below it.

[0050]

number

[0051] The correction unit 145 calculates the corrected pitch c'° by the following formula (4).

[0052] c'°=c°+p° (4)

[0053] The correction unit 145 rotates the three-dimensional object model M on the three-dimensional space model by p° so that the pitch of the line of sight direction of the three-dimensional object model M becomes c'° after correction. If the angle p° is positive, the three-dimensional object model M is rotated upward. If the angle p° is negative, the three-dimensional object model M is rotated downward. In the example of FIG. 10B, the angle p° is positive. Therefore, the correction unit 145 rotates the three-dimensional object model M upward so that the pitch matches the arrow Ar3p.

[0054] It should be noted that both the yaw correction and the pitch correction may be performed, or only one of them may be performed.

[0055] To summarize the above, correction unit 145 divides the area captured in two-dimensional image PC3 into a first area located in a first direction relative to optical axis K and a second area located in a second direction relative to optical axis K, using optical axis K and an orthogonal axis perpendicular to optical axis K. When analysis object H is located in the first area, correction unit 145 rotates the orientation of three-dimensional object model M on a first plane formed by optical axis K and the orthogonal axis in the first direction. Furthermore, when analysis object H is located in the second area, correction unit 145 rotates the orientation of three-dimensional object model M on the first plane in the second direction.

[0056] The optical axis K corresponds to the Zr axis, and the orthogonal axis corresponds to the Xr axis or the Yr axis. When the orthogonal axis is the Xr axis, the first plane is the Xr-Zr plane, the first direction is, for example, the right direction, and the second direction is, for example, the left direction. The first direction may be the left direction and the second direction may be the right direction. When the first direction is the right direction and the second direction is the left direction, the first region is the region Fr in FIG. 10A, and the second region is the region Fl in FIG. 10A. When the person H6, which is the analysis target H, is in the region Fr, the correction unit 145 rotates the three-dimensional object model M in the right direction. Also, when the person H6 is in the region Fl, the correction unit 145 rotates the three-dimensional object model M in the left direction.

[0057] In addition, the correction unit 145 uses the intersection point between the optical axis K and the orthogonal axis as a reference point, detects the angle that the line segment connecting the position on the first plane of the analysis object H and the reference point makes with respect to the optical axis K, and rotates the orientation of the three-dimensional object model M by that angle.

[0058] The reference point corresponds to the center point O of the lens. When the orthogonal axis is the Xr axis, the position of the analysis object H on the first plane is the coordinates (x1, z1) shown in FIG. 10A, and the line segment connecting the center point O and the coordinates (x1, z1) is the line segment Lc1. The angle that the line segment Lc1 makes with the optical axis K is y°. The correction unit 145 rotates the orientation of the three-dimensional object model M by y°.

[0059] The above-mentioned orthogonal axis is defined as the first orthogonal axis. The correction unit 145 further divides the area captured in the two-dimensional image PC3 into a third area located in a third direction with respect to the optical axis K and a fourth area located in a fourth direction with respect to the optical axis K by a second orthogonal axis perpendicular to the optical axis K and the first orthogonal axis. When the analysis object H is located in the third area, the correction unit 145 rotates the orientation of the three-dimensional object model M on the second plane formed by the optical axis K and the second orthogonal axis in the third direction. Furthermore, when the analysis object H is located in the fourth area, the correction unit 145 rotates the orientation of the three-dimensional object model M on the second plane in the fourth direction.

[0060] If the Xr axis is the first orthogonal axis, the second orthogonal axis is the Yr axis. When the second orthogonal axis is the Yr axis, the second plane is the Zr-Yr plane, the third direction is, for example, the upward direction, and the fourth direction is, for example, the downward direction. The third direction may be the downward direction, and the fourth direction may be the upward direction. When the third direction is the upward direction and the fourth direction is the downward direction, the third region is the region Fu in FIG. 10B, and the fourth region is the region Fd in FIG. 10B. When the person H6, who is the analysis target H, is in the region Fu, the correction unit 145 rotates the three-dimensional object model M in the upward direction. Also, when the person H6 is in the region Fd, the correction unit 145 rotates the three-dimensional object model M in the downward direction. Note that the first orthogonal axis may be the Yr axis, and the second orthogonal axis may be the Xr axis.

[0061] Furthermore, the correction angle is not limited to the angle (angle y° or angle p°) that the line segment connecting the position on the first plane or the second plane of the analysis object H and the reference point forms with respect to the optical axis K. For example, the correction angle may be an angle with a predetermined margin with respect to the angle y° or the angle p°.

[0062] Returning to the description of FIG. 1, the motion estimation unit 146 estimates the motion of the analysis target H in the real space SR based on the relative position between the object model Bm in the three-dimensional space model SV and the corrected three-dimensional object model M. When the analysis target H performs a motion in the real space SR, the relative position between the object B present in the real space SR changes at least at the start and end of the motion. For example, when the motion of the analysis target H is a motion that acts on the object B (for example, moving the object B, etc.), at least a part of the analysis target H (or another object such as a striking tool) is brought into contact with the object B to apply a force. That is, at least a part of the analysis target H is in a position where it contacts the object B. Also, for example, when the motion of the analysis target H is a motion that does not act on the object B (for example, viewing the surrounding object B, etc.), the relative position between at least a part of the body of the analysis target H (the head in the case of viewing the object B) and the surrounding object B changes. Therefore, the motion of the analysis target H in the real space SR can be estimated based on the relative position between the object model Bm in the three-dimensional space model SV and the three-dimensional object model M.

[0063] The motion estimation unit 146 estimates the motion based on, for example, a contact point between the object model Bm in the three-dimensional space model SV and the three-dimensional object model M. For example, when the hand of the three-dimensional object model M1 touches the object model Bm5 representing the stool B5, the motion estimation unit 146 estimates that the person H1 is moving the stool B5. Also, when the lower limbs of the three-dimensional object model M1 touch the object model Bm5, the motion estimation unit 146 estimates that the person H1 is sitting on the stool B5.

[0064] In addition, for example, in a three-dimensional space model SV that imitates a store, when the hand of a three-dimensional object model M that imitates a customer touches an object model Bm that indicates a product, the motion estimation unit 146 estimates that the customer has picked up the product in the store in the real space SR. When the three-dimensional space model SV reproduces the shelf layout of the store in the real space SR, the motion estimation unit 146 can identify the specific product that the customer has picked up. Even when the three-dimensional space model SV does not reproduce the shelf layout of the store in the real space SR, it can be identified that the customer has picked up the product.

[0065] Furthermore, the motion estimation unit 146 estimates the motion based on an object model Bm in the three-dimensional space model SV located in the direction of a specific part of the three-dimensional object model M. For example, by setting the specific part as the eye or face of the three-dimensional object model M, the direction in which the specific part is located can be regarded as the direction of the line of sight of the three-dimensional object model M. Therefore, the object model Bm located in this direction is estimated to correspond to the object B that the analysis target H is looking at.

[0066] As described above, the orientation of the three-dimensional object model M in the three-dimensional space model SV is corrected by the correction unit 145. Therefore, the three-dimensional object model M reproduces with high accuracy on the three-dimensional space model SV the body orientation, hand position, gaze direction, and the like of the analysis object H. By performing the correction by the correction unit 145, the accuracy of the motion estimation by the motion estimation unit 146 can be improved.

[0067] Fig. 4A and Fig. 4B are diagrams that show the results of motion estimation by the motion estimation unit 146. Fig. 4A is a side view of the three-dimensional space model SV, and Fig. 4B is a top view of the three-dimensional space model SV. Fig. 4A and Fig. 4B show the gaze directions L1 and L2 of the three-dimensional object models M1 and M2. The gaze directions L1 and L2 are obtained by regarding the facial directions of the three-dimensional object models M1 and M2 as the gaze directions.

[0068] An object model Bm1 corresponding to a wall B1 is located in the line of sight direction L1 of the three-dimensional object model M1 corresponding to the person H1. Therefore, the motion estimation unit 146 estimates that the person H1 is looking at the wall B1. Furthermore, an object model Bm6 corresponding to a monitor B6 is located in the line of sight direction L2 of the three-dimensional object model M2 corresponding to the person H2. Therefore, the motion estimation unit 146 estimates that the person H2 is looking at the monitor B6. By arranging the three-dimensional object models M1 and M2 in the three-dimensional space model SV, the direction of motion can be estimated three-dimensionally, and the accuracy of motion estimation is improved compared to estimating the motion of the analysis object H using only the two-dimensional image PC.

[0069] 11 is a flowchart showing the operation of image analyzing device 10. Processing device 140 of image analyzing device 10 functions as acquisition unit 141 and acquires a two-dimensional image PC in which real space SR and analysis object H are captured from imaging device C (step S10). Processing device 140 functions as object model generation unit 142 and generates a three-dimensional object model M that imitates analysis object H based on the two-dimensional image PC (step S11).

[0070] The processing device 140 functions as a depth estimation unit 143, and estimates the depth of the analysis object H relative to the imaging device C that captures the two-dimensional image PC based on the two-dimensional image PC (step S12). The processing device 140 functions as a placement unit 144, and estimates three-dimensional coordinates indicating the position of the analysis object H in the real space SR based on the depth estimated by the depth estimation unit 143, the position of the analysis object H on the two-dimensional image PC, and the position of the imaging device C in the real space SR (step S13). Note that the order of the processing of step S11 and the processing of steps S12 to S13 may be reversed, or they may be performed simultaneously.

[0071] The processing device 140 functions as a placement unit 144 and places the three-dimensional object model M at a position corresponding to the three-dimensional coordinates estimated in step S13 on the three-dimensional space model SV simulating the real space SR (step S14). The processing device 140 functions as a correction unit 145 and corrects the orientation of the three-dimensional object model M placed on the three-dimensional space model SV based on the positional relationship between the optical axis K of the imaging device C and the analysis object H (step S15).

[0072] The processing device 140 functions as a motion estimation unit 146, and estimates the motion of the analysis target H in the real space SR based on the relative position between the object model Bm and the three-dimensional target model M in the three-dimensional space model SV (step S16). After that, the processing device 140 returns the process to step S10 and repeats the subsequent processes.

[0073] As described above, the image analysis device 10 according to the embodiment generates a three-dimensional object model M of the analysis object H based on the two-dimensional image PC, and places the three-dimensional object model M in a three-dimensional space model SV that imitates the real space SR. The image analysis device 10 corrects the orientation of the three-dimensional object model M placed in the three-dimensional space model SV based on the positional relationship between the optical axis K of the imaging device C and the analysis object H. Thus, the state of the analysis object H in the real space SR can be reproduced with high accuracy by the three-dimensional object model M placed in the three-dimensional space model SV.

[0074] Furthermore, the image analysis device 10 divides the area captured in the two-dimensional image PC into an area Fr located to the right of the optical axis K and an area Fl located to the left of the optical axis K, using the optical axis K and the Xr axis perpendicular to the optical axis K. When the analysis object H is located in the area Fr, the image analysis device 10 rotates the orientation of the three-dimensional object model M on the Xr-Zr plane to the right, and when the analysis object H is located in the area Fl, the image analysis device 10 rotates the orientation of the three-dimensional object model M on the Xr-Zr plane to the left. Thus, the image shift in the two-dimensional image PC represented by perspective projection is corrected, and the three-dimensional object model M can more accurately reproduce the state of the analysis object H.

[0075] Furthermore, image analyzing device 10 sets the intersection point of optical axis K and Xr axis as a reference point (center point O), detects an angle y° that a line segment Lc1 connecting the position of analysis object H on the Xr-Zr plane and the reference point forms with respect to optical axis K, and rotates the orientation of the 3D object model by angle y°. Thus, the angle to be corrected can be calculated simply and accurately.

[0076] Furthermore, the image analysis device 10 divides the area shown in the two-dimensional image PC into an area Fu located above the optical axis K and an area Fd located below the optical axis K by the Yr axis perpendicular to the Xr axis and the optical axis K. When the analysis object H is located in the area Fu, the image analysis device 10 rotates the orientation of the three-dimensional object model M on the Zr-Yr plane upward, and when the analysis object H is located in the area Fd, the image analysis device 10 rotates the orientation of the three-dimensional object model M on the Zr-Yr plane downward. Thus, the image shift in the two-dimensional image PC expressed by perspective projection is corrected, and the three-dimensional object model M can reproduce the state of the analysis object H more accurately. Furthermore, by correcting the Yr axis, which is the second orthogonal axis, in addition to the Xr axis, which is the first orthogonal axis, the accuracy of the orientation of the three-dimensional object model M is further improved.

[0077] Furthermore, the image analyzing device 10 estimates the movement of the analysis object H in the real space SR based on the relative position between the object model Bm and the three-dimensional target model M in the three-dimensional space model SV. In other words, the analysis object H shown in the two-dimensional image PC is projected onto the three-dimensional space model SV. Therefore, the estimation accuracy of the movement of the analysis object H is improved compared to using only the two-dimensional image PC.

[0078] Furthermore, image analyzing device 10 corrects the orientation of three-dimensional object model M in three-dimensional space model SV by correction unit 145. Thus, three-dimensional object model M can reproduce the body orientation, hand position, gaze direction, and the like of analysis object H on the three-dimensional space model SV with high accuracy, thereby improving the accuracy of motion estimation by motion estimation unit 146.

[0079] B:Other (1) In the above-described embodiment, ROM, RAM, etc. are given as examples of storage device 130, but storage device 130 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.

[0080] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0081] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0082] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).

[0083] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0084] (6) Each function illustrated in FIG. 1 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0085] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0086] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0087] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.

[0088] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0089] (10) In the above-described embodiments, the portable device may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

[0090] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0091] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0093] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.

[0094] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0095] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0096] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information). [Explanation of symbols]

[0097] 10...image analysis device, 100...input device, 110...display device, 120...communication device, 130...storage device, 140...processing device, 141...acquisition unit, 142...object model generation unit, 143...depth estimation unit, 144...placement unit, 145...correction unit, 146...motion estimation unit, B...object, Bm...object model, C...imaging device, H...analysis target, H1 to H6...person, K...optical axis, M (M1, M2)...3D object model, PG...program, SR...real space, SV...3D space model.

Claims

1. an acquisition unit that acquires a two-dimensional image of a real space and an analysis target located in the real space, the two-dimensional image being captured by an imaging device; an object model generation unit that generates a three-dimensional object model that imitates the analysis object based on the two-dimensional image; a depth estimation unit that estimates a depth of the analysis object relative to the imaging device based on the two-dimensional image; a placement unit that places the three-dimensional object model on a three-dimensional space model that imitates the real space based on the depth estimated by the depth estimation unit and a position of the analysis object on the two-dimensional image; and a correction unit that corrects an orientation of the three-dimensional object model on the three-dimensional space model based on a positional relationship between an optical axis of the imaging device and the analysis object; An image analysis device comprising:

2. The correction unit corrects an orientation of the three-dimensional object model by rotating the three-dimensional object model based on a position of the analysis object relative to the optical axis.

2. The image analysis device according to claim 1.

3. the correction unit divides an area captured in the two-dimensional image into a first area located in a first direction relative to the optical axis and a second area located in a second direction relative to the optical axis by the optical axis and an orthogonal axis perpendicular to the optical axis, and when the analysis object is located in the first area, rotates an orientation of the three-dimensional object model on a first plane formed by the optical axis and the orthogonal axis in the first direction, and when the analysis object is located in the second area, rotates an orientation of the three-dimensional object model on the first plane in the second direction.

3. The image analysis device according to claim 2.

4. the correction unit detects an angle formed by a line segment connecting a position of the analysis object on the first plane and the reference point with respect to the optical axis, using an intersection point between the optical axis and the orthogonal axis as a reference point, and rotates an orientation of the three-dimensional object model by the angle.

4. The image analysis device according to claim 3.

5. the orthogonal axis is a first orthogonal axis, the correction unit divides an area captured in the two-dimensional image into a third area located in a third direction with respect to the optical axis and a fourth area located in a fourth direction with respect to the optical axis by a second orthogonal axis orthogonal to the optical axis and, when the analysis object is located in the third area, rotates an orientation of the three-dimensional object model on a second plane formed by the optical axis and the second orthogonal axis in the third direction, and, when the analysis object is located in the fourth area, rotates an orientation of the three-dimensional object model on the second plane in the fourth direction.

4. The image analysis device according to claim 3.

6. a motion estimation unit that estimates a motion of the analysis target in the real space based on a relative position between an object model indicating an object in the three-dimensional space model and the corrected three-dimensional target model, 2. The image analysis device according to claim 1.

Images