Information processing device and method for estimating newborn baby's weight

Abstract

This invention makes it possible to measure a subject's body volume without touching the subject. This invention comprises: an acquisition unit for acquiring image information indicating a three-dimensional image in which a subject lying down on a flat surface has been imaged from above; a recognition unit for recognizing, from among objects displayed in the three-dimensional image, a subject image representing the subject; and an estimation unit for using the subject image to estimate the body volume of the subject, inclusive of a lower portion of the subject that is missing in the subject image, assuming that the lower portion is in contact with the flat surface. The invention also comprises: a step for capturing a three-dimensional image of a neonate; a step in which a computer estimates the neonate's body volume from the three-dimensional image; and a step in which the computer estimates the neonate's body weight from the estimated body volume.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a method for estimating the weight of a newborn baby. [Background technology]

[0002] Conventionally, techniques have been proposed for acquiring various types of biometric information from a subject (e.g., a newborn) lying on a flat surface (e.g., the top surface of a mat). For example, Patent Document 1 discloses a technique for measuring the heart rate of a newborn inside an incubator. Depending on the newborn (especially a premature newborn), touching the newborn to acquire biometric information may be considered an invasive procedure. The technique disclosed in Patent Document 1 has the advantage of being able to measure the heart rate without touching the newborn, thereby reducing the number of invasive procedures. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-78703 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology of Patent Document 1, depending on the type of biological information, it is necessary to touch the newborn to measure it. For example, when measuring weight, it is necessary to touch the newborn. In consideration of the above circumstances, an object of the present invention is to make it possible to measure biological information without touching the object, which in conventional technologies requires touching the object to measure it. [Means for solving the problem]

[0005] In order to solve the above problems, the information processing device of the present invention comprises an acquisition unit that acquires image information showing a three-dimensional image of an object lying on a flat surface taken from above, a recognition unit that recognizes an object image representing the object among the objects displayed in the three-dimensional image, and an estimation unit that uses the object image to estimate the shape of the object, including the missing lower part in the object image, assuming that the object is in contact with the flat surface on the underside, and calculates the object's physical information using the estimated object shape. [Effects of the Invention]

[0006] According to the present invention, it is possible to measure biological information without touching the subject, whereas in the prior art it was necessary to touch the subject to measure biological information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating each configuration of the information processing system. [Figure 2] FIG. 2 is a hardware configuration diagram of a computer. [Figure 3] FIG. 2 is a functional block diagram of the information processing system. [Figure 4] FIG. 10 is a diagram for explaining a specific example of a target image. [Figure 5] FIG. 10 is another diagram for explaining a specific example of the target image. [Figure 6] FIG. 1 is a diagram illustrating a configuration for generating an object model. [Figure 7] 10A and 10B are diagrams for explaining head size approximation processing and the like. [Figure 8] 10A and 10B are diagrams for explaining a head model deformation process. [Figure 9] 10A and 10B are diagrams for explaining a body model deformation process. [Figure 10] FIG. 10 is a diagram for explaining a weight estimation process and a height estimation process. [Figure 11] 10 is a flowchart of a process during image capture. [Figure 12] FIG. 10 is a diagram for explaining a target model according to the second embodiment. [Figure 13]10A and 10B are diagrams for explaining ellipse approximation processing in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0008] FIG. 1 is a diagram for explaining each component of an information processing system 1. The information processing system 1 of this embodiment includes a computer 100 and a depth camera 200. The above components are connected so that they can communicate with each other. In this embodiment, an incubator is used as the computer 100. As shown in FIG. 1, a mat is provided inside the incubator. A newborn baby (hereinafter referred to as "subject S") lies on the upper surface of the mat (hereinafter referred to as "flat surface A"). Flat surface A is a plane with an approximately rectangular outer edge.

[0009] The depth camera 200 generates a three-dimensional image (distance image) including depth information indicating the distance to the subject. For example, the three-dimensional image may be a point cloud image captured using LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) technology. In this embodiment, the computer 100 estimates the weight or height of the subject S by capturing an image of the subject S with the depth camera 200.

[0010] Specifically, when the object S is photographed by the depth camera 200, image information Dg of a three-dimensional image showing the object S is transmitted to the computer 100. The computer 100 uses the image information Dg received from the depth camera 200 to generate an object model M showing the shape (size) of the photographed object S (see FIG. 6 described later). The volume of the object model M is estimated as the volume of the photographed object S, and the weight of the object S is estimated by multiplying the estimated result by the density. Furthermore, the height of the object model M is estimated as the height of the photographed object S. The above configuration will be described in detail later.

[0011] In this embodiment, an incubator is used as the computer 100, but the computer 100 may be configured separately from the incubator. Also, the depth camera 200 may be configured to have the functions of the computer 100. Also, the incubator may be configured to include the depth camera 200. Instead of the configuration in which the image information Dg is transmitted from the depth camera 200 to the computer 100 by wireless communication, the image information Dg may be transmitted by wired communication. g may be transmitted.

[0012] 2 is a diagram showing the hardware configuration of this embodiment. The computer 100 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a monitor 104, a memory 105, and a communication unit 106.

[0013] The ROM 102 of the computer 100 stores various data including programs in a non-volatile manner. The CPU 101 executes the programs to realize various functions (such as the estimation unit 13) described below. The RAM 103 temporarily stores, for example, various pieces of information referenced by the CPU 101 when executing the programs. The monitor 104 displays various pieces of information. For example, the monitor 104 displays biological information of the subject S in an incubator.

[0014] The memory 105 stores various types of information in a non-volatile manner. For example, a flash memory can be used as the memory 105. The memory 105 stores information necessary for estimating the weight and height of the target S. Specifically, information indicating a head model Mh, a torso model Mb, an arm model Ma, and a leg model Ml for generating a target model M described below is stored in the memory 105. The communication unit 106 receives various types of information from an external device. For example, the communication unit 106 receives image information Dg transmitted from the depth camera 200.

[0015] 3 is a functional block diagram of the information processing system 1 according to this embodiment. As shown in FIG. 3, the information processing system 1 includes an information processing device 10 and an image capturing device 20. For example, the computer 100 described above functions as the information processing device 10 by executing a program, and the depth camera 200 functions as the image capturing device 20. As described above, image information Dg indicating the captured three-dimensional image G is transmitted from the image capturing device 20 to the information processing device 10.

[0016] 3, the information processing device 10 includes an acquisition unit 11, a recognition unit 12, an estimation unit 13, a storage unit 14, a calculation unit 15, and a measurement unit 16. The acquisition unit 11 acquires image information Dg indicating a three-dimensional image G obtained by photographing an object S lying on a flat surface A from above. Specifically, the acquisition unit 11 acquires the image information Dg from the photographing device 20. As will be described in detail later, the three-dimensional image G obtained by photographing an object S lying on the flat surface A from above includes a flat image Ga representing the flat surface A in addition to an object image Gs representing the object S (see FIG. 5).

[0017] The recognition unit 12 recognizes a target image Gs representing a target S among the objects displayed in the three-dimensional image. Specifically, the recognition unit 12 separately recognizes the target image Gs representing the target S and the flat image Ga representing the flat surface A. As a technique for recognizing each object in the three-dimensional image G, for example, a segmentation technique is preferably adopted. AI (Artificial Intelligence) technology can be adopted for the above segmentation. For example, each object in the three-dimensional image G is segmented using a trained FCN (Fully Convolutional Network) (for example, the technology described in JP 2022-29169 A can be adopted).

[0018] The estimation unit 13 estimates the volume of the target S using the target image Gs. Let us assume that a target image Gs representing the entire target S (both the upper and lower parts) has been captured. In this case, the volume of the target S approximately matches the volume of the target image Gs. Therefore, the volume of the target S can be estimated as the volume of the target S.

[0019] However, in reality, when an image of a target S lying on a flat surface A is taken from above, only the upper part of the target is usually photographed, and the lower part (on the flat surface A side) is not photographed (see FIG. 5 described later). Therefore, the lower part of the target S is missing from the target image Gs. The volume of the target image Gs described above cannot be used as the actual volume of the target S.

[0020] In consideration of the above circumstances, this embodiment employs a configuration that can estimate with high accuracy the volume of the target S even from a target image Gs in which the lower portion of the target S is missing. Specifically, the estimation unit 13 of this embodiment assumes that the lower side of the target S is in contact with the flat surface A, and estimates the volume of the target S including the missing lower portion in the target image Gs, using the target image Gs.

[0021] More specifically, the storage unit 14 stores model information indicating the shape of each model (such as a head model Mh) representing each part (such as the head) of the target S (see FIG. 6 described later). The model information in this embodiment is image information indicating a head model Mh that is a model of the head of the target S, image information indicating a torso model Mb that is a model of the torso of the target S, image information indicating an arm model Ma that is a model of the arms of the target S, and image information indicating a leg model Ml that is a model of the legs of the target S.

[0022] The estimation unit 13 uses the target image Gs to deform the shape of each model (Mh, Mb, Ma, Ml) so that the shape approximates the shape of the part of the target S to which the model corresponds. Furthermore, the estimation unit 13 uses each model after deformation to generate a target model M that indicates the overall shape of the target S. The estimation unit 13 estimates the volume of the target model M as the volume of the target S.

[0023] Furthermore, when deforming the shapes of the head model Mh and the torso model Mb of the models, the estimation unit 13 deforms the shapes of the models assuming that the target S is in contact with the flat surface A on the underside. Specifically, the estimation unit 13 identifies a reference plane Fa that passes through the flat image Ga (see FIG. 5). Furthermore, the estimation unit 13 deforms the models (Mh, Mb) based on the distance from the target image Gs to the reference plane Fa (Lhz in FIG. 8(b)) (see FIGS. 7(a)(b), 8(a)-(c), and 9(a)(b)), and estimates the volume of the target S using the deformed models. The above configuration will be described in detail later.

[0024] The calculation unit 15 calculates the weight of the target S using the volume of the target S estimated by the estimation unit 13. Specifically, the information processing device 10 stores in advance the average density of the target S (hereinafter referred to as "average density D"). The calculation unit 15 calculates the weight W of the target S by multiplying the volume V of the target S estimated by the estimation unit 13 by the average density D (W = V × D). The measurement unit 16 measures the height of the target S using the model transformed by the estimation unit 13. Specifically, the measurement unit 16 estimates the length of a predetermined part of the target model M as the height of the target S (see FIG. 10).

[0025] FIG. 4 is a diagram for explaining a specific example of the configuration up to the generation of the target image Gs. In this embodiment, a mat is provided inside the incubator so that the flat surface A is approximately parallel to the horizontal direction. For the sake of explanation, the vertical direction may be referred to as the "z-axis direction." Furthermore, the longitudinal direction of the flat surface A may be referred to as the "x-axis direction," and the width direction of the flat surface A may be referred to as the "y-axis direction." In the above cases, the xy plane is approximately parallel to the horizontal direction. Note that in this embodiment, as shown in FIG. 4, it is assumed that the target S lies on the flat surface A with the top of its head facing the x-axis direction.

[0026] In this embodiment, as shown in the upper part of Fig. 4, a newborn baby lying on a flat surface A is photographed as the object S by the photographing device 20 (depth camera 200). In the specific example of Fig. 4, it is assumed that the object S is photographed from above (in the direction of arrow d). In this case, the flat surface A is photographed in addition to the object S. When a three-dimensional image G is photographed by the photographing device 20, image information Dg representing the three-dimensional image G is transmitted to the information processing device 10 (Sa1 in Fig. 4).

[0027] The central portion of FIG. 4 shows a simulated view of a three-dimensional image G captured by the imaging device 20. Hereinafter, for the sake of explanation, the direction in the three-dimensional image space corresponding to the z-axis direction (vertical direction) may be referred to as the "Z-axis direction." Similarly, the direction corresponding to the x-axis direction may be referred to as the "X-axis direction," and the direction corresponding to the y-axis direction may be referred to as the "Y-axis direction." As shown in FIG. 4, the three-dimensional image G represented by the image information Dg includes a target image Gs and a flat image Ga. The target image Gs represents the target S, and the flat image Ga represents the flat surface A. Note that, for the sake of explanation, the target image Gs and the flat image Ga are shown in different colors in FIG. 4 (the same applies to FIG. 5, which will be described later).

[0028] The information processing device 10 recognizes each image of the three-dimensional image G including the target image Gs and the flat image Ga by a segmentation technique (Sa2 in FIG. 4). For example, the target image Gs and the flat image Ga are recognized as images showing different objects.

[0029] The lower part of Figure 4 shows an excerpt of the target image Gs from the 3D image G. As shown in Figure 4, the target image Gs includes a head image Gsh, a chest image Gsc, a waist image Gsw, an arm image Gsa, and a leg image Gsl. The head image Gsh represents the head of the target S. Similarly, the chest image Gsc represents the chest of the torso of the target S from the neck to the navel, the waist image Gsw represents the torso of the target S from the navel to the legs, the arm image Gsa represents the arms of the target S, and the leg image Gsl represents the legs of the target S. Each of these images is recognized as an image showing a separate object using segmentation techniques.

[0030] FIG. 5 is a diagram for explaining the reference plane Fa. As will be described in detail later, the reference plane Fa is a plane that passes through the flat surface A and is used when estimating the volume of the target S. FIG. 5 shows a simulated diagram of a three-dimensional image G (target image Gs, flat image Ga). In the specific example of FIG. 5, it is assumed that the three-dimensional image G is viewed in the X-axis direction. Note that in FIG. 5, the head image Gsh and chest image Gsc are selected from the target image Gs and are shown, with other images (such as the arm image Gsa) omitted.

[0031] The specific example of FIG. 5, like the specific example of FIG. 4 described above, assumes that subject S lying on a flat surface A is photographed from above. In the above case, the target image Gs includes an image representing the upper part of the target S, but does not include (misses) the lower part of the target S. For example, the specific example of FIG. 5 assumes that subject S (a newborn baby) is lying on his back on the flat surface A. In the above case, the head image Gsh of the target image Gs represents the face side (eyes, nose, mouth) of the target S, but does not include the back side of the head of the target S. Furthermore, the chest image Gsc of the target image Gs represents the abdominal side (chest, navel) of the target S, but does not include the back side of the target S.

[0032] The information processing device 10 of this embodiment identifies a reference plane Fa from the flat image Ga (Sb in FIG. 5). Specifically, the flat image Ga is a point cloud image. In this embodiment, the reference plane Fa is calculated by plane approximation using the least squares method. That is, the plane that minimizes the sum of the squares of the distances from each point constituting the flat image Ga is identified as the reference plane Fa.

[0033] 5 shows a conceptual diagram of the reference plane Fa. As described above, the flat image Ga is perpendicular to the Z-axis direction (vertical direction). Therefore, the reference plane Fa is also perpendicular to the Z-axis direction. Furthermore, the flat image Ga is parallel to the XY plane (horizontal direction). Therefore, the reference plane Fa is also parallel to the XY plane.

[0034] As described above, when the subject S is photographed from above, the lower portion of the subject image Gs is missing. In Fig. 5, the lower portion (Sh, Sb) of the subject S missing from the subject image Gs is indicated by a dashed line. Specifically, the back of the head Sh missing from the head image Gsh and the back Sb missing from the chest image Gsc of the subject image Gs are indicated by a dashed line.

[0035] When subject S (newborn baby) lies on a flat surface A, the lower end Pah of the back of the head Sh of subject S usually touches the flat surface A. That is, the lower end Pah of the back of the head Sh of subject S is missing from the target image Gs, but the point in the three-dimensional image space corresponding to the lower end Pah is located on the reference plane Fa. Similarly, when subject S lies on a flat surface A, the lower end Pac of the back Sb of subject S usually touches the flat surface A. That is, the lower end Pac of the back Sb of subject S is missing from the target image Gs, but the point in the three-dimensional image space corresponding to the lower end Pac is located on the reference plane Fa.

[0036] 6 is a diagram for explaining a specific example of each process for generating the target model M. As described above, the target model M represents the overall shape of the target S. The information processing device 10 generates the target model M from a captured target image Gs, and estimates the volume of the target model M as the volume of the target S.

[0037] 6, model information indicating a head model Mh, a torso model Mb, an arm model Ma, and a leg model Ml is stored in advance in the storage unit 14 of the information processing device 10. The torso model Mb includes a chest model Mc and a waist model Mw.

[0038] The head model Mh (before deformation) stored in the memory unit 14 represents the average shape of the head of the subject S. Similarly, the torso model Mb stored in the memory unit 14 represents the average shape of the torso of the subject S, the waist model Mw stored in the memory unit 14 represents the average shape of the waist of the subject S, the arm model Ma stored in the memory unit 14 represents the average shape of the arms of the subject S, and the leg model Ml stored in the memory unit 14 represents the average shape of the legs of the subject S.

[0039] When the information processing device 10 acquires a target image Gs from the photographing device 20, it extracts a head image Gsh from the target image Gs. The information processing device 10 then executes a head size approximation process (S1 in FIG. 6) using the head image Gs and the head model Mh. In the head size approximation process, the rough shape (size) of the head model Mh is determined (see FIG. 7(a) described later). After executing the head size approximation process, the information processing device 10 then executes a head model rotation process (S2 in FIG. 6). In the head model rotation process, the orientation of the head model Mh is rotated (adjusted) in accordance with the orientation of the head image Gh (see FIG. 7(b)).

[0040] Thereafter, the information processing device 10 executes a head model deformation process (S3 in FIG. 6). As will be described in detail later, in the head model deformation process, the shape of the head model Mh is deformed so as to approximate the shape of the head of the target S (to the size of the head of the target S) assuming that the underside of the target S is in contact with the flat surface A (see FIGS. 8(a) to 8(c) described later).

[0041] The information processing device 10 performs the torso image rotation process (see FIG. 6 The torso image rotation process aligns the orientation of the torso model Mb with the orientation of the torso image Gb. In this embodiment, the torso model Mb is oriented in the X-axis direction beforehand. In the torso image rotation process, the torso image Gb is rotated so that it faces the X-axis direction. Specifically, the torso image Gb is a point cloud image. The information processing device 10 identifies the orientation of the torso image Gb using principal component analysis, and rotates the torso image Gb so that it is parallel to the X-axis direction. As a configuration for adjusting the orientation of a point cloud image, for example, the configuration described in Japanese Patent Application Laid-Open No. 2014-44078 can be adopted.

[0042] After executing the torso image rotation process, the information processing device 10 executes a torso model deformation process (S5 in FIG. 6). As will be described in detail later, in the torso model deformation process, the shape of the torso model Mb is deformed so as to approximate the shape of the torso of the target S, assuming that the target S is in contact with the flat surface A on the underside (see FIGS. 9(a) and 9(b) described later).

[0043] The information processing device 10 performs arm model deformation processing (S6 in FIG. 6) to deform the shape of the arm model Ma so that it approximates the shape of the arm of the target S. The arm image Gsa described above includes an image representing the area from the shoulder to the elbow of the target S (hereinafter referred to as the "first arm image Gsa1") and an image representing the area from the elbow to the fingertips (hereinafter referred to as the "second arm image Gsa2"). Each of these images is distinguished by segmentation technology. Furthermore, as shown in FIG. 6, the arm model Ma includes a first arm model Ma1 representing the area from the shoulder to the elbow and a second arm model Ma2 representing the area from the elbow to the fingertips.

[0044] In the arm model deformation process, the information processing device 10 deforms the first arm model Ma1 using the first arm image Gsa1. Similarly, in the arm model deformation process, the information processing device 10 deforms the second arm model Ma2 using the second arm image Gsa2. The target image Gs includes an arm image Ga representing the right arm of the target S and an arm image Ga representing the left arm. The arm model Ma includes an arm model Ma representing the right arm and an arm model Ma representing the left arm of the target S. In the arm model deformation process, the arm model Ma representing the right arm is deformed using the arm image Ga representing the right arm. Similarly, the arm model Ma representing the left arm is deformed using the arm image Ga representing the left arm. A specific example of the arm model deformation process will be described later as a modified example.

[0045] The information processing device 10 performs leg model deformation processing (S7 in FIG. 6) to deform the shape of the leg model Ml so that it approximates the shape of the leg of the target S. The leg image Gsl described above includes an image representing the leg of the target S from the groin to the knee (hereinafter referred to as the "first leg image Gsl1") and an image representing the leg from the knee to the toe (hereinafter referred to as the "second leg image Gsl2"). Each of the above images is distinguished by segmentation technology. Furthermore, as shown in FIG. 6, the leg model Ml includes a first leg model Ml1 representing the thigh of the target S and a second leg model Ml2 representing the part from the knee to the toe.

[0046] In the leg model deformation process, the information processing device 10 deforms the first leg model Ml1 using the first leg image Gsl1. Similarly, in the leg model deformation process, the information processing device 10 deforms the second leg model Ml2 using the second leg image Gsl2. The target image Gs includes a leg image Gsl representing the right leg of the target S and a leg image Gsl representing the left leg. The leg model Ml includes a leg model Ml representing the right leg of the target S and a leg model Ml representing the left leg. In the leg model deformation process, the leg model Ml representing the right leg is deformed using the leg image Gsl representing the right leg. Similarly, the leg model Ml representing the left leg is deformed using the leg image Gsl representing the left leg. A specific example of the leg model deformation process will be described later as a modified example.

[0047] As can be understood from the above description, the head model Mh, the torso model Mb (chest model Mc, waist model Mw), the arm model Ma, and leg The model Ml is deformed in accordance with the captured target image Gs. The information processing device 10 generates the target model M by appropriately combining each of the deformed models. The target model M described above approximates the actual shape of the target S. Below, a detailed description will be given of the configuration for deforming the head model Mh (see FIGS. 7(a) and (b) and FIGS. 8(a) to (c)) and the configuration for deforming the torso model Mb (see FIGS. 9(a) and (b)).

[0048] FIG. 7(a) is a diagram for explaining the details of the head size approximation process (S1 in FIG. 6). As described above, in the head size approximation process, the rough size of the head model Mh is determined. Specifically, in FIG. 7 (a) As shown in Fig. 1, the head image Gsh is approximated as a sphere by the least squares method using the point cloud that constitutes the head image Gsh. The diameter d1 of the spherically approximated head image Gsh is then calculated. Similarly, the head model Mh before deformation is approximated as a sphere, and the diameter d2 of the spherically approximated head model Mh is calculated.

[0049] The information processing device 10 calculates the ratio Rd (Rd=d1 / d2) between the diameter d1 of the spherically approximated head image Gsh and the diameter d2 of the spherically approximated head model Mh. The information processing device 10 then uses the ratio Rd to change the size of the head model Mh. Specifically, the head model Mh before deformation is multiplied by the ratio Rd. According to the above head size approximation process, the size of the head model Mh becomes closer to the actual size of the head of the target S. However, in this embodiment, by executing a head model deformation process (see FIG. 8) in addition to the head size approximation process, the shape of the head model Mh is deformed so as to approximate the shape of the head of the target S with high accuracy.

[0050] 7(b) is a diagram for explaining the details of the head model rotation process (S2 in FIG. 6). As described above, in the head model rotation process, the orientation of the head model Mh is rotated (adjusted) according to the orientation of the head image Gh. Specifically, if the coordinates of an arbitrary point in the head image Gh (hereinafter referred to as "target point Ps") are (xi, yi, zi), and the coordinates of a point in the head model Mh corresponding to the target point Ps (hereinafter referred to as "corresponding point Pm") (representing the same part in the target S) are (xi', yi', zi'), the information processing device 10 calculates the following rotational movement transformation formula (r, t):

[0051] (Number 1) TIFF0007756972000001.tif2879

[0052] To obtain the rotational movement transformation formula (r, t), the information processing device 10 substitutes a combination of a specific target point Ps and a corresponding point Pm corresponding to the target point Ps. Specifically, techniques for detecting each feature point (such as pupils) in a face from a face image have been conventionally known. As such a technique, for example, the technique described in Japanese Patent Application Laid-Open No. 2022-128652 can be adopted. The information processing device 10 detects the right eye Ps1, left eye Ps2, nose Ps3, mouth Ps4, and left ear Ps5 of the target S in the head image Gs as target points Ps. Similarly, the information processing device 10 detects the right eye Pm1, left eye Pm2, nose Pm3, mouth Pm4, and left ear Pm5 in the head model Mh as target points Ps. correspondence It is detected as point Pm.

[0053] The information processing device 10 calculates the elements r and t by substituting multiple pairs of combinations of the target point Ps and the corresponding point Pm, and obtains a rotational translation transformation formula. Specifically, the total number of elements r to be calculated (9) and the number of elements t (3) is 12. Furthermore, by substituting one pair of the target point Ps and the corresponding point Pm, three linear equations containing the elements r and t as coefficients are obtained. In this embodiment, 15 linear equations are obtained by substituting the above-mentioned five pairs of combinations of the target point Ps and the corresponding point Pm, and the elements r and t are obtained using some of the linear equations.

[0054] The information processing device 10 uses the rotational movement transformation formula to move the corresponding point Pm of the head model Mh to the target point Ps of the target image Gs. That is, the information processing device 10 rotates the orientation of the head model Mh to the orientation of the head image Gsh.

[0055] 8(a) to 8(c) are diagrams for explaining the details of the head model deformation process. As described above, the head model deformation process deforms the shape of the head model Mh so that it closely approximates the shape of the head of the target S with high accuracy.

[0056] 8(a) is a diagram for explaining the reference box Ch and the adjustment box Ci. As will be described in detail later, the reference box Ch is generated according to the size of the target image Gs. Furthermore, the adjustment box Ci is generated according to the size of the head model Mh immediately before the head model deformation process is executed. In the head model deformation process, the ratio Rhi between the size of the reference box Ch and the size of the adjustment box Ci is calculated, and the head model Mh is deformed (enlarged or reduced) according to the ratio Rhi.

[0057] As shown in Fig. 8(a), the reference box body Ch is a rectangular parallelepiped and is composed of a bottom face Fh1, side faces Fh2 to Fh5, and a top face Fh6. Also, as shown in Fig. 8(a), each face Fh of the reference box body Ch is parallel to one of the XY plane, the XZ plane, or the YZ plane. Specifically, of the faces Fh (1 to 6), the bottom face Fh1 and the top face Fh6 are parallel to the XY plane, the side faces Fh2 and Fh5 are parallel to the YZ plane, and the side faces Fh3 and Fh4 are parallel to the XZ plane.

[0058] As shown in Fig. 8(a), the adjustment box Ci is a rectangular parallelepiped and is composed of a bottom surface Fi1, side surfaces Fi2 to Fi5, and a top surface Fi6. Also, as shown in Fig. 8(a), each surface Fi is parallel to one of the XY plane, the XZ plane, or the YZ plane. Specifically, of the surfaces Fi (1 to 6), the bottom surface Fi1 and the top surface Fi6 are parallel to the XY plane, the side surfaces Fi2 and Fi5 are parallel to the YZ plane, and the side surfaces Fi3 and Fi4 are parallel to the XZ plane.

[0059] FIG. 8(b) is a diagram illustrating a configuration for generating a reference box body Ch. The left part of FIG. 8(b) shows a conceptual diagram of the reference box body Ch as seen from the Z-axis direction (top). The reference box body Ch of this embodiment is generated so that it can accommodate a head image Gsh. FIG. 8(b) shows a head image Gsh accommodated in the reference box body Ch. As shown in FIG. 8(b), the side surface Fh2 of the reference box body Ch passes through point Ph2 in the head image Gsh where the X coordinate is maximum. Furthermore, the side surface Fh5 of the reference box body Ch passes through point Ph5 in the head image Gsh where the X coordinate is minimum. The side surface Fh3 of the reference box body Ch passes through point Ph3 in the head image Gsh where the Y coordinate is maximum, and the side surface Fh4 passes through point Ph4 in the head image Gsh where the Y coordinate is minimum.

[0060] As can be understood from the above explanation, each side surface Fh (2 to 5) of the reference box body Ch is generated so as to surround the head image Gsh on all four sides when viewed from the Z-axis direction. Also, the right side of FIG. 8(b) shows a conceptual diagram of the head image Gsh when viewed from the X-axis direction. As shown in FIG. 8(b), the top surface Fh6 of the reference box body Ch passes through the point Ph6 of the head image Gsh where the Z coordinate is maximum. As described above, each of the side surfaces Fh (2 to 5) and the top surface Fh6 of the reference box body Ch is generated so as to be in contact with the target image Gh.

[0061] However, when the object S is photographed from above (Z-axis side), the lower part of the object S is not photographed. For these reasons, the lower part of the object S is missing from the head image Gsh. However, in this embodiment, in order to deform the shape of the head model Mh so that it closely approximates the shape of the object S with high accuracy, it is necessary to generate the bottom surface Fh1 of the reference box body Ch at a position corresponding to the lower end Pah of the head of the object S.

[0062] However, since the head image Gsh does not include the lower part of the target S, the point (lower end) where the Z coordinate of the head image Gsh is minimum is different from the actual lower end (Pah) of the target S. head If the bottom surface Fh1 of the reference box Ch is generated at the point in the image Gsh where the Z coordinate is minimum, it may not be possible to deform the shape of the head model Mh so that it closely approximates the shape of the target S. In such cases, there is the disadvantage that the volume and height of the target S cannot be estimated with high accuracy.

[0063] To prevent the above inconveniences, the applicant has focused on the fact that the lower end Pah of the object S lying on the flat surface A is located on the flat surface A. In FIG. 8(b), the lower part Sh of the head of the object S that is missing from the head image Gs is shown by a dashed line. As described above, the reference plane Fa is generated at a position corresponding to the flat surface A. Therefore, even if the lower part Sh of the object S is not included in the head image Gs, it can be assumed that the point corresponding to the lower end Pah of the lower part Sh in the three-dimensional image space is located on the reference plane Fa.

[0064] In consideration of the above circumstances, this embodiment employs a configuration in which the bottom surface Fh1 of the reference box Ch is located on the reference surface Fa. In other words, the above configuration is a configuration in which the lower portion Sh of the target S is assumed to be in contact with the flat surface A at the lower end Pah, and the volume of the target S including the lower portion Sh that is missing in the target image Gs is estimated using the target image Gs.

[0065] Fig. 8(c) is a diagram for explaining the configuration for generating the adjustment box Ci. Fig. 8(c) shows a conceptual diagram of the adjustment box Ci as viewed from the Z-axis direction side (top side). The adjustment box Ci is generated so that it can accommodate the head model Mh.

[0066] As shown in the right part of FIG. 8(c), the top surface Fi6 of the adjustment box body Ci passes through the point Qi6 of the head model Mh where the Z coordinate is maximum. Also, as shown in the left part of FIG. 8(c), the side surface Fi2 of the adjustment box body Ci passes through the point Qi2 of the head model Mh where the X coordinate is maximum, and the side surface Fi5 of the adjustment box body Ci passes through the point Qi5 of the head model Mh where the X coordinate is minimum. The side surface Fi3 of the adjustment box body Ci passes through the point Pi3 of the head model Mh where the Y coordinate is maximum, and the side surface Fi4 passes through the point Pi4 of the head model Mh where the Y coordinate is minimum. The head model Mh is a part of the target S including the lower part. head As shown in Fig. 8(c), the bottom surface Fi1 of the adjustment box body Ci passes through the point Qi1 of the head model Mh where the Z coordinate is the smallest.

[0067] In the head model deformation process, the information processing device 10 deforms the head model Mh according to the size (shape) of the reference box body Ch. Specifically, the information processing device 10 calculates the ratio Rhix (=Lhx / Lix) between the length Lhx of the reference box body Ch in the X-axis direction shown in FIG. 8(b) and the length Lix of the adjustment box body Ci in the X-axis direction shown in FIG. 8(c). The information processing device 10 also calculates the ratio Rhiy (=Lhy / Liy) between the length Lhy of the reference box body Ch in the Y-axis direction shown in FIG. 8(b) and the length Liy of the adjustment box body Ci in the Y-axis direction shown in FIG. 8(c). Similarly, the information processing device 10 calculates the ratio Rhiz (=Lhz / Liz) between the length Lhz of the reference box body Ch in the Z-axis direction shown in FIG. 8(b) and the length Liz of the adjustment box body Ci in the Z-axis direction shown in FIG. 8(c).

[0068] After determining the ratios Rhix, Rhiy, and Rhiz (sometimes collectively referred to as "ratio Rhi"), the information processing device 10 deforms (enlarges or reduces) the head model Mh using the ratio Rhi. Specifically, the information processing device 10 multiplies the head model Mh by Rhix in the X-axis direction, by Rhiy in the Y-axis direction, and by Rhiz in the Z-axis direction. According to the head model deformation process described above, even if the lower part of the target S is not photographed, by generating the reference box body Ch on the assumption that the lower end Pah of the target S is in contact with the flat surface A, the shape of the head model Mh can be more easily approximated to the shape of the head of the target S with high accuracy.

[0069] 9(a) and 9(b) are diagrams for explaining the details of the torso model deformation process. In the torso model deformation process, the shape of the torso model Mb is deformed using the reference box Cb and the adjustment box Cc, similar to the head model deformation process described above.

[0070] However, the subject S in this embodiment is assumed to be a newborn, and newborns often wear disposable diapers. In this case, the waist image Gw in the torso image Gb has a large area representing the disposable diaper covering the waist of the newborn. Taking the above into consideration, the reference box Cb in this embodiment is generated based on the chest image Gc in the torso image Gb.

[0071] FIG. 9(a) is a diagram illustrating the reference box Cb. The reference box Cb is a rectangular parallelepiped and is generated so that it can accommodate a chest image Gc. Specifically, the reference box Cb is composed of a bottom surface Fb1, side surfaces Fb2 to Fb5, and a top surface Fb6. As shown in FIG. 9(a), each of the above surfaces Fb (1 to 6) is parallel to one of the XY plane, the XZ plane, or the YZ plane. Specifically, of the surfaces Fb (1 to 6), the bottom surface Fb1 and the top surface Fb6 are parallel to the XY plane, the side surfaces Fb2 and Fb5 are parallel to the YZ plane, and the side surfaces Fb3 and Fb4 are parallel to the XZ plane.

[0072] 9(a), the top surface Fb6 contacts the point Pb6 on the chest image Gc where the Z coordinate is the maximum. The side surface Fb2 contacts the point Pb2 on the chest image Gc where the X coordinate is the minimum, and the side surface Fb5 contacts the point Pb5 on the chest image Gc where the X coordinate is the maximum. Similarly, the side surface Fb4 contacts the point Pb4 on the chest image Gc where the Y coordinate is the minimum, and the side surface Fb3 contacts the point Pb3 on the chest image Gc where the Y coordinate is the maximum.

[0073] As described above, when the subject S is photographed from above (Z-axis side), the lower part of the subject S is not photographed. For these reasons, the lower part of the subject S is missing from the chest image Gc. In Fig. 9(a), the lower part Sb of the subject S's chest that is missing from the chest image Gc is indicated by a dashed line.

[0074] However, when the subject S lies on the flat surface A, the lower part of the chest of the subject S is usually in contact with the flat surface A. Taking the above into consideration, in this embodiment, a reference box Cb is generated assuming that the lower end Pac of the chest of the subject S is located on the flat surface A (similar to the reference box Ch shown in FIG. 8(b) above).

[0075] 9(a), the lower portion Sc of the chest of the subject S is missing from the chest image Gc. In consideration of the above circumstances, in this embodiment, the bottom surface Fb1 of the reference box Cb is generated on the reference plane Fa (flat surface A) where the actual lower end Pac of the chest is located, instead of the lower end of the chest image Gc. With this configuration, the weight or height of the subject S can be estimated with higher accuracy, compared to, for example, a configuration in which the bottom surface of the reference box Cb passes through the lower end of the chest image Gc.

[0076] FIG. 9(b) is a diagram illustrating the adjustment box Cc. The adjustment box Cc is a rectangular parallelepiped and is generated so that it can accommodate the chest model Mc. Specifically, the adjustment box Cc is composed of a bottom surface Fc1, side surfaces Fc2 to Fc5, and a top surface Fc6. As shown in FIG. 9(b), each of the above surfaces Fc (1 to 6) is parallel to any of the XY plane, XZ plane, or YZ plane. Specifically, of the surfaces Fc (1 to 6), the top surface Fc6 and the bottom surface Fc1 are parallel to the XY plane, the side surfaces Fc2 and Fc5 are parallel to the YZ plane, and the side surfaces Fc3 and Fc4 are parallel to the XZ plane.

[0077] As shown in FIG. 9(b), of the surfaces Fc (1-6) of the adjustment box Cc, the top surface Fc6 is in contact with the point Qc6 of the chest model Mc where the Z coordinate is the maximum. In addition, the side surface Fc2 is in contact with the point Qc2 of the chest model Mc where the X coordinate is the minimum, and the side surface Fc5 is in contact with the point Qc5 of the chest model Mc where the X coordinate is the maximum. Similarly, the side surface Fc3 is in contact with the point Qc3 of the chest model Mc where the Y coordinate is the maximum, and the side surface Fc4 is in contact with the point Qc4 of the chest model Mc where the Y coordinate is the minimum. The chest model Mc is in contact with the object S including the lower part. chest Taking the above into consideration, the bottom surface Fc1 of the adjustment box body Cc passes through the lower end Qc1 where the Z coordinate of the chest model Mc is the smallest.

[0078] In the torso model deformation process, the information processing device 10 deforms the chest model Mc in accordance with the size (shape) of the reference box Cb. Specifically, the information processing device 10 calculates the ratio Rbcx (=Lbx / Lcx) between the length Lbx of the reference box Cb in the X-axis direction shown in FIG. 9(a) and the length Lcx of the adjustment box Cc in the X-axis direction shown in FIG. 9(b). The information processing device 10 also calculates the ratio Rbcy (=Lby / Lcy) between the length Lby of the reference box Cb in the Y-axis direction shown in FIG. 9(a) and the length Lcy of the adjustment box Cc in the Y-axis direction shown in FIG. 9(b). Similarly, the information processing device 10 calculates the ratio Rbcz (=Lbz / Lcz) between the length Lbz of the reference box Cb in the Z-axis direction shown in FIG. 9(a) and the length Lcz of the adjustment box Cc in the Z-axis direction shown in FIG. 9(b).

[0079] After determining the ratios Rbcx, Rbcy, and Rbcz (sometimes collectively referred to as "ratio Rbc"), the information processing device 10 deforms (enlarges or reduces) the torso model Mb (chest model Mc+waist model Mw) based on the ratio Rbc. Specifically, the information processing device 10 multiplies the torso model Mb by Rbcx in the X-axis direction, by Rbcy in the Y-axis direction, and by Rbcz in the Z-axis direction. According to the torso model deformation process described above, even if the lower part of the subject S is not photographed, by generating a reference box Cb assuming that the lower end Pac of the subject S's chest is in contact with the flat surface A, the shape of the torso model Mb can be deformed to closely approximate the shape of the subject S's torso with high accuracy.

[0080] FIG. 10(a) is a diagram for explaining the volume estimation process. In the volume estimation process, the volume of the target S is estimated. As described above, in this embodiment, the head model deformation process (see FIGS. 8(a) to (c)), the torso model deformation process (see FIGS. 9(a) and (b)), the arm model deformation process, and the leg model deformation process are performed to deform the head model Mh, the torso model Mb, the arm model Ma, and the leg model Ml so as to approximate the shapes of each part of the target S. The target model M is generated by combining each model after the above deformations. FIG. 10(a) shows a schematic diagram of the target model M.

[0081] In the volume estimation process, the information processing device 10 generates each cross section Mp shown in FIG. 10(a). Each cross section Mp is obtained by cutting the target model M parallel to the YZ plane while shifting the position in the X-axis direction. FIG. 10(a) shows two adjacent cross sections Mp. As shown in FIG. 10(a), adjacent cross sections Mp are separated by a "distance Δx" in the X-axis direction. The distance Δx is sufficiently small compared to the size of the target image Gs. When the distance Δx is sufficiently small, the volume V of the target model M is calculated (approximated) by the following equation 2. Note that "An" in equation 2 refers to the area of ​​the n-th cross section Mp from the tip of the target model M in the X-axis direction. Also, "N" in equation 2 refers to the total number of cross sections Mp.

[0082] (Number 2) JPEG0007756972000002.jpg2045

[0083] As described above, the shape (size) of the target model M is approximate to the shape of the actual target S. Therefore, the volume V of the target model M calculated in the volume estimation process can be estimated as the volume of the actual target S. Furthermore, the information processing device 10 calculates the weight W of the target S by multiplying the volume V of the target model M by the average density D.

[0084] FIG. 10(b) is a diagram for explaining the height estimation process. In the height estimation process, the height of the target model M is calculated, and the calculation result is estimated as the height of the target S. Specifically, in the height estimation process, the information processing device 10 detects predetermined feature points in the target model M. In the height estimation process of this embodiment, as shown in FIG. 10(b), the following points are detected in the target model M: point Pa corresponding to the top of the head of the target S, point Pb corresponding to the shoulder joints, point Pc corresponding to the hip joints, point Pd corresponding to the knee joints, and point Pe corresponding to the ankles. Note that, as a technology for detecting feature points in the target model M, for example, the technology described in Japanese Patent No. 6868875 can be adopted.

[0085] When the information processing device 10 detects feature points in the target model M, it calculates the horizontal distance Lab from point Pa corresponding to the top of the head to point Pb corresponding to the shoulder joint. The information processing device 10 also calculates the horizontal distance Lbc from point Pb corresponding to the shoulder joint to point Pc corresponding to the hip joint, the distance Lcd from point Pc corresponding to the hip joint to point Pd corresponding to the knee joint, and the distance Lde from point Pd corresponding to the knee joint to point Pe corresponding to the ankle.

[0086] Furthermore, the information processing device 10 calculates the sum of the distance Lab, the distance Lbc, the distance Lcd, and the distance Lde as the height H of the target model M. The height H of the target model M is estimated as the height of the target S represented by the target model M. The height estimation process can be modified as appropriate. For example, the feature points detected from the target model M are not limited to the above examples as long as they can measure the height of the target model M.

[0087] 11 is a flowchart of the image capturing process executed by the information processing device 10. As described above, the image capturing device 20 transmits image information Dg indicating a three-dimensional image G (including a target image Gs) to the information processing device 10. The information processing device 10 executes the image capturing process, for example, when it acquires (receives) the image information Dg. However, the trigger for executing the image capturing process can be changed as appropriate.

[0088] When the image capturing process starts, the information processing device 10 executes an image recognition process (S0). In the image recognition process, each object (including the target S) shown in the three-dimensional image G indicated by the image information Dg is recognized. After the image recognition process, the information processing device 10 executes a head size approximation process (S1). In the head size approximation process, the rough shape (size) of the head model Mh is determined (see FIG. 7(a) above). After executing the head size approximation process, the information processing device 10 also executes a head model rotation process. In the head model rotation process, the orientation of the head model Mh is rotated (adjusted) in accordance with the orientation of the head image Gh (see FIG. 7(b) above).

[0089] After executing the head model rotation process, the information processing device 10 executes the head model deformation process (S3). As described above, in the head model deformation process, the shape of the head model Mh is deformed so as to approximate the shape of the head of the target S with high accuracy (see FIGS. 8(a) to (c) above).

[0090] Furthermore, the information processing device 10 executes a torso image rotation process (S4). In the torso image rotation process, the torso image Gsb is rotated so as to be parallel to the X-axis direction. Thereafter, the information processing device 10 executes a torso model deformation process (S5). As described above, in the torso model deformation process, the shape of the torso model Mb is deformed so as to approximate the shape of the torso of the target S with high accuracy (see FIGS. 9(a) and 9(b) above).

[0091] The information processing device 10 executes an arm model deformation process (S6) to deform the shape of the arm model Ma so as to approximate the shape of the arm of the target S. In addition, the information processing device 10 executes a leg model deformation process (S7) to deform the shape of the leg model Ml so as to approximate the shape of the leg of the target S.

[0092] Thereafter, the information processing device 10 executes an object model generation process (S8), in which an object model M is generated by combining a head model Mh deformed in the most recent head model deformation process, a torso model Mb deformed in the torso model deformation process, an arm model Ma deformed in the arm model deformation process, and a leg model Ml deformed in the leg model deformation process.

[0093] After executing the object model generation process, the information processing device 10 executes a volume estimation process (S9). In the volume estimation process, the volume of the object model M generated in the most recent object model generation process is calculated (see FIG. 10(a) above). After executing the volume estimation process, the information processing device 10 executes a weight estimation process (S10). In the weight estimation process, the information processing device 10 multiplies the volume of the object model M calculated in the most recent volume estimation process by the average density, and stores the calculation result as the weight W of the object S.

[0094] After executing the weight estimation process, the information processing device 10 executes a height estimation process (S11). In the height estimation process, the height of the object model M generated in the most recent object model generation process is measured, and the measurement result is stored as the height H of the object S (see FIG. 10(b) above). After executing the height estimation process, the information processing device 10 executes a biological information display process (S12). In the biological information display process, the weight W of the object S calculated in the most recent weight estimation process and the height H of the object S calculated in the height estimation process are displayed on, for example, the monitor 104 of an incubator. After executing the biological information display process, the information processing device 10 ends the image capture process.

[0095] Second Embodiment Other embodiments of the present invention will be described below. In each of the following exemplary embodiments, elements whose actions and functions are equivalent to those of the first embodiment will be designated by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions thereof will be omitted where appropriate.

[0096] Medical procedures involving the insertion of a medical tube into a subject have been known for some time. For example, such medical procedures include the insertion of a gastric tube into a subject (including patients and newborns) who is unable to orally ingest food. In such medical procedures, the gastric tube is inserted through the nostril, passes through the esophagus, and reaches the inside of the stomach. Furthermore, in such medical procedures, it is necessary to use a gastric tube of an appropriate length so that the tip (the tube's exit) is positioned inside the stomach.

[0097] However, the length from the nostrils to the inside of the stomach varies depending on the subject. Therefore, the appropriate length of the gastric tube varies depending on the subject. Therefore, the appropriate length of the gastric tube has traditionally been estimated as the sum of the length from the ear canal to the space between the eyebrows and the length from the navel to the middle of the xiphoid process on the subject's surface.

[0098] However, when measuring the length on the surface of an object, it is necessary to touch the object. Furthermore, depending on the object, this action may be considered invasive. In consideration of the above, the second embodiment aims to make it possible to estimate the appropriate length of a medical tube (e.g., a gastric tube) to be inserted into the object without touching the object.

[0099] Fig. 12 is a conceptual diagram of a head model Mh and a torso model Mb according to the second embodiment. For the sake of explanation, Fig. 12 shows the head model Mh and the torso model Mb in combination. Fig. 12 is also a cross-sectional view of the head model Mh and the torso model Mb cut horizontally on the XZ plane.

[0100] In the second embodiment, similar to the first embodiment, a target image Gs (three-dimensional image) representing the target S is captured by the imaging device 20. Also, in the second embodiment, similar to the first embodiment, a head model Mh, a torso model Mb, an arm model Ma, and a leg model Ml are deformed based on the target image Gs to generate a target model M. Also, in the second embodiment, similar to the first embodiment, the volume of the target model M is estimated as the volume of the target S. Also, the weight of the target S is calculated from the estimated volume. In the second embodiment, similar to the first embodiment, the height of the target model M is estimated as the height of the target S.

[0101] As shown in FIG. 12 , the object model M in the second embodiment is configured to include a tube model Mt (Mth, Mtb). The tube model Mt represents a medical tube of an appropriate length to be inserted into the object S indicated by the combination of the head model Mh and the torso model Mb. Specifically, the tube model Mt represents the shape of a gastric tube (an example of a medical tube) when inserted into the object S. The gastric tube when inserted into the object S bends along the path from the nostrils through the esophagus to the stomach. The tube model Mt bends in the same way as the gastric tube when inserted into the object S.

[0102] The tube model Mt is provided in a region of the combination of the head model Mh and the torso model Mb that corresponds to a medical tube inserted into the subject S. For example, the tube model Mt representing a gastric tube is located in a region that corresponds to the path from the nostrils of the subject S through the esophagus to the inside of the stomach. Note that FIG. 12 shows a region St that corresponds to the stomach of the subject S. As shown in FIG. 12, the end of the tube model Mt on the torso model Mb side is located inside the region St.

[0103] As shown in FIG. 12, the tube model Mt includes a first tube model Mth provided inside the head model Mh and a second tube model Mtb provided inside the body model Mb. In the second embodiment, as in the first embodiment described above, a head model deformation process (see FIGS. 8(a) to 8(c)) is executed, and the head model Mh is deformed to the size of the head of the target S. In the head model deformation process, the first tube model Mth is deformed as an image integrated with the head model Mh. In the second embodiment, as in the first embodiment described above, a body model deformation process (see FIGS. 9(a) and 9(b)) is executed, and the body model Mb is deformed to the size of the body of the target S. In the body model deformation process, the second tube model Mtb is deformed as an image integrated with the body model Mb.

[0104] In the second embodiment described above, an object model M is generated that approximates the shape of the object S. The object model M also includes a tube model Mt (a combination of a first tube model Mth and a second tube model Mtb) that has been deformed (stretched) to the length to be inserted into the object S. Therefore, by calculating the length of the tube model Mt in the object model M, the length of the medical tube to be inserted into the object S can be determined.

[0105] <Third embodiment> In the first embodiment described above, the weight of the target S is estimated by generating the target model M and multiplying the volume of the target model M by the average density. However, the configuration for estimating the weight of the target S is not limited to the above example.

[0106] In the third embodiment, a target model M is generated in the same manner as in the first embodiment described above. Furthermore, in the third embodiment, the length of a predetermined portion of the target model M is identified as an explanatory variable X for estimating the weight of the target S. The explanatory variable X is the length of a predetermined portion of the target model M that has a causal relationship with the weight of the target S. In the third embodiment, n+1 (n is a positive integer) explanatory variables (X0, X1, X2, X3...Xn) are identified. For example, the body length, chest circumference, chest width, chest length, leg length, hand length, etc. of the target model M are identified as the explanatory variables X.

[0107] When the information processing device 10 of the third embodiment identifies each explanatory variable X, it substitutes the explanatory variable X into the following equation 3 to calculate (estimate) the weight W of the subject S. Note that the coefficients k0 to kn in equation 3 are determined by, for example, multiple regression analysis. However, the method for determining the coefficients k0 to kn is not limited to multiple regression analysis.

[0108] (Number 3) W=k0×X0+k1×X1+k2×X2+k3×X3+…+kn×Xn

[0109] According to the third embodiment described above, the same effects as those of the first embodiment described above can be achieved. Note that, in the third embodiment, as in the second embodiment, the object model M may also be configured to include a tube model Mt. Furthermore, the configuration for estimating the weight W of the object S using explanatory variables (X0, X1, X2, X3...Xn) is not limited to the above example. For example, a configuration may also be used that uses a machine learning model trained by machine learning using training data of the explanatory variable X and the weight W. Specifically, a random forest algorithm is used to generate machine learning models (multiple decision tree models). In the above configuration, when the object model M of the object S is generated, the explanatory variable X is identified from the object model M, and when the explanatory variable X is input to the machine learning model, the weight W of the object S is determined (estimated).

[0110] <Modification> The above embodiments can be modified in various ways. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples can be combined as appropriate.

[0111] (1) In each of the above-described embodiments, the arm model deformation process can be modified as appropriate. For example, as described above with reference to Fig. 6, the arm image Gsa includes a first arm image Gsa1 representing the area from the shoulder to the elbow of the target S and a second arm image Gsa2 representing the area from the elbow to the fingertips. Also, as shown in Fig. 6, the arm model Ma includes a first arm model Ma1 representing the area from the shoulder to the elbow of the target S and a second arm model Ma2 representing the area from the elbow to the fingertips.

[0112] In the arm model generation process, the information processing device 10 approximates the first arm image Gsa1 to a cylinder. Specifically, the information processing device 10 identifies the orientation of the first arm image Gsa1 (hereinafter referred to as "direction vector V") by principal component analysis. The information processing device 10 generates a cylinder (hereinafter referred to as "cylinder Cg") whose center line is parallel to the direction vector V and whose side surface substantially overlaps with the first arm image Gsa1, using the least squares method.

[0113] The information processing device 10 generates a cylinder (hereinafter referred to as "cylinder Cm") that approximates the first arm model Ma1 using a method similar to that used to generate the cylinder Cg from the first arm image Gsa1. The information processing device 10 calculates the ratio "Rg / Rm" by dividing the diameter Rg (thickness) of the base of the cylinder Cg by the diameter Rm of the base of the cylinder Cm, and enlarges or reduces the thickness of the first arm model Ma1 according to the ratio "Rg / Rm." The information processing device 10 also calculates the ratio "Tg / Tm" by dividing the height Tg (length) of the cylinder Cg by the height Tm of the cylinder Cm, and enlarges or reduces the length of the first arm model Ma1 according to the ratio "Tg / Tm." According to the above configuration, the shape of the first arm model Ma1 is deformed so as to approximate the shape of the upper arm of the subject S. Similarly, the shape of the second arm model Ma2 is deformed using the second arm image Gsa2.

[0114] (2) In each of the above-described embodiments, the leg model deformation process can be modified as appropriate. For example, as described above with reference to FIG. 6, the leg image Gsl includes a first leg image Gsl1 representing the leg from the groin to the knee and a second leg image Gsl2 representing the leg from the knee to the toe. These images are distinguished using segmentation techniques. Also, as shown in FIG. 6, the leg model Ml includes a first leg model Ml1 representing the thigh of the subject S and a second leg model Ml2 representing the leg from the knee to the toe.

[0115] The information processing device 10 deforms the shape of the first leg model Ml1 using the first leg image Gsl1 in a manner similar to the manner in which the shape of the first arm model Ma1 in the above-described modified example (1) was deformed. According to the above configuration, the shape of the first leg model Ml1 is deformed so as to approximate the shape of the thigh of the target S. Furthermore, the information processing device 10 deforms the shape of the second leg model Ml2 using the second leg image Gsl2 in a manner similar to the manner in which the shape of the first arm model Ma1 was deformed. According to the above configuration, the shape of the second leg model Ml2 is deformed so as to approximate the shape of the leg from the knee to the toe of the target S.

[0116] (3) In each of the above embodiments, the method for determining the reference plane Fa can be changed as appropriate. For example, assume a configuration in which the positional relationship between the imaging device 20 and the flat surface A is fixed. As an example of the above configuration, a configuration in which the imaging device 20 is fixed to an incubator is conceivable. In the above configuration, the distance from the imaging device 20 to the flat surface A is constant, and the reference plane Fa through which the flat image Ga passes is common regardless of the three-dimensional image G. In the above configuration, the reference plane Fa can be stored in advance in the information processing device 10. Therefore, there is an advantage in that the process for identifying the reference plane Fa from the flat image Ga can be omitted.

[0117] (4) In the first embodiment described above, the head size approximation process estimates the shape (size) of the head of the target S by approximating the head image Gsh to a sphere. Instead of the above configuration, a configuration may be adopted in which the head image Gsh is approximated to an ellipsoid.

[0118] 13(a) and 13(b) are diagrams illustrating a modified example in which the head image Gsh can be approximated by an ellipsoid. In this modified example, each cross section of the head image Gsh cut parallel to the ZX plane and at equal intervals in the Y-axis direction is approximated by an ellipse. The model Mh is then deformed (enlarged or reduced) to the size of the image generated by overlapping the approximated ellipses. FIG. 13(a) shows the head image Gsh before elliptical approximation. As described above, the portion of the head of the subject S facing the reference plane Fa (flat surface A) is not photographed and is therefore missing from the head image Gsh. Note that in FIG. 13(a), the shape of the head of the subject S missing from the head image Gsh is indicated by a dashed line.

[0119] Incidentally, subject S (newborn baby) may be touching his / her own head with his / her hand. Also, subject S inside an incubator may have medical tubes attached. In such cases, part of the head may be occluded by subject S's hand (arm) or tube, and the occluded part may be missing from the head image Gsh. In the specific example of Figure 13(a), a case is assumed in which part of the head is occluded by subject S's arm. In such cases, as shown in Figure 13(a), the head image Gsh below the arm image Gsa (on the opposite side in the Z-axis direction) is missing. Also, the arm image Gsa may be recognized as part of the head image Gsh.

[0120] If an image including an arm image Gsa and a head image Gsh is approximated by an ellipse, the approximated ellipse is likely to not match the actual shape of the head of the subject S. Therefore, in the above case, it is difficult to accurately approximate the actual shape of the head of the subject S. Estimate In consideration of the above circumstances, the modified example has a configuration that suppresses the above inconveniences. The above configuration will be described in detail below.

[0121] The ellipse is calculated by using five unknowns A to E. i 2 +AX i Y i +BY i 2 +CX i +DY i +E” (hereinafter referred to as “Equation E iTherefore, the coordinates of five or more points in the head image Gsh (X i ,Y i ) into the equation E i In this modified example, the coordinates of six points (X i ,Y i ) to find the equation E i In this modification, the combination of points in the head image Gsh is randomly changed to obtain n (for example, n=1000) ellipse equations E i Ask for.

[0122] Also, the equation of the ellipse is E i After calculating the number of points on the head image Gsh that are located in the ellipse (hereinafter referred to as the evaluation value V i In the above modification, the evaluation value V i The larger the ellipse, the more likely it is to approximate the shape of the head of the actual subject S. For example, in the specific example of FIG. 13(a), the combination of points Pa1 to Pa6 in the head image Gsh determines the ellipse equation E i is obtained, and the equation of another ellipse E is obtained by combining points Pb1 to Pb6. i 13(a), it is assumed that all of the points Pa1 to Pa6 are located on the head image Gsh, while the point Pb5 among the points Pb1 to Pb6 is located on the arm image Gsa, not on the head image Gsh. In this case, the ellipse obtained by combining the points Pa1 to Pa6 is likely to approximate the shape of the actual head of the target S. That is, the ellipse obtained by combining the points Pa1 to Pa6 has a large area that overlaps with the head image Gsh, and the evaluation value V i On the other hand, the ellipse obtained by combining the points Pb1 to Pb6 is difficult to approximate to the actual shape of the head of the target S. In other words, the ellipse obtained by combining the points Pb1 to Pb6 has a small area that overlaps with the head image Gsh, and the evaluation value V i tends to become smaller.

[0123] As can be understood from the above explanation, the evaluation value Vi The larger the ellipse, the more closely it resembles the shape of the head of the actual target S. Taking the above into consideration, in this modified example, the evaluation values ​​V of n ellipses are i Calculate the evaluation value V i The ellipse with the largest value is estimated to be the ellipse that represents the actual head shape of the subject S.

[0124] FIG. 13(b) is a flowchart of the ellipse approximation process. The ellipse approximation process calculates an equation E of an ellipse that approximates the shape of the head of the target S. i Specifically, when the ellipse approximation process starts, the information processing device 10 randomly determines six points P in the target image Gs (S101), and calculates an equation E of an ellipse that passes through the six points P. i (S102). In addition, the information processing device 10 calculates the equation E i The target image Gs is located on the ellipse i The number of points on the evaluation value V i (S103). Then, the information processing device 10 calculates the evaluation values ​​V i It is determined whether the evaluation value V i If the number of ellipses for which the evaluation value V is calculated is less than n (S104: No), the information processing device 10 repeatedly executes the above steps S101 to S104. i When the number of ellipses for which evaluation values ​​V i The equation of the ellipse with the largest i is saved (S105), and the ellipse approximation process is completed.

[0125] The length of the radius of the ellipse obtained by the ellipse approximation process should be half the length from the reference plane Fa to the vertex of the target image Gs (Lc in FIG. 13(a)). Therefore, the difference (error) between the length of the radius of the ellipse obtained by the ellipse approximation process and the length Lc may be calculated, and if the difference exceeds a predetermined threshold, an error may be determined. If an error is determined, the ellipse approximation process may be executed again. Furthermore, the equation E i The tilt of the ellipse may be calculated from the calculated tilt and the orientation of the ellipse may be adjusted according to the calculation result.

[0126] (5) In each of the above-described embodiments, the information of the target S to be estimated is not limited to weight and height. For example, the information processing device 10 (measurement unit 16) may be configured to estimate the head circumference, chest circumference, abdominal circumference, arm thickness, and leg thickness of the target S. Specifically, the information processing device 10 may be configured to measure the head circumference, chest circumference, abdominal circumference, arm thickness, and leg thickness of the generated target model M, and estimate each measurement result as the head circumference, chest circumference, abdominal circumference, arm thickness, and leg thickness of the target S.

[0127] <Summary of the functions and effects of the exemplary embodiment> <First aspect> The information processing device (10) of this embodiment includes an acquisition unit (11) that acquires image information (Dg) representing a three-dimensional image (G) of a target (S) lying on a flat surface (A) captured from above; a recognition unit (12) that recognizes a target image (Gs) representing the target among the objects displayed in the three-dimensional image; and an estimation unit (13) that uses the target image to estimate the shape of the target, including the missing lower portion of the target, assuming that the lower portion of the target is in contact with the flat surface. The estimated shape of the target is then used to calculate physical information (volume, weight, height, etc.). According to this embodiment, the physical information of the target can be estimated without touching the target. Furthermore, according to this embodiment, the physical information of the target can be estimated with high accuracy even from a target image in which the lower portion of the target is missing. Note that "estimating the shape of the target" in the present invention means that it is sufficient to be able to calculate the physical information of the target S, and is not limited to estimating the shape of the target S in detail.

[0128] <Second mode> The information processing device of this aspect includes a memory unit (14) that stores model information indicating the shape of a target model (such as a head model), a recognition unit recognizes a flat image (Ga) that represents a flat surface (A) in the three-dimensional image, and an estimation unit identifies a reference plane (Fa) that passes through the flat image, and deforms the model according to the ratio of the distance (Lhz, Lbz) in the vertical axis direction (Z axis direction) from a target point in the target image (Ph6 in FIG. 8(b), Pb6 in FIG. 9(a)) to the reference plane to the distance (Liz, Lcz) in the vertical axis direction from a corresponding point in the model that corresponds to the target point (Qi6 in FIG. 8(c), Qc6 in FIG. 9(b)) to the bottom end of the model, and estimates the shape of the deformed model as the shape of the target. According to this aspect, for example Compared to a configuration in which the physical information of a target is estimated using only an image of the target with the lower portion missing, this configuration makes it possible to estimate the physical information of the target with high accuracy.

[0129] <Third aspect> The information processing device of this aspect calculates the weight of the subject as physical information, and can estimate the weight of the subject without touching the subject.

[0130] <Fourth aspect> The information processing device of this aspect calculates the height of the target as physical information. According to this aspect, it is possible to estimate the height of the target without touching the target.

[0131] <Fifth aspect> The information processing device of this aspect (10) In the present embodiment, the model includes a tube model (Mt) representing a medical tube (e.g., a gastric tube) inserted into a subject, and the estimation unit is capable of deforming the model and the tube model together when deforming models other than the tube model (head model Mh, chest model Mb). According to this embodiment, it is possible to estimate the optimal length of the medical tube without touching the subject.

[0132] <Sixth aspect> The information processing device (10) of this embodiment includes an acquisition unit (11) that acquires image information (Dg) representing a three-dimensional image (G) of a subject (S) lying on a flat surface (A) captured from above, a recognition unit (12) that recognizes a target image (Gs) representing the subject and a flat image (Ga) representing the flat surface among images representing each object displayed in the three-dimensional image, a memory unit (14) that stores model information representing the shape of a model of the subject, and an estimation unit (13) that estimates the subject's weight. The estimation unit identifies a reference plane that passes through the flat image, deforms the model according to the ratio of the distance from a target point in the target image to the reference plane in the vertical direction to the distance from a corresponding point in the model corresponding to the target point to the bottom end of the model in the vertical direction, identifies the length of a predetermined portion of the deformed model as an explanatory variable for estimating the subject's weight, and estimates the subject's weight using the identified explanatory variable. This embodiment achieves the same effects as the third embodiment.

[0133] <Seventh aspect> The method for estimating the weight of a newborn according to this embodiment includes the steps of capturing a three-dimensional image of the newborn, estimating the shape of the newborn from the three-dimensional image using a computer (S1 to S9 in FIG. 11), and estimating the weight of the newborn from the estimated shape using a computer (S10 in FIG. 11). For a newborn, touching the newborn to measure its physical information using a scale can be considered an invasive procedure. According to this embodiment, the weight of the newborn can be estimated without touching the newborn, which has the advantage of reducing the number of invasive procedures. [Explanation of symbols]

[0134] 10...information processing device, 11...acquisition unit, 12...recognition unit, 13...estimation unit, 14...storage unit, 15...calculation unit, 16...measurement unit, 20...imaging device

Claims

1. an acquisition unit that acquires image information representing a three-dimensional image of an object lying on a flat surface captured from above; a recognition unit that recognizes a part of the three-dimensional image as an object image representing the object and recognizes another part of the three-dimensional image as a flat image representing the flat surface; calculating a reference plane that passes through the flat image and a position corresponding to the flat surface that is blocked by the object and missing from the flat image based on the position of the flat image; an estimation unit that estimates a shape of the object including a lower portion missing in the object image by using a distance in an up-down axis direction from the reference plane to an object point in the object image, Calculating physical information of the object using the estimated shape of the object; The object image and the flat image used in estimating the shape of the object are both obtained from one of the three-dimensional images. Information processing device.

2. a storage unit that stores model information indicating a shape of a model of the object; The estimation unit deforming the model according to a ratio of a distance in an up-down axis direction from a target point in the target image to the reference plane and a distance in an up-down axis direction from a corresponding point in the model corresponding to the target point to a lower end of the model; The shape of the model after deformation is estimated as the shape of the object. The information processing device according to claim 1 .

3. The body weight of the subject is calculated as the physical information.

3. The information processing device according to claim 1.

4. Calculating the height of the subject as the physical information 3. The information processing device according to claim 1.

5. the model includes a tube model representing a medical tube inserted into the subject; the estimation unit is capable of enlarging the tube model in accordance with enlarging the model other than the tube model, and is capable of reducing the tube model in accordance with reducing the model other than the tube model, The length of the medical tube required for the subject can be estimated from the length of the tube model after deformation. The information processing device according to claim 2 .

6. Identifying the length of a predetermined portion of the model after deformation as an explanatory variable for estimating the body weight of the subject; The identified explanatory variables are used to estimate the subject's weight. The information processing device according to claim 2 .

7. capturing a three-dimensional image of a newborn lying on a flat surface; recognizing a portion of the three-dimensional image as a target image representing the newborn and recognizing another portion of the three-dimensional image as a flat image representing the flat surface; A step of calculating, by a computer, a reference plane that passes through the flat image based on the position of the flat image and passes through a position corresponding to the flat surface that is blocked by the newborn and is missing from the flat image; a step of estimating, by a computer, the shape of the newborn, including a lower portion missing in the target image, using a distance in the up-down axis from the reference plane to the target point in the target image; and a step of estimating physical information of the newborn from the estimated shape by a computer, The object image and the flat image used in estimating the shape of the newborn are both obtained from one of the three-dimensional images. A method for estimating the physical information of newborns.

Images