Image processing device

Abstract

To detect the position and the posture of a probe with a high degree of precision while reducing a feeling of discomfort given by an outer appearance without deteriorating workability of a user.SOLUTION: An image processing device 100 includes: a probe 11 for scanning a subject to acquire a cross-sectional image; a relief part 30 formed by simulating the convexoconcave of the face of a person, which has the eye, ear, and nose, and is disposed in the probe in a non-movable manner; an imaging part 20 for imaging the subject and the relief part; a position calculation part 123 for calculating a subject feature point indicating a skeleton shape of the imaged subject and position coordinates of a relief feature point indicating a skeleton shape of the relief part respectively; a position and posture calculation part 124 for calculation a scanning position and a scanning posture of the probe relative to the subject on the basis of the calculated position coordinates; and an image conversion part 125 for converting the cross-sectional image acquired by the probe on the basis of the calculated scanning position and scanning posture of the probe relative to the subject to a three-dimensional image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device that processes cross-sectional images obtained by measuring a subject and converts them into three-dimensional images. [Background technology]

[0002] One known example of this type of device is one that converts a cross-sectional image obtained by ultrasound measurement into a three-dimensional image based on the three-dimensional position of a probe during ultrasound measurement (see, for example, Patent Document 1). In the device described in Patent Document 1, a doll is attached to the probe as a marker for detecting the position and orientation of the probe, and feature points of the body joints and face of the subject and the doll are detected in images of the subject and the doll, the position and orientation of the probe relative to the subject are estimated based on the position coordinates of these points, and the cross-sectional image is converted into a three-dimensional image based on the estimation results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-127629 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if a doll is attached to the probe as a marker, as in the device described in Patent Document 1, it may reduce the user's workability when performing ultrasound measurements, and the appearance of the probe may cause discomfort to the user or subject. [Means for solving the problem]

[0005] An image processing device according to one aspect of the present invention includes: a probe that scans a subject to obtain a cross-sectional image; a relief portion that is formed to resemble the contours of a human face and has at least a pair of eyes, a pair of ears, and a nose, and is immovably attached to the probe; an imaging unit that images the subject and the relief portion; a position calculation unit that calculates the position coordinates of subject feature points that indicate the skeletal shape of the subject imaged by the imaging unit and relief feature points that indicate the skeletal shape of the relief portion; a position and orientation calculation unit that calculates the scanning position and scanning orientation of the probe relative to the subject based on the position coordinates calculated by the position calculation unit; and an image conversion unit that converts the cross-sectional image obtained by the probe into a three-dimensional image based on the scanning position and scanning orientation of the probe relative to the subject calculated by the position and orientation calculation unit. [Effects of the Invention]

[0006] According to the present invention, the position and orientation of a probe can be detected with high accuracy without reducing the user's workability and while reducing the sense of discomfort caused by the appearance. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram schematically showing a measuring instrument to which an image processing apparatus according to an embodiment of the present invention is applied; [Figure 2A] FIG. 2 is a diagram for explaining a cross-sectional image acquired along a certain slice direction by a measuring device such as a CT. [Figure 2B] 2A to 2C are diagrams for explaining cross-sectional images acquired by changing the slice direction using the measuring instrument of FIG. 1; [Figure 3] 1 is a block diagram showing the configuration of a main part of an image processing apparatus according to an embodiment of the present invention; [Figure 4A] FIG. 4 is a front view of the imaging unit in FIG. 3. [Figure 4B] FIG. 4 is a rear view of the imaging unit in FIG. 3. [Figure 4C] FIG. 4 is a perspective view of the imaging unit in FIG. 3 . [Figure 5] FIG. 4 is a diagram showing an example of a node set by the two-dimensional position calculation unit in FIG. 3; [Figure 6A] FIG. 2 is a diagram for explaining how to hold the probe in FIG. 1. [Figure 6B] 2 is a diagram showing an example of a relief portion provided as a marker on the probe of FIG. 1; FIG. [Figure 6C] FIG. 6C is an enlarged view of the relief portion of FIG. 6B. [Figure 7A] 4 is an explanatory diagram of two-dimensional positions showing an example of a two-dimensional image acquired by the camera of FIG. 3. [Figure 7B] 4 is an explanatory diagram of three-dimensional positions showing an example of a two-dimensional image acquired by the camera of FIG. 3. [Figure 8] 4 is a flowchart showing an example of processing executed by the image processing apparatus according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] FIG. 1, which will be used to explain an embodiment of the present invention with reference to FIGS. 1 to 8, is a diagram schematically illustrating a measuring device 10 to which an image processing device 100 according to an embodiment of the present invention is applied. The measuring device 10 is, for example, an ultrasound measuring device that acquires cross-sectional information of the body of a subject A and generates and displays medical images such as cross-sectional images. As shown in FIG. 1, the measuring device 10 includes a probe 11 that transmits and receives ultrasound while scanning the surface of the body of the subject A, a computer 12 connected to the probe 11 by wire or wirelessly, and a display unit 13 connected to the computer 12.

[0009] The probe 11 converts the received waves containing cross-sectional information of the body of the subject A into electrical signals and transmits them to the computer 12. The computer 12 converts the electrical signals received from the probe 11 into a tomographic image (a cross-sectional image; a grayscale image according to the intensity of the electrical signal) showing the internal structure of the subject A, in particular the boundaries of the organs of the subject A, and transmits it to the display unit 13. The user performing the ultrasound measurement performs the ultrasound measurement by continuously changing the scanning position and scanning posture of the probe 11 while checking the cross-sectional image displayed on the display unit 13.

[0010] The probe 11 contains a transducer, an acoustic matching layer, a backing material, an acoustic lens, etc. in a case. A linear electronic scanning probe, a sector electronic scanning probe, or a mechanically scanned or manually scanned probe can be used as the probe 11. The cross-sectional images acquired by continuously changing the scanning position and scanning posture of the probe 11 will be described below as a plurality of time-series cross-sectional images IS.

[0011] FIG. 2A is a diagram for explaining a cross-sectional image IS acquired along a fixed slice direction by a measuring instrument such as a CT (Computed Tomography), and FIG. 2B is a diagram for explaining a cross-sectional image IS acquired by changing the slice direction (measurement direction) by a measuring instrument 10 such as an ultrasonic measuring instrument.

[0012] As shown in the example of FIG. 2A, a measuring device such as a CT scan acquires cross-sectional images IS along a certain slice direction as indicated by the arrow PM. In such a case, a number of feature points (eight in the example shown) BP1 to BP8 can be set on the boundaries BL of the organs of subject A in each cross-sectional image IS by a well-known superimposition process, and the feature points BP1 to BP8 of each cross-sectional image IS can be associated with each other. That is, as shown in FIG. 2A, the cross-sectional images IS, which are two-dimensional grayscale images, can be superimposed and displayed as three-dimensional perspective views in a virtual three-dimensional space, or can be converted into three-dimensional images showing the boundaries of the organs of subject A.

[0013] Furthermore, when measurements are taken while rotating the body of subject A in the direction of arrow RM in Fig. 2A, the feature points BP1 to BP8 of each cross-sectional image IS can be accurately associated with each other and superimposed by using, for example, affine transformation. Also, when the measurement target is an organ whose shape changes, such as the lungs, the feature points BP1 to BP8 of each cross-sectional image IS can be accurately associated with each other and superimposed by using, for example, nonlinear transformation.

[0014] On the other hand, a measuring device 10 such as an ultrasonic measuring device performs measurements by rotating the axis of a probe 11 along the irregularities of the surface of the body of subject A, or by scanning while pressing the probe 11 against the surface of the body of subject A. Therefore, as shown in the example of FIG. 2B, cross-sectional images IS are acquired in random slice directions (measurement directions).

[0015] In such a case, as shown in Fig. 2B, the cross-sectional images IS can be displayed superimposed as 3D perspective views after correcting the position and angle of the cross-sectional images IS in the virtual 3D space, taking into account the scanning position SP and scanning posture SN of the probe 11 relative to the subject A. This allows the feature points BP1 to BP8 of each cross-sectional image IS to be accurately associated with each other, and the superimposition process can be performed.

[0016] Ultrasound measurement is suitable for remote medical consultations, such as those conducted at the patient's home, because the measuring device 10 is relatively small and portable, and there is no risk of exposure to ionizing radiation. Furthermore, the high spatial resolution and high tissue contrast in the cross-sectional image IS make it convenient for users conducting remote medical consultations. However, the cross-sectional image IS obtained by ultrasound measurement has a limited field of view, making it difficult for inexperienced users to interpret. Even under similar measurement conditions, the cross-sectional image changes depending on the scanning position SP and scanning posture SN of the probe 11 relative to the patient A, as shown in Figure 2B. Therefore, even under similar measurement conditions, the actual cross-sectional image IS often differs from the typical cross-sectional image published in a medical textbook, for example.

[0017] By converting a plurality of actually acquired cross-sectional images IS into a three-dimensional image, it becomes possible to convert into cross-sectional images IS acquired from any scanning position SP and scanning posture SN, making it easy for even inexperienced users to understand. Therefore, the image processing device 100 according to the embodiment of the present invention is configured as follows to perform a process of superimposing the cross-sectional images IS taking into account the scanning position SP and scanning posture SN of the probe 11 relative to the subject A.

[0018] Fig. 3 is a block diagram showing the main configuration of an image processing device 100. As shown in Fig. 3, the image processing device 100 includes a measuring device 10 for ultrasound measurement and an imaging unit 20 that images a subject A and a probe 11 for ultrasound measurement. As shown in Fig. 3, the imaging unit 20 includes two cameras 21a and 21b, computers 22a and 22b connected to the cameras 21a and 21b, respectively, by wire or wirelessly, a display unit 23 connected to the cameras 21a and 21b by wire or wirelessly, and a case 24. Furthermore, the computers 22a and 22b of the imaging unit 20 are connected to the computer 12 of the measuring device 10 by wire or wirelessly.

[0019] Fig. 4A is a front view of the imaging unit 20, Fig. 4B is a rear view of the imaging unit 20, and Fig. 4C is a perspective view of the imaging unit 20. The cameras 21a and 21b are configured by compact cameras capable of capturing video. As shown in Fig. 4A, the cameras 21a and 21b are arranged as stereo cameras that capture images in the front direction of the imaging unit 20 from positions at the same height and spaced a predetermined distance L (approximately 10 cm or more, for example, about 20 cm) apart from each other in the horizontal direction (left and right direction).

[0020] The cameras 21a and 21b are attached to the case 24, for example, via holes 24a and 24b provided in the case 24. Two-dimensional images (image data) captured by the cameras 21a and 21b are transmitted to the computers 22a and 22b and the display unit 23. When an inexperienced user performs remote medical treatment, the image data can be transmitted to a user terminal of a skilled technician or the like in a remote location, and the technician can receive advice in real time on the appropriate scanning position and scanning posture of the probe 11.

[0021] The computers 22a and 22b are small computers. For example, single-board computers such as a Raspberry Pi (registered trademark) can be used as the computers 22a and 22b. The calculation results by the computers 22a and 22b are sent to the computer 12 of the measuring instrument 10. As shown in FIGS. 4A and 4C, the computers 22a and 22b are attached to the case 24 from inside the case 24, for example.

[0022] Display unit 23 is configured by, for example, the display of a commercially available tablet terminal. Display unit 23 displays images transmitted from cameras 21a and 21b. By checking the images displayed on display unit 23, the user can confirm the shooting range of imaging unit 20 and adjust the tilt angle and installation position of imaging unit 20 as necessary.

[0023] The case 24 is configured, for example, by a case for a commercially available tablet terminal, whose tilt angle can be adjusted. As shown in FIG. 4C, the imaging unit 20 can be installed at an appropriate tilt angle via the case 24. Furthermore, by installing the imaging unit 20 on a movable stand whose height can be adjusted, it can be installed in an appropriate position. The tilt angle and installation position of the imaging unit 20 are adjusted so that the subject A and the probe 11 are included in the imaging range, as shown in FIG. 1.

[0024] 3, the computers 22a and 22b are configured as arithmetic processing devices having CPUs 220a and 220b, memories 221a and 221b such as ROM and RAM, other peripheral circuits, etc. The CPUs 220a and 220b function as two-dimensional position calculation units 222a and 222b.

[0025] Based on the image data transmitted from the cameras 21a and 21b, the two-dimensional position calculation units 222a and 222b set a plurality of nodes corresponding to feature points indicating the skeletal shape of the subject A, etc. in a plurality of two-dimensional images in time series, and calculate the two-dimensional coordinates of each node. The setting of each node and the calculation of the coordinates are performed using, for example, a feature point detection algorithm based on deep learning, such as the OpenPose library (CVPR 2017, Carnegie Mellon University, Zhe Cao et al.).

[0026] The OpenPose library detects the position of each joint based on the relationship between feature points, such as the distance between feature points and their relative movement. This allows accurate feature detection even when there are multiple subjects in the same image, when body parts are hidden, when subjects enter and exit the screen, or when the subject is in a posture other than standing. The OpenPose library requires a relatively small amount of memory, and can be stored in the memories 221a and 221b (e.g., about 1 GB) of single-board computers 22a and 22b, such as Raspberry Pi®, allowing processing using the feature point detection algorithm to be performed on the edge.

[0027] Libraries other than OpenPose can be used for pose estimation using deep learning, but they are not suitable for use in remote medical consultations because of the communication costs involved in sending image data to a remote server where the library is stored. Furthermore, they are not suitable for use in remote medical consultations from the perspective of protecting the privacy of subject A. If it is not necessary to detect the positions of subject A's body joints, it is also possible to use a library (such as Dlib) that specializes in detecting facial feature points and requires a relatively small amount of memory.

[0028] Fig. 5 is a diagram showing an example of nodes set by the two-dimensional position calculation units 222a and 222b. As shown in the example of Fig. 5, the two-dimensional position calculation units 222a and 222b detect feature points that indicate the skeletal shape of a person in a two-dimensional image, including body joints and facial feature points such as eyes, ears, and nose, and set nodes Nn corresponding to these feature points (18 nodes N1 to N18 in the example of Fig. 5). Furthermore, the two-dimensional position calculation units 222a and 222b calculate the two-dimensional position of each node Nn in the two-dimensional image. The two-dimensional position calculation units 222a and 222b that detect the body joints and facial feature points of a person as shown in Fig. 5 can also similarly detect feature points for dolls, reliefs, etc. that resemble people and have these feature points.

[0029] FIG. 6A is a diagram illustrating how to hold the probe 11. When performing ultrasound measurement, the probe 11 is lightly held in one hand to ensure smooth scanning with the probe 11. Specifically, as shown in FIG. 6A, the side of the probe 11 is supported with the fingers, and the probe 11 is held so that there is a space between the crotch between the thumb and index finger and the handle 11a of the probe 11, and ultrasound measurement is performed while scanning the surface of the body of the subject A with the scanning surface 11b. The part that the user who performs ultrasound measurement supports and holds with their fingers is referred to as the grip portion 11c below. The grip portion 11c is a portion of the side of the probe 11 close to the scanning surface 11b, and has an inclined surface whose diameter decreases in the direction away from the scanning surface 11b (toward the handle 11a).

[0030] The scanning position SP of the probe 11 shown in Fig. 2B is the center point of the scanning plane 11b shown in Fig. 6A. The scanning attitude SN of the probe 11 shown in Fig. 2B is a direction vector along the central axis SA of the probe 11 that passes through the scanning position SP and points in a direction away from the probe 11. The scanning attitude SN of the probe 11 corresponds to the direction in which the probe 11 transmits and receives ultrasonic waves during ultrasonic measurement.

[0031] Fig. 6B is a diagram showing an example of a relief portion 30 provided on the probe 11 as a marker for detecting the scanning position SP and scanning attitude SN of the probe 11, and Fig. 6C is an enlarged view of the relief portion 30. As shown in Figs. 6B and 6C, the relief portion 30 is formed to resemble the contours of a person's face and is immovably provided on the probe 11. The relief portion 30 may be formed directly on the probe 11, or may be formed on an attachment 40 that is detachably attached to the probe 11 as shown in Figs. 6B and 6C.

[0032] The relief portion 30 includes a plurality of (for example, four) relief portions 30a to 30d formed to resemble human faces with different expressions. Each of the relief portions 30a to 30d is provided, for example, on each of a plurality of (for example, four) side surfaces of the probe 11. When the side surface of the probe 11 is configured with a single curved surface, the relief portions 30a to 30d are provided, for example, at equal intervals in the circumferential direction centered on the central axis SA of the probe 11. When the relief portion 30 includes a plurality of pairs (for example, two pairs) of the relief portions 30a to 30d, the relief portions 30a to 30d are provided, for example, so that each pair of relief portions is symmetrical with respect to the central axis SA of the probe 11.

[0033] 6C, the relief portion 30 has characteristic points of a person's face, including at least a nose portion 31, a pair of eyes 32 and 33, and a pair of ears 34 and 35. Since a plurality of the characteristic points 31 to 35 exist within a certain range corresponding to the size of the person's face, the orientation of the relief portion 30 can be estimated in real time with high accuracy by detecting the positions of the characteristic points 31 to 35 using the two-dimensional position calculation portions 222a and 222b.

[0034] As shown in FIGS. 6B and 6C, the attachment 40 is detachably attached to the grip portion 11c of the probe 11. The attachment 40 is made of, for example, a flexible resin member (thermoplastic resin) and is formed to fit into the grip portion 11c of the probe 11. The attachment 40 can be produced, for example, using a 3D printer. That is, the attachment 40 having the relief portions 30 (30a to 30d) as shown in FIG. 6C can be produced using a 3D printer. The attachment 40 is produced using a single-color 3D printer filament made of thermoplastic resin so that the entire attachment is a single color (for example, white), more specifically, so that it is visible as a single color in a normally bright place.

[0035] Preferably, the attachment 40 is formed by combining multiple parts (e.g., a first part including the relief portions 30a and 30b and a second part including the relief portions 30c and 30d) that can be separated in a direction along the central axis SA of the probe 11. In this case, the attachment 40 made of a flexible resin member can be attached to the grip portion 11c of the probe 11 while the probe 11 and the computer 12 are connected by a cable or the like. The attachment 40 may be made of an elastic member such as rubber. In this case, the attachment 40 can be attached to the grip portion 11c of the probe 11 while the probe 11 is connected to the computer 12 by a cable or the like, without separating it into multiple parts. Furthermore, the attachment 40 made of an elastic member can be attached to the grip portion 11c of probes 11 having different shapes.

[0036] Providing the concave-convex relief portion 30 on the grip portion 11c of the probe 11 improves the grip feeling when a user holding the probe 11 performing ultrasonic measurement, thereby improving operability. The attachment 40 is formed along the shape of the grip portion 11c and has an inclined surface that tapers away from the scanning surface 11b of the probe 11 and toward the handle 11a. In other words, the attachment 40 has an inclined surface that tapers away from the handle 11a of the probe 11 and toward the scanning surface 11b. During ultrasonic measurement, the attachment 40 is pressed toward the scanning surface 11b by the fingers of a user holding the attachment 40 to perform ultrasonic measurement. However, because the inclined surface tapers toward the scanning surface 11b, the attachment 40 does not shift out of position. In other words, movement of the attachment 40 along the central axis SA of the probe 11 is restricted by the inclined surface.

[0037] In the relief portion 30, either the feature points 31 to 35 or the portion other than the feature points 31 to 35 may be made of a material that is visible in the dark or a material that emits ultraviolet (near ultraviolet) light. The material that is visible in the dark may be, for example, a phosphorescent material that absorbs light energy with an excitation wavelength (approximately 250 to 450 nm) around the ultraviolet region in bright light, stores light, and emits light energy with a wavelength (approximately 400 to 550 nm) in the visible light region. In this case, for example, the attachment 40 is fabricated using a white non-phosphorescent filament for the portion other than the feature points 31 to 35, and a phosphorescent filament that appears white in bright light for the feature points 31 to 35. Alternatively, the attachment 40 is fabricated using a white non-phosphorescent filament for the feature points 31 to 35, and a phosphorescent filament that appears white in bright light for the portion other than the feature points 31 to 35.

[0038] Luminous paint may be used instead of the luminous filament. In this case, for example, after the attachment 40 is manufactured using only the non-luminous filament, luminous paint is applied to the characteristic points 31 to 35 or to the portions other than the characteristic points 31 to 35. Alternatively, after the attachment 40 is manufactured using only the luminous filament, non-translucent paint is applied to the characteristic points 31 to 35 or to the portions other than the characteristic points 31 to 35.

[0039] When a luminous material is used for the relief portion 30, a black light capable of irradiating ultraviolet light with a wavelength of about 365 nm may be used in combination. In this case, continuous visibility of the luminous material in a dark place can be ensured by irradiating the relief portion 30 with ultraviolet light from the black light. A tape-shaped black light (LED) may be interposed between the grip portion 11c of the probe 11 and the attachment 40, and ultraviolet light may be irradiated onto the relief portion 30 from the inside.

[0040] The ultraviolet ray emitting member may be, for example, a combination of a light-transmitting member and a tape-shaped black light. In this case, for example, the attachment 40 is fabricated using a non-light-transmitting filament for the portion other than the characteristic points 31 to 35, and a light-transmitting filament for the portion other than the characteristic points 31 to 35. Alternatively, the attachment 40 is fabricated using a light-transmitting filament for the portion other than the characteristic points 31 to 35, and a non-light-transmitting filament for the portion other than the characteristic points 31 to 35. The tape-shaped black light is then interposed between the grip portion 11c of the probe 11 and the attachment 40, and ultraviolet rays are irradiated onto the relief portion 30 from the inside.

[0041] The imaging unit 20 images the subject A during ultrasound measurement and the probe 11 including the relief portion 30. When either the feature points 31 to 35 or the portion other than the feature points 31 to 35 of the relief portion 30 are made of a material that emits ultraviolet light, the imaging unit 20 is configured to receive ultraviolet light and capture an image. That is, cameras having imaging elements compatible with the near-ultraviolet region are used as the cameras 21a and 21b of the imaging unit 20, and if a filter that blocks ultraviolet light is installed, the filter is removed before use.

[0042] Ultrasound measurement is usually performed in a dark place. Therefore, by forming either the feature points 31 to 35 of the relief portion 30 or the portion other than the feature points 31 to 35 from a material that is visible in a dark place or a material that emits ultraviolet light, the feature points 31 to 35 can be clearly photographed during ultrasound measurement. On the other hand, in a bright place, the entire attachment 40 appears monochromatic, making the feature points 31 to 35 of the relief portion 30 difficult for the user or subject performing the ultrasound measurement to see, thereby reducing the discomfort caused by the appearance. Note that using a material that emits infrared light instead of a material that emits ultraviolet light can also reduce the discomfort caused by the appearance in a bright place, but using a material that emits ultraviolet light rather than infrared light allows the imaging unit 20 to clearly photograph the feature points 31 to 35 in a dark place.

[0043] 7A is an explanatory diagram of two-dimensional positions showing an example of a two-dimensional image IMGa captured by camera 21a. In the example of FIG. 7A, the two-dimensional position calculation unit 222a detects subject feature points indicating the skeletal shape of subject A in the two-dimensional image IMGa, i.e., the body joints and facial feature points of subject A, sets corresponding nodes N1 to N10, and calculates the two-dimensional position of each node. Also, the two-dimensional position calculation unit 222a detects relief feature points indicating the skeletal shape of the relief portion 30 in the two-dimensional image IMGa, i.e., the facial feature points of the relief portion 30, sets corresponding nodes N31 to N35, and calculates the two-dimensional position of each node (enlarged view of part A). For example, the two-dimensional positions (x1a, y1a) to (x10a, y10a), (x31a, y31a) to (x35a, y35a) of each of the nodes N1 to N10 and N31 to N35 in the two-dimensional image IMGa are calculated in pixel units with the lower left corner of the two-dimensional image IMGa as the origin P0 (0, 0).

[0044] Similarly, two-dimensional position calculation unit 222b detects subject feature points that indicate the skeletal shape of subject A in two-dimensional image IMGb captured by camera 21b, sets corresponding nodes N1 to N10, and calculates the two-dimensional positions (x1b, y1b) to (x10b, y10b) of each node. Also, relief feature points that indicate the skeletal shape of relief portion 30 in two-dimensional image IMGb are detected, set corresponding nodes N31 to N35, and calculate the two-dimensional positions (x31b, y31b) to (x35b, y35b) of each node (enlarged view of part A).

[0045] The two-dimensional positions (x1a, y1a) to (x10a, y10a), (x31a, y31a) to (x35a, y35a), (x1b, y1b) to (x10b, y10b), (x31b, y31b) to (x35b, y35b) of each node N1 to N10, N31 to N35 calculated by the two-dimensional position calculation units 222a, 222b are transmitted to the computer 12.

[0046] 3, the computer 12 is configured as an arithmetic processing device having a CPU 120, a memory 121 such as a ROM and a RAM, and other peripheral circuits. The CPU 120 has, as functional components, a cross-sectional image generation unit 122, a three-dimensional position calculation unit 123, a position and orientation calculation unit 124, and an image conversion unit 125. Hereinafter, the two-dimensional position calculation units 222a and 222b of the computers 22a and 22b and the three-dimensional position calculation unit 123 of the computer 12 may be collectively referred to as a "position calculation unit."

[0047] The cross-sectional image generating unit 122 converts (generates) the electrical signal received from the probe 11 into a cross-sectional image IS (FIGS. 2A and 2B). The cross-sectional image IS generated by the cross-sectional image generating unit 122 is transmitted to the display unit 13.

[0048] Fig. 7B is an explanatory diagram of three-dimensional positions, and similarly to Fig. 7A, shows an example of two-dimensional image IMGa captured by camera 21a. Three-dimensional position calculation unit 123 calculates the three-dimensional positions (x1, y1, z1) to (x10, y10, z10), (x31, y31, z31) to (x35, y35, z35) of each node based on the two-dimensional positions of nodes N1 to N10 and N31 to N35 corresponding to the subject feature points and relief feature points transmitted from computers 22a and 22b.

[0049] That is, the three-dimensional position (xn, yn, zn) of each node Nn is calculated by the principle of triangulation based on the two-dimensional positions (xna, yna), (xnb, ynb) of each node Nn calculated individually from images taken by a pair of left and right cameras 21a, 21b configured as a stereo camera. More specifically, the left-right position xn and depth position zn of each node Nn in three-dimensional space are calculated based on the difference (parallax) between the left-right positions xna, xnb of each node Nn in images taken by cameras 21a, 21b that are spaced a predetermined distance L apart in the left-right direction at the same height. The height position yn of each node Nn in three-dimensional space is the same as the height positions yna, ynb of each node Nn in the images taken by cameras 21a, 21b.

[0050] It should be noted that it is also possible to estimate the posture of the probe 11 by providing a marker other than a human-shaped marker, such as a doll or relief, for example a triangular pyramid-shaped marker, as a marker for detecting the scanning position SP and scanning posture SN of the probe 11, but in this case the OpenPose library cannot be used. By using the relief portion 30 that resembles a person, it becomes possible to calculate the position and posture of the subject A and the probe 11 using a single algorithm, and the memory capacity and calculation load of the entire device can be reduced.

[0051] If posture estimation is performed by providing a marker on the probe 11 that does not have a shape resembling a person, a different algorithm must be used to detect the position of such a marker, which requires synchronization processing, etc. This increases the memory capacity and calculation load of the entire device. Also, although libraries other than OpenPose can be applied as libraries for posture estimation using deep learning, if multiple algorithms are used, it becomes necessary to redo the calculation of the positional relationship between subject A and the probe 11 each time the library is changed.

[0052] Instead of using a stereo camera and the OpenPose library, the three-dimensional positions of the subject A and the probe 11 can be detected using an infrared depth camera such as Kinect (registered trademark). However, this method involves projecting a dot pattern onto the subject (object) and detecting the reflected light, so accuracy decreases depending on environmental conditions such as the brightness of the room. In addition, multiple cameras cannot be used simultaneously, and the capture range is limited.

[0053] The position and orientation calculation unit 124 calculates the scanning position SP and scanning orientation SN of the probe 11 with respect to the subject A based on the three-dimensional positions of the nodes Nn corresponding to the subject feature points and relief feature points calculated by the three-dimensional position calculation unit 123. By providing multiple relief portions 30a to 30d as shown in FIG. 6C, it is possible to reliably photograph the relief portion 30 and calculate the two-dimensional and three-dimensional positions of the nodes N31 to N35 corresponding to the relief feature points, regardless of the positional relationship between the probe 11 and the imaging unit 20. That is, it is possible to photograph at least one of the multiple relief portions 30a to 30d and calculate the two-dimensional and three-dimensional positions of the relief feature points. Furthermore, since the expressions of the relief portions 30a to 30d are different from one another, it is possible to identify which of the relief portions 30a to 30d the photographed relief portion is, based on the relative positional relationships between the relief feature points. Therefore, the scanning position SP and scanning posture SN of the probe 11 with respect to the subject A can be calculated in real time with high accuracy based on the three-dimensional positions of the nodes Nn corresponding to the subject feature points and the relief feature points.

[0054] The image conversion unit 125 converts the cross-sectional images acquired by the probe 11 into three-dimensional images based on the scanning position SP and scanning orientation SN of the probe 11 relative to the subject A calculated by the position and orientation calculation unit 124. That is, as shown in Fig. 2B, the image conversion unit 125 corrects the position and angle of each cross-sectional image IS in the virtual three-dimensional space based on the scanning position SP and scanning orientation SN of the probe 11, and then displays the cross-sectional images IS superimposed as three-dimensional perspective views. As a result, the feature points (BP1 to BP8 in Fig. 2B) of each cross-sectional image IS are associated with each other, and superimposition processing is performed.

[0055] 8 is a flowchart showing an example of processing executed by the image processing device 100 in accordance with a pre-stored program. The processing shown in this flowchart is started, for example, when power is supplied to the image processing device 100, and is repeated at a predetermined interval. First, in step S1, image data of the cross-sectional image IS transmitted from the probe 11 is acquired by processing in the two-dimensional position calculation units 222a and 222b of the computers 22a and 22b. Next, in step S2, a plurality of nodes N1-N1-N10 and N31-N35 corresponding to the subject feature points and relief feature points in each cross-sectional image IS acquired in step S1 are set (detected).

[0056] Next, in step S3, the two-dimensional positions (x1a, y1a) to (x10a, y10a), (x31a, y31a) to (x35a, y35a), (x1b, y1b) to (x10b, y10b), and (x31b, y31b) to (x35b, y35b) of each of the nodes N1 to N1 to N10 and N31 to N35 detected in step S2 are calculated. Next, in step S4, the three-dimensional position calculation unit 123 of the computer 12 calculates the three-dimensional positions (x1, y1, z1) to (x10, y10, z10), (x31, y31, z31) to (x35, y35, z35) of each of the nodes N1 to N1 to N10 and N31 to N35 based on the two-dimensional positions of each node calculated in step S3.

[0057] Next, in step S5, the position and orientation calculation unit 124 performs processing to calculate the scanning position SP and scanning orientation SN of the probe 11 with respect to the subject A based on the three-dimensional position of each node calculated in step S4. Next, in step S5, the image conversion unit 125 performs processing to superimpose each cross-sectional image IS based on the scanning position SP and scanning orientation SN of the probe 11 calculated in step S4.

[0058] According to the embodiment of the present invention, the following advantageous effects can be achieved. (1) The image processing device 100 includes a probe 11 that scans a subject A to obtain a cross-sectional image IS, a relief portion 30 formed to resemble the contours of a human face and having at least a pair of left and right eyes 32, 33, a pair of left and right ears 34, 35, and a nose 31 and immovably provided on the probe 11, an imaging unit 20 that images the subject A and the relief portion 30, and position coordinates (x1, y1, z1) to (x10, y10, z2) of subject feature points N1 to N10 indicating the skeletal shape of the subject A imaged by the imaging unit 20 and relief feature points N31 to N35 indicating the skeletal shape of the relief portion 30. z10), (x31, y31, z31) to (x35, y35, z35), respectively; a position and orientation calculation unit 124 that calculates the scanning position SP and scanning orientation SN of the probe 11 with respect to the subject A based on the position coordinates calculated by the position calculation unit; and an image conversion unit 125 that converts the cross-sectional image IS acquired by the probe 11 into a three-dimensional image based on the scanning position SP and scanning orientation SN of the probe 11 with respect to the subject A calculated by the position and orientation calculation unit 124 (FIG. 3).

[0059] Because the scanning position SP and scanning posture SN of the probe 11 relative to the subject A are calculated, the cross-sectional image IS can be converted into a three-dimensional image even when ultrasound measurements are performed by rotating the axis of the probe 11 along the contours of the subject A's body surface or by continuously changing the subject A's posture. That is, the position and angle of the cross-sectional image IS in a virtual three-dimensional space are corrected taking into account the scanning position SP and scanning posture SN of the probe 11 relative to the subject A, and the cross-sectional image IS can be displayed superimposed as a three-dimensional perspective view. This allows the feature points BP1 to BP8 of each cross-sectional image IS to be accurately associated with each other (Figure 2B) and superimposed. Furthermore, the feature points of the subject A's body joints and face are not affected by the displacement caused by the pressing of the probe 11. Therefore, even when ultrasound measurements are performed by continuously displacing the surface of the subject A's body with the probe 11, the scanning position SP and scanning posture SN of the probe 11 relative to the subject A can be accurately calculated and the cross-sectional image IS can be converted into a three-dimensional image. Furthermore, since the positions and orientations of the subject A and the probe 11 are calculated using a single algorithm, the memory capacity and calculation load of the entire device can be reduced.

[0060] Furthermore, the provision of the uneven relief portion 30 improves the gripping feeling when the user performing the ultrasonic measurement holds the probe 11, thereby improving workability. Furthermore, the scanning position SP and scanning attitude SN of the probe 11 can be accurately detected in real time while reducing the sense of discomfort caused by the appearance.

[0061] (2) The relief portion 30 includes a plurality of relief portions 30a-30d formed to resemble human faces with different facial expressions ( FIG. 6B ). In this case, the relief portion 30 can be reliably photographed regardless of its position relative to the imaging unit 20, and the two-dimensional and three-dimensional positions of the nodes N31-N35 corresponding to the relief feature points can be calculated. That is, at least one of the plurality of relief portions 30a-30d can be photographed, and the two-dimensional and three-dimensional positions of the relief feature points can be calculated. Furthermore, based on the relative positional relationships between the relief feature points, it is possible to identify which of the relief portions 30a-30d the photographed relief portion is. Therefore, the scanning position SP and scanning posture SN of the probe 11 with respect to the subject A can be calculated with higher accuracy based on the subject feature points and the three-dimensional positions of the nodes Nn corresponding to the relief feature points.

[0062] (3) The image processing device 100 further includes an attachment 40 that is detachably attached to the grip portion 11c of the probe 11 (FIGS. 6B and 6C). The relief portion 30 is provided on the attachment 40. In this case, the relief portion 30 can be attached to the probe 11 as a marker for detecting the scanning position SP and scanning attitude SN of the probe 11 without processing the probe 11 itself.

[0063] (4) Either the eye portions 32, 33, ear portions 34, 35, and nose portion 31 of the relief portion 30 or the portion other than the eye portions 32, 33, ear portions 34, 35, and nose portion 31 of the relief portion 30 is made of a material that is visible in the dark or a material that emits ultraviolet light. In this case, the feature points 31 to 35 can be clearly photographed during ultrasound measurement performed in the dark. Furthermore, the feature points 31 to 35 of the relief portion 30 are difficult to see in bright places, reducing the discomfort caused by their appearance.

[0064] (5) The photographing unit 20 has a camera 21a that photographs the subject A and the relief portion 30 from a first position and a camera 21b that photographs the subject A and the relief portion 30 from a second position different from the first position (FIGS. 3 and 4A). The two-dimensional position calculation unit 222a calculates first coordinates (x1a, y1a) to (x10a, y10a), (x31a, y31a) to (x35a, y35a) of the subject feature points N1 to N10 and the relief feature points N31 to N35 photographed by the camera 21a. The two-dimensional position calculation unit 222b calculates second coordinates (x1b, y1b) to (x10b, y10b), (x31b, y31b) to (x35b, y35b) of the subject feature points N1 to N10 and the relief feature points N31 to N35 photographed by the camera 21b. The three-dimensional position calculation unit 123 calculates the position coordinates (x1, y1, z1) to (x10, y10, z10), (x31, y31, z31) to (x35, y35, z35) of the subject feature points N1 to N10 and the relief feature points N31 to N35 in three-dimensional space based on the first coordinates and the second coordinates.

[0065] That is, the three-dimensional position can be calculated with high accuracy by using the principle of triangulation to calculate the three-dimensional position of each node Nn based on the two-dimensional position of each node Nn calculated individually from the images of the pair of left and right cameras 21a and 21b configured as a stereo camera. Also, since multiple cameras spaced a predetermined distance L apart are used simultaneously, it is easy to capture the feature points of the subject A or the probe 11 even if a user performing ultrasound measurement is between the subject A or the probe 11 and the cameras 21a and 21b.

[0066] (6) The probe 11 is an ultrasonic probe that transmits and receives ultrasonic waves while scanning the subject A, so that the burden on the subject A during the examination is small.

[0067] In the above embodiment, the image processing device 100 is applied to a measuring device 10 that transmits and receives ultrasound while scanning a subject A with a probe 11, but the probe may be any type that scans a subject and acquires a cross-sectional image, and the measuring device 10 is not limited to an ultrasonic measuring device.

[0068] In the above embodiment, an example was described in which the image processing device 100 was used to calculate the scanning position SP and scanning orientation SN of the ultrasonic measurement probe 11 relative to the subject A. However, the image processing device can also be applied to calculating the position and orientation of various objects on which a relief portion is provided. For example, the image processing device can be applied to calculating the position and orientation of medical instruments such as a cone for intraoral X-ray imaging, a turbine head for cutting, and a handpiece for a dental laboratory engine. In this case, for example, when a skilled technician uses a medical instrument with a relief portion on a patient, the calculation results of the position and orientation of the medical instrument relative to the patient can be used for educational purposes in clinical training or the like in which the same medical instrument is used.

[0069] Alternatively, the present invention can be applied to calculating the position and orientation of devices other than medical instruments, such as cameras and lighting devices that photograph a target person, speakers that provide sound to the target person, etc. In this case, the calculation results of the position and orientation of the device relative to the target person when aligning the device with a relief portion can be used for installing the device next time and thereafter, thereby improving the reproducibility of the installation position and installation angle (orientation) of the device.

[0070] The image processing device may be applied to detect detailed body movements of a target person. For example, reliefs may be provided at positions corresponding to the target person's limbs on a bodysuit worn by the target person, and the movements of the target person's limbs (the position and posture of the limbs relative to the torso) may be detected (calculated). In this case, CG that reproduces the target person's movements may be created, or the movements of skilled dancers, such as Japanese dance, yoga, etc., may be used for practicing or teaching unskilled people.

[0071] In this way, the image processing device can be applied to calculate the position and orientation of various objects provided with a relief portion. Because the relief portion is formed to resemble the contours of a person's face, it can be provided on various objects relatively easily and at low cost, and even when a relief portion is provided, it is unlikely to cause discomfort to the user of the object, the intended user of the object, or those around them. By applying the image processing device, the position and orientation of various objects provided with a relief portion can be calculated accurately in real time.

[0072] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other. [Explanation of symbols]

[0073] 10 Measuring instrument, 11 Probe, 11a Handle, 11b Scanning surface, 11c Grip portion, 12, 22a, 22b Computer, 13 Display portion, 20 Imaging portion, 21a, 21b Camera, 23 Display portion, 24 Case, 24a, 24b Hole portion, 30 Relief portion, 31 Nose portion, 32, 33 Eye portion, 34, 35 Ear portion, 40 Attachment, 100 Image processing device, 120, 220a, 220b CPU, 121, 221a, 221b Memory, 122 Cross-sectional image generation unit, 123 3D position calculation unit, 124 Position and orientation calculation unit, 125 Image conversion unit, 222a, 22b 2D position calculation unit

Claims

1. a probe for scanning a subject to obtain cross-sectional images; a relief portion formed to resemble the contours of a person's face, having at least a pair of left and right eyes, a pair of left and right ears, and a nose, and being immovably provided on the probe; an imaging unit that images the subject and the relief portion; a position calculation unit that calculates position coordinates of subject feature points that indicate a skeletal shape of the subject photographed by the photographing unit and position coordinates of relief feature points that indicate a skeletal shape of the relief portion; a position and orientation calculation unit that calculates a scanning position and a scanning orientation of the probe with respect to the subject based on the position coordinates calculated by the position calculation unit; an image conversion unit that converts the cross-sectional image acquired by the probe into a three-dimensional image based on the scanning position and scanning orientation of the probe with respect to the subject calculated by the position and orientation calculation unit.

2. 2. The image processing device according to claim 1, The image processing device is characterized in that the relief portion includes a plurality of relief portions formed to resemble human faces with different facial expressions.

3. 2. The image processing device according to claim 1, Further provided is an attachment that is detachably attached to the grip portion of the probe, The image processing device is characterized in that the relief portion is provided on the attachment.

4. 4. The image processing device according to claim 1, An image processing device characterized in that either the eye portion, the ear portion, and the nose portion of the relief portion, or the portion of the relief portion other than the eye portion, the ear portion, and the nose portion, is made of a material that is visible in dark places or a material that emits ultraviolet light.

5. 4. The image processing device according to claim 1, the imaging unit includes a first imaging unit that images the subject and the relief portion from a first position, and a second imaging unit that images the subject and the relief portion from a second position different from the first position; The position calculation unit calculates first coordinates of the subject feature points and the relief feature points acquired by the first photographing unit, calculates second coordinates of the subject feature points and the relief feature points photographed by the second photographing unit, and calculates position coordinates of the subject feature points and the relief feature points in three-dimensional space based on the first coordinates and the second coordinates.

6. 4. The image processing device according to claim 1, The image processing apparatus is characterized in that the probe is an ultrasound probe that transmits and receives ultrasound waves while scanning the subject.

Images