Medical instrument, medical support device, and medical support program

Abstract

To provide a medical instrument, medical support device, and medical support program that can identify the position and posture of the medical instrument with good accuracy.SOLUTION: A medical instrument includes a first region 90 provided with a first marker M1, an intermediate region 92 not provided with a marker, and a second region 94 provided with a second marker M2, which are disposed in a longitudinal direction, wherein: the first marker M1 includes plural first pattern rows 60 arranged along a direction that intersects with the longitudinal direction, each first pattern row 60 being configured by plural symbols 64, 66 arranged along the longitudinal direction, and an arrangement of the symbols 64, 66 being different from at least other first pattern rows 60 adjacent thereto, and the second marker M2 includes plural second pattern rows 70 arranged along the direction that intersects with the longitudinal direction, and each second pattern row 70 being configured by plural symbols 74, 76 arranged along the longitudinal direction, and an arrangement of the symbols 74, 76 being different from at least other second pattern rows 70 adjacent thereto.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to medical devices, medical support devices, and medical support programs.

Background Art

[0002] Conventionally, techniques for specifying the position and orientation of a medical device inserted into the body have been known. For example, Patent Document 1 discloses obtaining the position of an ultrasonic probe in the surgical field using a magnetic sensor. However, such a method using a magnetic sensor may be affected by or affect metallic medical devices other than the ultrasonic probe, and thus there are also situations where it is not suitable for use.

[0003] Therefore, methods that do not use magnetic sensors have also been proposed. For example, Patent Document 2 discloses obtaining a surgical field image obtained by optically photographing an ultrasonic probe provided with a marker using an endoscope, and estimating the position and orientation of the ultrasonic probe by detecting the marker from the surgical field image. Also, for example, Patent Document 3 discloses providing a marker on an ophthalmic surgical instrument inserted into the eye, and determining the position and orientation of the ophthalmic surgical instrument by detecting the marker from a fundus image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] Due to the nature of medical instruments inserted into the body, it is sometimes unavoidable that markers used to identify the position and orientation of the medical instrument are difficult to optically detect. For example, the marker may be blocked by an organ or other object. Furthermore, the marker may appear overexposed in the optical image due to the reflection of illumination light used to capture an optical image inside the body on the surface of the marker (medical instrument). Furthermore, depending on the relative positions of the marker and the camera, the marker may not fit within the field of view of the optical image.

[0006] Whenever such a situation occurs, it is undesirable to move the medical instrument to a position and posture where the marker can be detected, from the viewpoints of efficiency and minimal invasiveness. With conventional techniques for optically detecting markers, if even a portion of the marker cannot be detected, the accuracy of identifying the position and posture of the medical instrument decreases or it becomes impossible to identify the position and posture, making the technique unsuitable for practical use.

[0007] The present disclosure provides a medical instrument, a medical support device, and a medical support program that can identify the position and orientation of the medical instrument with high accuracy. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a medical device including, along a longitudinal direction, a first region having a first marker, an intermediate region without a marker, and a second region having a second marker, wherein the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of which is composed of a plurality of symbols arranged along the longitudinal direction and whose arrangement of the symbols is different from at least other adjacent first pattern rows; and the second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, each of which is composed of a plurality of symbols arranged along the longitudinal direction and whose arrangement of the symbols is different from at least other adjacent second pattern rows.

[0009] In the above embodiment, the surface of the first region may be at least partially formed as a curved surface, and the surface of the second region may be at least partially formed as a curved surface.

[0010] In the above aspect, at least one of the first marker and the second marker may include a pattern sequence made up of an arrangement of codes that is not included in the other one.

[0011] In the above aspect, the plurality of first pattern sequences and the plurality of second pattern sequences may all have different code arrangements.

[0012] In the above aspect, at least one of the first marker and the second marker may include at least one pattern sequence configured to include a type of code that is not included in the other marker.

[0013] In the above aspect, if the direction from the first marker to the second marker is defined as the forward direction and the direction from the second marker to the first marker is defined as the reverse direction, the arrangement of codes along the forward direction in at least one first pattern sequence may be identical to the arrangement of codes along the reverse direction in any other first pattern sequence, and the arrangement of codes along the forward direction in at least one second pattern sequence may be identical to the arrangement of codes along the reverse direction in any other second pattern sequence.

[0014] In the above aspect, the first marker may be one in which a plurality of first pattern sequences are arranged so that only combinations of first pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction, and the second marker may be one in which a plurality of second pattern sequences are arranged so that only combinations of second pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction.

[0015] In the above aspect, the type of code forming the first pattern sequence and the type of code forming the second pattern sequence may be the same.

[0016] In the above aspect, the colors of the codes constituting the first pattern sequence and the colors of the codes constituting the second pattern sequence may all be the same.

[0017] In the above aspect, the number of symbols constituting the first pattern sequence may be the same as the number of symbols constituting the second pattern sequence.

[0018] In the above aspect, the first pattern sequence may be configured by arranging a total of four symbols of two types with different shapes, and the second pattern sequence may be configured by arranging a total of four symbols of two types that are the same as the types of symbols included in the first pattern sequence.

[0019] In the above aspect, information other than the marker may be added to the intermediate region.

[0020] In the above aspect, information indicating a predetermined direction may be added to the intermediate region.

[0021] In the above aspect, a scale may be provided in the intermediate region.

[0022] In the above aspect, information indicating the center of the scale may be further added.

[0023] In the above aspect, the medical instrument may be a medical probe capable of observing the internal structure of an organ.

[0024] A second aspect of the present disclosure is a medical support device that includes a processor and identifies the position and posture of a medical instrument according to the first aspect, wherein the processor acquires an optical image of a medical instrument inserted into a body, optically photographing the medical instrument using a camera, extracts at least a portion of a plurality of first pattern sequences and a plurality of second pattern sequences from the optical image, and identifies the position and posture of the medical instrument based on the extracted at least a portion of the pattern sequences.

[0025] In the above second aspect, the processor may determine the positional relationship between the first marker and the second marker based on the shape of the medical instrument included in the optical image, and determine the position and posture of the medical instrument based on at least a portion of the extracted pattern sequence and the determined positional relationship between the first marker and the second marker.

[0026] In the second aspect, the processor may use a trained model that has been trained in advance to use an optical image as input and a first region and a second region in the input optical image as output to detect a first region and a second region from the optical image, identify the positional relationship between the first marker and the second marker based on the detected first region and second region, and identify the position and posture of the medical instrument based on at least a portion of the extracted pattern sequence and the identified positional relationship between the first marker and the second marker.

[0027] A third aspect of the present disclosure is a medical support program for causing a computer to execute a process for identifying the position and posture of a medical instrument according to the first aspect, in which the process involves obtaining an optical image of a medical instrument inserted into a body, optically photographing the medical instrument using a camera, extracting at least a portion of a plurality of first pattern sequences and a plurality of second pattern sequences from the optical image, and identifying the position and posture of the medical instrument based on the extracted at least a portion of the pattern sequences. [Effects of the Invention]

[0028] According to the above aspects, the medical instrument, medical support device, and medical support program of the present disclosure can identify the position and orientation of the medical instrument with high accuracy. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram showing an outline of a medical support system including a medical support device. [Figure 2] 1 is a diagram showing the state of the body during laparoscopic surgery. FIG. [Figure 3] FIG. 4 is an explanatory diagram of a first guide groove of the ultrasonic probe. [Figure 4] FIG. 10 is an explanatory diagram of a second guide groove of the ultrasonic probe. [Figure 5] FIG. 10 is a view showing the inserted state of the puncture needle guided by the first guide groove. [Figure 6] 10 is a view showing the inserted state of the puncture needle guided by the second guide groove. FIG. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a marker. [Figure 8] FIG. 10 is a diagram showing another example of the configuration of the marker. [Figure 9] FIG. 2 is a diagram illustrating a hardware configuration of the medical support device. [Figure 10] 10A and 10B are diagrams illustrating the relationship between the positions and orientations of a marker and an ultrasound probe. [Figure 11] FIG. 10 is another diagram showing the relationship between the position and orientation of the marker and the ultrasound probe. [Figure 12] 10 is a flowchart illustrating an example of medical support processing. [Figure 13] FIG. 10 is a diagram showing another example of a code. [Figure 14] FIG. 10 is a diagram showing another example of a code. [Figure 15] FIG. 10 is a diagram showing another example of a code. [Figure 16] FIG. 10 is a diagram showing another example of a code. [Figure 17] FIG. 10 is a diagram illustrating another example of support information. [Figure 18] FIG. 10 is a diagram illustrating another example of support information. [Figure 19] FIG. 10 is a diagram showing another example of the configuration of the marker. DETAILED DESCRIPTION OF THE INVENTION

[0030] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are designated by the same reference numerals, and duplicate descriptions will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0031] As shown in FIGS. 1 and 2, a medical support system 10 is used, for example, when performing endoscopic surgery on a patient PT using an endoscope 13. Unlike open surgery, endoscopic surgery is a surgery performed by making a small hole in the patient PT's body and inserting medical instruments such as an endoscope 13 through the hole. The medical support system 10 not only provides medical staff ST, including doctors, with a view of the surgical field inside the patient PT's body, but also provides support information to support medical treatment such as surgery and examinations. As will be described later, the support information specifically includes the puncture path of a puncture needle 18. Such a medical support system 10 has the function of providing support information in real time during surgery, and is therefore also referred to as a surgical navigation system or the like.

[0032] 1, the medical support system 10 includes, for example, a medical support device 11, an endoscope 13, an ultrasound probe 14, and a display 16. The medical support device 11 is connected to the endoscope 13, the ultrasound probe 14, and the display 16 so as to be able to communicate with them.

[0033] 2 shows the state in which an endoscope 13 and an ultrasound probe 14 are being inserted into the abdomen of a patient PT. In endoscopic surgery, a portion of the endoscope 13 and the ultrasound probe 14, including their respective tips, is inserted into the body via a trocar 17. The trocar 17 is an insertion tool that has an insertion hole through which the endoscope 13 and the like are inserted and a valve that is provided within the insertion hole to prevent gas leakage. In endoscopic surgery, pneumoperitoneum is created by injecting carbon dioxide gas into the abdominal cavity, and therefore the trocar 17 is used to insert the endoscope 13, the ultrasound probe 14, and the like into the body.

[0034] The puncture needle 18 is a treatment tool that punctures a lesion, such as a tumor, contained in an organ as a puncture target. The puncture needle 18 is, for example, a cauterization puncture needle used to cauterize the lesion. The cauterization puncture needle has an electrode to which a high-frequency voltage is applied at its tip. When the high-frequency voltage is applied with the electrode punctured into the lesion, the heat generated by the electrode causes the lesion to necrotize. In this example, a tumor 27 in the liver LV is visualized using an ultrasound image 22, and the visualized tumor 27 is cauterized with the puncture needle 18, thereby causing the tumor 27 to necrotize (see also FIGS. 5 and 6). The puncture needle 18 has a needle portion 18A and a handle portion 18B provided on the proximal end side of the needle portion 18A.

[0035] The endoscope 13 has an insertion section 13A that is inserted into the body of the patient PT. A camera 13B and a light source for illumination (e.g., an LED (Light Emitting Diode)) are built into the tip of the insertion section 13A. As an example, the endoscope 13 has a rigid endoscope with a hard insertion section 13A, which is also called a laparoscope because it is often used for observing the abdominal cavity. The camera 13B has an image sensor such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor that can capture color images.

[0036] The endoscope 13 optically captures an operative field SF including a target site (liver LV in this example) inside the body of the patient PT using the camera 13B. The operative field SF is a space extending inside a body cavity defined by organs and the body wall. The endoscope 13 is connected to an image processing processor for the endoscope (not shown), and the image processing processor performs signal processing on the imaging signal output by the image sensor to generate an operative field image 21 of the operative field SF inside the body. The camera 13B is an example of a "camera" in the present disclosure. The operative field image 21 is an example of an "optical image" in the present disclosure.

[0037] Visible light such as white light is used as illumination light for the endoscope 13. However, special light such as ultraviolet light or infrared light may also be used as illumination light for the endoscope 13. The special light may be light limited to a specific wavelength, such as short-wavelength narrow-band light obtained by narrowing the band of light in a short wavelength range such as the ultraviolet region. The operative field image 21 captured by the endoscope 13 is transmitted in real time to the medical support device 11 via an image processing processor for the endoscope. In FIG. 2, the symbols Xin and Yin indicate the coordinate system of the operative field image 21.

[0038] The ultrasonic probe 14 has an insertion section 14A that is inserted into the body of the patient PT and an operation section 14D at the proximal end of the insertion section 14A. An ultrasonic transducer 14C is built into the distal end 14B of the insertion section 14A. The ultrasonic transducer 14C transmits ultrasonic waves to a target area and receives reflected waves from the target area. The ultrasonic probe 14 is connected to an image processing processor for the ultrasonic probe (not shown).

[0039] The image processor for the ultrasound probe performs image reconstruction processing based on signals corresponding to the reflected waves received by the ultrasound probe 14. The image reconstruction processing generates an ultrasound image 22 showing the internal structure of the target area scanned by the ultrasound probe 14. The ultrasound image 22 is a so-called B (Brightness) mode image that visualizes the internal structure of the target area, from the superficial layer to the deep layer reached by ultrasound, as brightness information. The ultrasound image 22 visualizes the internal structure of the target area that cannot be observed in the surgical field image 21 obtained by optical imaging. The ultrasound probe 14 is an example of a "medical instrument" and a "medical probe capable of observing the internal structure of an organ" of the present disclosure.

[0040] The ultrasonic probe 14 is, for example, a convex type that transmits ultrasonic waves radially, and acquires a fan-shaped image with the ultrasonic transducer 14C as the base point. By scanning the ultrasonic probe 14, multiple ultrasonic images 22 are captured along the scanning direction. The ultrasonic images 22 are transmitted in real time to the medical support device 11 via an image processor for the ultrasonic probe. In FIG. 2, the symbols Xpb and Ypb indicate the coordinate system of the ultrasonic image 22.

[0041] A guide groove 29 is provided in the tip 14B of the insertion section 14A. The guide groove 29 is a groove for guiding the insertion of the puncture needle 18 to a target position inside an organ by engaging with the puncture needle 18. In this example, two guide grooves, a first guide groove 29A and a second guide groove 29B, are provided as the guide groove 29. Hereinafter, when there is no need to distinguish between the first guide groove 29A and the second guide groove 29B, they will both be collectively referred to as the guide groove 29.

[0042] Fig. 3 shows the state of the puncture needle 18 guided by the first guide groove 29A. Fig. 4 shows the state of the puncture needle 18 guided by the second guide groove 29B. Fig. 5 shows a schematic diagram of the puncture needle 18 inserted into the body, guided by the first guide groove 29A, puncturing a tumor 27 in the liver LV. Fig. 6 shows a schematic diagram of the puncture needle 18 inserted into the body, guided by the second guide groove 29B, puncturing a tumor 27 in the liver LV.

[0043] The first guide groove 29A is a guide groove 29 provided on the distal end portion 14B closer to the base end than the ultrasonic transducer 14C. The first guide groove 29A is inclined at an angle of θ1 with respect to the direction of the axis AX of the distal end portion 14B. The first guide groove 29A is inclined backward so that the tip of the puncture needle 18 inserted from the proximal end side of the distal end portion 14B faces the distal end side of the distal end portion 14B.

[0044] The second guide groove 29B is a guide groove 29 formed at the very tip of the tip portion 14B and provided closer to the tip side than the ultrasonic transducer 14C. The second guide groove 29B is inclined at an angle of θ2 with respect to the direction of the axis AX of the tip portion 14B. The second guide groove 29B is inclined forward so that the tip of the puncture needle 18 inserted from the tip side of the tip portion 14B faces the base end side of the tip portion 14B.

[0045] The puncture needle 18 is inserted while checking the tumor 27 using the ultrasound image 22. The first guide groove 29A and the second guide groove 29B are each inclined toward the ultrasound transducer 14C, making it possible to direct the tip of the puncture needle 18 toward the area visualized by the ultrasound image 22.

[0046] As shown in Figures 3 and 4, a first marker M1 and a second marker M2 are attached to the tip 14B of the insertion section 14A. Hereinafter, when there is no need to distinguish between the first marker M1 and the second marker M2, both will be collectively referred to as marker M. The configuration of the marker M will be described later. Note that in order to avoid cluttering the drawings, the illustration of the marker M is simplified or omitted in some of the drawings.

[0047] The marker M is a marker that can be recognized from the surgical field image 21 optically captured by the camera 13B of the endoscope 13, that is, an optically detectable marker. The marker M is used by the medical support device 11 to identify the position and posture of the ultrasonic probe 14, more specifically, the position and posture of the tip 14B of the insertion section 14A. A method for identifying the position and posture of the ultrasonic probe 14 using the marker M will be described later.

[0048] The medical support device 11 acquires a surgical field image 21 from an endoscope 13, and acquires an ultrasound image 22 from an ultrasound probe 14. As shown in Fig. 1, two displays 16 are provided, for example, and two images, the surgical field image 21 and the ultrasound image 22, are displayed on each display 16. The surgical field image 21 and the ultrasound image 22 are each output as a video and displayed as a live view on the display 16.

[0049] When the ultrasonic probe 14 is inserted into the surgical field SF, the ultrasonic probe 14, more specifically the tip 14B of the insertion section 14A, appears in the surgical field image 21. The medical support device 11 outputs the surgical field image 21, in which the ultrasonic probe 14 appears, to the display 16. A view of the surgical field SF inside the body of the patient PT is provided to the medical staff ST via the screen of the display 16.

[0050] 5 and 6, a second puncture line NR2 indicating a path passing through the first guide groove 29A or the second guide groove 29B that guides the puncture needle 18 can be superimposed on the ultrasound image 22. There are two types of second puncture lines NR2: one indicating a puncture path along the inclination angle θ1 of the first guide groove 29A, and the other indicating a puncture path along the inclination angle θ2 of the second guide groove 29B. The second puncture line NR2 is generated as an extension of the guide groove 29 based on the known positional relationship between the ultrasonic transducer 14C and the guide groove 29.

[0051] The second puncture line NR2 is used as a guide when inserting the puncture needle 18 into the target position. For example, if the target position to be inserted with the puncture needle 18 is a tumor 27 in the liver LV, the medical staff ST positions the ultrasound probe 14 so that the second puncture line NR2 overlaps with the tumor 27 in the ultrasound image 22. In this state, the puncture needle 18 is inserted into the tumor 27 using the second puncture line NR2 as a guide. In this way, by passing the puncture needle 18 through the guide groove 29 with the ultrasound probe 14 positioned so that the second puncture line NR2 passes through the target position, the puncture needle 18 can be made to reach the target position.

[0052] However, because the ultrasound image 22 is an image of the inside of an organ, it is difficult to determine the position on the body surface BS and the posture in which the puncture needle 18 should be inserted when inserting it into the body from outside the body simply by superimposing the second puncture line NR2 on the ultrasound image 22. Specifically, as shown in Figures 5 and 6, it is difficult to determine the insertion angle of the puncture needle 18 from the body surface BS toward the organ and the insertion position NP on the body surface BS of the patient PT.

[0053] Therefore, the medical support device 11 superimposes the second puncture line NR2 on the ultrasound image 22, and also superimposes the first puncture line NR1 on the operative field image 21. The first puncture line NR1 indicates the puncture path from the body surface BS to the guide groove 29 of the ultrasound probe 14 in the operative field image 21. The second puncture line NR2 indicates the puncture path from the guide groove 29 to a target position in an organ, such as a tumor 27 in the liver LV, in the ultrasound image 22. This provides the medical staff ST with support information that serves as a guide for the insertion angle and insertion position NP of the puncture needle 18.

[0054] Furthermore, in order to provide support information, the medical support device 11 identifies the position and posture of the ultrasound probe 14, more specifically, the position and posture of the tip portion 14B of the insertion section 14A, by using the marker M. The posture of the tip portion 14B is expressed, for example, by the direction of the axis AX of the tip portion 14B. Then, based on the identified direction of the axis AX of the tip portion 14B, the position and posture of the guide groove 29, whose positional relationship relative to the tip portion 14B is known, are identified.

[0055] [Marker M configuration] However, due to the nature of the ultrasound probe 14 being a medical instrument inserted into the body, there are cases where it is difficult to optically detect the marker M. For example, the marker M may be blocked by an organ or the like. Furthermore, for example, the illumination light used to capture the operative field image 21 inside the body may be reflected on the surface of the marker M, causing the marker M to be overexposed in the operative field image 21. Furthermore, for example, depending on the positional relationship between the marker M and the camera 13B, it may not be possible to fit the entire marker M within the angle of view of the operative field image 21.

[0056] From the viewpoint of efficiency and minimal invasiveness, it is not desirable to move the endoscope 13, the ultrasound probe 14, etc. to a position and posture that allows the entire marker M to be detected every time such a situation occurs. Therefore, the marker M of the present disclosure is a so-called highly redundant marker M that can identify the position and posture of the ultrasound probe 14 with good accuracy even if part of it cannot be detected.

[0057] The configuration of the marker M will be described with reference to Fig. 7. Fig. 7 is an example of a developed view of the surface of the tip portion 14B. The tip portion 14B of the insertion section 14A includes, along the longitudinal direction, a first region 90 to which a first marker M1 is attached, an intermediate region 92 to which no marker M is attached, and a second region 94 to which a second marker M2 is attached. The "longitudinal direction" refers to the direction of the axis AX of the tip portion 14B (the X-axis direction in Fig. 7). In Fig. 7, the left side of the paper is the tip side, and the right side of the paper is the base side.

[0058] The first marker M1 includes a plurality of first pattern rows 60 arranged along a direction intersecting the longitudinal direction. Each first pattern row 60 is composed of a plurality of symbols 64, 66 arranged along the longitudinal direction, and the arrangement of the symbols 64, 66 is different from at least other adjacent first pattern rows 60. The "direction intersecting the longitudinal direction" refers to a direction intersecting the axis AX on the surface of the tip portion 14B (the Y-axis direction in FIG. 7), and is the circumferential direction of the ultrasonic probe 14.

[0059] As an example, in Fig. 7, 13 first pattern rows 60 are arranged along the circumferential direction. Hereinafter, when distinguishing between the 13 first pattern rows 60, the symbols A to M are assigned in order from the topmost first pattern row 60 on the paper surface of Fig. 7, and the symbol A is added to the end of the symbol. For example, the topmost first pattern row 60 on the paper surface of Fig. 7 (the first pattern row 60 surrounded by a two-dot chain line) will be referred to as "first pattern row 60A."

[0060] 7, the first pattern sequence 60 is configured by arranging a total of four symbols 64, 66 of two different shapes. For example, the first pattern sequence 60A is configured by a rectangular symbol 66, a circular symbol 64, a circular symbol 64, and a rectangular symbol 66 along the longitudinal direction. None of the other first pattern sequences 60B to 60M has the same arrangement as the first pattern sequence 60A. The arrangements of the symbols 64, 66 differ from one another in the thirteen first pattern sequences 60A to 60M.

[0061] The second marker M2 includes a plurality of second pattern rows 70 arranged in a direction intersecting the longitudinal direction. Each second pattern row 70 is composed of a plurality of symbols 74, 76 arranged in the longitudinal direction, and the arrangement of the symbols 74, 76 differs from at least other adjacent second pattern rows 70.

[0062] 7, for example, thirteen second pattern arrays 70 are arranged in the circumferential direction. As with the first pattern array 60, when the thirteen second pattern arrays 70 need to be distinguished, they are referred to as second pattern arrays 70A to 70M. Note that there may be overlap in the arrangement between the first pattern arrays 60A to 60M and the second pattern arrays 70A to 70M.

[0063] 7, the second pattern sequence 70 is configured by arranging a total of four symbols 74, 76, each of which is the same as the symbols 64, 66 included in the first pattern sequence 60. For example, the second pattern sequence 70A is configured by a circular symbol 74, a circular symbol 74, a circular symbol 74, and a rectangular symbol 76 along the longitudinal direction. None of the other second pattern sequences 70B to 70M has the same arrangement as the second pattern sequence 70A. The arrangements of the symbols 74, 76 differ from one another in the thirteen second pattern sequences 70A to 70M.

[0064] The method of applying each of the first pattern sequences 60 and the second pattern sequences 70 to the ultrasonic probe 14 is known. Therefore, if at least some of the multiple first pattern sequences 60A to 60M and the multiple second pattern sequences 70A to 70M can be detected, the position and posture of the ultrasonic probe 14 in the operative field image 21 can be identified based on the known positional relationship between the detected pattern sequences and the ultrasonic probe 14.

[0065] In this way, redundancy in the longitudinal direction can be ensured by arranging the first marker M1 and the second marker M2, each of which can independently identify the position and orientation of the ultrasound probe 14, at a distance in the longitudinal direction across the intermediate region 92. For example, even in a situation where only the first marker M1 appears in the operative field image 21 and the second marker M2 does not appear in the operative field image 21, the position and orientation of the ultrasound probe 14 can be identified based on the first marker M1.

[0066] Furthermore, by configuring the first marker M1 and the second marker M2 with multiple pattern rows arranged in the circumferential direction, redundancy in the circumferential direction can be ensured. For example, even in a situation where a stripe-like highlight that should appear in the operative field image 21 is lost due to reflection of illumination light from the endoscope 13, the position and orientation of the ultrasound probe 14 can be identified based on the other pattern rows.

[0067] As described above, the markers M according to this embodiment ensure redundancy in both the longitudinal and circumferential directions, and therefore the position and orientation of the ultrasonic probe 14 can be identified with good accuracy under various circumstances, improving practicality. Note that when both the first marker M1 and the second marker M2 can be detected, the position and orientation of the entire ultrasonic probe 14 can also be identified from their positional relationship, further improving the identification accuracy.

[0068] 7, the arrangements of the symbols 64, 66 in the multiple first pattern sequences 60 may be different from each other. With this configuration, the position and posture of the ultrasound probe 14 can be identified by detecting just one first pattern sequence 60 included in the first marker M1. This makes it easier to avoid situations where the position and posture of the ultrasound probe 14 become unidentifiable. Similarly, the arrangements of the symbols 74, 76 in the multiple second pattern sequences 70 may be different from each other. The effect in this case is also the same as that of the first marker M1.

[0069] [Preferred and Possible Configurations of Marker M] The above-described configuration of the marker M is the basic configuration of the marker M according to this embodiment. Preferred and possible configurations of the marker M will be described below.

[0070] It is preferable that at least a portion of the surface of the first region 90 is formed as a curved surface. Similarly, it is preferable that at least a portion of the surface of the second region 94 is formed as a curved surface. By making the surface to which the first marker M1 and the second marker M2 are attached a curved surface, it is possible to prevent the entire surface from being blown out, even if reflection of the illumination light from the endoscope 13 occurs. Specifically, if the surface is curved, even if reflection occurs, it will only result in streaky blown out highlights, so the position and orientation of the ultrasound probe 14 can be identified with high accuracy from the pattern sequence of the non-blown out highlight portions.

[0071] The surfaces of the first region 90 and the second region 94 are not limited to being entirely curved, and may be formed, for example, as a combination of curved and flat surfaces. That is, the first marker M1 and / or the second marker M2 may have at least a portion attached to a curved surface, and may also have a portion attached to a flat surface.

[0072] At least one of the first marker M1 and the second marker M2 preferably includes a pattern sequence configured with an arrangement of codes not included in the other. Specifically, the first marker M1 and the second marker M2 may each include a pattern sequence with a unique arrangement. Alternatively, only one of the first marker M1 and the second marker M2 may include a pattern sequence with a unique arrangement.

[0073] 7, the first pattern sequence 60D included in the first marker M1 is a unique sequence that does not overlap with any of the multiple second pattern sequences 70A-70M included in the second marker M2. Also, for example, the second pattern sequence 70B included in the second marker M2 is a unique sequence that does not overlap with any of the multiple first pattern sequences 60A-60M included in the first marker M1.

[0074] Since at least one of the first marker M1 and the second marker M2 includes a unique pattern sequence, when the unique pattern sequence is detected from the operative field image 21, it is possible to identify whether the detected marker M is the first marker M1 or the second marker M2. For example, when either the first marker M1 or the second marker M2 appears in the operative field image 21, it is possible to identify which one appears. Furthermore, when both the first marker M1 and the second marker M2 appear in the operative field image 21, it is possible to identify which is the first marker M1 and which is the second marker M2. Therefore, the position and orientation of the entire ultrasound probe 14 can be identified from the relative positional relationship between the first marker M1 and the second marker M2, thereby further improving the identification accuracy.

[0075] The plurality of first pattern sequences 60 and the plurality of second pattern sequences 70 may all have different code arrangements. As described above, the method of attaching each of the first pattern sequence 60 and the second pattern sequence 70 to the ultrasonic probe 14 is known. Therefore, by configuring the first pattern sequence 60 and the second pattern sequence 70 so that there is no overlap in the arrangement, the position and posture of the ultrasonic probe 14 can be identified by detecting only a part of the pattern sequence included in the first marker M1 or the second marker M2. This makes it easier to avoid situations where the position and posture of the ultrasonic probe 14 become impossible to identify.

[0076] 7 illustrates an example in which the types of the codes 64, 66 constituting the first pattern sequence 60 and the types of the codes 74, 76 constituting the second pattern sequence 70 are the same, but this is not limiting. At least one of the first marker M1 and the second marker M2 may include at least one pattern sequence that includes a type of code that is not included in the other marker.

[0077] Specifically, both the first pattern sequence 60 and the second pattern sequence 70 may be configured to include unique codes. For example, the first pattern sequence 60 may be configured with black circle codes and white circle codes, and the second pattern sequence 70 may be configured with rectangular codes and triangular codes.

[0078] Furthermore, both the first pattern sequence 60 and the second pattern sequence 70 may be configured to include common codes and unique codes. For example, the first pattern sequence 60 may be configured with black circle codes and white circle codes, and the second pattern sequence 70 may be configured with black circle codes (i.e., common codes) and rectangular codes (i.e., unique codes).

[0079] Furthermore, either the first pattern sequence 60 or the second pattern sequence 70 may be configured to include a unique code in part. For example, the first pattern sequence 60 may be configured with two types of codes, black circle codes and white circle codes, and the second pattern sequence 70 may be configured with three types of codes, black circle codes (i.e., common codes), white circle codes (i.e., common codes), and rectangular codes (i.e., unique codes). In this case, it is sufficient that at least one of the multiple second pattern sequences 70 includes a unique code, and the other second pattern sequences 70 do not need to include a unique code.

[0080] Since at least one of the first marker M1 and the second marker M2 includes a unique code, the first marker M1 and the second marker M2 can be distinguished when the unique code is detected from the operative field image 21. This allows the position and posture of the entire ultrasound probe 14 to be identified from the relative positional relationship between the first marker M1 and the second marker M2 in the operative field image 21, thereby further improving the identification accuracy.

[0081] As described above, the arrangements of the symbols 64, 66 in the multiple first pattern sequences 60 may be different from each other. Here, "different arrangements" means that the arrangements do not overlap when each first pattern sequence 60 is read from one direction. If the reading direction is changed for each of the multiple first pattern sequences 60, the arrangements may overlap. Hereinafter, the direction from the first marker M1 to the second marker M2 along the longitudinal direction (the X-axis direction in FIG. 7) is referred to as the forward direction, and the direction from the second marker M2 to the first marker M1 is referred to as the reverse direction.

[0082] Specifically, in the first marker M1, the arrangement of the symbols 64, 66 in the forward direction in at least one first pattern sequence 60 may be the same as the arrangement of the symbols 64, 66 in the reverse direction in any other first pattern sequence 60. For example, in FIG. 7 , when the first pattern sequence 60B is read in the forward direction, the symbols are arranged in the following order: circular symbols 64, rectangular symbols 66, circular symbols 64, rectangular symbols 66. When the first pattern sequence 60I is read in the reverse direction, the symbols are arranged in the following order: circular symbols 64, rectangular symbols 66, circular symbols 64, rectangular symbols 66. In other words, the arrangement of the symbols 64, 66 in the forward direction in the first pattern sequence 60B is the same as the arrangement of the symbols 64, 66 in the reverse direction in the first pattern sequence 60I.

[0083] In this way, by allowing overlapping of the code arrangement when a certain first pattern sequence 60 is read in the forward direction and when another first pattern sequence 60 is read in the reverse direction within the first marker M1, the number of codes constituting the first pattern sequence 60 can be reduced. This allows the first marker M to be made compact, thereby meeting the recent demand for miniaturization of medical instruments. In other words, even with a small ultrasound probe 14, the position and orientation of the ultrasound probe 14 can be determined with good accuracy.

[0084] Similarly, in the second marker M2, the arrangement of the codes 74, 76 along the forward direction in at least one second pattern sequence 70 may be the same as the arrangement of the codes 74, 76 along the reverse direction in any other second pattern sequence 70. The effect in this case is also the same as that of the first marker M1.

[0085] Furthermore, as described above, the markers M can be configured to enable identification of the first marker M1 and the second marker M2 from the operative field image 21. In this case, the forward and reverse directions in the operative field image 21 can also be identified from the relative positional relationship between the first marker M1 and the second marker M2, so that the predetermined (i.e., correct) reading direction of the pattern sequence can be identified. Therefore, even if the code arrangement when one pattern sequence is read in the forward direction and when another pattern sequence is read in the reverse direction overlap, the two pattern sequences will not be confused, ensuring identification accuracy.

[0086] The first marker M1 preferably has a plurality of first pattern sequences 60 arranged so that a predetermined range in the circumferential direction includes only combinations of first pattern sequences 60 whose Hamming distance is 2 or greater. The Hamming distance here refers to the number of positions where the codes 64 and 66 at the same position differ in a combination of two first pattern sequences 60. For example, in FIG. 7, the Hamming distance between first pattern sequence 60A and first pattern sequence 60B is 2.

[0087] Specifically, it is preferable that each first pattern sequence 60 has a Hamming distance of 2 or more from other first pattern sequences 60 that are located within a predetermined range from itself (for example, two in front and two behind in the circumferential direction). With the above configuration, first pattern sequences 60 that are close in Hamming distance (i.e., Hamming distance of 1 or less) are less likely to appear simultaneously in the operative field image 21. Therefore, it is possible to prevent erroneous detection when detecting the first pattern sequence 60 from the operative field image 21, and to further improve the identification accuracy.

[0088] Similarly, the second marker M2 preferably has a plurality of second pattern sequences 70 arranged in a predetermined range in the circumferential direction so that only combinations of second pattern sequences 70 having a Hamming distance of 2 or more are included. The effect in this case is also the same as that of the first marker M1.

[0089] It is preferable that the colors of the symbols 64 and 66 constituting the first pattern sequence 60 and the symbols 74 and 76 constituting the second pattern sequence 70 are all the same. By making the first markers M1 and the second markers M2 all the same color (monochrome), manufacturing costs can be reduced. Furthermore, by not distinguishing between the symbols 64 and 66 and between the symbols 74 and 76 by color, it is possible to avoid being affected by the characteristics of the camera 13B (e.g., spectral sensitivity) and the characteristics of the illumination light (e.g., color temperature and illuminance). Therefore, the position and orientation of the ultrasound probe 14 can be identified with high accuracy under various circumstances.

[0090] 7 illustrates an example in which the number of first pattern sequences 60 included in the first marker M1 and the number of second pattern sequences 70 included in the second marker M2 are the same (13), but this is not limiting. For example, the number of first pattern sequences 60 included in the first marker M1 and the number of second pattern sequences 70 included in the second marker M2 may be different.

[0091] 7 illustrates an example in which the number of symbols 64, 66 constituting the first pattern sequence 60 and the number of symbols 74, 76 constituting the second pattern sequence 70 are the same (four), but this is not limiting. For example, the number of symbols 64, 66 constituting the first pattern sequence 60 and the number of symbols 74, 76 constituting the second pattern sequence 70 may be different.

[0092] It is preferable that the number of first pattern sequences 60 included in the first marker M1 and the number of second pattern sequences 70 included in the second marker M2 are larger because this allows for more precise identification of the position and posture of the ultrasound probe 14, but in practice there are various restrictions. For example, the maximum number M of pattern sequences that can be arranged is determined depending on the shape of the ultrasound probe 14 and the resolution of the surgical field image 21.

[0093] For example, suppose the diameter of the ultrasonic probe 14 is designed to be 12 mm for insertion into the trocar 17. Also, suppose it is known from experiments that in order to detect the marker M from the surgical field image 21 with sufficient accuracy while the ultrasonic probe 14 assumes various positions and postures, the size of the code needs to be 1.2 mm square or more, and the spacing between the codes in the longitudinal and circumferential directions needs to be 1.2 mm or more. In this case, the maximum number M of pattern rows that can be arranged in the circumferential direction of the ultrasonic probe 14 is calculated using the following formula: M = ((diameter) × (pi)) / ((sign size) + (space between signs)) =(12×3.14) / (1.2+1.2)=15.7

[0094] Furthermore, once the maximum number M of pattern sequences is determined, the number of codes constituting the pattern sequence can also be determined. As described above, all of the first pattern sequences 60 in the first marker M1 are unique, and all of the second pattern sequences 70 in the second marker M2 are unique. In other words, the number of codes constituting the pattern sequence must be a number that can express the maximum number M of unique pattern sequences.

[0095] Specifically, if the number of types of codes is a and the total number of codes included in one pattern sequence is n, then a n is equal to or greater than the maximum number M of pattern sequences. For example, if the number of code types a is 2 and the maximum number M of pattern sequences is 15, then 2 n To satisfy the condition ≧15, n is required to be 4 or more.

[0096] Furthermore, for example, the minimum number m of pattern sequences arranged in the circumferential direction of the ultrasonic probe 14 is preferably 7. This is because, from the viewpoint of redundancy in the circumferential direction, it is preferable that at least three pattern sequences are captured in the operative field image 21. If at least three first pattern sequences 60 can be detected from the first marker M1 in the operative field image 21, the orientation of the ultrasonic probe 14 in the circumferential direction can be identified from the first marker M1 alone. Similarly, if at least three second pattern sequences 70 can be detected from the second marker M2 in the operative field image 21, the orientation of the ultrasonic probe 14 in the circumferential direction can be identified from the second marker M2 alone. In this case, if the number a of types of codes is 2, then n To satisfy the condition ≧7, n is required to be 3 or more.

[0097] Note that the larger the number of code types a and the total number of codes n, the more pattern sequences that can be expressed, making it possible to ensure the Hamming distance between different pattern sequences and prevent erroneous detection. On the other hand, the smaller the number of code types a and the total number of codes n, the simpler the detection process becomes, contributing to faster speeds. Furthermore, the smaller the total number of codes n, the more space is required, which is advantageous for miniaturizing the ultrasonic probe 14. Therefore, the number of code types a and the total number of codes n may be set appropriately taking into consideration the maximum number M and minimum number m of pattern sequences, as well as the above-mentioned circumstances.

[0098] The intermediate region 92 may be provided with information other than the markers M. The information other than the markers M may be, for example, position information indicating the position of the ultrasonic transducer 14C, product information indicating the identification information and specifications of the ultrasonic probe 14, and directional information indicating a predetermined direction in the ultrasonic probe 14. This information is expressed, for example, by symbols, figures, colors, characters, etc.

[0099] 7, as an example, a scale 80 is provided in the intermediate region 92. The scale 80 is an example of position information indicating the position of the ultrasonic transducer 14C. The scale 80 is arranged corresponding to the position of the ultrasonic transducer 14C, that is, the distance from one end to the other end of the ultrasonic transducer 14C in the longitudinal direction. Furthermore, the scale 80 is further provided with information 80A indicating the center of the scale 80, as an example of position information indicating the center position of the ultrasonic transducer 14C.

[0100] 7, as an example, product information 82 indicating the identification information of the ultrasonic probe 14 is added in the text format of "FUJI123" to the intermediate region 92. Also, direction information 84 indicating the direction (longitudinal direction) of the axis AX of the tip portion 14B is added as a straight line to the intermediate region 92.

[0101] Fig. 8 shows a modified example of Fig. 7. In the example of Fig. 8, directional information 86 indicating the reading direction of the first marker M1 and the second marker M2 is attached as a graphic to the intermediate region 92. In the directional information 86 of Fig. 8, the direction from the rectangular graphic to the circular graphic indicates the reading direction of the first marker M1 and the second marker M2.

[0102] In this way, convenience can be improved by utilizing the intermediate area 92 to add various information related to the ultrasonic probe 14. In particular, by printing direction information 86 that indicates the reading direction, the accuracy of identifying the position and posture of the ultrasonic probe 14 can be improved.

[0103] Furthermore, adding text information such as "FUJI123" to the intermediate region 92 makes it easier for the user to determine whether the camera 13B has properly captured the marker M. As described below, the medical support device 11 performs a process of extracting the marker M from the operative field image 21. Here, for example, if the marker M is not within the field of view of the camera 13B or if the operative field image 21 is blurred due to focus shift or lens dirt, it is difficult to properly extract the marker M from the operative field image 21. Therefore, the user must determine whether the operative field image 21 is clear enough to extract the marker M. If the operative field image 21 is not clear, the user must take measures such as adjusting the field of view, adjusting the focus, and cleaning the lens. In this case, adding highly visible information such as "FUJI123" to the intermediate region 92 makes it easier for the user to determine the clarity of the operative field image 21. This contributes to improving the accuracy of identifying the position and orientation of the ultrasound probe 14.

[0104] [Configuration of medical support device 11] Next, the configuration of the medical support device 11 will be described. Fig. 9 shows an example of the hardware configuration of the medical support device 11. The medical support device 11 includes a processor 41, a display 16, a reception device 42, a RAM (Random Access Memory) 43, a storage 44, a communication I / F (Interface) 45, and an external I / F 46. Each of these components is connected to a bus 48 such as a system bus and a control bus, and can communicate with each other.

[0105] The medical support device 11 is operated by an operator such as a medical staff member ST through a reception device 42. The reception device 42 has a keyboard, a mouse, etc. (not shown) and receives instructions from the operator. The reception device 42 may be a device that receives touch input such as a touch panel, a device that receives voice input such as a microphone, or a device that receives gesture input such as a camera.

[0106] Examples of the display 16 include a liquid crystal display and an EL (Electro-Luminescence) display. As described above, there are two displays 16, and each display 16 displays various information in addition to the surgical field image 21 and the ultrasound image 22.

[0107] The processor 41 is, for example, a CPU (Central Processing Unit), which comprehensively controls each part of the medical support device 11 according to a control program, and executes various processes according to various application programs.

[0108] The storage 44 is a non-volatile storage device that stores various programs, various parameters, etc. Examples of the storage 44 include a hard disk drive (HDD) and a solid state drive (SSD). The storage 44 also stores a medical support program 49 that causes the computer to function as the medical support device 11.

[0109] The RAM 43 is a memory that temporarily stores information and is used as a work memory by the processor 41. The RAM 43 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0110] The communication I / F 45 is connected to a network (not shown) such as a LAN (Local Area Network) and / or a WAN (Wide Area Network), and performs transmission control in accordance with communication protocols defined by various wired or wireless communication standards.

[0111] The external I / F 46 is, for example, a USB (Universal Serial Bus) interface, and is used to connect to peripheral devices such as a printer and a memory card.

[0112] The processor 41 performs medical support processing by reading the medical support program 49 from the storage 44 and executing the medical support program 49 on the RAM 43. The medical support processing is realized by the processor 41 operating as an image acquisition unit 41A, an extraction unit 41B, an identification unit 41C, and a display control unit 41D.

[0113] The storage 44 also stores dimension information 50. The dimension information 50 includes information representing the dimensions of the ultrasonic probe 14, specifically, the positional relationship of the marker M at the tip 14B of the ultrasonic probe 14 and the positional relationship of the guide groove 29 at the tip 14B.

[0114] The positional relationship of the marker M in the tip portion 14B is, for example, information on the position and attitude of the arrangement of the pattern sequence included in the marker M relative to the axial and circumferential directions of the tip portion 14B. The positional relationship of the guide groove 29 in the tip portion 14B is, for example, the linear distance in the axial direction along the axis AX of the tip portion 14B between the reference point of the tip portion 14B and the guide groove 29, the inclination angle θ (including θ1 and θ2) of the guide groove 29 relative to the axial direction of the tip portion 14B, etc.

[0115] The image acquiring unit 41A acquires a surgical field image 21 and an ultrasound image 22. For example, the image acquiring unit 41A acquires the surgical field image 21 from a device including a processor of the endoscope 13 via the external I / F 46 or the communication I / F 45. For example, the image acquiring unit 41A acquires the ultrasound image 22 from a device including a processor of the ultrasound probe 14 via the external I / F 46 or the communication I / F 45. Note that the processor of the endoscope 13 and / or the processor of the ultrasound probe 14 may be included in the medical support device 11. The surgical field image 21 is an example of an optical image optically captured by the camera 13B of the ultrasound probe 14 inserted into the body.

[0116] The extraction unit 41B extracts at least some of the plurality of first pattern sequences 60 and the plurality of second pattern sequences 70 from the operative field image 21. Specifically, the extraction unit 41B preferably extracts a total of at least three pattern sequences from the plurality of first pattern sequences 60 and the plurality of second pattern sequences 70. By extracting three or more pattern sequences, the identification unit 41C can properly identify the position and posture of the ultrasound probe 14 in the longitudinal direction and circumferential direction.

[0117] For example, the extraction unit 41B may extract the pattern sequence using an image processing method such as pattern matching. The processor 41 may extract the pattern sequence by searching for morphological features of the markers M, such as the symbols 64, 66, 74, 76, etc., in the surgical field image 21.

[0118] Furthermore, for example, instead of a rule-based method such as pattern matching, a pattern sequence may be extracted using an AI (Artificial Intelligence) technique using a machine learning model. As such a machine learning model, for example, a neural network model such as a CNN (Convolutional Neural Network) that is trained in advance to input the surgical field image 21 and output the pattern sequence in the input surgical field image 21 can be applied.

[0119] Specifically, this machine learning model may output the probability that each pixel in the surgical field image 21 is a marker M as a heat map. In this case, by identifying an area where the probability is equal to or greater than a predetermined threshold as one of the codes 64, 66, 74, or 76, the type of the code 64, 66, 74, or 76 can be determined from the shape of the area. Furthermore, the coordinates of the center or center of gravity of the area can be identified as the coordinates of the code 64, 66, 74, or 76. From the positional relationship between the multiple codes 64, 66, 74, or 76 obtained in this way, the position and orientation (e.g., extension direction) of the pattern sequence can be identified.

[0120] The determination unit 41C determines the position and posture of the ultrasonic probe 14, more specifically, the position and posture of the tip 14B in the surgical field SF, based on at least a portion of the pattern sequence extracted from the surgical field image 21, as shown below.

[0121] 10 and 11 conceptually illustrate the correspondence between the position and orientation of the tip 14B of the ultrasound probe 14 having a marker M in a surgical field image 21 and the position and orientation of the tip 14B in a surgical field SF defined as a three-dimensional space. The Z-axis of the surgical field SF, which is a three-dimensional space, is parallel to the imaging optical axis of the camera 13B, and the X-Y plane (hereinafter simply referred to as the XY plane) is parallel to the imaging surface of the camera 13B and perpendicular to the imaging optical axis. In FIGS. 10 and 11, the XY plane is parallel to the screen of the surgical field image 21. The surgical field image 21 is a projected image of the surgical field SF projected from a single viewpoint. In FIGS. 10 and 11, a grid is displayed in which each symbol corresponds to a lattice point to make it easier to understand the positional relationship of the symbols included in each pattern sequence.

[0122] 10 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13B within the surgical field SF in three-dimensional space (more specifically, a state in which the axis AX of the tip 14B is parallel to the Xin axis). When the tip 14B is in this position, in the surgical field image 21, the longitudinal direction (i.e., the direction in which the symbols in each pattern row are arranged) is parallel to the Xin axis, and the circumferential direction (i.e., the direction in which the multiple pattern rows are arranged) is parallel to the Yin axis. Therefore, the symbols appear at equal intervals in the surgical field image 21.

[0123] On the other hand, Fig. 11 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 within the surgical field SF in three-dimensional space is not perpendicular to the imaging optical axis of the camera 13B, but is tilted in the depth direction parallel to the imaging optical axis. The posture shown in Fig. 11 is a state in which the axis AX of the tip 14B is rotated about the Yin axis by approximately -15° from the posture shown in Fig. 10. When the tip 14B is in this posture, in the surgical field image 21, the farther away from the camera 13B in the depth direction, the shorter the intervals between pattern sequences, and the shorter the intervals between symbols appear to be.

[0124] In this way, the shape of the marker M reflected in the operative field image 21 changes depending on the posture of the tip portion 14B. The processor 41 identifies the posture of the tip portion 14B of the ultrasound probe 14 in the operative field SF based on the posture of the marker M in the operative field image 21. Specifically, the processor 41 detects the orientation of the axis AX of the tip portion 14B in the operative field SF, i.e., the axial direction of the tip portion 14B, as the posture of the tip portion 14B. For example, the extension direction of one pattern sequence extracted by the extraction unit 41B coincides with the orientation of the axis AX.

[0125] Furthermore, if the position of the tip portion 14B changes within the operative field SF, the position of the marker M reflected in the operative field image 21 also changes. The processor 41 identifies the position of the tip portion 14B within the operative field SF based on the position of the marker M. The position of the tip portion 14B is, for example, the position of a reference point of the tip portion 14B, such as the tip position of the tip portion 14B. Furthermore, the shooting distance from the camera 13B to the marker M within the operative field SF (i.e., the distance in the Zin axis direction parallel to the shooting optical axis) can be calculated based on the focal length of the camera 13B and the size of the marker M reflected in the operative field image 21. The processor 41 derives the position coordinates of the reference point of the tip portion 14B within the operative field SF based on the shooting distance and dimensional information 50 of the tip portion 14B, which includes the known dimensions of the marker M.

[0126] The identification unit 41C identifies the position and posture of the guide groove 29 of the ultrasonic probe 14 within the surgical field SF based on the identified position and posture of the tip 14B and the positional relationship of the guide groove 29 in the tip 14B contained in the dimensional information 50 of the ultrasonic probe 14.

[0127] 10, the position of the tip 14B of the ultrasonic probe 14 is specified as information such as the position coordinates (X01, Y01, Z01) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the axial direction of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is specified as information that it is parallel to the XY plane and the XZ plane and perpendicular to the YZ plane.

[0128] The position of the first guide groove 29A is specified as information such as the position coordinates (X11, Y11, Z11) of a reference point within the surgical field SF. The position of the second guide groove 29B is specified as information such as the position coordinates (X21, Y21, Z21) of a reference point within the surgical field SF. The orientations of the first guide groove 29A and the second guide groove 29B are defined as orientations relative to the axis AX, and these become known inclination angles θ1 and θ2 as dimensional information 50.

[0129] 11, the position of the tip 14B of the ultrasonic probe 14 is specified as information such as the position coordinates (X02, Y02, Z02) of the reference point of the tip 14B within the surgical field SF. The orientation of the tip 14B is defined as the axial direction of the tip 14B within the surgical field SF. For example, the axis AX of the tip 14B is specified as information such as -15° with respect to the XY plane, parallel to the XZ plane, and 75° with respect to the YZ plane.

[0130] The position of the first guide groove 29A is specified as information such as the position coordinates (X12, Y12, Z12) of a reference point within the surgical field SF. The position of the second guide groove 29B is specified as information such as the position coordinates (X22, Y22, Z22) of a reference point within the surgical field SF. The orientations of the first guide groove 29A and the second guide groove 29B are defined as orientations relative to the axis AX, and these become known inclination angles θ1 and θ2 as dimensional information 50.

[0131] The above-described identification method can also be applied to identifying the position and orientation of the ultrasound probe 14 when only one of the first marker M1 and the second marker M2 appears in the surgical field image 21. On the other hand, when both the first marker M1 and the second marker M2 appear in the surgical field image 21, the position and orientation of the ultrasound probe 14 can also be identified by utilizing the positional relationship between them.

[0132] For example, the identification unit 41C may identify the positional relationship between the first marker M1 and the second marker M2 based on the shape of the ultrasonic probe 14 included in the operative field image 21. For example, the positional relationship between the shape of the ultrasonic probe 14, such as the first guide groove 29A and the second guide groove 29B, and the first marker M1 and the second marker M2 is known. Therefore, if the shape of the ultrasonic probe 14 can be detected from the operative field image 21, the first marker M1 and the second marker M2 can be identified. Note that, when detecting the shape of the ultrasonic probe 14 from the operative field image 21, an image processing method such as pattern matching can be used.

[0133] Furthermore, for example, instead of a rule-based method such as pattern matching, an AI technique using a machine learning model may be used. As such a machine learning model, for example, a trained model such as CNN that receives the surgical field image 21 as input and outputs the first region 90 and the second region 94 in the input surgical field image 21 may be applied. In other words, this trained model classifies the ultrasound probe 14 in the surgical field image 21 into the first region 90, the second region 94, and other regions.

[0134] The identification unit 41C may use this trained model to detect the first region 90 and the second region 94 from the surgical field image 21, and identify the positional relationship between the first marker M1 and the second marker M2 based on the detected first region 90 and second region 94. Specifically, it is assumed that both the first marker M1 and the second marker M2 are located on a straight line connecting the first region 90 and the second region 94 detected by the trained model. Therefore, the identification unit 41C may identify the first marker M1 and the second marker M2 configured by a pattern arrangement such that their orientations with respect to the straight line are aligned.

[0135] The identification unit 41C identifies the position and orientation of the ultrasound probe 14 based on at least a portion of the extracted pattern sequence and the identified positional relationship between the first marker M1 and the second marker M2. This allows the position and orientation of the entire ultrasound probe 14 to be identified not only from the pattern sequence but also from the positional relationship between the first marker M1 and the second marker M2 captured in the surgical field image 21, thereby further improving the identification accuracy.

[0136] Furthermore, the identification unit 41C derives a first puncture line NR1 of the puncture needle 18 in the operative field image 21 based on the identified position and posture of the guide groove 29. Specifically, the identification unit 41C derives the first puncture line NR1, which is a straight line in the operative field SF that follows the inclination angle θ1 of the first guide groove 29A, based on the position coordinates of the first guide groove 29A in the operative field SF defined as a three-dimensional space and the inclination angle θ1 of the tip portion 14B with respect to the axis AX.

[0137] For example, the identification unit 41C sets the position of the first guide groove 29A as a base point and determines a line segment angled at the inclination angle θ1 of the first guide groove 29A from there as the first puncture line NR1. Then, the identification unit 41C converts this first puncture line NR1 into the two-dimensional coordinate system of the operative field image 21, thereby deriving the first puncture line NR1 in the operative field image 21. The same applies to the second guide groove 29B.

[0138] The display controller 41D executes control to display the operative field image 21 and the ultrasound image 22 on the display 16. The display controller 41D also controls to superimpose the derived first puncture line NR1 on the operative field image 21. Specifically, the "superimposed display" refers to control to generate a composite image in which the first puncture line NR1 is superimposed on the operative field image 21, and display the generated composite image on the display 16. As a result, the operative field image 21 with the first puncture line NR1 superimposed thereon is displayed on the display 16.

[0139] Since the surgical field image 21 is output to the display 16 as a moving image, the position and posture of the first puncture line NR1 in the surgical field image 21 displayed on the display 16 are also updated to reflect the positions of the endoscope 13 and ultrasound probe 14, etc. in real space photographed by the camera 13B.

[0140] The effects of the technology of the present disclosure will be described below using as an example a case where a puncture needle 18 is inserted into the liver LV, which is an organ to be punctured, with a tumor 27 in the liver LV as the target position. As shown in Figures 1 and 5, for example, the medical staff ST adjusts the position and orientation of the ultrasonic transducer 14C provided at the tip 14B of the ultrasonic probe 14 while observing the ultrasound image 22 on which the second puncture line NR2 is superimposed. Specifically, the position and orientation of the ultrasonic transducer 14C are adjusted to a position where the second puncture line NR2, which is an extension of the guide groove 29, passes through the tumor 27.

[0141] The position and orientation of the first puncture line NR1 superimposed on the surgical field image 21 changes in accordance with the change in the position and orientation of the ultrasonic probe 14. When the second puncture line NR2 is displayed in the ultrasonic image 22 so as to pass through the tumor 27, the second puncture line NR2 indicates an ideal puncture path for the puncture needle 18 within the liver LV. In this state, the position and orientation of the ultrasonic probe 14 are fixed while the puncture is being performed.

[0142] In this state, the first puncture line NR1 in the operative field image 21 indicates, for example, an ideal puncture path of the puncture needle 18 within the body cavity from the body surface BS of the patient PT to the surface of the liver LV to be punctured. While observing the operative field image 21 on which the ideal first puncture line NR1 is superimposed, the medical staff ST adjusts the position and posture of the puncture needle 18 outside the body so that the puncture needle 18 is inserted along the ideal first puncture line NR1.

[0143] Specifically, the operator finds an approximate insertion position NP corresponding to the first puncture line NR1, and inserts the puncture needle 18 into the body surface BS of the patient PT from the insertion position NP. Then, while watching the display of the first puncture line NR1, the operator adjusts the position and posture of the puncture needle 18 and inserts the puncture needle 18 into the body cavity toward the first guide groove 29A of the ultrasound probe 14. Because the first puncture line NR1 is displayed with the first guide groove 29A as the base point, the puncture needle 18 can be easily inserted along the first puncture line NR1 to reach the first guide groove 29A.

[0144] After the puncture needle 18 reaches the first guide groove 29A, the puncture needle 18 can be inserted into the liver LV along the second puncture line NR2 while checking the second puncture line NR2 on the ultrasound image 22 as shown in Figures 1, 2, and 5, thereby inserting the puncture needle 18 into the tumor 27, which is the target position. With the puncture needle 18 inserted into the tumor 27, the tumor 27 is cauterized, for example.

[0145] Next, the operation of the medical support device 11 will be described with reference to Fig. 12. In the medical support device 11, the processor 41 executes the medical support program 49, thereby executing the medical support process shown in Fig. 12. This process is executed, for example, when a command to start execution is given by the user via the reception device 42.

[0146] In step S10, the image acquisition unit 41A acquires an optical image (e.g., surgical field image 21) of a medical instrument (e.g., ultrasound probe 14) inserted into the body, optically photographed by the camera 13B. In step S12, the extraction unit 41B extracts at least some of the plurality of first pattern sequences 60 and the plurality of second pattern sequences 70 from the optical image acquired in step S10.

[0147] In step S14, the identification unit 41C identifies the position and orientation of the medical instrument based on at least a portion of the pattern sequence extracted in step S12. In step S16, the identification unit 41C identifies the puncture path of the puncture needle 18 in the optical image acquired in step S10 as support information based on the position and orientation of the medical instrument (e.g., guide groove 29) identified in step S14. In step S18, the display control unit 41D controls the display 16 to present the support information identified in step S16, and this process ends.

[0148] As described above, the medical device (ultrasound probe 14) according to this embodiment includes, along the longitudinal direction, a first region 90 to which first markers M1 are attached, an intermediate region 92 to which no markers M are attached, and a second region 94 to which second markers M2 are attached. The first markers M1 include a plurality of first pattern rows 60 arranged along a direction intersecting the longitudinal direction, and each first pattern row 60 is composed of a plurality of symbols 64, 66 arranged along the longitudinal direction, and the arrangement of the symbols 64, 66 differs from at least other adjacent first pattern rows 60. The second markers M2 include a plurality of second pattern rows 70 arranged along a direction intersecting the longitudinal direction, and each second pattern row 70 is composed of a plurality of symbols 74, 76 arranged along the longitudinal direction, and the arrangement of the symbols 74, 76 differs from at least other adjacent second pattern rows 70.

[0149] That is, according to the medical instrument of this embodiment, the markers M for identifying the orientation and position of the medical instrument ensure redundancy in both the longitudinal and circumferential directions. Therefore, the position and orientation of the medical instrument can be identified with good accuracy under various circumstances, improving practicality. Furthermore, when both the first marker M1 and the second marker M2 can be detected, the position and orientation of the entire medical instrument can also be identified from their positional relationship, further improving identification accuracy.

[0150] Furthermore, in the technology of the present disclosure, the hardware configuration for identifying the position and posture of a medical instrument such as an ultrasound probe 14 simply requires providing a marker M in the insertion portion of the medical instrument. Therefore, a large-scale device such as a magnetic system is not required. In addition, even when a conventional medical instrument is used, only the exterior of the insertion portion needs to be modified, which is simpler than a configuration that incorporates a light source for the marker M. Therefore, the medical support device 11 can display support information (for example, the first puncture line NR1 of the puncture needle 18 at an appropriate position) according to the position or posture of the medical instrument with a simple configuration.

[0151] Furthermore, in the above embodiment, the medical instrument inserted into the body of the patient PT is an ultrasound probe 14 (an example of a medical probe) capable of observing the internal structure of an organ. The procedure of puncturing an internal organ with a puncture needle 18 is often performed using a medical probe capable of observing the internal structure of the organ. Therefore, when a medical probe is used as the medical instrument, as in the above embodiment, the technology of the present disclosure is particularly effective.

[0152] Furthermore, the ultrasound probe 14 is relatively often used in combination with the puncture needle 18. Therefore, the technology of the present disclosure is even more effective when the ultrasound probe 14 is used as a medical probe. Note that the medical probe capable of observing the internal structure of an organ may be other than the ultrasound probe 14, and may be, for example, an optical coherence tomography (OCT) probe or the like.

[0153] Note that the medical instrument may be anything other than a medical probe capable of observing the internal structure of an organ. For example, the medical instrument may be a treatment instrument that does not have the function of observing the internal structure and only has a guide groove 29 for the puncture needle 18 at its tip. For example, when a tumor is present on the surface of an organ and a treatment is performed in which the puncture needle 18 is inserted into the surface tumor, even a treatment instrument that does not have the function of observing the internal structure can properly guide the puncture needle 18 as long as it has the guide groove 29. In this case, for example, the medical staff ST aligns the guide groove 29 of the treatment instrument with a position corresponding to the tumor on the surface of the organ, and then inserts the puncture needle 18 into the tumor through the guide groove 29.

[0154] Furthermore, the medical instrument may be a simple rod-like treatment instrument that does not have a guide groove 29. If a tumor is present on the surface of an organ, even such a treatment instrument can point to the tumor. Simply superimposing the first puncture line NR1 on the operative field image 21 in which the tumor on the surface of the organ is pointed to with the treatment instrument can serve as a guide for checking the puncture direction of the puncture needle 18, etc. Therefore, the technology of the present disclosure is effective even if the medical instrument does not have a guide groove 29.

[0155] Furthermore, in the above embodiment, the shapes of the reference numerals 64, 66, 74, and 76 have been described using examples such as circles, rectangles, and triangles, but are not limited to these. Figures 13 to 18 show other examples of the shapes of the reference numerals 64, 66, 74, and 76. The reference numerals 64 and 74 shown in Figure 13 have a shape in which a circle is cut out (i.e., annular), or a shape in which circles of different sizes and colors are overlapped. The reference numerals 66 and 76 shown in Figure 14 have a shape in which a circle is cut out, or a shape in which circles of different colors are overlapped on a rectangle.

[0156] The symbols 64 and 74 shown in Fig. 15 have a shape in which a circle has been cut out and a cross mark has been placed in the cutout. The symbols 66 and 76 shown in Fig. 16 have a shape in which a rectangle has been cut out and a cross mark has been placed in the cutout. In this way, the symbols 64, 66, 74, and 76 may be represented by a combination of multiple shapes. Furthermore, for example, the symbols may be represented by shapes such as polygons, stars, and various marks, or by letters, etc.

[0157] In the above embodiment, the marker M is attached to the tip 14B of the insertion section 14A of the ultrasonic probe 14, but the present invention is not limited to this. The marker M may be attached to a portion of the various medical instruments that is inserted into the body of the patient PT, and may be attached to, for example, the middle part of the insertion section 14A or the base end side of the insertion section 14A.

[0158] Furthermore, in the above embodiment, cauterization has been described as an example of the function of the puncture needle 18, but the function of the puncture needle 18 is not limited to this. Furthermore, although the puncture needle 18 has been described as an example of a treatment tool, other than the puncture needle 18, treatment tools for injecting a fluorescent agent such as ICG (Indocyanine Green), biopsy needles and forceps used for collecting tissue for biopsy, etc. may also be applied.

[0159] In addition, in the above embodiment, the inside of the body has been described using body cavities such as the abdominal cavity and the thoracic cavity as examples, but the inside of the body may also be the inside of an upper digestive tract such as the esophagus, a lower digestive tract such as the intestines, or a duct such as the bronchi. When the technology of the present disclosure is applied to a surgical field inside a duct, for example, a marker M is provided at the base end of a flexible endoscope inserted into the duct.

[0160] In the above embodiment, an example has been described in which the puncture path of the puncture needle 18 (the first puncture line NR1 and the second puncture line NR2) is used as support information corresponding to the position or posture of the medical instrument, but this is not limiting. For example, the medical support device 11 may provide support information corresponding to the position or posture of the ultrasound probe 14 by superimposing an ultrasound image 22 corresponding to the position or posture of the ultrasound probe 14 on the surgical field image 21 in addition to or instead of the puncture path of the puncture needle 18.

[0161] Here, the imaging range of the ultrasound image 22 corresponds to the position and orientation of the ultrasound transducer 14C, more specifically, is a sector-shaped range with the ultrasound transducer 14C as the base point. The position of the ultrasound transducer 14C in the ultrasound probe 14 is known and is included in the dimensional information 50 of the ultrasound probe 14. Therefore, if the position and orientation of the ultrasound probe 14 are known, the position and orientation of the ultrasound transducer 14C can also be identified, and therefore the imaging range of the ultrasound image 22 in the surgical field image 21 can be identified.

[0162] 17 and 18 show examples in which an ultrasound image 22 corresponding to the position and posture of the ultrasound probe 14 is superimposed on a surgical field image 21. Similar to Fig. 10, Fig. 17 shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is perpendicular to the imaging optical axis of the camera 13B. In this state, the Xin-Yin plane on which the surgical field image 21 is formed in the surgical field SF and the Xpb-Ypb plane on which the ultrasound image 22 is formed in the surgical field SF are parallel to each other.

[0163] 18, similar to FIG. 11, shows a state in which the axial direction of the tip 14B of the ultrasound probe 14 is not perpendicular to the imaging optical axis of the camera 13B but is tilted in the depth direction parallel to the imaging optical axis. In this state, the Xin-Yin plane on which the operative field image 21 is formed in the operative field SF is not parallel to the Xpb-Ypb plane on which the ultrasound image 22 is formed in the operative field SF. Taking into account the difference in coordinate systems between the operative field image 21 and the ultrasound image 22, the medical support device 11 may perform transformations such as projective transformation, affine transformation, translation, rotation, enlargement, and reduction on the ultrasound image 22 and then superimpose it on the operative field image 21.

[0164] Specifically, as described above, the identification unit 41C uses the marker M to identify the position and orientation of the tip 14B of the ultrasonic probe 14 in the operative field image 21. Furthermore, the identification unit 41C identifies the position and orientation of the ultrasonic transducer 14C in the operative field SF based on the identified position and orientation of the tip 14B and the positional relationship of the ultrasonic transducer 14C included in the dimension information 50. Then, based on the identified position and orientation of the ultrasonic transducer 14C, the identification unit 41C performs transformation processing such as projective transformation so that the imaging range (Xpb-Ypb plane) of the ultrasonic image 22 is formed within the operative field image 21 (Xin-Yin plane). The display control unit 41D controls the transformation-processed ultrasonic image 22 to be superimposed on the operative field image 21.

[0165] Such support information makes it easier to grasp the position and orientation of the ultrasonic transducer 14C even if it is difficult to visually recognize the position and orientation in the surgical field image 21 due to, for example, obstruction by organs or the positional relationship with the camera 13B. Therefore, it becomes easier to adjust the position and orientation of the ultrasonic probe 14 so that a desired area (for example, a tumor 27) is captured in the ultrasonic image 22, which contributes to improving the operability of the ultrasonic probe 14.

[0166] In the above embodiment, the example of FIG. 7 shows a configuration in which the arrangements of the symbols 64 and 66 in the multiple first pattern sequences 60A to 60M are different from each other, but this is not limited to this. In the medical device according to this embodiment, it is sufficient that the arrangements of the symbols 64 and 66 in the multiple first pattern sequences 60 are different from at least the arrangements of the symbols 64 and 66 of other adjacent first pattern sequences 60. In other words, there may be combinations of the multiple first pattern sequences 60 having the same arrangements of the symbols 64 and 66. FIG. 19 shows an example configuration including combinations having the same arrangements of the symbols 64 and 66 as a modification of FIG. 7. In FIG. 19, the symbol arrangements of the first pattern sequence 60B and the first pattern sequence 60L are the same, and both are composed of four symbols 64.

[0167] Similarly, the arrangement of the symbols 74, 76 among the multiple second pattern sequences 70 may be different from at least the arrangement of the symbols 74, 76 among the adjacent second pattern sequences 70. In other words, among the multiple second pattern sequences 70, there may be a combination having the same arrangement of the symbols 74, 76.

[0168] When a pattern sequence having the same arrangement as another pattern sequence is extracted, the identifying unit 41C may distinguish the pattern sequence having the same arrangement from other pattern sequences within a predetermined range from the extracted pattern sequence. For example, suppose a pattern sequence having the same arrangement as another pattern sequence is extracted, but it is unclear whether it is the first pattern sequence 60B or the first pattern sequence 60L (hereinafter referred to as a duplicated pattern sequence). In this case, when the first pattern sequence 60A and / or 60C is extracted as a pattern sequence adjacent to the duplicated pattern sequence in the circumferential direction, the identifying unit 41C may distinguish the extracted duplicated pattern sequence as the first pattern sequence 60B (i.e., not the first pattern sequence 60L). Furthermore, when at least one of the second pattern sequences 70A to 70C is extracted as the second pattern sequence 70 circumferentially close to the duplicated pattern sequence, the identifying unit 41C may distinguish the extracted duplicated pattern sequence as the first pattern sequence 60B.

[0169] Furthermore, for example, when overlapping pattern sequences are attached to the left and right side surfaces of the ultrasonic probe 14, such as the first pattern sequences 60B and 60L, the identification unit 41C may identify which side surface, left or right, the portion of the ultrasonic probe 14 reflected in the operative field image 21 is. Specifically, the identification unit 41C may identify which side surface, left or right, the portion of the ultrasonic probe 14 reflected in the operative field image 21 is, based on at least one of the shape of the ultrasonic probe 14 included in the operative field image 21 and the product information 82 (e.g., the text "FUJI123") attached to the intermediate region 92. For example, the side surface to which the first pattern sequences 60A to 60F are attached is the right side surface, and the side surface to which the first pattern sequences 60H to 60M are attached is the left side surface. In this case, when the identification unit 41C identifies that the right side surface is reflected, the overlapping pattern sequence reflected in the operative field image 21 can be identified as the first pattern sequence 60B.

[0170] In the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the image acquisition unit 41A, extraction unit 41B, identification unit 41C, and display control unit 41D. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0171] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0172] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0173] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0174] In the above embodiment, the medical support program 49 is pre-stored (installed) in the storage 44, but this is not limiting. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network. Furthermore, the technology of the present disclosure extends to not only programs but also storage media that non-temporarily store programs.

[0175] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be substituted from the above-described description and illustrations, within the scope of the gist of the technology of the present disclosure.

[0176] The following additional notes are further disclosed regarding the above embodiment. [Appendix 1] The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; The second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, and each of the second pattern rows is composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols is different from at least other adjacent second pattern rows. Medical equipment. [Appendix 2] the surface of the first region is at least partially formed as a curved surface, At least a portion of the surface of the second region is formed as a curved surface. 1. A medical device as described in Appendix 1. [Appendix 3] At least one of the first marker and the second marker includes a pattern sequence configured with an arrangement of the codes that is not included in the other marker. 1. A medical device according to claim 1 or 2. [Appendix 4] The plurality of first pattern sequences and the plurality of second pattern sequences all have different arrangements of the codes. 1. A medical device as described in Appendix 3. [Appendix 5] At least one of the first marker and the second marker includes at least one pattern sequence including the code of a type not included in the other marker. 1. A medical device according to claim 3 or 4. [Appendix 6] When the direction from the first marker to the second marker is defined as a forward direction and the direction from the second marker to the first marker is defined as a reverse direction, an arrangement of the codes along the forward direction in at least one of the first pattern sequences is the same as an arrangement of the codes along the reverse direction in any other one of the first pattern sequences; The arrangement of the codes along the forward direction in at least one of the second pattern sequences is the same as the arrangement of the codes along the reverse direction in any other one of the second pattern sequences. 10. The medical device of any one of claims 1 to 5. [Appendix 7] the first marker is configured such that the plurality of first pattern sequences are arranged such that only combinations of the first pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction; The second marker has the plurality of second pattern sequences arranged so that only combinations of the second pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction. 10. The medical device of any one of claims 1 to 6. [Appendix 8] The type of the code constituting the first pattern sequence and the type of the code constituting the second pattern sequence are the same. 10. The medical device of any one of claims 1 to 7. [Appendix 9] The colors of the codes constituting the first pattern sequence and the colors of the codes constituting the second pattern sequence are all the same. 10. The medical device of any one of appendices 1 to 8. [Appendix 10] The number of the codes constituting the first pattern sequence is the same as the number of the codes constituting the second pattern sequence. 10. The medical device of any one of claims 1 to 9. [Appendix 11] the first pattern sequence is configured by arranging a total of four of the two types of symbols having different shapes, The second pattern sequence is configured by arranging a total of four of the two types of codes that are the same as the types of codes included in the first pattern sequence. 11. The medical device of any one of claims 1 to 10. [Appendix 12] The intermediate region is provided with information other than the marker. 12. The medical device of any one of claims 1 to 11. [Appendix 13] The intermediate region is provided with information indicating a predetermined direction. 13. The medical device of claim 12. [Appendix 14] The intermediate region is marked with a scale. 14. The medical device of claim 12 or 13. [Appendix 15] Information indicating the center of the scale is further added. 14. A medical device as described in Appendix 14. [Appendix 16] The medical instrument is a medical probe capable of observing the internal structure of an organ. 16. The medical device of any one of claims 1 to 15. [Appendix 17] A medical support device including a processor for identifying a position and a posture of a medical instrument to be inserted into a body, The medical device comprises: The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; the second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, each of the second pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent second pattern rows; The processor: Acquiring an optical image by optically photographing the medical instrument inserted into the body with a camera; extracting at least some of the first pattern arrays and the second pattern arrays from the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence. Medical support equipment. [Appendix 18] The processor: determining a positional relationship between the first marker and the second marker based on a shape of the medical instrument included in the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence and the identified positional relationship between the first marker and the second marker. 18. The medical support device of claim 17. [Appendix 19] The processor: Detecting the first region and the second region from the optical image using a trained model that has been trained in advance to use the optical image as an input and output the first region and the second region in the input optical image; Identifying a positional relationship between the first marker and the second marker based on the detected first region and the detected second region; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence and the identified positional relationship between the first marker and the second marker. 19. The medical support device according to claim 17 or 18. [Appendix 20] A medical support program for causing a computer to execute a process for identifying the position and posture of a medical instrument to be inserted into a body, The medical device comprises: The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; the second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, each of the second pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent second pattern rows; The process comprises: Acquiring an optical image by optically photographing the medical instrument inserted into the body with a camera; extracting at least some of the first pattern arrays and the second pattern arrays from the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence. Medical assistance program. [Explanation of symbols]

[0177] 11 Medical support equipment 13 Endoscopy 13A Insertion section 13B Camera 14 Ultrasound probe 14A Insertion section 14B Tip 14C Ultrasonic Transducer 14D Control unit 16 Display 17 Trocar 18 Puncture needle 18A needle part 18B Gripping part 21 Surgical field image 22 Ultrasound images 27 Tumor 29 Guide groove 29A First guide groove 29B Second guide groove 41 processors 41A Image acquisition unit 41B Extraction part 41C Specific part 41D Display control unit 42 Reception Device 43 RAM 44 Storage 45 Communication I / F 46 External I / F 48 Bus 49 Medical Assistance Program 50 Dimensional Information 60, 60A~60M First pattern row 64, 66, 74, 76 sign 70, 70A~70M Second pattern row 80 scales 80A Scale center information 82 Product Information 84, 86 Direction information 90 1st area 92 Intermediate area 94 Second area θ, θ1, θ2 Tilt angle AX axis BS body surface LV liver M marker M1 First marker M2 Second marker NP insertion position NR1 First puncture line NR2 Second puncture line PT patient SF field ST Medical Staff Xin, Yin, Xpb, Ypb coordinates

Claims

1. The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; The second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, and each of the second pattern rows is composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols is different from at least other adjacent second pattern rows. Medical equipment.

2. At least a portion of the surface of the first region is formed as a curved surface, At least a portion of the surface of the second region is formed as a curved surface. The medical device of claim 1 .

3. At least one of the first marker and the second marker includes a pattern sequence configured with an arrangement of the codes that is not included in the other marker. The medical device of claim 1 .

4. The plurality of first pattern sequences and the plurality of second pattern sequences all have different arrangements of the codes. The medical device of claim 3 .

5. At least one of the first marker and the second marker includes at least one pattern sequence including the code of a type not included in the other marker. The medical device of claim 3 .

6. When the direction from the first marker to the second marker is defined as a forward direction and the direction from the second marker to the first marker is defined as a reverse direction, an arrangement of the codes along the forward direction in at least one of the first pattern sequences is the same as an arrangement of the codes along the reverse direction in any other one of the first pattern sequences; The arrangement of the codes along the forward direction in at least one of the second pattern sequences is the same as the arrangement of the codes along the reverse direction in any other of the second pattern sequences. The medical device of claim 1 .

7. the first marker is configured such that the plurality of first pattern sequences are arranged such that only combinations of the first pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction; The second marker has the plurality of second pattern sequences arranged so that only combinations of the second pattern sequences having a Hamming distance of 2 or more are included in a predetermined range in a direction intersecting the longitudinal direction. The medical device of claim 1 .

8. The type of the code constituting the first pattern sequence is the same as the type of the code constituting the second pattern sequence. The medical device of claim 1 .

9. The colors of the codes constituting the first pattern sequence and the colors of the codes constituting the second pattern sequence are all the same. The medical device of claim 1 .

10. The number of the codes constituting the first pattern sequence is the same as the number of the codes constituting the second pattern sequence. The medical device of claim 1 .

11. the first pattern sequence is configured by arranging a total of four of the two types of symbols having different shapes, The second pattern sequence is configured by arranging a total of four of the two types of codes that are the same as the types of codes included in the first pattern sequence. The medical device of claim 1 .

12. The intermediate region is provided with information other than the marker. The medical device of claim 1 .

13. The intermediate region is provided with information indicating a predetermined direction. The medical device of claim 12.

14. The intermediate region is marked with a scale. The medical device of claim 12.

15. Information indicating the center of the scale is further added.

15. The medical device of claim 14.

16. The medical instrument is a medical probe capable of observing the internal structure of an organ. The medical device of claim 1 .

17. A medical support device including a processor for identifying a position and a posture of a medical instrument to be inserted into a body, The medical device comprises: The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; the second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, each of the second pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent second pattern rows; The processor: Acquiring an optical image by optically photographing the medical instrument inserted into the body with a camera; extracting at least some of the first pattern arrays and the second pattern arrays from the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence. Medical support equipment.

18. The processor: determining a positional relationship between the first marker and the second marker based on a shape of the medical instrument included in the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence and the identified positional relationship between the first marker and the second marker. The medical support device according to claim 17.

19. The processor: Detecting the first region and the second region from the optical image using a trained model that has been trained in advance to use the optical image as an input and output the first region and the second region in the input optical image; determining a positional relationship between the first marker and the second marker based on the detected first region and the detected second region; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence and the identified positional relationship between the first marker and the second marker. The medical support device according to claim 17.

20. A medical support program for causing a computer to execute a process for identifying the position and posture of a medical instrument to be inserted into a body, The medical device comprises: The device includes a first region having a first marker, an intermediate region having no marker, and a second region having a second marker along the longitudinal direction, the first marker includes a plurality of first pattern rows arranged along a direction intersecting the longitudinal direction, each of the first pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent first pattern rows; the second marker includes a plurality of second pattern rows arranged along a direction intersecting the longitudinal direction, each of the second pattern rows being composed of a plurality of symbols arranged along the longitudinal direction, and the arrangement of the symbols being different from at least other adjacent second pattern rows; The process comprises: Acquiring an optical image by optically photographing the medical instrument inserted into the body with a camera; extracting at least some of the first pattern arrays and the second pattern arrays from the optical image; Identifying the position and orientation of the medical instrument based on the extracted at least part of the pattern sequence. Medical assistance program.

Images