Information processing equipment, medical imaging diagnostic equipment, programs and storage media

Abstract

To enable estimation of an inspection part robust to occlusion.SOLUTION: An information processing device according to embodiment comprises acquisition means, cable estimation means, estimation means, and inspection part estimation means. The acquisition means acquires an image including an inspection device which scans an analyte. The cable estimation means estimates the position of a cable of the inspection device based on an image. The estimation means estimates the position and direction of the inspection device based on the position of the cable of the inspection device. The inspection part estimation means estimates the inspection part by the inspection device based on the position and direction of the inspection device.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] , , , , , , , ,

[0005] , , , ,

[0001] The embodiments disclosed in this specification and the like relate to an information processing apparatus, a medical image diagnostic apparatus, a program, and a storage medium.

Background Art

[0002] In the medical field, doctors perform diagnoses using medical images taken by various modalities (medical image diagnostic apparatuses). Examples of modalities include ultrasonic diagnostic apparatuses and photoacoustic imaging apparatuses (hereinafter referred to as PAT (PhotoAcoustic Tomography) apparatuses). Other modalities include magnetic resonance imaging apparatuses (hereinafter referred to as MRI (Magnetic Resonance Imaging) apparatuses), computed tomography apparatuses (hereinafter referred to as X-ray CT (Computed Tomography) apparatuses), and the like.

[0003] There is known a system for discriminating (identifying) which part of a subject is imaged by the medical images used for these diagnoses based on the positional relationship between the medical image diagnostic apparatus and the subject. Specifically, the system identifies the positions of the subject and the ultrasonic probe from an external image of the subject and the ultrasonic probe taken during the examination by template matching with respective template images, and calculates the examination site from their positional relationship.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] One of the problems that the embodiments disclosed herein aim to solve is to enable robust estimation of the inspection site against occlusion. However, the problems solved by the embodiments disclosed herein are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems solved by the embodiments disclosed herein. [Means for solving the problem]

[0006] The information processing apparatus of the embodiment comprises an acquisition means, a cable estimation means, an estimation means, and an inspection site estimation means. The acquisition means acquires an image including an inspection device that scans a subject. The cable estimation means estimates the position of the cable of the inspection device based on the image. The estimation means estimates the position and orientation of the inspection device based on the position of the cable of the inspection device. The inspection site estimation means estimates the inspection site by the inspection device based on the position and orientation of the inspection device. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the configuration of an ultrasound diagnostic apparatus according to the first embodiment. [Figure 2] Figure 2 shows an example of the configuration of an ultrasonic probe according to the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of the main body of an ultrasound diagnostic device according to the first embodiment. [Figure 4] Figure 4 is a flowchart showing the overall flow of the ultrasound diagnostic apparatus according to the first embodiment. [Figure 5] Figure 5 is a flowchart showing the processing flow of the measurement process according to the first embodiment. [Figure 6] Figure 6 is a flowchart showing the processing flow of ultrasonic image processing according to the first embodiment. [Figure 7] Figure 7 is a flowchart showing the processing flow of a modified example 1 of the measurement process according to the first embodiment. [Figure 8] Figure 8 is a flowchart showing the processing flow of the human body position and posture estimation process according to the first embodiment. [Figure 9] Figure 9 is a flowchart showing the processing flow of the inspection site identification process according to the first embodiment. [Figure 10] Figure 10 is a flowchart showing the operation after the measurement process according to the first embodiment. [Figure 11] Figure 11 is an illustrative diagram of a measurement using an ultrasound diagnostic device according to the first embodiment. [Figure 12A] Figure 12A is an illustrative diagram of the output results obtained during measurement according to the first embodiment. [Figure 12B] Figure 12B is an illustrative diagram of the output results obtained during measurement according to the first embodiment. [Figure 12C] Figure 12C is an illustrative diagram of the output results obtained during measurement according to the first embodiment. [Figure 12D] Figure 12D is an illustrative diagram of the output results obtained during measurement according to the first embodiment. [Figure 13] Figure 13 is an image showing the external appearance of the first embodiment and an image showing the ultrasonic probe contained within the external appearance. [Figure 14A] Figure 14A is an image diagram obtained during the position estimation process of the ultrasonic probe according to the first embodiment. [Figure 14B] Figure 14B is an image diagram obtained during the position estimation process of the ultrasonic probe according to the first embodiment. [Figure 14C] Figure 14C is an image diagram obtained during the position estimation process of the ultrasonic probe according to the first embodiment. [Figure 14D] Figure 14D is an image diagram obtained during the position estimation process of the ultrasonic probe according to the first embodiment. [Figure 15] Figure 15 is an image showing the superimposition of skeletal information as the result of estimating the position and posture of the human body and crosshairs as the result of estimating the probe position according to the first embodiment. [Figure 16]FIG. 16 is a diagram showing an example of a screen displayed on a display when measuring an ultrasonic image according to the first embodiment. [Figure 17A] FIG. 17A is an image diagram of a screen displayed on a display when specifying an examination site according to the first embodiment. [Figure 17B] FIG. 17B is an image diagram of a screen displayed on a display when specifying an examination site according to the first embodiment. [Figure 18] FIG. 18 is an image diagram showing an example of a cable region of interest according to Modification 4.

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of the information processing apparatus, medical image diagnostic apparatus, program, and storage medium according to the present application will be described in detail with reference to the accompanying drawings. Note that the information processing apparatus, medical image diagnostic apparatus, program, and storage medium according to the present application are not limited to the embodiments shown below. In the following description, the same components are given common reference numerals and redundant descriptions are omitted.

[0009] (First Embodiment) An embodiment will be described. Here, in the first embodiment, an ultrasonic diagnostic apparatus as an example of the medical image diagnostic apparatus according to the present application will be described. That is, in the first embodiment, an example of an ultrasonic diagnostic apparatus including the information processing apparatus according to the present application will be described. Note that the information processing apparatus according to the present application is applicable not only to a medical image diagnostic apparatus but also to any electronic device capable of processing an imaged image. These electronic devices may include, for example, a mobile phone, a tablet terminal, a personal computer, a watch-type or glasses-type information terminal, and the like.

[0010] Figure 1 shows an example configuration of an ultrasound diagnostic apparatus 100 according to the first embodiment. As shown in Figure 1, the ultrasound diagnostic apparatus 100 comprises an ultrasound diagnostic apparatus body 1, an ultrasound probe 2, a camera 3, an arm 4, a display 5, and a control panel 6. The ultrasound diagnostic apparatus body 1 has a computer built into its casing as an information processing device, equipped with various control units, a power supply, and a communication I / F (interface).

[0011] The ultrasonic probe 2 is an example of an inspection device according to this embodiment, and is an ultrasonic transducer that transmits and receives ultrasonic waves while its tip surface is in contact with the surface of the subject. The ultrasonic probe 2 has a plurality of piezoelectric transducers and is connected to the ultrasonic diagnostic device body 1. The ultrasonic probe 2 generates ultrasonic waves in the plurality of piezoelectric transducers based on control signals supplied from the ultrasonic diagnostic device body 1, receives reflected waves from the subject, and converts them into electrical signals (echo signals). For example, the ultrasonic probe 2 may be any type of ultrasonic probe, such as sector type, linear type, or convex type. Furthermore, the ultrasonic probe 2 may be a one-dimensional ultrasonic probe in which a plurality of piezoelectric transducers are arranged in a line, an ultrasonic probe in which the plurality of piezoelectric transducers of a one-dimensional ultrasonic probe are mechanically oscillated, or a two-dimensional ultrasonic probe in which a plurality of piezoelectric transducers are arranged in a grid in two dimensions.

[0012] Figure 2 shows an example of the configuration of the ultrasonic probe 2 according to the first embodiment. As shown in Figure 2, the ultrasonic probe 2 comprises a probe body 201, a cable 202, a cable bush 203, a freeze button 6a, and a confirmation button 6b.

[0013] The probe body 201 incorporates multiple piezoelectric transducers and transmits and receives ultrasound waves based on control signals received via the cable 202, and transmits echo signals based on reflected waves to the ultrasound diagnostic device body 1 via the cable 202. The cable 202 is a cable that electrically connects the ultrasound diagnostic device body 1 and the probe body. The cable bush 203 is a protective member that protects the cable 202. The freeze button 6a is a button for temporarily pausing ultrasound transmission and reception in the probe body 201. The confirm button 6b is a button for saving the ultrasound image.

[0014] Camera 3 is installed at the tip of an arm 4 attached to the main body 1 of the ultrasound diagnostic device and can be used to image the area around the ultrasound diagnostic device 100. In this embodiment, camera 3 is mainly used to acquire an external image for identifying the examination site when examining a subject using the ultrasound probe 2. Specifically, camera 3 captures an external image including the examination site and the ultrasound probe 2 when examining a subject using the ultrasound probe 2.

[0015] Camera 3 has the configuration of a typical camera, including an imaging optical system, an image sensor, a CPU, an image processing circuit, ROM, RAM, and at least one communication interface. Light beams from a subject are imaged onto an image sensor, such as a CCD or CMOS sensor, by the imaging optical system, which consists of optical elements such as lenses, to perform imaging. The imaging optical system includes a lens group, and Camera 3 also includes a lens drive control circuit that controls zoom and focus by driving the lens group in the optical axis direction. The electrical signal output from the image sensor is converted into digital image data by an A / D converter, various image processing is performed by the image processing circuit, and the result is output to an external device. At least a portion of the image processing performed by the image processing circuit may be output to an external device via a communication interface and then processed by the processing circuit of the external device.

[0016] In this embodiment, camera 3 primarily uses an image sensor that receives and captures light in the visible light region. However, the camera 3 is not limited to this; it may also be a camera that receives and captures light in the infrared region, or a camera that receives and captures light in multiple wavelength regions, such as visible light and infrared light. Furthermore, it may be a stereo camera that enables distance measurement in addition to external image capture, or a camera equipped with a TOF (Time of Flight) sensor for distance measurement. Hereafter, images captured by camera 3 will be referred to as camera images.

[0017] Arm 4 is installed on the main body 1 of the ultrasound diagnostic device and is used to position the camera 3 in a location and orientation that allows for the acquisition of an external image including the examination site of the subject and the ultrasound probe 2. In this embodiment, arm 4 is a serial link mechanism arm with five joints. The joint to which the camera 3 is connected at the tip of arm 4 is a ball joint, which allows for easy adjustment of the camera 3's orientation.

[0018] Display 5 includes a display device such as an LCD and displays images, menu screens, and graphical user interfaces (GUIs) input from the ultrasound diagnostic device main unit 1. Specifically, Display 5 displays images stored in the memory of the ultrasound diagnostic device main unit 1 and images recorded in the non-volatile memory. Display 5 also displays ultrasound images, camera images, body mark images, probe mark images, and site identification results. Here, the body mark image is a simplified representation of the body shape and is commonly used in ultrasound diagnostic devices. The probe mark image is a mark displayed superimposed on the body mark image and is added to allow for quick identification of the angle at which the ultrasound probe 2 is in contact with the tangent plane of the body.

[0019] The control panel 6 consists of a keyboard, trackball, switches, dials, touch panel, etc. Using these operating components, the control panel 6 receives various input operations from the examiner, such as instructions for imaging using the ultrasound probe 2 or camera 3, instructions for displaying various images, image switching, mode selection, and instructions for various settings. The received input operation signals are input to the ultrasound diagnostic device main unit 1 and reflected in various controls. If the control panel 6 is a touch panel, it may be integrated with the display 5, and the examiner can perform various settings and operations on the ultrasound diagnostic device main unit 1 by touching or dragging buttons displayed on the display.

[0020] When the ultrasound diagnostic device 1 is updating the ultrasound image in its memory based on a signal received from the ultrasound probe 2, if the examiner operates the freeze button 6a, the signal from the ultrasound probe 2 stops, and the updating of the ultrasound image in memory is temporarily paused. At this time, the signal from the camera 3 also stops, and the updating of the camera image in memory is temporarily paused. If the freeze button 6a is operated again while the updating of the camera image in memory is paused, the ultrasound probe 2 starts receiving a signal again, and the updating of the ultrasound image in memory starts, and the updating of the camera image also starts in the same way. When the ultrasound image has been fixed to a single image by pressing the freeze button 6a, if the examiner operates the confirm button 6b, that ultrasound image is saved to non-volatile memory. The freeze button 6a and confirm button 6b may also be provided on the control panel 6 instead of the ultrasound probe 2.

[0021] Figure 3 is a block diagram showing an example configuration of the ultrasound diagnostic apparatus body 1 according to the first embodiment. The ultrasound diagnostic apparatus body 1 includes a transmit / receive circuit 12, a signal processing circuit 13, an image generation circuit 14, a camera control circuit 15, a processing circuit 7, a memory 8, a non-volatile memory 9, a communication I / F 10, and a power supply 11 connected to an internal bus 17. Each component connected to the internal bus 17 is configured to exchange data with each other via the internal bus 17.

[0022] In the ultrasound diagnostic device 100, each processing function is stored in memory 154 in the form of a program that can be executed by a computer. The transmitting / receiving circuit 12, signal processing circuit 13, image generation circuit 14, camera control circuit 15, and processing circuit 7 are processors that realize the functions corresponding to each program by reading and executing the program from the non-volatile memory 9. In other words, each circuit, when it has read a program, has the function corresponding to the read program.

[0023] In the above description, the term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). The processor performs its functions by reading and executing a program stored in memory. Alternatively, instead of storing the program in memory, the processor may be configured to directly incorporate the program into its circuitry. In this case, the processor performs its functions by reading and executing the program incorporated into the circuitry. In this embodiment, each processor is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor, and its functions may be achieved through this combination.

[0024] Memory 8 consists of, for example, RAM (volatile memory using semiconductor elements). Processing circuit 7 controls each component of the ultrasound diagnostic device body 1 using memory 8 as work memory, according to a program stored in non-volatile memory 9, for example. For example, as shown in Figure 3, processing circuit 7 performs various processes in the ultrasound diagnostic device 100 by executing control function 7a, image acquisition function 7b, and estimation function 7c. Here, control function 7a is an example of control means. Image acquisition function 7b is an example of acquisition means. Estimation function 7c is an example of cable estimation means, estimation means, examination site estimation means, first posture estimation means, and second posture estimation means.

[0025] The control function 7a performs various processes related to the acquisition and display of ultrasound images. The image acquisition function 7b performs various processes related to the acquisition of external images. The estimation function 7c performs various processes related to the estimation of the scanning position of the subject scanned by the ultrasound probe 2. Details of the processes of the control function 7a, image acquisition function 7b, and estimation function 7c will be described later.

[0026] The non-volatile memory 9 stores image data, subject data, and various programs necessary for the operation of each circuit, including the processing circuit 7. The non-volatile memory 9 is composed of, for example, an HD or ROM.

[0027] The transmitting / receiving circuit 12 includes at least one communication interface for supplying power to the ultrasonic probe 2, transmitting control signals, and receiving echo signals. The transmitting / receiving circuit 12 supplies control signals to the ultrasonic probe 2 to transmit an ultrasonic beam, for example, based on control signals from the processing circuit 7. Furthermore, the transmitting / receiving circuit 12 receives the reflected wave signal, i.e., the echo signal, from the ultrasonic probe 2, performs phase-correcting summation on the received signal, and outputs the signal obtained by the phase-correcting summation to the signal processing circuit 13.

[0028] The signal processing circuit 13 includes a B-mode processing circuit (or Bc-mode processing circuit), a Doppler mode processing circuit, a color Doppler mode processing circuit, and the like. The B-mode processing circuit visualizes the amplitude information of the received signal supplied from the transmitting / receiving circuit 12 using known processing and generates B-mode signal data. The Doppler mode processing circuit extracts the Doppler shift frequency component from the received signal supplied from the transmitting / receiving circuit 12 using known processing, and further applies FFT (Fast Fourier Transform) processing to generate Doppler signal data of blood flow information. The color Doppler mode processing circuit visualizes blood flow information based on the received signal supplied from the transmitting / receiving circuit 12 using known processing and generates color Doppler mode signal data. The signal processing circuit 12 outputs the various generated data to the image generation circuit 14.

[0029] The image generation circuit 14 generates two-dimensional and three-dimensional ultrasound images of the scan area by known processing based on the data supplied from the signal processing circuit 13. For example, the image generation circuit 14 generates volume data of the scan area from the supplied data. From the generated volume data, the image generation circuit 14 generates two-dimensional ultrasound image data by MPR processing (multi-plane reconstruction method) or three-dimensional ultrasound image data by volume rendering processing. Examples of ultrasound images include B-mode images, Doppler-mode images, color Doppler-mode images, and M-mode images.

[0030] The camera control circuit 15 is equipped with at least one communication interface (I / F) for supplying power to the camera 3, transmitting and receiving control signals, and transmitting and receiving image signals. The camera 3 may not receive power from the ultrasound diagnostic device body 1 and may have its own power supply for independent operation. Furthermore, the camera control circuit 15 can control various imaging parameters of the camera 3, such as zoom, focus, and aperture value, by transmitting control signals to the camera 3 via the communication interface. The camera 3 may be equipped with a pan-tilt head that can automatically pan and tilt, and may be configured to receive pan-tilt control signals and control its position and attitude by pan-tilt drive. Alternatively, the tip of the arm 4 may be equipped with a drive unit and drive control circuit for electrically controlling the position and attitude of the camera 3, and the position and attitude of the camera 3 may be controlled based on control signals from the camera control circuit 15 or the processing circuit 7.

[0031] The configuration of the ultrasound diagnostic device 100 according to this embodiment has been described above. With this configuration, the ultrasound diagnostic device 100 enables the estimation of the examination site (scanning position) that is robust to occlusion. In examinations using the ultrasound diagnostic device 100, information on the examination site (contact position of the ultrasound probe) by the ultrasound probe can be input as a probe mark each time an ultrasound image is acquired. However, when there are many examination sites or a large number of subjects, the burden on the examiner is great, and there is a risk of input errors.

[0032] Therefore, as a method for automating the input of the examination site, for example, the method using the external image described above has been proposed. However, the shape and design of the ultrasound probe have been determined with operability in mind and through long-term use by medical professionals. For this reason, it is preferable to have a shape and design that does not change as much as possible and has little impact on operability. Consequently, when estimating the examination site of the ultrasound probe using an external image under such constraints, because the ultrasound probe is held and operated by hand, it may not be possible to stably capture the probe in the image in a way that allows the probe itself to be recognized due to occlusion by the hand.

[0033] The size of the examiner's hands and the way they hold the ultrasound probe vary. Furthermore, even the same examiner typically changes their grip on the ultrasound probe depending on the area being examined. Therefore, this embodiment enables robust estimation of the examination area against ultrasound probe occlusion.

[0034] The following describes the processing performed by the ultrasound diagnostic device 100. Figure 4 is a flowchart showing the overall flow of the ultrasound diagnostic device 100 according to the first embodiment. Here, each step shown in Figure 4 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0035] In step S401, the control function 7a turns on the power in response to the examiner's operation, loads the OS (operating system) stored in the non-volatile memory 9, and then in step S402, automatically starts the ultrasound diagnostic application. At this time, the control function 7a sends an image signal of the startup screen to the display 5 for display.

[0036] Then, after the ultrasound diagnostic application is launched, the control function 7a performs an initialization process and switches the display screen of the display 5 to the subject information registration screen. In step S403, it accepts instructions to register subject information in response to the examiner's operation on the control panel 6. Here, subject information includes the examination site according to the subject's medical condition (mammary gland, heart, artery, abdomen, carotid artery, thyroid gland, vein, etc.), subject ID, name, gender, date of birth, age, height, weight, whether the patient is hospitalized or outpatient, etc. After the subject information is entered, when the start button on the control panel 6 (on the display or on the operation panel) is pressed by the examiner, the control function 7a saves the subject information to memory 8 or non-volatile memory 9. After that, the control function 7a switches the display screen of the display 5 to the measurement screen of the ultrasound diagnostic application.

[0037] Furthermore, in step S403, the control function 7a accepts a setting to determine whether the inspection site is set manually or automatically. The flow when the inspection site is set manually will be described later using Figure 5. The flow when the inspection site is set automatically (modified example 1) will be described later using Figure 7.

[0038] Then, after switching to the measurement screen of the ultrasound diagnostic application, the control function 7a performs ultrasound diagnostic measurement processing in step S404 according to the examiner's operation. Details of the measurement processing will be described later.

[0039] Once all parts have been inspected, the control function 7a saves the inspection data obtained in step S405 to the non-volatile memory 9 or an external media (not shown), or transfers it to an external device (external server) via the communication I / F 10.

[0040] Once all processing is complete and the power is turned off by the examiner, the control function 7a terminates the ultrasound examination application and OS in step S406, and ends the series of processes.

[0041] Next, the details of the process in step S404 will be described. Figure 5 is a flowchart showing the processing flow of the measurement process according to the first embodiment. Here, Figure 5 shows the process when the inspection area is set to be manually set by the inspector. Furthermore, each step shown in Figure 5 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0042] In step S501, the control function 7a generates an ultrasound image by applying signal processing and image processing to the echo signal received from the probe 2 and displays it on the display 5. Details of the ultrasound image processing in step S501 will be described later.

[0043] The examiner checks the ultrasound image displayed on the display 5 and presses the confirmation button 6b when the desired ultrasound image is obtained. In step S502, the control function 7a confirms the ultrasound image according to the examiner's operation and stores the ultrasound image in the memory 8.

[0044] Here, the control function 7a causes the display 5 to display a screen for setting the examination site. Figure 16 is a diagram showing an example of a screen displayed on the display when measuring an ultrasound image according to the first embodiment. For example, as shown in Figure 16, the control function 7a causes the display 5 to display a screen that shows the ultrasound image 2101, the examination site 2102, and an external image 2103 including the ultrasound probe 2.

[0045] The examiner performs various operations while referring to the screen in Figure 16. In response to the examiner's operations, the control function 7a sets information about the inspected area, such as the name of the inspected area, body marks, and probe marks, in order to record information about which area was inspected in step S503. The control function 7a can also add annotations such as comments and arrows to the display 5 in response to the examiner's operations.

[0046] When the confirm button is pressed by the examiner, the control function 7a stores the information of the examination site in the memory 8 in step S504.

[0047] When the measurement process for a particular inspection area is completed, in step S505, the processing circuit 7 determines whether all inspection areas predetermined according to the inspection content have been measured. If there are still areas that have not been inspected (step S505, No), the control function 7a returns to step S501 and continues processing. The inspection content information is selected and set according to the operator's operation from a collection of information that has been pre-classified and recorded in the non-volatile memory 9 according to the inspection area and symptoms. On the other hand, if it is determined that the measurement process for all inspection areas has been completed (step S505, Yes), the control function 7a terminates the process in step S404.

[0048] Next, the details of the process in step S501 will be described. Figure 6 is a flowchart showing the processing flow of ultrasonic image processing according to the first embodiment. Here, each step shown in Figure 6 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0049] As described above, the ultrasonic probe 2 is an example of an examination device, and it scans the scan area while transmitting ultrasound into the subject using each piezoelectric transducer, and receives the reflected waves from the subject as echo signals. In this embodiment, the ultrasonic probe 2 is designed to be held and operated by the examiner. In step S601, the echo signals sent from the ultrasonic probe 2 are processed by the signal processing circuit 13 and the image generation circuit 14 to generate an ultrasonic image, which is then displayed on the display 5 by the processing circuit 7.

[0050] To obtain the desired image, the examiner can adjust and correct various processing parameters using the control panel 6 while checking the ultrasound image displayed on the display 5. Specifically, in step S602, various parameters (mode, gain, focus, echo level, etc.) are changed according to the operation signals received by the control panel 6, and the modified ultrasound image is regenerated and displayed on the display 5.

[0051] The ultrasound probe 2 is equipped with a freeze button 6a, and in step S603, the control function 7a determines whether or not the freeze button 6a has been pressed. If the freeze button 6a is not pressed (step S603, No), steps S601 and S602 are repeated. On the other hand, if the freeze button 6a is pressed (step S603, Yes), the control function 7a determines whether or not the desired ultrasound image has been acquired, displays the acquired and generated ultrasound image on the display 5, and in step S604, determines whether or not the confirmation button 6b has been pressed.

[0052] If the confirmation button 6b is not pressed (step S604, No), steps S601 and S602 are repeated. On the other hand, if the confirmation button 6b is pressed (step S604, Yes), the control function 7a completes the ultrasonic image processing flow.

[0053] (Variation 1) Here, a modified version of the measurement process in step S404 will be described. Figure 7 is a flowchart showing the processing flow of Modified Version 1 of the measurement process according to the first embodiment. Here, Figure 7 shows the processing when the inspection area is set to be automatically set by the inspector. That is, in Modified Version 1, the setting of the inspection area, which was performed in response to the inspector's operation in step S503 of Figure 5, is automated. Note that each step shown in Figure 7 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0054] In step S701, the control function 7a causes the camera control circuit 15 to activate and control the camera 3, thereby capturing an image including the subject. Figure 11 is an image diagram of a measurement using the ultrasound diagnostic device 100 according to the first embodiment. As shown in Figure 11, in an examination using the ultrasound diagnostic device 100, the examiner grasps the ultrasound probe 2, brings the transmitting and receiving surface of the ultrasound probe into contact with the body surface of the subject P, and transmits and receives ultrasound to scan the examination area within the subject.

[0055] In this embodiment, the examiner moves the arm 4 to position the camera 3 appropriately, including a portion of the subject and the probe 2 in the field of view, and the camera 3 captures an external image by operating the control panel 6 or an operating member such as a shutter button pre-installed on the camera 3. Note that the capture of the external image is not limited to this, and at least one of the steps for controlling the position and orientation of the camera 3 and controlling the image capture may be automated. For example, if a camera 3 with a pan-tilt head equipped with a pan-tilt mechanism is attached to the tip of the arm 4, the camera control circuit 15 drives and controls the position and orientation of the camera 3 to an appropriate position and orientation. For example, the camera control circuit 15 detects the subject P and the ultrasound probe 2 from the captured image at the current camera position and orientation using image analysis, and controls the position and orientation of the camera 3 by pan-tilt control so that at least one of the two is included in the field of view. Alternatively, the arm 4 may be a 6-axis articulated robot, and the camera control circuit 15 may control the position and orientation of the camera 3 by controlling the 6-axis articulated robot.

[0056] If, in this case, part of the subject cannot be detected from the image captured at the current position and orientation of camera 3, the camera control circuit 15 repeats the movement and imaging a predetermined number of times by panning and tilting the camera to a different field of view until detection is achieved. After camera 3 is controlled to an appropriate position and orientation (or after a camera with an appropriate arrangement has been determined), the image acquisition function 7b causes camera 3 to capture an external image including part of the subject and acquires the external image. Furthermore, if images from multiple fields of view (multiple viewpoints, multiple positions and orientations) are required for the estimation process of the subject's position and orientation, camera 3 may be driven and controlled to multiple positions and orientations and imaging may be performed multiple times.

[0057] The estimation function 7c estimates the position and orientation of a subject lying on a bed or the like from the external image acquired from the camera 3, and displays the estimation result on the display 5. When the estimation result of the subject's position and orientation is finalized in response to the examiner's operation on the control panel 6, the subject's position and orientation at that time is stored in the memory 8. Details of the estimation process for the subject's position and orientation within the field of view, the estimation process for the position and orientation of the ultrasound probe 2, and the process for identifying the examination site based on both estimation results will be described later.

[0058] Subsequently, the ultrasound image processing flow in step S702 and the examination site identification A flow in step S703 are processed in parallel. Note that step S702 is equivalent to the ultrasound image processing flow in step S501, which was explained earlier using Figure 6.

[0059] In step S703, the estimation function 7c automatically estimates the examination site using the external image acquired from the camera 3 and displays the estimation result on the display 5. Details of the examination site estimation process will be described later. When the examiner presses the freeze button 6a on the ultrasound probe 2, the updating of the ultrasound image on the display 5 based on the echo signal from the ultrasound probe 2 is stopped and the image is finalized. Once the ultrasound image is finalized, or immediately thereafter, the control function 7a displays the estimated examination site result at the time the freeze button 6a was pressed on the display 5 and exits steps S702 and S703.

[0060] In step S704, the control function 7a determines whether the ultrasound image has been confirmed or not. For example, if the ultrasound image is desired, the examiner confirms the ultrasound image by pressing the confirmation button 6b. On the other hand, if the ultrasound image is not desired, the examiner does not confirm the ultrasound image by not pressing the confirmation button 6b. If the confirmation button 6b is pressed by the examiner (step S704, Yes), the control function 7a proceeds to step S705 and performs post-measurement processing such as recording and display. On the other hand, if the examiner does not press the confirmation button 6b or performs another predetermined operation (step S704, No), the control function 7a returns to the parallel processing of steps S702 and S703. Other predetermined operations include, for example, pressing the freeze button 6a again.

[0061] In step S706, the control function 7a determines whether all the inspection areas predetermined according to the inspection content have been measured. If there are still areas that have not been inspected (step S706, No), the control function 7a returns to the parallel processing of steps S702 and S703. On the other hand, if the measurement of all areas has been completed (step S706, Yes), the control function 7a terminates the measurement processing of step S404. In this embodiment, the inspection content for inspecting multiple inspection areas is stored in the non-volatile memory 9 in advance, and the determination is made in step S706. However, it is also possible to have a configuration in which the flow ends without performing the above determination after the imaging and saving of one inspection area is completed.

[0062] Next, the details of the process in step S701 will be explained using Figures 8 and 12A to 12D. Figure 8 is a flowchart showing the processing flow of the human body position and posture estimation process according to the first embodiment. Figures 12A to 12D are illustrative diagrams of the output results obtained during measurement according to the first embodiment. Each step shown in Figure 8 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0063] In step S801, the image acquisition function 7b acquires an image including the examination device that scans the subject. For example, the image acquisition function 7b controls the camera control circuit 15 to take images of the subject lying on a bed or the like with the camera 3, as shown in Figure 12A. The camera 3 takes images sequentially at a predetermined frame rate, and the control function 7a receives the external images via the communication I / F of the camera control circuit 15 and displays them sequentially on the display 5.

[0064] The examiner adjusts the position of the arm 4 while checking the external image displayed on the display 5, so that at least a portion of the subject being examined is within the field of view of the camera 3. The display 5 may also display lines to guide the examiner in positioning the camera 3 so that the subject's examination area is in a predetermined position within the displayed external image. In this case, the guiding lines (for example, GUI data superimposed on the displayed image) are stored in the non-volatile memory 9 in advance, associated with information about the examination area. The control function 7a displays guide lines for adjusting the field of view of the camera 3 based on the information about the examination area and the information stored in the non-volatile memory 9.

[0065] In step S802, the estimation function 7c estimates the position and posture of the human body by image analysis processing based on the image from camera 3 acquired as an external image. In this embodiment, the position and posture information of the human body is output as skeletal information consisting of the position coordinates of feature points such as each joint. Joints include the nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, central hip, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, left thumb, left little finger, left heel, right thumb, right little finger, right heel, etc. As a means of obtaining skeletal information from images, a learner trained using machine learning (deep learning) is used. For example, in this embodiment, a learning model (learner) that has been pre-trained with a set of multiple training images containing the human body as a subject and correct information of skeletal information (probability distribution of joints, etc.) in each training image is used. This method makes it possible to acquire skeletal information in 2D or 3D coordinates, regardless of whether the information obtained from the camera (including stereo cameras, infrared cameras, and TOF cameras) consists only of luminance images, only depth images, or both. Examples of such learning systems include OpenPose® from Carnegie Mellon University.

[0066] In this embodiment, positional orientation information (skeletal information) estimated by a machine learning learner is stored in memory in the form of an array or list. Specifically, for multiple parts such as joints as described above, information showing the probability distribution of the presence of each part in the image is output as estimated positional orientation information. If Rn(x,y) is the probability distribution of the confidence level of the presence of part n (where n is an integer) in the image, then the output R as skeletal information is expressed as R={Rn(x,y)|n=1,2,...,N,N is an integer}. Furthermore, Rn(x,y) does not have to be the confidence level distribution of the entire region in the image, but may be the confidence level distribution of only the region with a confidence level greater than a threshold. Alternatively, only the peak value of the confidence level may be stored as Rn(x,y) in association with its coordinates (for example, part 3: right shoulder, confidence level: 0.5, coordinates: (x,y)=(122, 76)).

[0067] Figure 12B shows an example of visualizing skeletal information by extracting the peak confidence value for each part (i.e., the location where each part is detected with the highest probability of being present) from the output R, which is output as skeletal information. For example, as shown in Figure 12B, skeletal information can be visualized by showing the location of each joint as a point and connecting the joints with straight lines for the human body included in the external image.

[0068] The means for obtaining skeletal information from an image are not limited to the examples described above, and other known means can be arbitrarily applied. In this embodiment, as preprocessing for obtaining skeletal information from an image using a trained learner, image quality corrections such as noise reduction, distortion correction, color conversion, brightness and color gradation correction, and image rotation and flipping are performed. The parameters for correction are compiled into a table according to the model of the camera 3 used for imaging and the imaging conditions at the time of imaging, and stored in the non-volatile memory 9. The estimation function 7c applies correction processing to the input appearance image using these correction parameters, and performs inference with higher accuracy by bringing it closer to the imaging conditions of the images in the dataset used for training. For example, when correcting the brightness of an image taken in a dark room, high-sensitivity noise may occur, and the trend may differ from that of the dataset used for training. In such cases, it is preferable to apply a process to remove high-sensitivity noise to the input appearance image. Similarly, if the lens of the camera 3 is wide-angle and the peripheral distortion is large, it is preferable to perform distortion correction on the input appearance image. Also, if all the images in the dataset have the head facing upwards, it is preferable to rotate or flip the image so that the head is facing upwards before inputting it. Furthermore, if the model is being trained using images that have been transformed in some way, it is preferable to transform the input images in the same manner before inputting them into the model.

[0069] As shown in Figure 12B, in step S802, it is possible that skeletal information of the examiner and surrounding people, in addition to the subject, is also acquired. Therefore, in step S803, the estimation function 7c identifies the subject's skeletal information from the skeletal information image obtained in step S802.

[0070] One possible method for identifying the skeletal information of a subject is to combine it with facial recognition processing. The estimation function 7c authenticates the examiner's face, which is pre-registered in the non-volatile memory 9, from the appearance image. For example, the estimation function 7c identifies the examiner's skeletal information and the subject's skeletal information based on the distance relationship between the location where the examiner's face is authenticated and detected, and the parts of the face detected in the skeletal information detected in step S802 (eyes, nose, ears, etc.).

[0071] Alternatively, the ultrasound diagnostic device 100 may be identified, and the examiner and the subject may be identified based on their planar (image XY direction) or three-dimensional (Z direction) distance from the ultrasound diagnostic device 100. Another method involves identifying the type of posture based on the positional relationship of joint points in the skeletal information, and then identifying the examiner and the subject. Yet another method involves procedurally adjusting the field of view of the camera 3 so that the subject is captured within a predetermined area, and then identifying the subject based on their position in the skeletal information.

[0072] In this embodiment, the estimation function 7c performs at least one of the above identification methods to identify the examiner and the subject, and visualizes the positional orientation information of the identified subject as shown in Figure 12C. That is, the estimation function 7c generates positional orientation information that includes only the skeletal information of the subject, as shown in Figure 12C.

[0073] Then, in step S804, the control function 7a displays on the display 5 an image on which the estimated position and orientation information of the subject is superimposed on the display image (appearance image) from the camera 3, for example, as shown in Figure 12D. Here, the image that the control function 7a displays on the display 5 may not be the image acquired by the camera 3 directly, but may be replaced with an avatar or animation, or converted into a 3D model, using known image processing, in consideration of privacy.

[0074] Next, in step S805, if the freeze button 6a is pressed in response to the examiner's operation (step S805, Yes), the estimation function 7c finishes updating the estimation results of the subject's skeletal information (position and orientation information) based on the appearance images that were sequentially displayed on the display 5, and proceeds to step S806. If the freeze button 6a is not pressed (step S805, No), the estimation function 7c returns to step S801 and continues estimating the position and orientation information.

[0075] If the freeze button 6a is pressed in step S805, and the examiner confirms in step S806 that there are no problems with the estimated position and posture information displayed on the display 5 and operates the confirm button (step S806, Yes), the estimation function 7c terminates the human body position and posture estimation process. On the other hand, if the desired position and posture result is not obtained in step S806 and the confirm button is not pressed (step S806, No), the image acquisition function 7b and the estimation function 7c return to step S801 and repeatedly perform position and posture estimation.

[0076] The reason for waiting for confirmation by the examiner in step S806 is to address situations such as when the subject's posture is not as desired, the camera angle is unfavorable, or the position and posture detection results deviate significantly from what is seen by the human eye. In other embodiments, steps S805 and S806 may be omitted (ignored). That is, the position and posture information of the human body may continue to be updated at predetermined time intervals until the examiner determines that the subsequent process of identifying the inspection area is appropriate and the confirmation operation is performed.

[0077] Furthermore, the subject's body shape may be estimated by selecting one or more pre-modeled human body models based on information obtained from external images (positional relationships of each joint, size of body regions, etc.) or registered subject information (gender, height, weight, etc.), or by parametrically deforming these models. Such technology can also be realized using machine learning.

[0078] Next, the details of the process in step S703 will be explained using Figures 9, 13, 14A to 14D, 15, 17A, and 17B. Figure 9 is a flowchart showing the processing flow of the inspection area identification process according to the first embodiment. Figure 13 is an image showing the external appearance image and the ultrasound probe contained within the external appearance image according to the first embodiment. Figures 14A to 14D are image diagrams obtained during the ultrasound probe position estimation process according to the first embodiment. Figure 15 is an image diagram superimposed with skeletal information as the estimation result of human body position and posture and crosshairs as the estimation result of probe position according to the first embodiment. Figures 17A and 17B are image diagrams of the screen displayed on the display when the inspection area is identified according to the first embodiment. Note that each step shown in Figure 9 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0079] In step S901, the image acquisition function 7b acquires an external image (image data) including the ultrasound probe 2 from the camera 3 via the communication I / F of the camera control circuit 15. Since the field of view of the camera 3 has been adjusted in the previous step to include the subject, imaging can be performed as is, but at least one of the following may be performed to obtain a field of view that makes it easier to detect the ultrasound probe 2: pan control, tilt control, zoom control, etc.

[0080] In such cases, the image acquisition function 7b controls the camera control circuit 15 to adjust the field of view so that the ultrasound probe 2 has a more easily detectable field of view. For example, as shown in Figure 13, the image acquisition function 7b adjusts the field of view of the camera 3 from an external image mainly of the subject to a field of view mainly of the ultrasound probe 2, as shown by the dotted rectangle.

[0081] The estimation function 7c estimates the position of the cable of the ultrasonic probe 2 based on the image. Specifically, the estimation function 7c performs a process to estimate the position of the cable of the ultrasonic probe 2, including the cable bush (a general term for components such as bearing cylinders and protrusions provided at the cable end). For example, the estimation function 7c performs two different processes on the acquired appearance image and then combines the results of each. In the first process, in step S902, the estimation function 7c analyzes the acquired appearance image and performs a segmentation process on the cable bush 203 of the ultrasonic probe 2 to obtain a cable bush segmentation map. Specifically, the estimation function 7c infers the likelihood of each pixel being a cable bush using a learner that performs cable bush segmentation and is pre-stored in the non-volatile memory 9.

[0082] Here, the learning model is trained using external images of the ultrasonic probe 2 being operated and data in which the cable bushing areas within those images have been annotated. Various learning models have been proposed, such as Google's DeepLab.

[0083] To facilitate segmentation, the cable bush 203 of the ultrasound probe 2 can be given easily distinguishable image features, which makes learning easier and improves detection accuracy. For example, the cable bush 203 can be given a distinctive color (e.g., blue) or a distinctive texture (e.g., a mesh pattern). Alternatively, an infrared coating can be applied to the cable bush 203 so that when imaging with infrared illumination and an infrared camera, the brightness value of the cable bush area is high and a prominent appearance image can be obtained. Ultraviolet light may be used instead of infrared light.

[0084] For segmentation processing, machine learning is robust to environmental changes and shows good performance. However, if environmental conditions can be controlled, characteristic images can be acquired by applying such modifications to the cable bush 203, and then the segmentation can be performed using rule-based thresholding or similar methods.

[0085] In the second step, step S903, the estimation function 7c analyzes the acquired appearance image and performs segmentation processing on the cable portion connected to the probe body 201 to obtain a cable segmentation map. The specific method is the same as described in step S902. Also, similar to step S902, by giving the cable visual features that make it easy to distinguish from others, learning becomes easier and detection accuracy can be improved.

[0086] Although steps S902 and S903 are described as separate steps in Figure 9, a single learning model trained to output segmentation maps for both the cable bush and the cable may also be used.

[0087] For example, the estimation function 7c obtains the segmentation processing result shown in Figure 14B by inputting the appearance image shown in Figure 14A into the learning device described above. That is, as shown in Figure 14B, the estimation function 7c estimates the cable bush region R1 and the cable region R2 in the appearance image.

[0088] Then, in step S904, the estimation function 7c performs blob analysis based on the segmentation map obtained in step S902. Specifically, the estimation function 7c smooths the image of the segmentation map by blurring it using a Gaussian filter or the like, and then performs binarization. Furthermore, it performs morphological processing to remove noise and combine the fragmented regions to obtain a region of connected pixels (blob).

[0089] Subsequently, in step S905, the estimation function 7c determines whether the area of ​​the extracted blob is greater than or equal to a threshold previously stored in the non-volatile memory 9. If the area of ​​the blob is greater than or equal to the threshold (step S905, Yes), the estimation function 7c decides to use the extracted blob as the cable bushing area and proceeds to step S906. On the other hand, if the area of ​​the extracted blob is less than the predetermined threshold (step S905, No), the estimation function 7c proceeds to step S909.

[0090] In step S906, the estimation function 7c calculates the moment of the acquired cable bushing region to determine the major and minor axes of the blob and obtains the ratio a of the minor axis to the major axis. The estimation function 7c then determines whether the ratio a of the major axis to the minor axis is greater than or equal to a threshold t (t is a real number greater than or equal to 1, which is a parameter) that is set in the non-volatile memory 9. If the ratio a of the major axis to the minor axis is greater than or equal to the threshold t (step S906, Yes), the estimation function 7c proceeds to step S907. On the other hand, if the ratio a of the major axis to the minor axis is less than the set threshold t (step S906, No), the estimation function 7c proceeds to step S909.

[0091] In steps S907 and S908, the estimation function 7c estimates the position and orientation of the probe body 201 based on the external image. That is, the estimation function 7c estimates the position and orientation of the probe body based on the positional relationship between the cable bush and the cable of the ultrasonic probe in the image. Specifically, in step S907, the estimation function 7c finds two cable-related regions in the image that are of interest regarding the presence or absence of the cable. For example, the estimation function 7c first finds the coordinates of the two endpoints P(x1, y1) and Q(x2, y2) of the major axis of the cable bush region. Furthermore, the estimation function 7c finds the centroid coordinate G(x0, y0) of the cable bush region.

[0092] Next, the estimation function 7c determines two isosceles triangular regions with the centroid G of the cable bush region as its vertex. One of these isosceles triangles is defined as the isosceles triangular region (Cable Region of Interest 1) which is symmetrical with respect to the line segment GP' obtained by multiplying the line segment GP by n in the direction of P (n is a real number parameter). The other isosceles triangular region (Cable Region of Interest 2) is defined similarly with respect to the line segment GQ' obtained by multiplying the line segment GQ by n in the direction of Q (n is a real number). In other words, the estimation function 7c sets up cable regions of interest on both sides of the cable bush region along its long axis to determine the presence or absence of cables. Note that the cable regions of interest can have any shape as long as it is suitable for the purpose, and one of its vertices does not have to be the centroid of the cable bush region. For example, instead of the centroid of the cable bush, the position of the examiner's hand obtained from the estimated position and orientation of the subject (R) may be used.

[0093] In step S908, the estimation function 7c determines the area occupied by the cable region determined in step S903 within each of the two cable interest regions determined in step S907, and determines that the tip center of the probe body 201 is in the direction with the smaller cable region area along the long axis of the cable bush region. That is, the estimation function 7c determines that the cable body 201 is on the opposite side of the cable on both sides along the long axis of the cable bush region.

[0094] For example, the estimation function 7c sets two cable regions of interest R3 along the long axis of the cable bushing region, as shown in Figure 14C. Then, the estimation function 7c determines that the center of the probe tip is on the side of the two cable regions of interest R3 that does not include cable region R2 (the lower right side of the cable bushing).

[0095] In steps S907 and S908, the cable was recognized using image processing to estimate the direction of the tip center position of the probe body 201. However, it is also possible to estimate the direction of the tip center position of the probe body 201 based on landmarks that are easily identifiable using image processing, such as landmarks near the connection point between the probe body 201 and the cable 202, the cable 202 itself, or the examiner's hand. Landmarks can be, for example, a distinctive color (e.g., green), a distinctive shape, or an LED.

[0096] Next, in step S909, the estimation function 7c estimates the probe tip position, which is the center of the tip of the probe body (the center of the ultrasonic transmitting and receiving surface). Here, the estimation method in step S909 differs depending on the conditions obtained in the previous step.

[0097] In one case, if the area of ​​the blob in step S905 is less than a preset threshold, and the area of ​​the cable obtained in step S903 is also less than a preset threshold, the estimation function 7c estimates that the probe body 201 is not visible on the screen.

[0098] In such cases, the estimation function 7c can estimate the position and orientation of the probe body 201 based on the shape of the examiner's hand depicted in the external image. Here, the way an ultrasound probe is held varies even for probes of the same type, and the shape of the hand at that time also varies. Therefore, first, (1) the type of probe, (2) information on how the hand is held (information on the shape of the hand in the external image when the probe is hidden by the hand), and (3) information on the position and orientation of the probe (the central position of the tip of the probe body) are pre-associated and stored in the memory 8. It is desirable that the hand shape in (2) be associated for both the right hand and the left hand.

[0099] The estimation function 7c acquires the type of ultrasound probe used in the examination and obtains corresponding information for that type. Then, the estimation function 7c extracts the shape of the examiner's hand from the external image and estimates the position and orientation of the probe body based on the extracted hand shape and the acquired corresponding information.

[0100] In another case, if the area of ​​the blob in step S905 is less than a preset threshold, and the area of ​​the cable obtained in step S903 is greater than or equal to the set threshold, the estimation function 7c determines that no cable bush was found and only the cable was found, and estimates the endpoint of the cable region as the center position of the tip of the probe body. Here, a situation in which only the cable is found but no cable bush is found is, for example, when the cable bush is considered as a cylinder and photographed at a small angle from the direction of its bottom surface. Therefore, the position of the cable endpoint on the image and the position of the center of the tip of the probe on the image will be close to each other.

[0101] For example, the estimation function 7c determines the endpoints of the cable region by performing blob analysis equivalent to step S904 after cable segmentation in step S903, then thinning the cable region, and finally determining the endpoints closest to the center of the screen as the cable endpoints. If no cable bushing is found and only the cable is found, the position of the examiner's hand, obtained from the estimated position and orientation of the subject (R), may be used as the estimated value of the probe tip's center position.

[0102] Furthermore, if the cable bush is not found and only the cable is found, the estimation function 7c can also estimate the position and orientation of the probe body 201 based on the direction of the cable. For example, in situations where the cable bush is hidden by the inspector's hand and is not depicted in the external image, the estimation function 7c estimates the position and orientation of the probe body 201 based on the direction of the cable. In this case, when using the direction of the cable, a landmark indicating the direction of the cable is attached to the cable in order to detect the direction of the cable.

[0103] For example, the circumferential coloring of the cable is made different at each angle near the cable bushing. Alternatively, a striped pattern is applied along the long axis of the cable near the cable bushing. Here, since the area near the cable bushing is less prone to bending and twisting, it is assumed that the relationship between the circumferential angle of the cable and the orientation of the probe body 201, as well as the distance between lines in the striped pattern, remain constant even during ultrasonic scanning.

[0104] The estimation function 7c estimates the orientation of the probe body 201 based on the color of the cable detected in the external image. In other words, the estimation function 7c estimates the orientation of the piezoelectric vibrators relative to the subject. Furthermore, the estimation function 7c estimates the tilt of the probe body 201 based on the density (distance between lines) of the cable's stripe pattern detected in the external image. In other words, the estimation function 7c estimates the tilt of the probe body 201 based on the change in distance between multiple lines applied along the long axis of the cable.

[0105] The estimation function 7c then estimates the center position of the tip of the probe body 201 based on the correspondence between the distance between the lines and the distance from the tip of the probe body 201 to any of the lines in the striped pattern. The higher-level correspondence is determined in advance and stored in memory 8.

[0106] In another case, if the area of ​​the blob in step S905 is greater than or equal to a preset threshold, and the ratio a of the major axis to the minor axis of the blob obtained in step S906 is less than the threshold t, the estimation function 7c estimates the center of gravity of the blob to be the center of the tip of the probe body. A situation in which there is no difference in the length of the major axis and the minor axis of the blob is assumed to be when the cable bush is considered as a cylinder and photographed at a small angle from the direction of its bottom surface. Therefore, the position of the center of gravity of the cable bush on the image and the position of the center of the tip of the probe body on the image will be close to each other.

[0107] In another case, if the ratio a of the major axis to the minor axis obtained in step S906 is greater than or equal to the set threshold t, the estimation function 7c estimates the probe tip position as follows. Specifically, the estimation function 7c obtains the ratio r (design value) of the length of the cable bush 203 and the probe body 201. Then, the estimation function 7c finds a coordinate at a distance of r times the length of the major axis of the cable bush obtained in step S906, toward the probe tip center obtained in step S908, and estimates the obtained coordinate as the tip center of the probe body.

[0108] Here, the cable bush 203 may deform during operation of the ultrasonic probe 2. If the deformation cannot be ignored, the tip center position of the probe body 201 may be determined as follows. In such cases, first, the estimation function 7c finds a suitable approximation curve from the cable bush region. Next, the estimation function 7c finds the tangent T of the approximation curve at the endpoint of the cable bush on the probe tip center side, which was found in step S908. When the length of the approximation curve in the cable bush region is L, the coordinate of the tangent T is found at a distance of length L multiplied by r from the cable bush endpoint on the probe tip center side toward the probe tip center side, and this is estimated as the tip center.

[0109] Generally, the probe appears in the image at an angle relative to the camera, but since the optical magnification of an object in the image changes depending on the distance between the camera and the subject, simply multiplying the long axis of the cable bush by r will result in errors in estimating the distance.

[0110] Assuming the cable bush length is 50mm, the distance between the probe body 201 and camera 3 is 500mm, and the probe body 201 is tilted at a 30-degree angle with respect to the camera's optical axis, an error of approximately 10% (5mm) will occur in the estimated length of the cable bush due to the optical magnification factor. This corresponds to an estimated error of approximately 10mm in the center of the probe body's tip if the distance from the cable bush to the center of the probe body's tip is 100mm. On the other hand, when the distance between the probe body 201 and the camera is 1000mm, the estimated error in the cable bush length due to the optical magnification factor is approximately 5%, and the estimated error in the center of the probe body's tip is approximately 5mm.

[0111] Therefore, depending on the required accuracy for estimating the tip position of the probe body, in the case of an ultrasound diagnostic device, it is preferable that the distance between the camera 3 and the ultrasound probe 2 be approximately 500 mm or more from the viewpoint of optical magnification.

[0112] Furthermore, when the distance between the camera 3 and the ultrasound probe 2 is short (for example, less than 500 mm), the optical magnification may be corrected based on the ratio of the major axis to the minor axis of the cable bush to reduce the estimation error of the tip center of the probe body. That is, the estimation function 7c detects the major axis and minor axis of the area indicating the cable bush, and estimates the position and direction of the inspection device by changing the ratio (r times as described above) of extending the length of the major axis of the cable bush according to the ratio (ratio a above). The extent to which the length of the major axis of the cable is extended in relation to the ratio of the major axis to the minor axis can be set arbitrarily.

[0113] Figure 14D shows an example of the result of estimating the tip center of the probe using the method described above. For example, the estimation function 7c obtains the ratio r (design value) of the length of the cable bush 203 and the probe body 201. Then, as shown in Figure 14D, the estimation function 7c finds the coordinate (coordinate of the arrowhead in the figure) at a distance of r times the length of the major axis of the cable bush region R1 toward the probe tip center, and estimates the obtained coordinate as the tip center of the probe body.

[0114] Then, in step S910, the estimation function 7c estimates the area to be examined by the probe body 201 based on the position and orientation of the probe body 201. Specifically, the estimation function 7c estimates the area to be examined by the probe body 201 based on the position and orientation of the probe body 201 and the position of the subject. For example, the estimation function 7c identifies the examination area from the relationship between the estimated position and orientation of the subject (R) obtained in step S701 and stored in memory 8 and the tip center position information (x,y) of the probe body 201 (hereinafter, position information (x,y)) obtained in step S909.

[0115] In step S901, the field of view is adjusted, and if the field of view of the image differs between the estimation of the subject's position and orientation and the acquisition of the probe body 201's position information, a correspondence is made between the two images. For example, if the external image used to estimate the subject's position and orientation differs from the external image used to estimate the probe body's position, the correspondence may be made using the pan control value, tilt control value, and zoom control value of the camera 3 controlled when acquiring each image, or feature points may be found from the images, and the correspondence may be made by transforming the position and size through projection changes based on the relationships between the feature points, or both may be done.

[0116] In this embodiment, multiple candidate examination sites are output as identified results, starting with those with high evaluation values ​​as examination sites in each skeletal information. When skeletal information R is obtained as information on the position and orientation of the subject, the following methods can be used to identify the examination site. To aid understanding, Figure 15 visualizes and illustrates the relationship between the estimated position and orientation result R of the subject and the position information (x,y) of the probe body 201. Note that in Figure 15, the skeletal information of the subject and the position information indicating the center of the tip of the probe body 201 with a cross are superimposed.

[0117] For example, the estimation function 7c extracts the coordinates of the position where the confidence level peaks in the confidence distribution Rn(x,y) for each part, i.e., the position with the highest confidence level for each part (simply called the coordinates of each part). Then, the estimation function 7c identifies the examination site based on the coordinates of each part and the distance from the position information of the tip center of the probe body 201 (Euclidean / Mahalanobis distance, etc.). In other words, the estimation function 7c calculates evaluation values ​​such that the closer the distance between the coordinates of each part and the tip center of the probe body 201, the higher the evaluation value. Furthermore, the estimation function 7c calculates evaluation values ​​for each part such that the greater the confidence level corresponding to the coordinates of each part, the greater the weight assigned to it. Then, the estimation function 7c extracts candidate examination sites in order from those with relatively high evaluation values ​​among multiple parts. Note that when calculating evaluation values, only one of the following may be used: the distance from the tip center of the probe body 201 or the confidence distribution of each part.

[0118] Alternatively, evaluation values ​​may be calculated for each region in the image using the confidence distribution Rn(x,y) for each part. That is, the estimation function 7c calculates the evaluation value distribution for each part using Rn(x,y) × (weight based on the distance from the center of the tip of the probe body 201), and identifies the position and part with the highest evaluation value as the examination area.

[0119] Another method involves calculating an evaluation value based on the distance between the straight line connecting the locations of each of the aforementioned parts and the position information (x,y) of the tip center of the probe body 201, and the confidence distribution of the two points corresponding to the straight line, thereby identifying the inspection area.

[0120] Another method involves dividing the locations of multiple body parts at a certain ratio to obtain coordinates, which are then used to identify the inspection area. This can be done by calculating an evaluation value based on the distance between these coordinates and the position information (x,y) of the tip center of the probe body 201, and the confidence distribution of the two points from which the division was made.

[0121] Another method involves creating a 2D or 3D closed region from points determined at a certain ratio based on the relationships between multiple joints, and then determining whether the position information (x,y) of the tip center of the probe body 201 is inside or outside that closed region to calculate an evaluation value and identify the inspection site.

[0122] Furthermore, evaluation values ​​may be calculated by combining some or all of the methods described above.

[0123] In step 910, if the subject moves after step 701, the position and orientation of the subject stored in memory 8 may differ from the current position and orientation of the subject. In this case, the examination area identified using a different position and orientation may yield incorrect results. Therefore, it is possible to determine if the subject is moving, and if movement is detected, return to step 701 and perform the body position and orientation estimation again. Methods for determining whether the subject is moving include, for example, using difference images or optical flow.

[0124] When using difference images, for example, the estimation function 7c detects the subject's movement by masking the examiner's hand and probe and checking for variations in brightness values ​​and hue in the remaining portion. Differences may also be detected by statistical processing.

[0125] Furthermore, when using optical flow, for example, the estimation function 7c registers the human body in the camera image acquired in step S801 when human body position and orientation estimation is performed as a pattern, and detects movement by performing template matching on the camera image in step S901. Alternatively, the estimation function 7c temporarily stores the camera image acquired in step S801 in memory 8, and calculates the amount of movement of feature points obtained by SHIFT, AKAZE, etc., between the two images to detect the movement of the subject in the image. Other known tracking methods such as KCF tracker may also be used.

[0126] In step S911, the control function 7a displays the inspection site identified in step S910 on the display 5.

[0127] In step S912, the control function 7a determines whether the inspector has performed an operation corresponding to OK on the control panel 6, or whether the freeze button 6a on the probe body 201 has been pressed. For example, the control function 7c displays on the display 5 either the measurement-related screen shown in Figure 17A, or, in addition to the measurement-related screen, the screen for approval of the identification results of the inspection area shown in Figure 17B.

[0128] For example, as shown in Figure 17B, the control function 7a displays on the display 5 the probe mark 1902 of the GUI corresponding to the ultrasound probe 2 superimposed on the body mark 1901 of the GUI corresponding to the subject's body at the position of the corresponding examination site. If the name of the examination site has also been identified, it may be displayed together with the body mark and probe mark (in the figure, it is labeled "Central Point"). The control function 7a then displays a confirmation window for the examination results on the screen. The confirmation window displays OK and redo icon buttons. The examiner confirms the examination site or instructs a redo by selecting the OK or redo button using the control panel 6, or by pressing the freeze button 6a on the probe body 201.

[0129] The control function 7a terminates the series of processes if the OK button is selected or the freeze button 6a is pressed. If the redo button is not selected or the freeze button 6a is not pressed, the process for identifying the inspection site A is repeated from step S901. Note that in step S912, the process for estimating the inspection site may be interrupted while the window prompting confirmation of the inspection results shown in Figure 17B is displayed.

[0130] Next, the process of step S705 in Figure 7 will be explained using Figure 10. Figure 10 is a flowchart showing the operation after the measurement process according to the first embodiment. Here, each step shown in Figure 10 is realized by the processing circuit 7 reading the corresponding function from the non-volatile memory 9 and executing it.

[0131] In step S1001, the control function 7a saves the ultrasound image confirmed in step S704 to the non-volatile memory 9 or external media, or transfers the data externally, and also displays the ultrasound image confirmed in step S704 on the display 5.

[0132] In step S1002, the control function 7a confirms the examination site identified in the examination site identification operation in step S703, in response to the examiner's operation. At this time, the ultrasound image confirmed in step S704 and the body marks and probe marks displayed from step S904 are displayed on the display 5 simultaneously or switchably. The examiner confirms the examination site displayed on the display 5, and if it is correct, presses the confirmation button 6b on the control panel 6 or on the probe body 201. If it is incorrect, the examiner's operation on the control panel 6 displays second and third site candidates on the display 5, and after the examiner selects the corresponding examination site, it is confirmed by the confirmation operation described above.

[0133] In step S1003, the control function 7a saves the examination site determined in step S1002 to the non-volatile memory 9 or external media, or transfers it to an external source, in association with the ultrasound image determined in step S704.

[0134] It should be noted that the processing flow of each flowchart described above is merely an example, and there are countless variations in the order and parallelism of the execution of each process, such as the human body position and pose estimation process and the probe tip position estimation process being performed in parallel.

[0135] As described above, according to the first embodiment, the image acquisition function 7b acquires an image including the ultrasound probe 2 scanning the subject. The estimation function 7c estimates the position of the cable of the ultrasound probe 2 based on the image, estimates the position and direction of the probe body based on the position of the cable of the ultrasound probe 2, and estimates the examination area by the ultrasound probe 2 based on the position and direction of the probe body. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment can identify the scanning position by the probe body even when occlusion occurs where the probe body is hidden by the examiner's hand, and enables robust estimation of the examination area against occlusion.

[0136] Furthermore, according to the first embodiment, the estimation function 7c performs a process to estimate the position of the cable, including the cable bush of the ultrasound probe 2. Based on the positional relationship between the cable bush and the cable of the ultrasound probe 2 in the image, the estimation function 7c estimates the position and orientation of the probe body. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to easily implement a method that is robust to occlusion.

[0137] Furthermore, according to the first embodiment, the estimation function 7c detects the major axis and minor axis of the region indicating the cable bush, and estimates the position and direction of the probe body by changing the ratio by which the length of the major axis of the cable bush is extended according to the ratio of the major axis to the minor axis. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to reduce estimation errors due to optical magnification factors.

[0138] Furthermore, according to the first embodiment, the estimation function 7c performs a process to estimate the position of the cable, including the cable bush of the ultrasound probe 2. If the cable bush is not detected in the image, the estimation function 7c estimates the position and orientation of the probe body based on the position of the cable's endpoint. Also, if the cable bush is not detected in the image, the estimation function 7c estimates the position and orientation of the probe body based on the position or shape of the examiner's hand holding the ultrasound probe. Also, if the cable bush is not detected in the image, the estimation function 7c estimates the position and orientation of the inspection device based on the direction of the cable. Also, the estimation function 7c estimates the position and orientation of the ultrasound probe 2 based on a landmark indicating the direction of the cable attached to the cable. Also, the estimation function 7c estimates the position and orientation of the probe based on the positional relationship between the ultrasound probe cable in the image and a landmark attached to the cable or the probe body. Also, the estimation function 7c estimates the position and orientation of the probe body based on the position of the examiner's hand or a landmark attached to the examiner's hand included in the image. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to estimate the position and direction of the probe body according to various conditions during the examination.

[0139] Furthermore, according to the first embodiment, the image acquisition function 7b acquires an image including the ultrasound probe by detecting infrared light irradiated onto the ultrasound probe 2. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to acquire external images using a method more suitable for the hospital environment.

[0140] Furthermore, according to the first embodiment, the estimation function 7c estimates the examination site by the probe body 201 based on the position and orientation of the probe body 201 and the position of the subject. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to estimate the examination site with high accuracy.

[0141] Furthermore, according to the first embodiment, the control function 7a causes the display 5 to display guide lines for adjusting the field of view of the camera that captures images. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to acquire an external image that is more suitable for processing.

[0142] Furthermore, according to the first embodiment, the inspection device has features on at least one of the cable or cable bushing that correspond to image processing. The features are color, texture, or a material that yields a relatively high brightness value when illuminated with light outside the visible light spectrum. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment enables more accurate segmentation processing.

[0143] (Modification 2) As described above, the tip position of the probe body can be estimated not only from the segmented information of the cable (and cable bush), but also by combining it with other methods.

[0144] Specifically, the estimation function 7c estimates the orientation of the probe body based on the position of the cable, estimates the orientation of the probe body based on the shape of the end of the cable of the ultrasound probe 2, and estimates the area to be examined by the ultrasound probe based on the estimation result based on the cable position and the estimation result based on the shape of the end of the cable.

[0145] For example, if there is a structurally distinctive shape on the side to which the cable of the probe body 201 is connected, or if it is possible to attach such a structure later, it is possible to determine the position and orientation of the probe body 201 by image processing using that structure as a clue. Specifically, the estimation function 7c detects the distinctive structure at the end of the probe from the image using an object detection learner such as SSD (Single Shot Multibox Detector) or YOLO (You only look once), and then estimates the most likely position and orientation of the probe by comparing the appearance of that structure with a database. Alternatively, the estimation function 7c may estimate the position and orientation of the probe using an object orientation estimation learner such as SSD6D, with the distinctive structure at the end of the probe as a clue.

[0146] The estimation function 7c cross-checks the estimation result of the probe tip position using the result obtained using the cable in the previous embodiment. Here, as a method of cross-checking, for example, the cosine similarity of the direction vectors of the probe body obtained by both methods is calculated. If the similarity is high, the estimation result of the probe body tip position is adopted according to the priority predetermined in the non-volatile memory 9. Alternatively, the result of processing such as the average value may be adopted. If the similarity is low, an alert is issued to prompt the operator to make a manual setting, or the process is repeated.

[0147] As mentioned above, by estimating the tip position of the probe body in combination with other methods, it becomes possible to perform more accurate estimations.

[0148] (Variation 3) In the example described above, the ultrasound diagnostic device 100 was connected to one camera, but the ultrasound diagnostic device 100 is not limited to one arm; it may be connected to multiple arms, or to multiple cameras mounted on walls, ceilings, etc.

[0149] In such cases, the placement of each camera should be predetermined using configuration plates or similar means. Several methods can be used to determine which camera's image to use.

[0150] For example, the control function 7a displays the images captured by each camera on the display 5. The inspector compares the images displayed on the display 5 and uses the buttons on the control panel 6 to select the camera with the best field of view. As a result, the image acquisition function 7b acquires external images from this camera.

[0151] Alternatively, the estimation function 7c performs the processing shown in Figure 7 on each of the images from multiple cameras, and in step S910 in Figure 9, it integrates the results to estimate the inspection area. Specifically, the estimation function 7c estimates the cable bush area and prioritizes the results from the camera that was able to estimate the probe tip position. If there are multiple such results, for example, the estimation function 7c adopts the result where the area of ​​the cable bush area is greater than or equal to a threshold predetermined in the non-volatile memory 9, and the ratio of the long axis to the short axis of the cable bush area is larger. If the cable bush area is not found in any of the cameras, the estimation function 7c estimates the position of the cable end or the position of the hand as the probe position according to the camera priority predetermined in the non-volatile memory 9.

[0152] As described above, the image acquisition function 7b selects an image from among multiple images captured by multiple cameras 3 that depicts the inspection device including the cable. This makes it possible to estimate the position of the probe body with greater accuracy.

[0153] (Modification 4) In the example described above, the case where the region of interest for cables is set as an isosceles triangle was explained, but the region of interest for cables is not limited to an isosceles triangle and can be set in any shape and size. Figure 18 is an illustrative diagram showing an example of a region of interest for cables according to Modification 4.

[0154] For example, the estimation function 7c determines the coordinates of the two endpoints of the major axis of the cable bush region R1, similar to how the isosceles triangle cable region of interest is set up. Next, as shown in Figure 18, the estimation function 7c sets up a rectangular region (cable region of interest R3) for each endpoint of the cable bush region R1, with the major axis of the cable bush region R1 as its longitudinal direction. The subsequent processing using the cable region of interest is the same as in the case of the isosceles triangle cable region of interest.

[0155] For example, during operation of the ultrasonic probe 2, the cable 202 may bend to a position nearly perpendicular to the longitudinal direction of the probe body 201. In such a case, if the shape of the region of interest of the cable is an isosceles triangle, the area of ​​the cable region within the region of interest becomes small, which may lead to an incorrect estimation of the position of the tip center of the probe body 201. Therefore, as shown in Figure 18, by making the shape of the region of interest of the cable a rectangle, it becomes possible to accurately estimate the position of the tip center of the probe body 201 even in such a situation.

[0156] (Variation 5) In the example described above, the case in which the ultrasonic probe 2 is used with the cable 202 and cable bush 203 exposed was explained. However, this embodiment is also applicable when the cable 202 and cable bush are covered by a cover during the inspection.

[0157] In such cases, a learner trained using external images of the ultrasonic probe 2 being operated, with the cable 202 and cable bush covered by the cover, and data annotated with the cover area, (if identifiable) the cable bush area, and the cable area within those images is used. That is, the estimation function 7c analyzes the external images using the above learner, performs segmentation processing of the cover area, segmentation processing of the cable bush area, and segmentation processing of the cable area, and obtains a segmentation map.

[0158] Furthermore, the learning device used with and without the cover can be switched as needed. For example, the estimation function 7c uses different learning devices depending on the operator's specified operation. Alternatively, the estimation function 7c may perform segmentation processing on the appearance image using both learning devices, and then switch the processing result used for subsequent processing depending on the accuracy of the segmentation processing.

[0159] Furthermore, the learning device used in the segmentation processing described above may be trained separately for each type of examination. In examinations using the ultrasound probe 2, the way in which the ultrasound probe 2 is brought into contact with the subject may differ depending on the type of examination. Therefore, a learning device is provided for each type of examination, and the estimation function 7c uses the appropriate learning device for segmentation processing depending on the type of examination.

[0160] (Other embodiments) In the example described above, the camera 3 is connected to the ultrasound diagnostic device 100, and the camera 3 acquires external images in accordance with the control of the ultrasound diagnostic device 100. However, the camera 3 may also be controlled independently. In such a case, the image acquisition function 7b acquires external images from the camera 3 via the communication I / F 10.

[0161] Furthermore, although the above example described a case where the arm 4 is fixed to the ultrasound diagnostic device body 1, the arm 4 may also be configured to be movable independently of the ultrasound diagnostic device body 1. In such a case, for example, the arm 4 may be made up of a stand with casters and hold the camera 3.

[0162] Furthermore, while the above example used an ultrasound diagnostic device as an example of a medical imaging diagnostic device, the medical imaging diagnostic device is not limited to this and may also be a PAT device. In such cases, the examination device will be a photoacoustic measurement probe.

[0163] The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or CPU, MPU, etc.) of that system or device then reads and executes the program code stored on the storage medium.

[0164] In this case, the program code read from the storage medium itself realizes the novel function of this embodiment, and the storage medium and program that store the program code constitute the present invention.

[0165] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0166] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0167] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the CPU or other components of that function expansion board or function expansion unit perform some or all of the actual processing.

[0168] Furthermore, the components of each device illustrated in the above description of the embodiments are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those illustrated, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0169] As described above, the embodiment makes it possible to estimate the inspection area in a way that is robust to occlusion.

[0170] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0171] 1. Ultrasound diagnostic device main unit 2. Ultrasound probe 3 cameras 4 Arms 5 displays 6. Control Panel 7 Processing Circuit 7a Control Function 7b Image acquisition function 7c Estimation function 8 memory 9. Non-volatile memory 10 Communication I / F 11 Power supply 12 Transmit / Receive Circuit 13 Signal Processing Circuits 14 Image generation circuit 15 Camera control circuit

Claims

1. An acquisition means that acquires an image including an ultrasound probe that scans the subject, A cable estimation means for estimating the position of the cable, including the cable bush of the ultrasonic probe, based on the aforementioned image, Estimation means for estimating the position and direction of the ultrasonic probe based on the image and the positional relationship between the cable bush and the cable of the ultrasonic probe cable, A position estimation means for estimating the scanning position or contact position by the ultrasonic probe based on the position and direction of the ultrasonic probe, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the estimation means detects the major axis and minor axis of the region representing the cable bush, and estimates the position and direction of the ultrasonic probe by changing the ratio by which the length of the major axis of the cable bush is extended according to the ratio of the major axis to the minor axis.

3. The information processing apparatus according to claim 1, wherein the estimation means estimates the position and direction of the ultrasonic probe based on the position of the cable endpoint when the cable bush is not detected in the image.

4. The information processing apparatus according to claim 1, wherein the estimation means estimates the position and direction of the ultrasonic probe based on the position or shape of the examiner's hand holding the ultrasonic probe when the cable bush is not detected in the image.

5. The information processing apparatus according to claim 1, wherein the estimation means estimates the position and direction of the ultrasonic probe based on the direction of the cable when the cable bush is not detected in the image.

6. The information processing apparatus according to claim 5, wherein the estimation means estimates the position and direction of the ultrasonic probe based on landmarks indicating the direction of the cable attached to the cable.

7. The information processing apparatus according to claim 1, wherein the estimation means estimates the position and direction of the ultrasonic probe based on the positional relationship between the cable of the ultrasonic probe in the image and a landmark attached to the cable or the body of the ultrasonic probe.

8. The information processing apparatus according to claim 7, wherein the landmark is an LED.

9. The information processing apparatus according to claim 1, wherein the estimation means estimates the position and direction of the ultrasonic probe based on the position of the cable of the ultrasonic probe in the image and the position of a landmark attached to the hand of the examiner holding the ultrasonic probe.

10. The information processing apparatus according to any one of claims 1 to 9, wherein the acquisition means acquires an image including the ultrasonic probe by detecting infrared light irradiated onto the ultrasonic probe.

11. An acquisition means that acquires an image including an ultrasound probe that scans the subject, A cable estimation means for estimating the position of the cable, including the cable bush of the ultrasonic probe, based on the aforementioned image, A first posture estimation means for estimating the posture of the ultrasonic probe based on the aforementioned image and the positional relationship between the cable bush and the cable of the cable, A second posture estimation means for estimating the posture of the ultrasonic probe based on the shape of the end of the cable of the ultrasonic probe in the aforementioned image, A position estimation means that estimates the scanning position or contact position of the ultrasonic probe based on the estimation result by the first posture estimation means and the estimation result by the second posture estimation means, An information processing device equipped with the following features.

12. The information processing apparatus according to any one of claims 1 to 9, wherein the position estimation means estimates the area to be examined by the ultrasound probe based on the position and direction of the ultrasound probe and the position of the subject estimated based on the image.

13. The information processing apparatus according to any one of claims 1 to 12, wherein the acquisition means selects an image from among a plurality of images captured by a plurality of imaging means in which the ultrasonic probe including the cable is depicted.

14. The information processing apparatus according to any one of claims 1 to 13, further comprising a control means for displaying guide lines on a display unit for adjusting the field of view of an imaging means for capturing the aforementioned image.

15. The information processing apparatus according to any one of claims 1 to 14, wherein the ultrasonic probe has features corresponding to image processing in at least one of the cable or cable bush.

16. The information processing apparatus according to claim 15, wherein the aforementioned feature is a material that can obtain a relatively high brightness value in response to light outside the visible light spectrum, in terms of color, texture, or light irradiation.

17. The ultrasonic probe and, An information processing device according to any one of claims 1 to 16, A medical imaging diagnostic device equipped with [a specific feature].

18. A computer-executable program describing procedures for realizing the functions of each control means of the information processing apparatus described in any one of claims 1 to 16.

19. A computer-readable storage medium in which a program is stored that causes the computer to perform the functions of each means of the information processing apparatus described in any one of claims 1 to 16.

Images