Multi-modal fusion target area pose prediction method and device

Abstract

The invention discloses a multi-modal fusion target area pose prediction method and device, and belongs to the field of medical position prediction.The method comprises the steps that digital speckles are transmitted to the surface of a measured object through a projector, a binocular camera collects a rough depth map, and a polarization image of the surface of the measured object is obtained; performing three-dimensional reconstruction according to the polarization image and the rough depth map to generate point cloud data; detecting the expansion of the measured object caused by breathing to measure the breathing frequency; a multi-channel array flexible sensor is adopted to collect pressure signals of the back; fusing the point cloud data, the respiratory rate and the pressure signal by adopting a multi-modal fusion model based on feature matching to form a body surface information fusion feature; body surface information fusion features are used as input data, a bidirectional recurrent neural network model is adopted for target region position prediction, and the problem that precision is not high when texture information of a measured object is weak is avoided; and the position of the tumor target region at a certain time point in the future when the tumor target region changes along with breathing can be predicted.

Description

Technical Field

[0001] The present invention relates to the field of medical position prediction, and in particular to a multi-modal fusion target area posture prediction method and device. Background Art

[0002] Respiratory movement of the chest and abdomen has a direct impact on changes in organ shape and position, which in turn affects surgical positioning and the target accuracy of tumor radiotherapy. Existing tumor target monitoring methods usually rely on real-time CT scanning. Although it can provide real-time data monitoring, patients are exposed to higher radiation doses. In addition, the gold standard implantation monitoring method is invasive and increases the risk of complications such as pneumothorax in patients. In order to cope with the interference of respiratory movement on treatment accuracy, surface-based monitoring and prediction technology has become particularly important.

[0003] Most of the existing surface monitoring uses binocular vision ranging systems, which are mainly composed of binocular cameras. They calculate the depth information of the object by capturing the left and right views, and have the advantage of real-time three-dimensional measurement. However, when the texture information of the object being measured is weak, the difficulty of stereo matching increases, which is easy to cause matching errors, resulting in low accuracy of depth information. It is difficult to obtain high-precision three-dimensional information by binocular cameras alone, and using only binocular vision ranging systems cannot predict the changes in the tumor target area position with breathing, that is, it is impossible to predict the tumor target area position at a certain point in the future. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, one of the objectives of the present invention is to provide a multi-modal fusion target area posture prediction method that can predict the position of the tumor target area at a certain point in the future when it changes with breathing.

[0005] In order to overcome the shortcomings of the prior art, the second purpose of the present invention is to provide a multi-modal fusion target area posture prediction device that can predict the position of the tumor target area at a certain point in the future when it changes with breathing.

[0006] One of the purposes of the present invention is achieved by the following technical solution:

[0007] A multi-modal fusion target area posture prediction method comprises the following steps:

[0008] Collecting point cloud data of the object under test: The projector transmits digital speckle to the surface of the object under test, and the binocular camera collects the rough depth map and obtains the polarization image of the surface of the object under test. The point cloud data is generated by three-dimensional reconstruction based on the polarization image and the rough depth map;

[0009] Monitoring breathing: Detecting the expansion of the object caused by breathing to measure the breathing rate;

[0010] Collecting back pressure signals: Using a multi-channel array flexible sensor to collect back pressure signals;

[0011] Multimodal data fusion: using a multimodal fusion model based on feature matching to fuse the point cloud data, the respiratory rate and the pressure signal to form a body surface information fusion feature;

[0012] Position prediction: The body surface information fusion features are used as input data, and a bidirectional recurrent neural network model is used to predict the target area position.

[0013] Furthermore, in the step of collecting point cloud data of the object to be measured, the projector transmits the digital speckle to the surface of the object to be measured specifically: the digital speckle pattern is pre-burned into the digital projector, and the projector projects different light fields according to the input digital signal.

[0014] Furthermore, the digital speckle pattern generation process is specifically as follows: creating a basic matrix, using random numbers to generate noise images with different amplitudes and phases, multiplying the noise image by the basic matrix element by element, performing Fourier transform on the generated image, and moving the spectrum center to the image center through a translation operation, performing inverse Fourier transform to generate the final speckle image, and normalizing the speckle image.

[0015] Furthermore, the pixel value range of the speckle image is between 0 and 1.

[0016] Furthermore, a polarization image of the surface of the object under test is obtained, and three-dimensional reconstruction is performed based on the polarization image and the rough depth map to generate point cloud data. Specifically, multiple images of the object under test at different angles are taken through a rotating polarization filter to capture the intensity of reflected light on the surface of the object under test at different polarization angles, the surface normal is calculated through the depth map, the surface normal is used to correct the normal obtained from the polarization clues, the three-dimensional shape is restored by integrating the normal, a sparse linear system is used to combine the depth map with the integration result, and finally the point cloud data is generated.

[0017] Furthermore, in the breathing monitoring step, a fiber optic sensor is attached to the abdomen, and the light emitted by the light source is transmitted to the spectrometer through the fiber optic sensor, and deformation analysis is performed based on the intensity distribution of the analytical light beam.

[0018] Furthermore, in the step of collecting the back pressure signal, the multi-channel array flexible sensor is made of graphene material.

[0019] Furthermore, the multimodal fusion target area posture prediction method also includes a verification step, and the verification step is specifically: when the respiratory frequency is abnormal data, turning on CBCT to verify the target area position.

[0020] The second object of the present invention is achieved by adopting the following technical solution:

[0021] A multi-modal fusion target area posture prediction device, used to implement any of the above multi-modal fusion target area posture prediction methods, the multi-modal fusion target area posture prediction device comprising

[0022] An imaging system, wherein the imaging system comprises a binocular vision imaging structure and a polarization imaging structure, wherein the binocular vision imaging structure collects a roughness depth map of the surface of the object being measured, and the polarization imaging structure collects a polarization image of the surface of the object being measured;

[0023] Fiber optic sensor, which monitors the expansion of the object caused by breathing to measure the breathing rate;

[0024] Multi-channel array flexible sensor to collect pressure signals on the back;

[0025] The processor performs three-dimensional reconstruction based on the polarization image and the rough depth map to generate point cloud data, adopts a multimodal fusion model based on feature matching, fuses the point cloud data, the respiratory frequency and the pressure signal to form a body surface information fusion feature, uses the body surface information fusion feature as input data, and adopts a bidirectional recurrent neural network model to predict the target area position.

[0026] Furthermore, the D-shaped region of the optical fiber sensor is coated with a PDMS film, and the optical fiber is bent into two layers, both of which are horizontal S-shaped waves, and the S-shapes of the upper and lower layers are opposite, and the upper and lower layers overlap to form an O-shape.

[0027] Compared with the prior art, the target area posture prediction method of the multimodal fusion of the present invention transmits digital speckle to the surface of the object to be measured through a projector, collects a rough depth map with a binocular camera, obtains a polarization image of the surface of the object to be measured, and performs three-dimensional reconstruction based on the polarization image and the rough depth map to generate point cloud data; detects the expansion of the object to be measured caused by breathing to measure the respiratory frequency; uses a multi-channel array flexible sensor to collect the pressure signal of the back; uses a multimodal fusion model based on feature matching to fuse the point cloud data, the respiratory frequency and the pressure signal to form a body surface information fusion feature; uses the body surface information fusion feature as input data, and uses a bidirectional recurrent neural network model to predict the target area position, which avoids the problem of low accuracy when the texture information of the object to be measured is weak; and can predict the position of the tumor target area at a certain point in the future when it changes with breathing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the target area posture prediction method of multi-modal fusion of the present invention;

[0029] Figure 2 It is a schematic diagram of binocular vision imaging structure;

[0030] Figure 3 is a schematic diagram of the imaging system;

[0031] Figure 4 It is a schematic diagram of the structure of the optical fiber sensor;

[0032] Figure 5 This is a schematic diagram of the use status of the optical fiber sensor;

[0033] Figure 6 Schematic diagram of collecting back pressure signals using a multi-channel array flexible sensor. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0036] Due to the human body's breathing, the position of the target on the human body's surface will move in real time, resulting in inaccurate operation during medical operations. The present application provides a multimodal fusion target area posture prediction method for predicting the real-time position of the target on the body surface, which is used for determining the point during human surgery, such as puncture point, skin cancer radiotherapy point, etc.

[0037] See also Figure 1 The present invention provides a multi-modal fusion target area posture prediction method, comprising the following steps:

[0038] Collecting point cloud data of the object under test: The projector transmits digital speckle to the surface of the object under test, and the binocular camera collects the rough depth map and obtains the polarization image of the surface of the object under test. The point cloud data is generated by three-dimensional reconstruction based on the polarization image and the rough depth map;

[0039] Monitoring breathing: Detecting the expansion of the object caused by breathing to measure the breathing rate;

[0040] Collecting back pressure signals: Using a multi-channel array flexible sensor to collect back pressure signals;

[0041] Multimodal data fusion: using a multimodal fusion model based on feature matching to fuse the point cloud data, the respiratory rate and the pressure signal to form a body surface information fusion feature;

[0042] Position prediction: The body surface information fusion features are used as input data, and a bidirectional recurrent neural network model is used to predict the target area position.

[0043] Please continue reading Figure 2 as well as Figure 3 , collecting the point cloud data of the measured object is specifically as follows:

[0044] When the texture information of the measured object is weak, it will increase the difficulty of stereo matching and cause matching errors. This application introduces digital speckle, which is projected onto the surface of the measured object by a projector to generate texture, changing the passive measurement method to an active measurement method. With the help of a customized texture method, the difficulty of stereo matching is reduced and the speed and accuracy of measurement are improved.

[0045] The binocular vision measurement system consists of two industrial cameras and a projector located between the two cameras. The generated digital speckle pattern is pre-burned into the digital projector, and the projector projects different light fields according to the input digital signal. After the stripes in the light field are irradiated on the surface of the object to be measured, they are modulated by the surface of the object and captured by the image sensor of the binocular camera. Then, the captured image data is transmitted to the computer, and the preliminary rough depth map is finally generated through the demodulation and triangulation method of the light field information.

[0046] The digital speckle generation method is as follows: first, a basic matrix is ​​created, and then a noise image with different amplitudes and phases is generated using random numbers and multiplied element by element with the basic matrix. Next, the generated image is Fourier transformed, and the center of the spectrum is moved to the center of the image through a translation operation. Subsequently, an inverse Fourier transform is performed to generate the final speckle image, and finally the image is normalized so that its pixel value range is between 0 and 1.

[0047] Polarization imaging uses the polarization information of light reflected from the target surface to analyze the three-dimensional structure. Compared with binocular vision, it is robust in passive capture, lighting conditions and materials, and can well restore the subtle surface details of objects without texture dependence. However, polarization imaging has the problem of azimuth ambiguity. This application uses a joint three-dimensional reconstruction model that combines binocular vision with polarization imaging to maximize the advantages of polarization imaging and binocular vision to achieve high-quality three-dimensional reconstruction. First, a "rough depth map" of the object surface with real depth information is obtained through binocular vision. The polarization three-dimensional imaging results containing huge texture details on the target surface are then combined with the "rough depth map" to achieve high-precision three-dimensional imaging in various scenarios.

[0048] Specifically, multiple images at different angles are taken by rotating the polarization filter to capture the intensity of reflected light from the surface at different polarization angles. These images contain information about the surface normal of the object. At each image point, the light intensity It can be expressed by the following formula:

[0049]

[0050] Among them, I max and I min are the maximum and minimum light intensities, is the polarization angle, is the azimuth angle. By sampling different polarization angles (usually at least three angles), the phase, amplitude, and offset of the received signal can be inferred.

[0051] By analyzing images at different polarization angles, the azimuth It can be determined by the phase of the received signal.

[0052] The equation of polarization degree ρ and zenith angle θ is:

[0053]

[0054] n is the refractive index.

[0055] Normal maps obtained directly from polarization have low-frequency and high-frequency ambiguities. This application corrects systematic distortions by introducing a coarse depth map based on binocular vision. The surface normal is calculated from the depth map and then used to correct the normal obtained from polarization cues. Low-frequency ambiguities in polarization normals are corrected through a binary operator.

[0056] Zenith angle distortion affects the reconstruction of three-dimensional shapes. This application corrects zenith angle distortion by identifying areas in the depth map where shape estimates are more accurate. The normals can be updated by rotationally adjusting local areas in the depth map. Combined with the corrected normals, the three-dimensional shape can be restored by integrating the normals. Normal integrals are easily affected by depth discontinuities. This patent uses a minimum spanning tree integration scheme to integrate only in areas where the surface normals are credible.

[0057] Since normal integral can only obtain relative 3D shape (i.e. relative offset and scaling of shape), depth fidelity constraint is required to maintain global coordinate consistency. This patent combines the depth map with the integral result through a sparse linear system to ensure that the reconstructed 3D shape is consistent with the original depth map, especially in the area where the depth map is accurate. The final generated 3D reconstructed image contains the global geometric information of the depth map and the surface details of the polarization map, thus achieving high-quality 3D reconstruction.

[0058] The specific steps for monitoring breathing are:

[0059] The chest wall movement caused by breathing is an important indicator for monitoring respiratory rate. This application installs a fiber optic sensor in a wearable device, which works by creating periodic refractive index modulation in the optical fiber, and measures the respiratory rate by detecting the expansion of the chest or abdomen caused by breathing. The fiber optic sensor has the advantages of high sensitivity, small size, and strong anti-electromagnetic interference ability.

[0060] Please continue reading Figure 4 Considering the poor fit of the optical fiber sensor when directly attached to the abdomen of the human body, this application combines macrobending with the D-type structure to design an O-type structure respiratory sensor to study the respiratory sensing characteristics. The D-shaped area of ​​the optical fiber sensor is coated with a PDMS film. PDMS not only enhances the toughness and elasticity of the D-shaped optical fiber, but also provides a stable refractive index environment for the D-shaped optical fiber. At the same time, since the D-shaped optical fiber is encapsulated in the PDMS film, the orientation of the polished surface of the D-shaped optical fiber can be calibrated during encapsulation. Therefore, the sensor can not only respond to the size of the curvature during curvature sensing, but also distinguish the direction of the bend. This makes the sensor have the potential to realize flexible sensing functions. The arrangement of the optical fiber in this patent is that the optical fiber is bent into two layers, the upper and lower layers, both of which are horizontal S-shaped waves, and the S-shapes of the upper and lower layers are opposite. The upper and lower layers overlap to form an O-shape, which can increase the evanescent field around the curved structure without destroying the optical fiber structure.

[0061] Please continue reading Figure 5 ,The fiber optic sensor is attached to the abdomen in a distributed manner, and multiple optical fibers are integrated for better detection of respiratory movements.,The light emitted by the light source is transmitted to the spectrum analyzer through the macrobend plastic optical fiber.,Based on the intensity distribution of the analyzed light beam, the computer data processing system performs deformation analysis to obtain the results.

[0062] Please continue reading Figure 6 , collecting back pressure signals specifically: Considering that traditional pressure sensors have poor adhesion and that a few sensors alone cannot collect complete muscle activity of the back, this application designs a multi-channel array flexible collection sensor suitable for back pressure signal detection. The flexible sensor in this application is deformable and can be deformed within a certain range, with better adhesion and biocompatibility. The pressure sensor is made of graphene material,

[0063] The specific steps of multimodal data fusion are as follows: adopt a multimodal fusion model based on feature matching, fuse the back array pressure change signal, the abdominal respiratory deformation based on fiber optic sensing, and the human chest and abdomen point cloud based on polarization imaging and binocular vision to form the body surface information fusion feature. Construct a multimodal fusion model and prediction model containing 4D-CT image information and body surface information, and establish a mapping relationship from CT images to external detection information. Use the Bi-LSTM model to predict the tumor target area position, and use the fusion feature as input data for prediction.

[0064] Before surgery, 4D-CT image data, back array pressure change signals, abdominal respiratory deformation based on fiber optic sensing, and human chest and abdomen point cloud data based on polarization imaging and binocular vision are collected to build a mapping model from 4D-CT image information to surface monitoring information. Based on the Bi-LSTM model, a tumor position prediction model based on back array pressure change signals, abdominal respiratory deformation based on fiber optic sensing, and human chest and abdomen point cloud data based on polarization imaging and binocular vision is established. During the operation, surface detection information is collected and the position of the active target area is predicted through the prediction model. When respiratory abnormalities are detected through surface data, CBCT is turned on for secondary verification to ensure the accuracy of the tumor target position and achieve real-time and accurate tracking of the tumor target area with low radiation dose.

[0065] Compared with the prior art, the target area posture prediction method of the multimodal fusion of the present invention transmits digital speckle to the surface of the object to be measured through a projector, collects a rough depth map with a binocular camera, obtains a polarization image of the surface of the object to be measured, and performs three-dimensional reconstruction based on the polarization image and the rough depth map to generate point cloud data; detects the expansion of the object to be measured caused by breathing to measure the respiratory frequency; uses a multi-channel array flexible sensor to collect the pressure signal of the back; uses a multimodal fusion model based on feature matching to fuse the point cloud data, the respiratory frequency and the pressure signal to form a body surface information fusion feature; uses the body surface information fusion feature as input data, and uses a bidirectional recurrent neural network model to predict the target area position, which avoids the problem of low accuracy when the texture information of the object to be measured is weak; and can predict the position of the tumor target area at a certain point in the future when it changes with breathing.

[0066] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A multi-modal fusion target area posture prediction method, characterized in that: The following steps are involved: Collecting point cloud data of the object under test: The projector transmits digital speckle to the surface of the object under test, and the binocular camera collects the rough depth map and obtains the polarization image of the surface of the object under test. The point cloud data is generated by three-dimensional reconstruction based on the polarization image and the rough depth map; Monitoring breathing: Detecting the expansion of the object caused by breathing to measure the breathing rate; Collecting back pressure signals: Using a multi-channel array flexible sensor to collect back pressure signals; Multimodal data fusion: using a multimodal fusion model based on feature matching to fuse the point cloud data, the respiratory rate and the pressure signal to form a body surface information fusion feature; Position prediction: The body surface information fusion features are used as input data, and a bidirectional recurrent neural network model is used to predict the target area position.

2. The target area posture prediction method of multimodal fusion according to claim 1, characterized in that: In the step of collecting point cloud data of the object to be measured, the projector transmits the digital speckle to the surface of the object to be measured specifically: the digital speckle pattern is pre-burned into the digital projector, and the projector projects different light fields according to the input digital signal.

3. The target area posture prediction method of multimodal fusion according to claim 2, characterized in that: The digital speckle pattern generation process is specifically as follows: creating a basic matrix, using random numbers to generate noise images with different amplitudes and phases, multiplying the noise image by the basic matrix element by element, performing Fourier transform on the generated image, and moving the spectrum center to the image center through translation operation, performing inverse Fourier transform to generate the final speckle image, and normalizing the speckle image.

4. The target area posture prediction method of multimodal fusion according to claim 3, characterized in that: The pixel value of the speckle image ranges from 0 to 1.

5. The target area posture prediction method of multimodal fusion according to claim 1, characterized in that: A polarization image of the surface of the object to be measured is obtained, and three-dimensional reconstruction is performed based on the polarization image and the rough depth map to generate point cloud data. Specifically, multiple images of the object to be measured at different angles are taken by rotating the polarization filter to capture the intensity of reflected light on the surface of the object to be measured at different polarization angles, and the surface normal is calculated through the depth map. The surface normal is used to correct the normal obtained from the polarization clues, and the three-dimensional shape is restored by integrating the normal. A sparse linear system is used to combine the depth map with the integration result to finally generate point cloud data.

6. The target area posture prediction method of multimodal fusion according to claim 1, characterized in that: In the breathing monitoring step, an optical fiber sensor is attached to the abdomen, and light emitted by a light source is transmitted to a spectrometer through the optical fiber sensor, and deformation analysis is performed based on the intensity distribution of the analytical light beam.

7. The target area posture prediction method of multimodal fusion according to claim 1, characterized in that: In the step of collecting the back pressure signal, the multi-channel array flexible sensor is made of graphene material.

8. The target area posture prediction method of multimodal fusion according to claim 1, characterized in that: The multimodal fusion target area posture prediction method also includes a verification step, and the verification step is specifically: when the respiratory frequency is abnormal data, turning on CBCT to verify the target area position.

9. A multi-modal fusion target area posture prediction device, used to implement the multi-modal fusion target area posture prediction method according to any one of claims 1 to 8, characterized in that: The multi-modal fusion target area posture prediction device includes An imaging system, wherein the imaging system comprises a binocular vision imaging structure and a polarization imaging structure, wherein the binocular vision imaging structure collects a roughness depth map of the surface of the object being measured, and the polarization imaging structure collects a polarization image of the surface of the object being measured; Fiber optic sensor, which monitors the expansion of the object caused by breathing to measure the breathing rate; Multi-channel array flexible sensor to collect pressure signals on the back; The processor performs three-dimensional reconstruction based on the polarization image and the rough depth map to generate point cloud data, adopts a multimodal fusion model based on feature matching, fuses the point cloud data, the respiratory frequency and the pressure signal to form a body surface information fusion feature, uses the body surface information fusion feature as input data, and adopts a bidirectional recurrent neural network model to predict the target area position.

10. The multi-modal fusion target area posture prediction method according to claim 9, characterized in that: The D-shaped region of the optical fiber sensor is coated with a PDMS film, and the optical fiber is bent into two layers, both of which are horizontal S-shaped waves, and the S-shapes of the upper and lower layers are opposite, and the upper and lower layers overlap to form an O-shape.

Images