Multi-modal fusion body surface target real-time displacement prediction method and device

Abstract

The invention discloses a multi-modal fusion body surface target real-time displacement prediction method and device, and belongs to the field of medical path navigation. Real-time position information of a body surface target is obtained through a binocular distance measurement system; the respiratory movement monitoring belt measures the respiratory movement data of the chest and abdomen; the pressure sensor measures myoelectricity and pressure data of the back; the acceleration sensor measures acceleration data of the chest and abdomen during respiratory movement; establishing a target spot real-time displacement prediction model according to the real-time position information of the body surface target spot, the thoracic and abdominal respiratory movement data, the back myoelectricity and pressure data and the thoracic and abdominal acceleration data; predicting the displacement of the body surface target spot at a certain moment in the future by adopting a model; comparing the predicted displacement with the position information at the moment obtained by the binocular ranging system, and when the error exceeds a threshold value, adjusting the target spot real-time displacement prediction model until the error is less than or equal to the threshold value; and the adjusted target spot real-time displacement prediction model is adopted to predict the real-time displacement of the body surface target spot, and the prediction accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of medical path navigation, and in particular to a multi-modal fusion real-time displacement monitoring and prediction method for body surface targets. Background Art

[0002] The displacement monitoring and prediction of surface targets plays an important role in respiratory monitoring, cancer radiotherapy such as skin cancer radiotherapy, and surgical puncture assistance. Surface targets move with human breathing. Although the camera can obtain the real-time position of the surface target, the real-time position obtained by the camera has a lag, and during the treatment process, the position of the surface target needs to be predicted in advance. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, one of the objectives of the present invention is to provide a multi-modal fusion body surface target real-time displacement prediction method that can predict the position of the body surface target in advance.

[0004] In order to overcome the deficiencies of the prior art, a second object of the present invention is to provide a multi-modal fusion body surface target real-time displacement prediction device that can predict the position of the body surface target in advance.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A multi-modal fusion method for real-time displacement prediction of body surface targets comprises the following steps:

[0007] Use binocular ranging system to obtain real-time position information of surface targets;

[0008] A respiratory motion monitoring belt is used to measure the respiratory motion data of the chest and abdomen of the subject;

[0009] Use pressure sensors to measure the electromyographic data and pressure data of the back;

[0010] An acceleration sensor is used to measure the acceleration data of the chest and abdomen during breathing movement;

[0011] Establishing a target point real-time displacement prediction model according to the real-time position information of the body surface target point, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen;

[0012] Using the target point real-time displacement prediction model to predict the displacement of the body surface target point at a certain moment in the future;

[0013] Comparing the predicted displacement of the body surface target at a certain moment with the position information at that moment acquired by the binocular ranging system, and when the error exceeds a threshold, adjusting the target real-time displacement prediction model until the error is less than or equal to the threshold;

[0014] The adjusted target real-time displacement prediction model is used to predict the real-time displacement of the surface target.

[0015] Furthermore, the steps of using a binocular ranging system to obtain the real-time position information of the target point are specifically as follows: using two cameras to build a binocular ranging platform, calibrating the binocular camera, determining the internal and external parameters of the camera to improve the accuracy of ranging, adjusting the parameters of feature recognition, recording the position information of the surface target in real time, and monitoring the respiratory movement status of the subject.

[0016] Furthermore, the target real-time displacement prediction model is constructed using deep learning and multimodal fusion algorithms.

[0017] Furthermore, there are multiple body surface targets, which are distributed in different parts of the human body, and different body surface targets have different displacement amplitudes with breathing.

[0018] Furthermore, the breathing type is determined according to the different displacement amplitudes of the different body surface targets as they move with breathing.

[0019] Furthermore, the breathing types include chest breathing and abdominal breathing.

[0020] Furthermore, the error threshold is 2%-5%.

[0021] The second object of the present invention is achieved by adopting the following technical solution:

[0022] A multi-modal fusion body surface target real-time displacement prediction device, used to implement any of the above multi-modal fusion body surface target real-time displacement prediction methods, the multi-modal fusion body surface target real-time displacement prediction device comprising

[0023] A binocular ranging platform, which obtains real-time position information of a target point on the body surface;

[0024] A motion monitoring belt, which is worn on the chest and abdomen of the subject and measures respiratory motion data of the chest and abdomen of the subject;

[0025] Pressure sensor, measuring electromyographic data and pressure data of the back;

[0026] Acceleration sensor, which measures the acceleration data of the chest and abdomen during breathing movement;

[0027] The processor analyzes the real-time position information of the body surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen to predict the real-time displacement of the body surface target.

[0028] Furthermore, the binocular ranging platform includes two industrial cameras, and the two industrial cameras are symmetrically arranged about the body surface target.

[0029] Compared with the prior art, the multimodal fusion real-time displacement prediction method of the present invention establishes a target real-time displacement prediction model by fusing the real-time position information of the surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen; the target real-time displacement prediction model is used to predict the displacement of the surface target at a certain moment in the future, with high prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the method for real-time displacement prediction of body surface targets by multi-modal fusion of the present invention;

[0031] Figure 2 A front view schematic diagram of a human body when acquiring data in the multi-modal fusion body surface target real-time displacement prediction method of the present invention;

[0032] Figure 3 This is a schematic diagram of the back of a human body when acquiring data in the multimodal fusion body surface target real-time displacement prediction method of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0035] Due to the human body's breathing, the position of the body's surface target will move in real time, resulting in inaccurate operation during medical operations. The present application provides a multi-modal fusion body surface target real-time displacement prediction method for predicting the real-time position of the body surface target, which is used for determining the point during human surgery, such as puncture point, skin cancer radiotherapy point, etc.

[0036] See also Figures 1 to 3 The present invention provides a multi-modal fusion body surface target real-time displacement prediction method, comprising the following steps:

[0037] Use binocular ranging system to obtain real-time position information of surface targets;

[0038] A respiratory motion monitoring belt is used to measure the respiratory motion data of the chest and abdomen of the subject;

[0039] Use pressure sensors to measure the electromyographic data and pressure data of the back;

[0040] An acceleration sensor is used to measure the acceleration data of the chest and abdomen during breathing movement;

[0041] A target real-time displacement prediction model is established based on the real-time position information of the surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen;

[0042] The real-time displacement prediction model of the target is used to predict the displacement of the surface target at a certain moment in the future;

[0043] Compare the predicted displacement of the body surface target at a certain moment with the position information at that moment obtained by the binocular ranging system. When the error exceeds a threshold, adjust the target real-time displacement prediction model until the error is less than or equal to the threshold.

[0044] The adjusted target real-time displacement prediction model is used to predict the real-time displacement of the surface target.

[0045] Specifically, before the step of using a binocular ranging system to obtain the real-time position information of the body surface target, the method further includes placing a target on the body surface of the subject. In the present embodiment, the body surface target is a plurality of gold discs with a diameter of 5 mm adhered to the body surface. The size and color of the body surface target can be changed as needed. There are multiple body surface targets, and the multiple body surface targets are distributed in different parts of the human body. Different body surface targets have different displacement amplitudes with breathing. The breathing type is determined according to the different displacement amplitudes of different body surface targets with breathing. Breathing types include chest breathing and abdominal breathing.

[0046] The specific steps for using the binocular ranging system to obtain the real-time position information of the target are as follows: use two cameras to build a binocular ranging platform, calibrate the binocular camera, determine the internal and external parameters of the camera to improve the accuracy of ranging, adjust the parameters of feature recognition, record the position information of the surface target in real time, and monitor the respiratory movement status of the subject.

[0047] The specific steps of using a respiratory movement monitoring belt to measure the respiratory movement data of the chest and abdomen of the subject are: use an existing respiratory movement monitoring belt, wear it on the chest and abdomen of the subject, monitor the respiratory movement of the subject and record the data, and draw a respiratory movement curve of the subject.

[0048] The specific steps of using pressure sensors to measure the myoelectric data and pressure data of the back are: using multiple pressure sensors to monitor and record the myoelectric data and pressure data of the breathing movement of the subject when the subject lies down.

[0049] The specific steps of using an acceleration sensor to measure the acceleration data of the chest and abdomen during respiratory movement are: using the acceleration sensor to monitor and record the acceleration data of the chest and abdomen during respiratory movement of the subject.

[0050] The specific steps for establishing a real-time displacement prediction model for the target are as follows: using deep learning-related algorithms, and constructing a multimodal fusion real-time monitoring and prediction model for the displacement of the target on the surface of the body based on the real-time position information of the target on the surface, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen.

[0051] The purpose of comparing the predicted displacement of the body surface target at a certain moment with the position information at that moment obtained by the binocular ranging system is to ensure the reliability of the model prediction. The error threshold is 2%-5%.

[0052] The present application also discloses a multi-modal fusion body surface target real-time displacement prediction device, which is used to implement the multi-modal fusion body surface target real-time displacement prediction method. The multi-modal fusion body surface target real-time displacement prediction device includes:

[0053] A binocular ranging platform, which obtains real-time position information of a target point on the body surface;

[0054] A motion monitoring belt, which is worn on the chest and abdomen of the subject and measures respiratory motion data of the chest and abdomen of the subject;

[0055] Pressure sensor, measuring electromyographic data and pressure data of the back;

[0056] Acceleration sensor, which measures the acceleration data of the chest and abdomen during breathing movement;

[0057] The processor analyzes the real-time position information of the body surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen to predict the real-time displacement of the body surface target.

[0058] Specifically, the binocular ranging platform includes two industrial cameras, and the two industrial cameras are symmetrically arranged about a body surface target.

[0059] Compared with the prior art, the multimodal fusion real-time displacement prediction method of the body surface target of the present invention establishes a real-time displacement prediction model of the target by fusing the real-time position information of the body surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen; the real-time displacement prediction model of the target is used to predict the displacement of the body surface target at a certain moment in the future, and the prediction accuracy is high; the displacement of the target area of ​​the patient can be predicted in cancer radiotherapy, such as skin cancer radiotherapy, to increase the accuracy of radiotherapy and improve the treatment effect. During surgery or puncture, the surgical or puncture position can be adjusted according to the real-time respiratory curve and the displacement prediction value to improve the accuracy of the surgical or puncture position. The required equipment is relatively simple, the cost is low, and the accuracy is high.

[0060] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A multi-modal fusion method for real-time displacement prediction of body surface targets, characterized in that: The following steps are involved: Use binocular ranging system to obtain real-time position information of surface targets; A respiratory motion monitoring belt is used to measure the respiratory motion data of the chest and abdomen of the subject; Use pressure sensors to measure the electromyographic data and pressure data of the back; An acceleration sensor is used to measure the acceleration data of the chest and abdomen during breathing movement; Establishing a target point real-time displacement prediction model according to the real-time position information of the body surface target point, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen; Using the target point real-time displacement prediction model to predict the displacement of the body surface target point at a certain moment in the future; Comparing the predicted displacement of the body surface target at a certain moment with the position information at that moment acquired by the binocular ranging system, and when the error exceeds a threshold, adjusting the target real-time displacement prediction model until the error is less than or equal to the threshold; The adjusted target real-time displacement prediction model is used to predict the real-time displacement of the surface target.

2. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 1, characterized in that: The steps of using the binocular ranging system to obtain the real-time position information of the target point are specifically as follows: using two cameras to build a binocular ranging platform, calibrating the binocular camera, determining the internal and external parameters of the camera to improve the accuracy of ranging, adjusting the parameters of feature recognition, recording the position information of the surface target in real time, and monitoring the respiratory movement status of the subject.

3. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 1, characterized in that: The target real-time displacement prediction model is constructed using deep learning and multimodal fusion algorithms.

4. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 1, characterized in that: There are multiple body surface targets, which are distributed in different parts of the human body, and different body surface targets have different displacement amplitudes with breathing.

5. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 4, characterized in that: The breathing type is determined according to the different displacement amplitudes of the target points on the body surface as the breathing progresses.

6. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 5, characterized in that: The breathing types include chest breathing and abdominal breathing.

7. The method for real-time displacement prediction of body surface targets based on multimodal fusion according to claim 1, characterized in that: The threshold of the error is 2%-5%.

8. A multi-modal fusion body surface target real-time displacement prediction device, used to implement the multi-modal fusion body surface target real-time displacement prediction method according to any one of claims 1 to 7, characterized in that: The multi-modal fusion body surface target real-time displacement prediction device includes A binocular ranging platform, which obtains real-time position information of a target point on the body surface; A motion monitoring belt, which is worn on the chest and abdomen of the subject and measures respiratory motion data of the chest and abdomen of the subject; Pressure sensor, measuring electromyographic data and pressure data of the back; Acceleration sensor, which measures the acceleration data of the chest and abdomen during breathing movement; The processor analyzes the real-time position information of the body surface target, the respiratory movement data of the chest and abdomen, the electromyographic data and pressure data of the back, and the acceleration data of the chest and abdomen to predict the real-time displacement of the body surface target.

9. The multi-modal fusion body surface target real-time displacement prediction device according to claim 8, characterized in that: The binocular ranging platform includes two industrial cameras, and the two industrial cameras are symmetrically arranged about a body surface target.

Images