Multi-sensor data fusion positioning method and system and storage medium

Abstract

The invention provides a multi-sensor data fusion positioning method and system and a storage medium, and belongs to the technical field of medical detection. The method comprises the following steps: acquiring image data, point cloud data and three-dimensional data of a target area of a detected person in a medical detection process; determining a three-dimensional bounding box of the target area according to the image data and the point cloud data; determining a reference bounding box of the target area according to the three-dimensional data; performing positioning correction on the three-dimensional bounding box according to the reference bounding box to obtain a corrected bounding box of the target area; and adjusting the position of the testee in the medical detection process according to the correction bounding box of the target area. Through the method, different types of image data, point cloud data and three-dimensional data are fused, the advantages of different types of data are fused, detection and observation are carried out from multiple different angles, the positioning accuracy is improved, and the problem that the positioning accuracy of a patient is low due to a traditional method is solved.

Description

Technical Field

[0001] The present application belongs to the field of medical detection technology, and in particular, relates to a multi-sensor data fusion positioning method, system and storage medium. Background Art

[0002] With the rapid development of science and technology, more and more medical devices are widely used to assist patients in diagnosing lesions. For example, computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), etc. When using the above medical devices to detect patients, the position of the patient needs to be adjusted to position the patient during the detection process or to avoid collision between the patient and the device.

[0003] The traditional method uses a 2D camera to capture the patient's planar image and adjust the patient's position based on the patient's planar image. However, the planar image captured by the 2D camera is easily affected by external environments such as light or shadow, and cannot perceive the patient's spatial information, resulting in low accuracy in positioning the patient. Summary of the invention

[0004] The purpose of the present application is to provide a multi-sensor data fusion positioning method, system and storage medium, aiming to solve the problem of low positioning accuracy of patients existing in traditional methods.

[0005] The present application provides a multi-sensor data fusion positioning method, comprising:

[0006] Acquire image data, point cloud data and three-dimensional data of the target area of ​​the subject during the medical examination;

[0007] Determine a three-dimensional bounding box of the target area according to the image data and the point cloud data;

[0008] Determining a reference bounding box of the target area according to the three-dimensional data;

[0009] According to the reference bounding box, positioning and correcting the stereoscopic bounding box to obtain a corrected bounding box of the target area;

[0010] The position of the subject during the medical examination is adjusted according to the correction bounding box of the target area.

[0011] In one embodiment, the step of determining a three-dimensional bounding box of the target area according to the image data and the point cloud data includes:

[0012] Convert the image data and the point cloud data to the same coordinate system to obtain a mapping calibration frame and point cloud mapping data;

[0013] According to the mapping calibration frame, feature extraction is performed on the point cloud mapping data to obtain target point cloud data of the target area;

[0014] The three-dimensional bounding box is constructed according to the target point cloud data.

[0015] In one embodiment, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area includes:

[0016] In the same coordinate system, calculating the center distance between the geometric center of the three-dimensional bounding box and the geometric center of the reference bounding box;

[0017] If the center distance is less than or equal to the distance threshold, then the overlap degree between the enclosing area of ​​the 3D enclosing frame and the reference enclosing frame is calculated.

[0018] In one embodiment, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area further includes:

[0019] If the overlap of the enclosed areas is greater than the overlap threshold, correcting multiple depth values ​​in the target area enclosed by the reference enclosing frame according to multiple reference depth values ​​in the target area enclosed by the reference enclosing frame to obtain multiple corrected depth values;

[0020] Acquire a plurality of horizontal values ​​and a plurality of vertical values ​​in the target area surrounded by the mapping calibration frame;

[0021] A plurality of corrected coordinate points and the corresponding corrected bounding boxes are constructed in the target area according to the plurality of corrected depth values, the plurality of lateral values, and the plurality of longitudinal values.

[0022] In one embodiment, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area further includes:

[0023] If the degree of overlap of the enclosed areas is less than or equal to the degree of overlap threshold, edge calibration is performed on the three-dimensional enclosing frame.

[0024] In one embodiment, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area further includes:

[0025] If the center distance is greater than the distance threshold, correcting the geometric center of the mapping calibration frame to obtain a corrected mapping calibration frame;

[0026] According to the corrected mapping calibration frame, feature extraction is performed on the point cloud mapping data to obtain target point cloud data in the target area.

[0027] In one embodiment, after the step of extracting features from the point cloud mapping data according to the corrected mapping calibration frame to obtain target point cloud data within the target area, the method further includes:

[0028] Acquire a plurality of corrected transverse values ​​and a plurality of corrected longitudinal values ​​in the target area surrounded by the corrected mapping calibration frame;

[0029] A plurality of corrected coordinate points in the target area and the corresponding corrected bounding boxes are constructed according to the plurality of corrected transverse values, the plurality of corrected longitudinal values, and the plurality of corrected depth values.

[0030] In one embodiment, the step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area includes:

[0031] Perform feature extraction on a plurality of calibration coordinate points in the target area enclosed by the calibration enclosing frame to obtain a plurality of calibration coordinate points corresponding to the joints of the subject and a corresponding joint enclosing frame;

[0032] The joint part is tracked according to a plurality of correction coordinate points corresponding to the joint part and / or a joint bounding box corresponding to the joint part to obtain a movement trajectory.

[0033] In one embodiment, the step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area includes:

[0034] According to the movement trajectory, the position of the subject is adjusted or the anti-collision adjustment is performed.

[0035] The present application provides a multi-sensor data fusion positioning system, which is applied to medical equipment, wherein the medical equipment includes an examination bed and a frame, and the multi-sensor data fusion positioning system includes:

[0036] An image data acquisition module is disposed on the frame and within the aperture of the frame. The image data acquisition module is disposed opposite to the examination bed and is used to acquire image data of a target area of ​​a subject located on the examination bed during a medical examination.

[0037] A point cloud data acquisition module is arranged on the frame, and the point cloud data acquisition module is arranged outside the aperture of the frame, and the point cloud data acquisition module is arranged opposite to the examination bed, and is used to collect point cloud data of the target area of ​​the subject during the medical examination process;

[0038] A three-dimensional data acquisition module, disposed near the examination bed of the medical device, for acquiring three-dimensional data of a target area of ​​the subject during the medical examination;

[0039] A processing module, connected to the image data acquisition module, the point cloud data acquisition module and the three-dimensional data acquisition module respectively;

[0040] The processing module is used to determine a three-dimensional bounding box of the target area based on the image data and the point cloud data; determine a reference bounding box of the target area based on the three-dimensional data; perform positioning correction on the three-dimensional bounding box based on the reference bounding box to obtain a corrected bounding box of the target area; and adjust the position of the subject during the medical examination process based on the corrected bounding box of the target area.

[0041] The present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0042] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0043] Through the multi-sensor data fusion positioning method provided by the present application, different types of image data are fused with point cloud data, fully integrating the characteristic stability advantages of 2D image data and the ranging precision advantages of point cloud data, and obtaining an accurate three-dimensional bounding box of the target area. The reference bounding box obtained by the three-dimensional data has accurate depth measurement information and rich three-dimensional details. Based on the reference bounding box obtained by the three-dimensional data, the three-dimensional bounding box is positioned and corrected, so that the reference bounding box and the three-dimensional bounding box of the same target area can be complementary references to obtain a more accurate corrected bounding box of the target area. The corrected bounding box of the target area can characterize the spatial position of the target area. Through the corrected bounding box of the target area, the spatial position of the target area can be accurately located, and then the spatial position of the subject during the medical examination process can be accurately located.

[0044] Therefore, through the multi-sensor data fusion positioning method provided by the present application, different types of image data, point cloud data and three-dimensional data are fused, the advantages of different types of data are integrated, and detection and observation are performed from multiple different angles, which reduces the uncertainty of the data and improves the positioning accuracy. Therefore, through the multi-sensor data fusion positioning method provided by the present application, the spatial position of the patient can be more accurately located to realize medical detection such as CT detection, MRI detection, PET detection, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the steps of a multi-sensor data fusion positioning method in an embodiment provided in the present application;

[0046] Figure 2 A schematic diagram of the steps of a multi-sensor data fusion positioning method in another embodiment provided in the present application;

[0047] Figure 3 This is a schematic diagram of the structure of the multi-sensor data fusion positioning system provided in this application. DETAILED DESCRIPTION

[0048] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0050] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0051] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0052] See also Figure 1 , the present application provides a multi-sensor data fusion positioning method, comprising:

[0053] Step S10, acquiring image data, point cloud data and three-dimensional data of a target area of ​​a subject during a medical examination;

[0054] Step S20, determining a three-dimensional bounding box of the target area according to the image data and the point cloud data;

[0055] Step S30, determining a reference bounding box of the target area according to the three-dimensional data;

[0056] Step S40, positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area;

[0057] Step S50, adjusting the position of the subject during the medical examination according to the correction bounding box of the target area.

[0058] In this embodiment, in step S10, the medical test may be a CT test, an MRI test, a PET test, etc. The testee may be understood as a patient or other person who needs medical testing. The target area may also be understood as an area of ​​interest, which is a local area and may be selected according to the actual application scenario, such as an area corresponding to an organ or a lesion area. By focusing on the target area, a certain area of ​​the testee may be analyzed more deeply without analyzing the entire area, which is more targeted and can achieve more accurate positioning. The image data is two-dimensional data obtained by a 2D camera. The point cloud data is discrete data obtained by a 3D sensor. In one embodiment, the 3D sensor may be a device such as a laser radar sensor or a time-of-flight sensor that can obtain three-dimensional information of an object or scene. Three-dimensional data is data obtained by a 3D camera that can characterize a three-dimensional space. In one embodiment, the 3D camera may be a binocular vision 3D camera that calculates depth information based on the parallax principle.

[0059] In step S20, the image data represents the coordinate information of the two-dimensional space of the target area, and has a high image resolution. Point cloud data is important data for obtaining three-dimensional information, especially in terms of ranging, which is more accurate and can accurately represent the depth information of the target area, making up for the lack of depth information in the image data, thereby further enriching the position description of the target area. Point cloud data is discrete data. Based on the coordinate information of the two-dimensional space of the target area, the point cloud range of the target area in the point cloud data is determined, and then a three-dimensional bounding box of the target area is obtained. The three-dimensional bounding box is a geometric shape that surrounds the target area in three-dimensional space, which can represent the position information of the surrounded target area in three-dimensional space, and integrates the two-dimensional coordinate information in the image data and the depth information in the point cloud data, so as to more accurately represent the position information of the target area. The three-dimensional bounding box can also be understood as a spatial geometric structure used to surround and define the geometric range of the target area in three-dimensional space, which can determine the position and size of the target area of ​​the subject in three-dimensional space.

[0060] In step S30, the three-dimensional data can accurately describe the shape, position and size of the target area in the three-dimensional space, including the geometric shape information, surface texture information, color information, etc. of the object, and has the advantages of accurate depth information, rich three-dimensional details, and strong adaptability and flexibility. The geometric shape of the target area can be determined by the x, y, and z coordinates in the three-dimensional data (which can also be understood as the three dimensions of length, width, and height, and can also be understood as the three dimensions of horizontal, vertical, and depth), that is, the reference bounding box is obtained. The reference bounding box is a three-dimensional bounding box formed by the three-dimensional data collected by the three-dimensional data acquisition module 30, which contains rich three-dimensional information such as depth, and can more accurately describe the information of the target area of ​​the subject in the three-dimensional space. In the present application, the reference bounding box obtained by the three-dimensional data is used as a reference basis to position and correct the three-dimensional bounding box in step S40.

[0061] During the medical examination process, the image data, point cloud data and three-dimensional data of the target area of ​​the subject are obtained through three different methods, which can complement each other and obtain the location information of the target area more accurately.

[0062] In step S40, the three-dimensional bounding box is positioned and corrected based on the reference bounding box, and the positioning error can be identified and corrected to obtain a corrected three-dimensional bounding box of the target area, that is, a corrected bounding box. The image data, point cloud data, and three-dimensional data obtained by different measurement methods can be complementary references, integrating the advantages of different measurement methods or instruments, observing from multiple different angles, reducing data uncertainty, and improving positioning accuracy.

[0063] In step S50, by tracking the corrected bounding box of the target area, the corrected three-dimensional coordinate information of the target area can be obtained in real time, and then the moving trajectory of the subject can be obtained. Therefore, according to the moving trajectory of the subject, the position of the subject in the medical examination process can be known in real time, and the position of the subject can be adjusted, so as to position the subject or avoid collision between the subject and the equipment, and more accurately assist the subject to perform CT examination, MRI examination, PET examination, etc.

[0064] Through the multi-sensor data fusion positioning method provided by the present application, different types of image data are fused with point cloud data, fully integrating the characteristic stability advantages of 2D image data and the ranging precision advantages of point cloud data, and obtaining an accurate three-dimensional bounding box of the target area. The reference bounding box obtained by the three-dimensional data has accurate depth measurement information and rich three-dimensional details. Based on the reference bounding box obtained by the three-dimensional data, the three-dimensional bounding box is positioned and corrected, so that the reference bounding box and the three-dimensional bounding box of the same target area can be complementary references to obtain a more accurate corrected bounding box of the target area. The corrected bounding box of the target area can characterize the spatial position of the target area. Through the corrected bounding box of the target area, the spatial position of the target area can be accurately located, and then the spatial position of the subject during the medical examination process can be accurately located.

[0065] Therefore, through the multi-sensor data fusion positioning method provided by the present application, different types of image data, point cloud data and three-dimensional data are fused, the advantages of different types of data are integrated, and detection and observation are performed from multiple different angles, which reduces the uncertainty of the data and improves the positioning accuracy. Therefore, through the multi-sensor data fusion positioning method provided by the present application, the spatial position of the patient can be more accurately located to realize medical detection such as CT detection, MRI detection, PET detection, etc.

[0066] See also Figure 2 In one embodiment, step S20, the step of determining a stereoscopic bounding box of the target area according to the image data and the point cloud data, comprises:

[0067] Step S210, converting the image data and the point cloud data to the same coordinate system to obtain a mapping calibration frame and point cloud mapping data;

[0068] Step S220, extracting features from the point cloud mapping data according to the mapping calibration frame to obtain target point cloud data of the target area;

[0069] Step S230: construct a three-dimensional bounding box according to the target point cloud data.

[0070] In this embodiment, in step S210, the image data is data obtained in the image coordinate system. The point cloud data is data obtained in the 3D sensor coordinate system, such as data obtained in the laser radar coordinate system. The image data and the point cloud data are located in different coordinate systems. Therefore, the image data and the point cloud data are respectively converted to the same coordinate system, and the image data and the point cloud data are unified to facilitate subsequent data processing. The same coordinate system can be a camera coordinate system or a world coordinate system. In one embodiment, the image data and the point cloud data are respectively converted to the camera coordinate system. The camera is calibrated to obtain internal and external parameters. The internal parameters include parameters such as focal length, principal point coordinates, and distortion coefficients.

[0071] In one embodiment, a calibration plate (e.g., a checkerboard calibration plate) is used for a calibration experiment. By shooting images of the calibration plate at different angles, the intrinsic parameter matrix of the camera is calculated using a calibration algorithm (e.g., Zhang Zhengyou calibration method). Through the camera intrinsic parameters, the four vertex coordinates of the two-dimensional rectangular box in the image data are converted from image coordinates to camera coordinates, thereby obtaining a mapping calibration frame in the camera coordinate system. The extrinsic parameters are the external parameters of the camera relative to the world coordinate system, i.e., the rotation matrix and the translation vector. In one embodiment, the extrinsic parameters can be calculated by placing markers with known coordinates in the scene and detecting the positions of these markers in the image. A projection mapping relationship between the laser radar coordinate system and the camera coordinate system is established, and the point cloud data is converted to the camera coordinate system to obtain point cloud mapping data.

[0072] The laser radar coordinate system takes the center of the laser radar as the origin, and the direction of the coordinate axis defines the position of the point detected by the laser radar in three-dimensional space. For example, in the laser radar coordinate system, the X-axis points to the front of the laser radar, the Y-axis points to the left, and the Z-axis points vertically upward. The position of the point is determined by the distance, azimuth, and elevation. The camera coordinate system takes the optical center of the camera as the origin. In the camera coordinate system, the x-axis and y-axis are parallel to the horizontal and vertical directions of the image plane, respectively, and the z-axis points to the shooting direction of the camera, which determines the correspondence between the spatial position of the object in the camera's field of view and the image pixel.

[0073] Assume that the coordinates of a point in the point cloud data in the laser radar coordinate system are P l =(x l ,y l ,z l ) T According to the projection mapping relationship, it can also be understood that the coordinate transformation formula is P c =R×P l +T for transformation. Where R is the rotation matrix, T is the translation vector, P l is the coordinate in the laser radar coordinate system, P cis the coordinate in the camera coordinate system. Then, the point cloud data is converted from the spatial position description of the laser radar to the spatial position description of the camera, and the point cloud data is converted to the camera coordinate system to obtain the point cloud mapping data.

[0074] In step S220, in the same coordinate system, based on the mapping calibration frame, the point cloud corresponding to the mapping calibration frame is screened out. It can also be understood as screening out the point cloud of the target area corresponding to the mapping calibration frame to obtain the target point cloud data corresponding to the target area. The target point cloud data is discrete data. The distribution of the point cloud in the direction perpendicular to the camera plane can be determined by the depth information in the screened target point cloud data. According to the depth value of the target point cloud, the minimum and maximum depth values ​​are determined to construct the range of the three-dimensional bounding box in the depth direction.

[0075] In step S230, principal component analysis is performed on the screened target point cloud data to determine the directions of the three coordinate axes of the point cloud. Further, ensure that the direction of the three-dimensional bounding box is consistent with the direction of the two-dimensional rectangular box in the image data on the image plane. According to the coordinate range of the target point cloud data in three directions, the length, width and height of the three-dimensional bounding box are determined respectively. Thus, three-dimensional construction is performed based on the mapping calibration frame and the target point cloud data to construct a three-dimensional bounding box corresponding to the target area. The three-dimensional bounding box combines the respective advantages of image data and point cloud data, obtains location information from multiple different angles, and improves positioning accuracy.

[0076] In one embodiment, in step S30, multiple images of the target area of ​​the subject are collected in real time by a binocular stereo vision module, and the binocular stereo vision module is binocularly calibrated to obtain calibration parameters of the binocular stereo vision module. Then, binocular stereo matching is performed on the multiple images to obtain image parallax. According to the calibration parameters and the image parallax, the multiple images are triangulated to obtain three-dimensional data. Thus, according to the three-dimensional data, the reference bounding box parameters are calculated, and according to the reference bounding box parameters, the reference bounding box of the target area is determined.

[0077] In one embodiment, step S40, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area, includes:

[0078] Step S410, calculating the center distance between the geometric center of the 3D bounding box and the geometric center of the reference bounding box in the same coordinate system;

[0079] Step S411, determining whether the center distance is less than or equal to a distance threshold;

[0080] Step S420, if the center distance is less than or equal to the distance threshold, then calculating the overlap between the enclosing area of ​​the stereoscopic enclosing frame and the reference enclosing frame;

[0081] Step S421, determining whether the degree of overlap of the enclosed areas is greater than a threshold value of overlap.

[0082] In this embodiment, in step S410, the geometric center of the stereoscopic enclosing frame can also be understood as the center point of the stereoscopic enclosing frame, and the distance to each vertex is the same. The geometric center of the reference enclosing frame can also be understood as the center point of the reference enclosing frame, and the distance to each vertex is the same. In the same coordinate system, the center point distance of the two geometric centers is calculated. The center point distance can be decomposed into distance components in three directions: the x-axis, the y-axis, and the z-axis.

[0083] In step S411, the distance threshold is used as a reference to determine whether the center distance is less than or equal to the distance threshold. In one embodiment, the distance threshold can be a specific value between 1 mm and 3 mm, such as 1 mm, 2 mm or 3 mm. The distance threshold can also be adjusted according to the actual application scenario. If the center distance is less than or equal to the distance threshold, it indicates that the geometric center of the stereoscopic bounding box is close to the geometric center of the reference bounding box and has consistency, which further indicates that the obtained geometric center position has high reliability and high accuracy.

[0084] In step S420, the overlap of the enclosed area can be understood as the overlap volume of the enclosed area of ​​the three-dimensional enclosing frame and the reference enclosing frame. Then, in step S421, it is determined whether the overlap of the enclosed area is greater than the overlap threshold. The overlap threshold can be a specific value between 95% and 98%, such as 95%, 96%, 97% or 98%, and can also be set according to the actual application scenario.

[0085] The data of the stereo bounding box comes from the fusion of image data and point cloud data. The data of the reference bounding box comes from three-dimensional data. In the process of medical testing, the image data, point cloud data and three-dimensional data of the target area of ​​the subject are obtained in three different ways, with the advantages of different angles. Through steps S410, S411, S420 and S421, the center point distance and the degree of overlap of the enclosed area are calculated, and the distance threshold and the degree of overlap of the enclosed area are compared and judged respectively, thereby realizing the fusion of image data, point cloud data and three-dimensional data, so that multiple different data can complement each other for reference, solving the problem of low positioning accuracy caused by the use of single data, and more accurately obtaining the location information of the target area.

[0086] In one embodiment, step S40, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area, further includes:

[0087] Step S430, if the overlap of the enclosed areas is greater than the overlap threshold, correcting multiple depth values ​​in the target area enclosed by the stereo enclosing frame according to multiple reference depth values ​​in the target area enclosed by the reference enclosing frame to obtain multiple corrected depth values;

[0088] Step S440, obtaining a plurality of horizontal values ​​and a plurality of vertical values ​​in a target area surrounded by a mapping calibration frame;

[0089] Step S450 , constructing a plurality of calibrated coordinate points and corresponding calibrated bounding boxes in the target area according to the plurality of calibrated depth values, the plurality of lateral values, and the plurality of longitudinal values.

[0090] In this embodiment, in step S430, when the overlap of the enclosed area is greater than the overlap threshold, it indicates that the stereoscopic enclosing frame is consistent with the reference enclosing frame, which further indicates that the obtained data reliability is high, and also means that the target area enclosed by the two is the same target area. Thus, multiple reference depth values ​​in the target area enclosed by the reference enclosing frame are used to replace multiple depth values ​​in the target area enclosed by the stereoscopic enclosing frame as multiple corrected depth values, thereby achieving correction of the depth values ​​in the target area enclosed by the stereoscopic enclosing frame. Through step S430, correction of multiple depth values ​​in the target area enclosed by the stereoscopic enclosing frame is achieved, which can eliminate errors, improve the accuracy and reliability of data, and make multiple corrected depth values ​​more accurate.

[0091] In step S440, multiple horizontal values ​​and multiple vertical values ​​in the target area surrounded by the mapping calibration frame are all from image data and have high image resolution. The horizontal value can be understood as the value in the x-coordinate direction. The vertical value can be understood as the value in the y-coordinate direction. Then, in step S450, multiple correction coordinate points corresponding to the target area and the corresponding correction bounding box are jointly constructed based on the multiple correction depth values, the multiple horizontal values ​​and the multiple vertical values.

[0092] In one embodiment, step S40, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area, further includes:

[0093] Step S460: If the overlap of the enclosing regions is less than or equal to the overlap threshold, edge calibration is performed on the three-dimensional enclosing frame.

[0094] In this embodiment, when the center distance in step S420 is less than or equal to the distance threshold, it indicates that the position data accuracy of the geometric center of the three-dimensional enclosing frame is high, and there is no need to adjust the position of the geometric center. When the center distance is less than or equal to the distance threshold, step S460 is executed. When the overlap of the enclosed area is less than or equal to the overlap threshold, it indicates that the overlap of the reference enclosing frame and the three-dimensional enclosing frame is low, and the edge calibration of the three-dimensional enclosing frame is required. Edge calibration is the process of adjusting and correcting the boundaries of the three-dimensional enclosing frame to make the edge positioning of the three-dimensional enclosing frame more accurate. Through edge calibration, the vertex coordinates, side length, angle and other parameters of the three-dimensional enclosing frame can be adjusted to more accurately represent the range of the target area in three-dimensional space.

[0095] When edge calibration is performed on the stereo bounding box, the position data of the geometric center of the stereo bounding box has high accuracy and can be maintained unchanged. The edge data of the stereo bounding box is adjusted, and some point cloud data is added to achieve edge calibration, so that more data enters the stereo bounding box, and the volume of the enclosed area of ​​the stereo bounding box is expanded until the overlap of the enclosed area is greater than the overlap threshold. Through step S460, the overlap of the enclosed area of ​​the stereo bounding box and the reference bounding box is adjusted so that the overlap of the enclosed area is greater than the overlap threshold, thereby making the areas enclosed by the two approach the same. Based on the overlap threshold as a reference, fine-tuning the stereo bounding box can improve the accuracy of the data enclosed by the stereo bounding box, so that subsequent steps S430 to S450 are further executed to obtain more accurate positioning information.

[0096] In one embodiment, step S40, the step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area, further includes:

[0097] Step S470, if the center distance is greater than the distance threshold, the geometric center of the mapping calibration frame is corrected to obtain a corrected mapping calibration frame;

[0098] Step S480, according to the corrected mapping calibration frame, feature extraction is performed on the point cloud mapping data to obtain the target point cloud data in the target area, that is, according to the corrected mapping calibration frame, step S220 is executed to obtain the target point cloud data corresponding to the corrected mapping calibration frame, and subsequent steps S230 to S460 are executed in sequence.

[0099] In this embodiment, in step S470, if the center distance is greater than the distance threshold, it indicates that the geometric center of the stereo bounding box is far away from the geometric center of the reference bounding box, and the obtained geometric center position has low reliability and accuracy, and the position of the geometric center of the stereo bounding box needs to be adjusted. The stereo bounding box is derived from the point cloud data corresponding to the mapping calibration frame. Furthermore, by correcting the geometric center of the mapping calibration frame, the correction of the geometric center of the stereo bounding box can be achieved, that is, the corrected mapping calibration frame is obtained. Based on the corrected mapping calibration frame, feature extraction is performed on the point cloud mapping data again to obtain the target point cloud data in the target area, that is, according to the corrected mapping calibration frame, step S220 is re-executed. For related descriptions, please refer to the related descriptions in the above embodiments. When the center distance is greater than the distance threshold, the geometric center of the mapping calibration frame is corrected through steps S470 and S480, and then the geometric center of the stereo bounding frame is corrected. Therefore, based on the corrected mapping calibration frame, the point cloud mapping data is re-extracted to obtain more accurate target point cloud data in the target area, and the location information of the target area is determined more accurately, thereby improving the patient's positioning accuracy.

[0100] In one embodiment, after the step S480 of extracting features from the point cloud mapping data according to the corrected mapping calibration frame to obtain the target point cloud data within the target area, the subsequent steps S230 to S421 are sequentially executed, and the multi-sensor data fusion positioning method further includes:

[0101] Step S490, obtaining a plurality of corrected transverse values ​​and a plurality of corrected longitudinal values ​​in the target area surrounded by the corrected mapping calibration frame, that is, executing step S440 according to the corrected mapping calibration frame to obtain a plurality of corrected transverse values ​​and a plurality of corrected longitudinal values ​​corresponding to the corrected mapping calibration frame;

[0102] Step S491, construct multiple correction coordinate points and corresponding correction bounding boxes in the target area according to multiple correction horizontal values, multiple correction vertical values ​​and multiple correction depth values, that is, according to multiple correction horizontal values, multiple correction vertical values ​​and multiple correction depth values, execute step S450 to construct multiple correction coordinate points and corresponding correction bounding boxes in the target area.

[0103] In this embodiment, the description of step S490 can refer to the description of step S440 in the above embodiment. The description of step S491 can refer to the description of step S450 in the above embodiment.

[0104] In step S490, the geometric center of the corrected mapping calibration frame changes, so that the corresponding horizontal value (which can also be understood as the value in the x-axis direction) and the longitudinal value (which can also be understood as the value in the y-axis direction) are also transformed, and the corresponding corrected horizontal value and corrected longitudinal value are obtained. According to the corrected horizontal value, the corrected longitudinal value and the corrected depth value, a correction coordinate point is formed, and then a plurality of correction coordinate points and a corresponding correction bounding frame in the target area are obtained. By correcting the geometric center of the mapping calibration frame, the center distance between the geometric center of the stereo bounding frame and the geometric center of the reference bounding frame is less than or equal to the distance threshold, so that the stereo bounding frame and the reference bounding frame are consistent, so that the obtained data is highly accurate, which also means that the target area surrounded by the two is the same target area.

[0105] In one embodiment, step S50, the step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area, includes:

[0106] Step S510, extracting features of a plurality of calibration coordinate points in a target area surrounded by a calibration bounding box, and obtaining a plurality of calibration coordinate points corresponding to joints of the subject and a corresponding joint bounding box;

[0107] Step S520 , tracking the joint part according to the multiple correction coordinate points corresponding to the joint part and / or the joint bounding box corresponding to the joint part to obtain the movement trajectory.

[0108] In this embodiment, in step S510, the target area surrounded by the correction bounding box includes multiple correction coordinate points. Each correction coordinate point represents a specific position in space. Each correction coordinate point can be represented by (x, y, z), which respectively represents the position of the correction coordinate point in three mutually perpendicular directions. The joint parts of the subject can be understood as the shoulder joint, elbow joint, wrist joint, knee joint, ankle joint, etc. of the subject. One joint part corresponds to a joint bounding box, which corresponds to multiple correction coordinate points. By performing feature extraction on multiple correction coordinate points in the target area surrounded by the correction bounding box, the joint bounding box of a certain joint part and the corresponding multiple correction coordinate points can be obtained.

[0109] Furthermore, in step S520, the multiple correction coordinate points corresponding to the joint parts and / or the joint bounding boxes corresponding to the joint parts are tracked, and the specific spatial position of the joint parts at each moment can be known, so that the movement trajectory of the subject can be known, and the spatial position of the subject can be located and tracked. Through steps S510 and S520, the spatial position of the joint parts of the subject can be accurately located, and the spatial position of the subject can be known in real time, so that the operator in the operating room can take countermeasures in time, and perform CT detection, MRI detection or PET detection on the subject.

[0110] In one embodiment, step S50, the step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area, further includes:

[0111] Step S530: adjusting the position or anti-collision of the subject according to the moving trajectory.

[0112] In this embodiment, the position of the subject is adjusted so that the subject or the part being tested / treated is in the best operating position of the device to ensure the accuracy, effectiveness and safety of the medical test or treatment. In one embodiment, in radiotherapy, the tumor of the subject is accurately placed within the irradiation range of the radiation beam through the multi-sensor data fusion positioning method provided by the present application. In medical imaging examination, the body part of the subject is placed in a suitable imaging plane.

[0113] During dynamic processes such as CT, MRI or PET testing, anti-collision adjustments are performed on the subject to prevent accidental collisions between the medical equipment and the subject during movement, operation or operation, thereby ensuring the safety of the subject and the operator.

[0114] According to the movement trajectory, when adjusting the position of the subject or anti-collision adjustment, the relative position of the subject and the equipment can be adjusted to the optimal detection position by adjusting the position of the subject or adjusting the mechanical structure of the equipment (such as moving the bed, rotating the frame, adjusting the detector position, etc.) to complete subsequent CT detection, MRI detection or PET detection, etc.

[0115] See also Figure 3 , the present application provides a multi-sensor data fusion positioning system 100, which is applied to a medical device 200. The medical device 200 includes an examination bed 201 and a frame 202. The multi-sensor data fusion positioning system 100 includes an image data acquisition module 10, a point cloud data acquisition module 20, a three-dimensional data acquisition module 30 and a processing module 40. The image data acquisition module 10 is arranged on the frame 202. The image data acquisition module 10 is arranged in the aperture of the frame 202. The image data acquisition module 10 is arranged opposite to the examination bed 201. The image data acquisition module 10 is used to collect image data of a target area of ​​a subject located on the examination bed 201 during a medical examination, for example Figure 3 The target area A and the target area B are shown. The target area can be set according to the actual application scenario. In one embodiment, the image data acquisition module 10 is set just above the center of the aperture, which can more clearly collect information such as the position and state view of the subject.

[0116] The point cloud data acquisition module 20 is arranged on the frame 202. And the point cloud data acquisition module 20 is arranged outside the aperture of the frame 202. The point cloud data acquisition module 20 is arranged relative to the examination bed 201, and is used to collect point cloud data of the target area of ​​the subject during the medical examination process. The number and position of the point cloud data acquisition module 20 can be set according to actual needs. In one embodiment, the position of the point cloud data acquisition module 20 can be set to an angle of 0°, 60°, 120° or 180° with the line connecting the center of the aperture.

[0117] In one embodiment, the number of point cloud data acquisition modules 20 can be set according to demand. The multi-sensor data fusion positioning system 100 includes multiple point cloud data acquisition modules 20. In one embodiment, the multi-sensor data fusion positioning system 100 includes 4 point cloud data acquisition modules 20. The angles between the position of each point cloud data acquisition module 20 and the line connecting the center of the aperture are 0°, 60°, 120°, and 180°, respectively, and point cloud data can be collected from different angles on the target area of ​​the subject on the examination bed 201. The data collected from each other can be supplemented and referenced to each other, so as to more accurately locate the position of the subject.

[0118] The three-dimensional data acquisition module 30 is arranged near the examination bed 201 of the medical device 200, and is used to collect three-dimensional data of the target area of ​​the subject during the medical examination. The three-dimensional data acquisition module 30 is arranged near the examination bed 201 of the medical device 200, so that the three-dimensional data acquisition module 30, the image data acquisition module 10, and the point cloud data acquisition module 20 are respectively arranged at different positions, and then the data of the subject during the medical examination is collected from different angles. The data collected from each other can be supplemented and referenced to each other, and the position of the subject is more accurately located.

[0119] The processing module 40 is connected to the image data acquisition module 10, the point cloud data acquisition module 20 and the three-dimensional data acquisition module 30 respectively. The processing module 40 is used to determine the stereoscopic bounding box of the target area according to the image data and the point cloud data. The processing module 40 is used to determine the reference bounding box of the target area according to the three-dimensional data. The processing module 40 is used to position and correct the stereoscopic bounding box according to the reference bounding box to obtain the corrected bounding box of the target area. The processing module 40 is used to adjust the position of the subject in the medical detection process according to the corrected bounding box of the target area.

[0120] In this embodiment, the description of the image data acquisition module 10 can refer to the description of step S10 in the above embodiment. The description of the point cloud data acquisition module 20 can refer to the description of step S20 in the above embodiment. The description of the three-dimensional data acquisition module 30 can refer to the description of step S30 in the above embodiment. The description of the processing module 40 can refer to the description of steps S40 and S50 in the above embodiment.

[0121] In one embodiment, the image data acquisition module 10 may be a 2D camera, the point cloud data acquisition module 20 may be a 3D sensor, and the three-dimensional data acquisition module 30 may be a 3D camera.

[0122] In one embodiment, the medical device 200 is a CT device. The image data acquisition module 10 and the point cloud data acquisition module 20 are arranged in the aperture of the CT device. The three-dimensional data acquisition module 30 is arranged outside the aperture of the CT device. The image data acquisition module 10, the point cloud data acquisition module 20 and the three-dimensional data acquisition module 30 respectively collect data from different angles of the subject, so that the image data, point cloud data and three-dimensional data obtained by different measurement methods can be complementary references, integrating the advantages of different measurement methods or instruments, and performing detection and observation from multiple different angles, reducing the uncertainty of the data and improving the positioning accuracy.

[0123] In one embodiment, the image data acquisition module 10 and the point cloud data acquisition module 20 are embedded in the housing of the medical device 200, so that the integration between the medical device 200, the image data acquisition module 10 and the point cloud data acquisition module 20 is higher, which is more conducive to saving space. The three-dimensional data acquisition module 30 is arranged outside the medical device 200 and can be arranged on the top of the detection room, which is conducive to saving space in the detection room. Furthermore, the data collected by the image data acquisition module 10, the point cloud data acquisition module 20 and the three-dimensional data acquisition module 30 can complement each other for reference, so as to improve the positioning accuracy of the patient, more accurately position the patient or avoid collision between the patient and the equipment.

[0124] In one embodiment, the processing module 40 may be a computer, a microcontroller, a central processing unit, or a digital signal processor.

[0125] In one embodiment, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0126] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0127] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0128] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0130] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0134] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-sensor data fusion positioning method, characterized in that: include: Acquire image data, point cloud data and three-dimensional data of the target area of ​​the subject during the medical examination; Determine a three-dimensional bounding box of the target area according to the image data and the point cloud data; Determining a reference bounding box of the target area according to the three-dimensional data; According to the reference bounding box, positioning and correcting the stereoscopic bounding box to obtain a corrected bounding box of the target area; The position of the subject during the medical examination is adjusted according to the correction bounding box of the target area.

2. The multi-sensor data fusion positioning method according to claim 1, characterized in that: The step of determining a stereoscopic bounding box of the target area according to the image data and the point cloud data comprises: Convert the image data and the point cloud data to the same coordinate system to obtain a mapping calibration frame and point cloud mapping data; According to the mapping calibration frame, feature extraction is performed on the point cloud mapping data to obtain target point cloud data of the target area; The three-dimensional bounding box is constructed according to the target point cloud data.

3. The multi-sensor data fusion positioning method according to claim 2, characterized in that: The step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area includes: In the same coordinate system, calculating the center distance between the geometric center of the three-dimensional bounding box and the geometric center of the reference bounding box; If the center distance is less than or equal to the distance threshold, then the overlap degree between the enclosing area of ​​the 3D enclosing frame and the reference enclosing frame is calculated.

4. The multi-sensor data fusion positioning method according to claim 3, characterized in that: The step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area also includes: If the overlap of the enclosed areas is greater than the overlap threshold, correcting multiple depth values ​​in the target area enclosed by the reference enclosing frame according to multiple reference depth values ​​in the target area enclosed by the reference enclosing frame to obtain multiple corrected depth values; Acquire a plurality of horizontal values ​​and a plurality of vertical values ​​in the target area surrounded by the mapping calibration frame; Constructing a plurality of correction coordinate points and the corresponding correction bounding boxes in the target area according to the plurality of correction depth values, the plurality of horizontal values, and the plurality of vertical values; If the degree of overlap of the enclosed areas is less than or equal to the degree of overlap threshold, edge calibration is performed on the three-dimensional enclosing frame.

5. The multi-sensor data fusion positioning method according to claim 3, characterized in that: The step of positioning and correcting the stereoscopic bounding frame according to the reference bounding frame to obtain a corrected bounding frame of the target area also includes: If the center distance is greater than the distance threshold, correcting the geometric center of the mapping calibration frame to obtain a corrected mapping calibration frame; According to the corrected mapping calibration frame, feature extraction is performed on the point cloud mapping data to obtain target point cloud data in the target area.

6. The multi-sensor data fusion positioning method according to claim 5, characterized in that: After the step of extracting features from the point cloud mapping data according to the corrected mapping calibration frame to obtain target point cloud data within the target area, the method further includes: Acquire a plurality of corrected transverse values ​​and a plurality of corrected longitudinal values ​​in the target area surrounded by the corrected mapping calibration frame; A plurality of corrected coordinate points in the target area and the corresponding corrected bounding boxes are constructed according to the plurality of corrected transverse values, the plurality of corrected longitudinal values, and the plurality of corrected depth values.

7. The multi-sensor data fusion positioning method according to any one of claims 1 to 6, characterized in that: The step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area includes: Perform feature extraction on a plurality of calibration coordinate points in the target area enclosed by the calibration enclosing frame to obtain a plurality of calibration coordinate points corresponding to the joints of the subject and a corresponding joint enclosing frame; The joint part is tracked according to a plurality of correction coordinate points corresponding to the joint part and / or a joint bounding box corresponding to the joint part to obtain a movement trajectory.

8. The multi-sensor data fusion positioning method according to claim 7, characterized in that: The step of adjusting the position of the subject during the medical examination according to the correction bounding box of the target area includes: According to the movement trajectory, the position of the subject is adjusted or the anti-collision adjustment is performed.

9. A multi-sensor data fusion positioning system, characterized in that: Applied to a medical device (200), the medical device (200) comprising an examination bed (201) and a frame (202), the multi-sensor data fusion positioning system comprising: An image data acquisition module (10) is arranged on the frame (202), and the image data acquisition module (10) is arranged in the aperture of the frame (202), and the image data acquisition module (10) is arranged opposite to the examination bed (201), and is used to acquire image data of a target area of ​​a subject located on the examination bed (201) during a medical examination; A point cloud data acquisition module (20) is arranged on the frame (202), and the point cloud data acquisition module (20) is arranged outside the aperture of the frame (202), and the point cloud data acquisition module (20) is arranged opposite to the examination bed (201), and is used to collect point cloud data of the target area of ​​the subject during the medical examination process; A three-dimensional data acquisition module (30), arranged close to the examination bed (201), for acquiring three-dimensional data of a target area of ​​the subject during a medical examination; A processing module (40) connected to the image data acquisition module (10), the point cloud data acquisition module (20) and the three-dimensional data acquisition module (30) respectively; The processing module (40) is used to determine a three-dimensional bounding box of the target area based on the image data and the point cloud data; determine a reference bounding box of the target area based on the three-dimensional data; perform positioning correction on the three-dimensional bounding box based on the reference bounding box to obtain a corrected bounding box of the target area; and adjust the position of the subject during the medical examination process based on the corrected bounding box of the target area.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images