Method for predicting collision between examination object and imaging device, and imaging device

Abstract

The invention relates to a method for predicting collision between an examination object and an imaging device and the imaging device. The prediction method may include: obtaining an image package of the inspection object through a multi-modal camera system, the image package including a depth image and a thermal image of the inspection object, the multi-modal camera system including a depth camera module and a thermal camera module; obtaining a 2D contour of the inspection object based on segmentation processing of the thermal image; generating a 3D contour of the inspection object based on the 2D contour of the inspection object and the depth image of the inspection object; and estimating whether the inspection object collides with an imaging device for scanning the inspection object on the moving path of the inspection object based on the 3D contour of the inspection object. The imaging device provided by the invention can realize the same prediction effect.

Description

Technical Field

[0001] The present invention generally relates to the medical field, and more particularly to a method for predicting a collision between an object to be examined and an imaging device, and an imaging device. Background Art

[0002] In a medical institution, it is often necessary to perform a scan and imaging on an object to be examined, such as a human body or an animal body, by an imaging device such as a CT (Computed Tomography) or an MR (Magnetic Resonance). During this process, when an operator controls the movement of the scan bed or the scan bed automatically adjusts its movement, it is very likely that the object to be examined will collide with the scan rack. For example, as Figure 16 shown, if the elbow joint of the human body being examined remains stationary, it will collide with the scan rack. In order to avoid such collision accidents, it is desirable to predict such a collision.

[0003] In actual operation, the areas that may collide with the scan rack include not only the parts on the object to be examined (such as the elbow joint, legs, head, etc. of the human body), but also the accessories on the scan bed such as sheets and blankets, and also "noises" such as an operator standing very close to the object to be examined. However, even if a collision occurs to the accessories on the scan bed, it will not affect the scan, nor will it cause harm to the object to be examined. The operator will not enter the scan rack hole together with the scan bed, so there will be no real collision with the scan rack. Interference factors such as the above-mentioned accessories and "noises" will greatly affect the accuracy of predicting whether the object to be examined will collide with the imaging device.

[0004] Therefore, there is a great need for a technology that can accurately predict the collision between the object to be examined and the imaging device while excluding other interference factors. Summary of the Invention

[0005] The present invention aims to overcome the above and / or other problems in the prior art. According to the present invention, there is provided a method for predicting a collision between an object to be examined and an imaging device, and an imaging device capable of realizing such prediction, which can accurately predict whether the object to be examined will collide with the imaging device while completely excluding interference factors such as accessories and "noises", thereby effectively ensuring that the imaging device scans and images the object to be examined efficiently and safely.

[0006] According to a first aspect of the present invention, there is provided a method for predicting a collision between an object to be inspected and an imaging device, which may include: a) obtaining an image packet of the object to be inspected through a multimodal camera system, the image packet including a depth image and a thermal image of the object to be inspected, the multimodal camera system including a depth camera module and a thermal camera module; b) obtaining a 2D contour of the object to be inspected based on segmentation processing of the thermal image; c) generating a 3D contour of the object to be inspected based on the 2D contour of the object to be inspected and the depth image of the object to be inspected; and d) predicting whether the object to be inspected will collide with the imaging device that scans the object to be inspected on its moving path based on the 3D contour of the object to be inspected.

[0007] According to a second aspect of the present invention, there is provided an imaging device, which may include a frame, a multimodal camera system, and a processing unit. The frame may include a frame hole for accommodating an object to be inspected. The multimodal camera system may include a depth camera module and a thermal camera module, and the multimodal camera system may be used to obtain an image packet of the object to be inspected, the image packet may include a depth image and a thermal image of the object to be inspected, and the multimodal camera system may include a depth camera module and a thermal camera module. The processing unit may be configured to: obtain a 2D contour of the object to be inspected based on segmentation processing of the thermal image; generate a 3D contour of the object to be inspected based on the 2D contour of the object to be inspected and the depth image of the object to be inspected; and predict whether the object to be inspected will collide with the imaging device that scans the object to be inspected on its moving path based on the 3D contour of the object to be inspected.

[0008] The present invention innovatively obtains the 2D contour of the object to be inspected through segmentation processing of the thermal image and further combines depth information, thereby excluding other interference factors that do not belong to the object to be inspected from the obtained 2D contour based on the thermal temperature information. On this basis, by further combining the depth information of the object to be inspected, a 3D contour that can more comprehensively and accurately reflect the position and posture of the object to be inspected can be obtained, so as to more accurately predict whether the object to be inspected will collide with the imaging device.

[0009] The above step b) may include: performing segmentation processing on the thermal image based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and extracting the 2D contour of the object to be inspected from the thermal contour image that most conforms to the contour of the object to be inspected among the plurality of thermal contour images. Correspondingly, the above processing unit may be further configured to: perform segmentation processing on the thermal image based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and extract the 2D contour of the object to be inspected from the thermal contour image that most conforms to the contour of the object to be inspected among the plurality of thermal contour images.

[0010] The above implementation method is to find the thermal contour image that best matches the inspection object from the thermal contour images obtained by segmenting and processing multiple temperature thresholds, and extract the 2D contour of the inspection object from it. The temperature threshold corresponding to the thermal contour image that best matches the inspection object is the temperature closest to the temperature of the inspection object sensed by the thermal camera module.

[0011] Alternatively, the image packet of the inspection object can also be obtained in real time through the multi-modal camera system. The above step b) may include: segmenting the current thermal image based on a preselected temperature threshold to obtain a thermal contour image; and extracting the 2D contour of the inspection object from the thermal contour image. Accordingly, the above processing unit can be further configured to: segment the current thermal image based on a preselected temperature threshold to obtain a thermal contour image; and extract the 2D contour of the inspection object from the thermal contour image.

[0012] If the temperature of the inspection object sensed by the thermal camera module can be determined, the temperature threshold segmentation processing can also be directly performed on the thermal image obtained in real time with this temperature to directly obtain the thermal contour image corresponding to the inspection object and extract the 2D contour of the inspection object from it.

[0013] The above temperature threshold corresponding to the temperature of the inspection object sensed by the thermal camera module can be obtained in various ways. For example, it can be obtained through the following steps: segmenting the thermal image obtained at a previous moment based on multiple predetermined temperature thresholds to obtain multiple thermal contour images; and selecting the thermal contour image that best matches the contour of the inspection object from the multiple thermal contour images, and determining the corresponding temperature threshold as the preselected temperature threshold. Accordingly, the above processing unit can be further configured to: segment the thermal image obtained at a previous moment based on multiple predetermined temperature thresholds to obtain multiple thermal contour images; and select the thermal contour image that best matches the contour of the inspection object from the multiple thermal contour images, and determine the corresponding temperature threshold as the preselected temperature threshold.

[0014] The above-mentioned thermal profile image that best matches the profile of the object to be inspected can be obtained in various ways. For example, the thermal profile image that best matches the profile of the object to be inspected can be selected by comparing it with a pre-obtained prior template image, where the prior template image can be obtained through the following steps: collecting multiple thermal images of different objects to be inspected under different conditions in advance; performing segmentation processing on each of the multiple thermal images based on multiple predetermined temperature thresholds to obtain multiple prior thermal profile images, and selecting the best thermal profile image that best matches the profile of the object to be inspected from the multiple prior thermal profile images; and extracting features from the best thermal profile images corresponding to all the multiple thermal images, and establishing the prior template image based on the extracted features. Correspondingly, the above-mentioned processing unit can be further configured to: collect multiple thermal images of different objects to be inspected under different conditions in advance; perform segmentation processing on each of the multiple thermal images based on multiple predetermined temperature thresholds to obtain multiple prior thermal profile images, and select the best thermal profile image that best matches the profile of the object to be inspected from the multiple prior thermal profile images; and extract features from the best thermal profile images corresponding to all the multiple thermal images, and establish the prior template image based on the extracted features.

[0015] The above-mentioned comparison with the pre-obtained prior template image may include, for example, comparing the features in the multiple thermal profile images with the features in the prior template image to obtain the thermal profile image that best matches the profile of the object to be inspected. Correspondingly, the above-mentioned processing unit can be further configured to: obtain the thermal profile image that best matches the profile of the object to be inspected by comparing the features in the multiple thermal profile images with the features in the prior template image.

[0016] The above-mentioned step c) may include: calculating the 3D coordinate values of each point on the object to be inspected based on the depth information in the depth image and the pixel distance information in the 2D profile, and obtaining the 3D profile based on all the 3D coordinate values. Correspondingly, the above-mentioned processing unit can be further configured to: calculate the 3D coordinate values of each point on the object to be inspected based on the depth information in the depth image and the pixel distance information in the 2D profile, and obtain the 3D profile based on all the 3D coordinate values.

[0017] The above-mentioned depth information includes the vertical depth of each point on the object to be inspected from the focal point of the depth camera module or the thermal camera module. The above-mentioned pixel distance information includes the pixel distance of each pixel in the 2D profile from the focal point of the depth camera module or the thermal camera module. Each pixel in the 2D profile corresponds to a point on the object to be inspected.

[0018] In the above step c), the thermal image or the 2D contour can be converted into the depth camera coordinate system or the depth image can be converted into the thermal camera coordinate system. Accordingly, the above processing unit can be further configured to: convert the thermal image or the 2D contour into the depth camera coordinate system or convert the depth image into the thermal camera coordinate system.

[0019] The thermal image or the 2D contour can be converted into the depth camera coordinate system or the depth image can be converted into the thermal camera coordinate system through a thermal image conversion matrix.

[0020] The thermal image conversion matrix can be obtained through the following steps: Place a calibration tool so that the calibration tool is located in the field of view of the depth camera and the field of view of the thermal camera at the same time; Image the calibration tool through the depth camera and the thermal camera respectively, and calculate the depth image inner angle coordinate values of the inner angles on the calibration tool in the depth camera coordinate system and the thermal image inner angle coordinate values in the thermal camera coordinate system, where the calibration tool is heated to form a thermal difference from its original temperature; and Calculate the thermal image conversion matrix based on the depth image inner angle coordinate values and the thermal image inner angle coordinate values.

[0021] Multiple rows of neatly arranged rectangular holes can be provided on the calibration tool so that the inner angle coordinates of the rectangular holes can be read from the thermal image obtained by the thermal camera module after the calibration tool is heated.

[0022] The above step d) may include: calculating the 3D contour coordinate values of the 3D contour of the inspection object in the frame coordinate system of the imaging device, where the 3D contour coordinate values include the 3D contour coordinate values of the inspection object at various positions during the scanning process; and When the 3D contour coordinate values overlap with the coordinate values of the frame holes of the imaging device, it is determined that the inspection object will collide with the frame holes on its moving path. Accordingly, the above processing unit can be further configured to: calculate the 3D contour coordinate values of the 3D contour of the inspection object in the frame coordinate system of the imaging device, where the 3D contour coordinate values include the 3D contour coordinate values of the inspection object at various positions during the scanning process; and When the 3D contour coordinate values overlap with the coordinate values of the frame holes of the imaging device, it is determined that the inspection object will collide with the frame holes on its moving path.

[0023] According to the third aspect of the present invention, there is provided a computer-readable storage medium on which encoded instructions are recorded, and when the instructions are executed, the method for predicting the collision between an inspection object and an imaging device of the present invention as described above can be implemented.

[0024] Through the following detailed description in conjunction with the drawings, other features and aspects of the present invention will become clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention can be better understood by describing exemplary embodiments of the present invention in conjunction with the accompanying drawings. In the drawings:

[0026] Figure 1 A flowchart showing a method for predicting a collision between an object to be examined and an imaging device according to an embodiment of the present invention is shown;

[0027] Figure 2 A schematic diagram showing imaging of an object to be examined according to an embodiment of the present invention;

[0028] Figure 3 A schematic diagram showing processing of each image according to an embodiment of the present invention;

[0029] Figure 4 is Figure 1 A flowchart of an embodiment of the shown prediction method;

[0030] Figure 5 A schematic diagram showing temperature threshold segmentation processing of a thermal image according to an embodiment of the present invention is shown;

[0031] Figure 6 is Figure 1 A flowchart of another embodiment of the shown prediction method;

[0032] Figure 7 An exemplary schematic diagram showing obtaining a prior template image according to an embodiment of the present invention;

[0033] FIG. 8(a) shows a schematic diagram of collecting a 2D image of an object;

[0034] FIG. 8(b) schematically shows the corresponding geometric relationship of relevant parameters in FIG. 8(a);

[0035] Figure 9 Exemplarily shows how to obtain a thermal image conversion matrix;

[0036] Figure 10 Exemplarily shows how to obtain a depth image - RGB image conversion matrix;

[0037] Figure 11 is Figure 1 A flowchart of yet another embodiment of the shown prediction method;

[0038] Figure 12 Exemplarily shows how to obtain a gantry - multi - modal camera conversion matrix;

[0039] Figure 13 Schematically shows the path along which the object to be examined will move with the scanning bed during the scanning process;

[0040] Figure 14 Schematically shows different situations where collisions will and will not occur;

[0041] Figure 15 Shows a schematic diagram of an imaging device according to an embodiment of the present invention; and

[0042] Figure 16 Shows a schematic diagram where an object to be inspected may collide with the imaging device. Detailed implementation manners

[0043] The present invention will be further described below in conjunction with specific embodiments and the accompanying drawings. More details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention is clearly capable of being implemented in many other ways different from this description. Those skilled in the art can make similar generalizations and deductions according to the actual application situation without departing from the spirit of the present invention. Therefore, the protection scope of the present invention should not be limited by the content of this specific embodiment.

[0044] Unless otherwise defined, the technical terms or scientific terms used in the claims and the specification should have the ordinary meaning understood by those of ordinary skill in the technical field to which the present invention pertains. The "first", "second" and similar terms used in the specification and claims of this application do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "a" or "one" do not denote a quantity limitation, but mean that there is at least one. The terms such as "comprising" or "including" mean that the elements or items appearing before "comprising" or "including" cover the elements or items listed after "comprising" or "including" and their equivalent elements, and do not exclude other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0045] According to an embodiment of the present invention, a method for predicting a collision between an object to be inspected and an imaging device is provided.

[0046] Figure 1 Shows a flowchart of a method 100 for predicting a collision between an object to be inspected and an imaging device according to an embodiment of the present invention. As Figure 1 shown, the method 100 may include step 120 to step 180.

[0047] In step 120, an image packet of the object to be inspected may be obtained through a multi-modal camera system. As Figure 2 shown, the multi-modal camera system may include a depth camera module 220 and a thermal camera module 240 for obtaining a depth image and a thermal image of the object to be inspected 260 in the image packet respectively.

[0048] In step 140, the 2D contour of the object to be inspected can be obtained based on the segmentation processing of the thermal image. The shade of each pixel in the thermal image represents the temperature of the object corresponding to that pixel. By performing segmentation processing on the thermal image, pixels below a certain temperature threshold can be excluded. As Figure 3 shown, after performing segmentation processing on the thermal image, a thermal contour image of the object to be inspected can be obtained. The gray-scale information in this thermal contour image only represents a part of the object to be inspected. In other words, other interfering factors that do not belong to the object to be inspected have been excluded. The 2D contour of the object to be inspected can be clearly extracted from this thermal contour image.

[0049] Next, in step 160, as Figure 3 shown, the 3D contour of the object to be inspected can be generated based on the 2D contour of the object to be inspected and the depth image of the object to be inspected. This 3D contour can more comprehensively and accurately reflect the position and posture of the object to be inspected, thus facilitating more accurate collision prediction in the subsequent process.

[0050] Finally, in step 180, it can be predicted whether the object to be inspected will collide with the imaging device that scans the object to be inspected based on the 3D contour of the object to be inspected. Since the 3D coordinate values of the imaging device are known, for example, if the imaging device is a CT scanner, the 3D coordinate values of the gantry hole can be directly obtained from the CT scanning system. Thus, based on the 3D contour of the object to be inspected and the 3D coordinate values of the gantry hole, it can be determined whether the object to be inspected will collide with the gantry hole.

[0051] Compared with the prior art that detects whether all items on / near the scanning bed will collide with the scanning imaging device, the present invention cleverly introduces the segmentation processing of the thermal image, excluding all interfering factors that do not belong to the object to be inspected outside the obtained 2D contour, and then also excluding them from the scope where it is necessary to predict whether there will be a collision. Thereby, the efficiency and accuracy of collision prediction are greatly improved. The collision prediction method of the present invention also particularly introduces the depth information of the object to be inspected. The 3D contour obtained by combining the 2D contour of the object to be inspected on this basis can more comprehensively and accurately reflect the position and posture of the object to be inspected, thereby further improving the accuracy of collision prediction.

[0052] Optionally, step 140 described above may include sub-steps 1412 and 1414 as Figure 4 shown.

[0053] In sub-step 1412, the thermal image may be segmented based on multiple predetermined temperature thresholds to obtain multiple thermal contour images. Segmenting the thermal image based on a certain temperature threshold means that the object information corresponding to the temperature below this temperature threshold will be excluded, and the obtained thermal contour image will no longer include the gray-scale information of the object corresponding to the temperature below this temperature threshold. The multiple predetermined temperature thresholds may be determined according to the ambient temperature in combination with the actual situation of the inspection object. For example, a temperature range may be determined first, and then multiple temperature thresholds may be selected from this temperature range.

[0054] For example, as Figure 5 shown, the inspection object is a human body, and the human body temperature is generally below 38°C, which is an empirical value of the average human body temperature. After measurement by the thermal camera module at different room temperatures, it can be determined that the temperature range of the human body sensed by the thermal camera module is from 24°C to 30°C. The temperature that other objects that are not part of the human body can be sensed by the thermal camera module must be less than this temperature range. Segmenting the thermal image based on the temperature thresholds of 24°C, 25°C, 26°C, 27°C, 28°C, 29°C, and 30°C selected from this temperature range, multiple corresponding thermal contour images can be obtained, and these thermal contour images only contain the gray-scale information corresponding to the human body part.

[0055] Next, in sub-step 1414, the 2D contour of the inspection object may be extracted from the thermal contour image that best matches the contour of the inspection object among the multiple thermal contour images.

[0056] Still taking Figure 5 as an example, among the multiple obtained thermal contour images, the human body contours in the thermal contour images obtained by threshold segmentation based on 26°C, 27°C, 28°C, 29°C, and 30°C are all missing to varying degrees, while the contour in the thermal contour image obtained by threshold segmentation based on 24°C, although not missing, only magnifies a part of the human body. Relatively speaking, only the contour in the thermal contour image obtained by threshold segmentation based on 25°C is both complete and can basically reflect the main contour of the human body. Therefore, it can be used as the thermal contour image that best matches the human body contour, and the 2D contour of the human body can be extracted from this thermal contour image.

[0057] Optionally, in the above step 120, the multi-modal camera system may obtain the image packet of the inspection object in real time.

[0058] If the temperature of the inspection object sensed by the thermal camera module is unknown, Figure 4The implementation method shown finds the thermal contour image that best matches the inspection object, and the temperature threshold corresponding to this thermal contour image is the one closest to the temperature of the inspection object sensed by the thermal camera module. However, if the temperature of the inspection object sensed by the thermal camera module can be determined, especially when the multi-modal camera system obtains the image packet of the inspection object in real time for collision prediction, the temperature of the inspection object sensed by this thermal camera module can also be directly used to perform temperature threshold segmentation processing on the thermal image obtained in real time.

[0059] Optionally, step 140 above may include sub-steps 1422 and 1424 as Figure 6 shown.

[0060] In sub-step 1422, the current thermal image may be segmented based on a preselected temperature threshold to obtain a thermal contour image.

[0061] In sub-step 1424, the 2D contour of the inspection object may be extracted from the thermal contour image.

[0062] Still taking the inspection of the human body as an example, when it is determined that the human body temperature sensed by the thermal camera module is 25 °C, the current thermal image can be directly segmented based on the temperature threshold of 25 °C. The obtained thermal contour image will surely exclude the gray-scale information of other interfering factors that do not belong to the human body part and only contain the gray-scale information corresponding to the human body part. The 2D contour of the human body can be extracted from this thermal contour image.

[0063] The above-mentioned preselected temperature threshold can be obtained in various ways. For example, as in the previous sub-step 1412, the thermal image obtained at a previous moment may be segmented based on multiple predetermined temperature thresholds to obtain multiple thermal contour images, and then as in the previous sub-step 1414, the thermal contour image that best matches the contour of the inspection object is selected from these multiple thermal contour images. Finally, the temperature threshold corresponding to the thermal contour image that best matches the contour of the inspection object is determined as the preselected temperature threshold.

[0064] Still taking Figure 5 the inspection of the human body as an example, after segmenting the thermal image obtained at a certain moment based on the temperature thresholds of 24 °C, 25 °C, 26 °C, 27 °C, 28 °C, 29 °C, and 30 °C, the thermal contour image obtained by segmenting the thermal image based on 25 °C is selected as the one that best matches the human body contour. 25 °C can be determined as the preselected temperature threshold, that is, the human body temperature sensed by the thermal camera module. Subsequently, all the human body thermal images obtained in real time can be directly segmented based on 25 °C, and the 2D contour of the human body can be extracted from the thermal contour image after this threshold segmentation processing.

[0065] Optionally, the thermal contour image that best matches the contour of the object to be inspected can also be selected by comparing it with a pre-obtained prior template image.

[0066] Still taking the inspection of the human body as an example, in order to obtain the prior template image of the human body, multiple thermal images can be collected from different human bodies under different conditions in advance, and then each of these thermal images is subjected to segmentation processing based on multiple predetermined temperature thresholds. Figure 7 Shows multiple prior thermal contour images obtained after segmenting a thermal image based on multiple predetermined temperature thresholds. Similarly, the multiple predetermined temperature thresholds can be selected according to the ambient temperature combined with the normal human body temperature. Select the best thermal contour image that best matches the human body contour from these multiple prior thermal contour images. For example, among the Figure 7 7 prior thermal contour images shown in which threshold segmentation is performed based on 24°C, 25°C, 26°C, 27°C, 28°C, 29°C, and 30°C, the thermal contour image obtained after threshold segmentation based on 25°C can relatively most completely reflect the main contour of the human body, and this thermal contour image is selected as the best thermal contour image that best matches the human body contour. Process all thermal images in the same way to obtain multiple best thermal contour images. Extract features from all these best thermal contour images, and based on these extracted features, a prior template image of the human body can be established. The extracted features can include, for example, area, aspect ratio, projection in the horizontal or vertical direction, etc. After that, after segmenting the human body thermal image based on multiple temperature thresholds, the human body thermal contour images obtained after each segmentation can be directly compared with the above prior template image, and the one with the closest comparison result is the thermal contour image that best matches the human body contour.

[0067] Optionally, when comparing with the pre-obtained prior template image, the features in the multiple thermal contour images can be compared with the features in the prior template image to obtain the thermal contour image that best matches the contour of the object to be inspected.

[0068] Still taking Figure 5Taking the human body inspection as an example, after segmenting the obtained thermal images based on temperature thresholds of 24°C, 25°C, 26°C, 27°C, 28°C, 29°C, and 30°C, the contour areas in the thermal contour images obtained after these segmentations can be respectively compared with the contour areas in the human body prior template image to find the closest one, and its thermal contour image is the thermal contour image that most conforms to the human body contour. Or the aspect ratios in the thermal contour images obtained after these segmentations can be respectively compared with the aspect ratios in the human body prior template image to find the closest one, and its thermal contour image is the thermal contour image that most conforms to the human body contour. It is also possible to respectively compare the projections in the horizontal / vertical directions in the thermal contour images obtained after these segmentations with the projections in the horizontal / vertical directions in the human body prior template image to find the closest one, and its thermal contour image is the thermal contour image that most conforms to the human body contour.

[0069] Optionally, step 160 above may include: calculating the 3D coordinate values of each point on the inspection object based on the depth information in the depth image and the pixel distance information in the 2D contour, and obtaining the 3D contour based on all the 3D coordinate values.

[0070] Figure 8(a) shows a schematic diagram of using a camera to collect a 2D image of an object. To obtain the 3D contour of the object, it is necessary to know the 3D coordinate values of each point on the object, that is, the 3D coordinate values (x, y, z) of each point relative to the center of the camera (i.e., the focus of the camera) in Figure 8(a). Figure 8(b) schematically shows the corresponding geometric relationship of the relevant parameters in Figure 8(a). In Figure 8(b), the straight line a corresponds to the focus of the camera, and the length of the line segment AB corresponds to the focal length f of the camera. These are all inherent to the camera and are known. For point P in Figure 8(b), its depth information in the depth image corresponds to the vertical depth h from point P to the camera focus, that is, z in the above 3D coordinate values. The x and y in the 3D coordinate values are the vertical distances d from point P to the camera focus in the other two directions (i.e., the X direction in the figure and the Y direction perpendicular to the X-Z plane). x and d y . The line segment EF in Figure 8(b) corresponds to the 2D image (2D contour) of the object, and P' on EF can reflect the 2D pixel distance information of point P, including the pixel distance (BP') from point P' to the camera focus in the X direction and the pixel distance (not shown in Figure 8(b)) from point P' to the camera focus in the Y direction. From the geometric relationship in the figure, f / h = BP' / d x , from which d can be obtained x , that is, x in the 3D coordinate values. It can be understood that for the sake of clear illustration, Figure 8(b) is only a schematic diagram of the X-Z plane. Similarly, in the schematic diagram of the Y-Z plane, d can be obtained in the same way. y, that is, y in the 3D coordinate values.

[0071] The cameras involved in the above calculations can be various cameras. If it is a depth camera, then the depth information in the depth image and the 2D pixel distance information in the 2D image need to be converted into the depth camera coordinate system; if it is a thermal camera, then the depth information in the depth image and the 2D pixel distance information in the 2D image need to be converted into the thermal camera coordinate system. In short, the depth information in the depth image and the 2D pixel distance information in the 2D image need to be converted into a camera coordinate system.

[0072] Returning to the above step 160, since the depth image and the thermal image are obtained by the depth camera module and the thermal camera module respectively, the thermal image or the 2D contour obtained based on the thermal image can be converted into the depth camera coordinate system, and based on the depth information (the vertical depth of each point on the inspection object to the focus of the depth camera module) and the pixel distance information in the 2D contour (the pixel distance of each pixel in the 2D contour to the focus of the depth camera module), combined with the focal length of the depth camera, the 3D coordinate values of each point on the inspection object can be calculated. Alternatively, the depth image can be converted into the thermal camera coordinate system, and based on the depth information (the vertical depth of each point on the inspection object to the focus of the thermal camera module) and the pixel distance information in the 2D contour (the pixel distance of each pixel in the 2D contour to the focus of the thermal camera module), combined with the focal length of the thermal camera, the 3D coordinate values of each point on the inspection object can also be calculated. Each pixel in the above 2D contour corresponds to each point on the inspection object.

[0073] Optionally, the thermal image or the 2D contour can be converted into the depth camera coordinate system or the depth image can be converted into the thermal camera coordinate system through a thermal image conversion matrix. The following will be combined with Figure 9 Examples will be given to illustrate how to obtain the thermal image conversion matrix.

[0074] As Figure 9 shown, first place the calibration tool, which can be placed on the scanning bed, for example, as long as the calibration tool is simultaneously located in the field of view of the depth camera and the field of view of the thermal camera. Regarding this calibration tool, if a checkerboard-type calibration tool (a flat plate with black and white square checkerboards) as shown in Figure 10 is used, the inner corner coordinates of each in the thermal image of the black and white checkerboard cannot be located. Therefore, for obtaining the thermal image conversion matrix between the thermal camera coordinate system and other camera coordinate systems, the present invention specifically designs a new open-hole type calibration tool, as shown in Figure 9As shown in [figure]. The open-hole calibration tool is a flat plate, on which multiple rows of neatly arranged rectangular holes are provided. After heating the calibration tool, a thermal difference will be formed between the rectangular hollowed-out part and the remaining non-hollowed-out part. Therefore, the inner corner coordinates of the rectangular holes can be clearly read from the obtained thermal image.

[0075] After placing the calibration tool as Figure 9 shown, the calibration tool can be imaged by the depth camera and the thermal camera respectively. The size of the collected images can be adjusted, for example, by interpolation and padding.

[0076] Next, the depth image inner corner coordinate values of the inner corners on the calibration tool in the depth camera coordinate system and the thermal image inner corner coordinate values in the thermal camera coordinate system can be calculated. To collect the thermal image, the calibration tool needs to be heated to increase its temperature and form a thermal difference with the original temperature. The depth image inner corner coordinate values are the inner corner coordinate values of each rectangular hole on the calibration tool in the collected depth image, and the thermal image inner corner coordinate values are the inner corner coordinate values of each rectangular hole on the calibration tool in the collected thermal image. All the inner corner coordinate values in the depth image and the thermal image can be found, for example, by separately finding the inner corner fractions of the rectangular holes in the depth image and the thermal image.

[0077] Finally, the thermal image transformation matrix is calculated based on the depth image inner corner coordinate values and the thermal image inner corner coordinate values. For example, the transformation matrix H for coordinate value conversion between the depth camera coordinate system and the thermal camera coordinate system can be calculated by using the homography algorithm.

[0078] Optionally, the multi-modal camera system used in the present invention can also introduce an RGB camera module for additionally collecting RGB images of the inspection object, and the thermal image or the 2D contour obtained based on the thermal image and the depth image can all be transformed into the RGB camera coordinate system. In this way, the user can perform more intuitive monitoring in the above collision prediction method and can directly perform operations such as stopping scanning according to needs.

[0079] The thermal image or the 2D contour obtained based on the thermal image can be transformed into the RGB camera coordinate system through the thermal image - RGB image transformation matrix. The acquisition of this thermal image - RGB image transformation matrix is similar to the acquisition of the above thermal image transformation matrix. The difference is that the depth camera in the process of obtaining the thermal image transformation matrix is replaced by an RGB camera, and what needs to be calculated are the RGB image inner corner coordinate values of the inner corners on the calibration tool in the RGB camera coordinate system, and finally the thermal image - RGB image transformation matrix is calculated based on the RGB image inner corner coordinate values and the thermal image inner corner coordinate values.

[0080] The depth image can be transformed into the RGB camera coordinate system through a depth image - RGB image transformation matrix. The acquisition of this depth image - RGB image transformation matrix is similar to that of the above - mentioned thermal image transformation matrix. The difference lies in that the thermal camera in the process of obtaining the thermal image transformation matrix is replaced by an RGB camera. What needs to be calculated are the RGB image inner - angle coordinate values of the inner angles on the calibration tool in the RGB camera coordinate system, and finally, the depth image - RGB image transformation matrix is calculated based on the RGB image inner - angle coordinate values and the depth image inner - angle coordinate values. Since no thermal image needs to be acquired during the process of obtaining this depth image - RGB image transformation matrix, there is no need to heat the calibration tool. The calibration tool used during this period can still be the flat plate with multiple neatly arranged rectangular holes as described above, or it can be a checkerboard - type calibration tool. As Figure 10 shown in

[0081] Optionally, step 180 above can be Figure 11 composed of sub - steps 1820 and 1840 as shown.

[0082] In sub - step 1820, the 3D contour coordinate values of the 3D contour of the inspection object in the frame coordinate system of the imaging device can be calculated. These 3D contour coordinate values include the 3D contour coordinate values of the inspection object at various positions during the scanning process. That is to say, these 3D contour coordinate values not only include the current position of the inspection object in the frame coordinate system, but also include the positions of the inspection object in the frame coordinate system during subsequent scanning processes.

[0083] Since the 3D contour of the inspection object obtained in step 160 is based on the thermal image and the depth image, that is to say, this 3D contour is in the multi - modal camera system coordinate system, it is necessary to first transform this 3D contour into the frame coordinate system of the imaging device. This transformation can be achieved through a frame - multi - modal camera transformation matrix. The following will combine Figure 12 to illustrate how to obtain the frame - multi - modal camera transformation matrix.

[0084] First, a calibration tool can be placed in the frame coordinate system. For example, a calibration tool can be placed on the scanning bed as Figure 12 shown, so that the calibration tool is within the field of view of the multi - modal camera system.

[0085] Since the transformation matrix between the camera coordinate systems in the multi-modal camera system can be obtained as described above, in practice, it is only necessary to place the calibration tool within the field of view of one of the cameras in the multi-modal camera system. This camera can be regarded as the calibration camera. After obtaining the transformation matrix between the coordinate system of this calibration camera and the gantry coordinate system, the coordinate transformation between the gantry coordinate system and the coordinate systems of other cameras can be achieved through the transformation matrices between the coordinate systems of other cameras and this calibration camera coordinate system.

[0086] Figure 12 The multi-modal camera system in [description] includes a depth camera and a thermal camera, and any one of them can be selected as the calibration camera. Of course, the multi-modal camera system can further include an RGB camera, and this RGB camera can also be selected as the calibration camera. When the depth camera or the RGB camera is selected as the calibration camera, the calibration tool can be either the checkerboard calibration tool or the perforated calibration tool; while when the thermal camera is selected as the calibration camera, the perforated calibration tool needs to be used.

[0087] After placing the calibration tool, the calibration camera can image the calibration tool, and calculate the calibration camera inner angle coordinates of the inner angles on the calibration tool in the coordinate system of the calibration camera.

[0088] Next, the gantry inner angle coordinates of the inner angles on the calibration tool in the gantry coordinate system can be measured by means of the laser beam emitted by the laser lamp in the gantry coordinate system and the moving distance of the scanning bed.

[0089] Finally, a set of rotation matrix R, translation matrix T, and scaling matrix S between the multi-modal camera coordinate system and the gantry coordinate system can be calculated based on the calibration camera inner angle coordinates (i.e., multi-modal camera inner angle coordinates) and the gantry inner angle coordinates.

[0090] Returning to sub-step 1820, after transforming the 3D contour of the inspection object obtained in step 160 into the gantry coordinate system through the above set of rotation matrix R, translation matrix T, and scaling matrix S, the 3D contour coordinate values of the inspection object at each position during the scanning process can be further obtained based on the movement offset coordinate value of the scanning bed and the 3D contour of the inspection object in the gantry coordinate system. The inspection object is located on the scanning bed, so the movement offset trajectory of the scanning bed corresponds to the movement offset trajectory of the inspection object. Figure 13The path along which the object to be inspected moves with the scanning table during the scanning process is shown therein. The movement offset trajectory of the scanning table is known. For example, if the imaging device is a CT scanner, the movement offset trajectory of its scanning table and the corresponding coordinate values can be directly obtained from the CT scanning system. Thus, based on the 3D contour of the object to be inspected and the movement offset coordinate values of the scanning table in the same gantry coordinate system, the 3D contour coordinate values of the object to be inspected at various positions during the scanning process can be obtained.

[0091] Next, in sub-step 1840, when the 3D contour coordinate values overlap with the coordinate values of the gantry hole of the imaging device, it can be determined that the object to be inspected will collide with the gantry hole on its movement path. Similarly, the coordinate values of the gantry hole of the imaging device are also known. For example, if the imaging device is a CT scanner, the coordinate values of its gantry hole can be directly obtained from the CT scanning system. Figure 14 The cases where the 3D contour coordinate values overlap and do not overlap with the coordinate values of the gantry hole are respectively shown therein.

[0092] So far, a method for predicting the collision between an object to be inspected and an imaging device according to the present invention has been described. It innovatively obtains the 2D contour of the object to be inspected based on the thermal image of the object to be inspected, and further predicts the collision based on the 3D contour of the object to be inspected by introducing the depth information of the object to be inspected, thereby providing a more accurate and efficient collision prediction, greatly improving the safety of the imaging device when imaging the object to be inspected while ensuring the efficiency.

[0093] According to an embodiment of the present invention, there is also provided a computer-readable storage medium on which encoded instructions are recorded, and when the instructions are executed, the method for predicting the collision between an object to be inspected and an imaging device according to the present invention can be implemented. The computer-readable storage medium may include a hard disk drive, a floppy disk drive, a CD read / write (CD-R / W) drive, a digital versatile disk (DVD) drive, a flash drive, and / or a solid-state storage device, etc.

[0094] According to an embodiment of the present invention, there is also provided an imaging device accordingly.

[0095] Reference Figure 15 shows an imaging device 1500 according to the present invention, which includes a gantry 1520, a multi-modal camera system 1540, and a processing unit 1560.

[0096] The gantry 1520 may include a gantry hole for accommodating the object to be inspected.

[0097] The multi-modal camera system 1540 may include a depth camera module 1542 and a thermal camera module 1544. The multi-modal camera system 1540 may be used to obtain an image package of the inspection object, which includes a depth image and a thermal image of the inspection object.

[0098] The processing unit 1560 may be configured to: obtain a 2D contour of the inspection object based on segmentation processing of the thermal image; generate a 3D contour of the inspection object based on the 2D contour of the inspection object and the depth image of the inspection object; and estimate whether the inspection object will collide with the imaging device that scans the inspection object on its moving path based on the 3D contour of the inspection object.

[0099] Optionally, the processing unit 1560 may be further configured to: perform segmentation processing on the thermal image based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and extract the 2D contour of the inspection object from the thermal contour image that best matches the contour of the inspection object among the plurality of thermal contour images.

[0100] Optionally, the multi-modal camera system 1540 may obtain the image package of the inspection object in real time, where the processing unit 1560 may be further configured to: perform segmentation processing on the current thermal image based on a preselected temperature threshold to obtain a thermal contour image; and extract the 2D contour of the inspection object from the thermal contour image.

[0101] Optionally, the processing unit 1560 may be further configured to: perform segmentation processing on the thermal image obtained at a previous moment based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and select the thermal contour image that best matches the contour of the inspection object from the plurality of thermal contour images, and determine the corresponding temperature threshold as the preselected temperature threshold.

[0102] Optionally, the thermal contour image that best matches the contour of the inspection object may be selected by comparing with a pre-obtained prior template image, where the processing unit 1560 may be further configured to: collect a plurality of thermal images of different inspection objects under different conditions in advance; perform segmentation processing on each of the plurality of thermal images based on a plurality of predetermined temperature thresholds to obtain a plurality of prior thermal contour images, and select the best thermal contour image that best matches the contour of the inspected object from the plurality of prior thermal contour images; and extract features from the best thermal contour images corresponding to all the plurality of thermal images, and establish the prior template image based on the extracted features.

[0103] Optionally, the processing unit 1560 may be further configured to: obtain the thermal contour image that best matches the contour of the inspection object by comparing the features in the plurality of thermal contour images with the features in the prior template image.

[0104] Optionally, the processing unit 1560 may be further configured to: calculate 3D coordinate values of each point on the inspection object based on the depth information in the depth image and the pixel distance information in the 2D contour, and obtain the 3D contour based on all the 3D coordinate values.

[0105] Optionally, the depth information may include the vertical depth of each point on the inspection object to the focal point of the depth camera module 1542 or the thermal camera module 1544, and the pixel distance information includes the pixel distance of each pixel in the 2D contour to the focal point of the depth camera module 1542 or the thermal camera module 1544, where each pixel in the 2D contour corresponds to each point on the inspection object.

[0106] Optionally, the processing unit 1560 may be further configured to: convert the thermal image or the 2D contour into the depth camera coordinate system or convert the depth image into the thermal camera coordinate system.

[0107] Optionally, the processing unit 1560 may be further configured to: calculate the 3D contour coordinate values of the 3D contour of the inspection object in the frame coordinate system of the imaging device 1500, where the 3D contour coordinate values include the 3D contour coordinate values of the inspection object at various positions during the scanning process; and when the 3D contour coordinate values overlap with the coordinate values of the frame holes of the imaging device 1500, determine that the inspection object will collide with the frame holes on its moving path.

[0108] The above imaging device can implement the method for predicting the collision between the inspection object and the imaging device according to the present invention as described above. Many design concepts and details applicable in the prediction method of the present invention also apply to the above imaging device, and the same beneficial technical effects can be obtained, which will not be elaborated here.

[0109] The various aspects of the present invention have been described above through some exemplary embodiments. However, it should be understood that various modifications can be made to the above exemplary embodiments without departing from the spirit and scope of the present invention. For example, if the described technology is executed in a different order and / or if the components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents, appropriate results can also be achieved. Accordingly, these modified other embodiments also fall within the protection scope of the claims.

Claims

1. A method for predicting a collision between an object to be inspected and an imaging device, comprising: a) obtaining an image packet of the object to be inspected through a multi-modal camera system, the image packet including a depth image and a thermal image of the object to be inspected, the multi-modal camera system including a depth camera module and a thermal camera module; b) obtaining a 2D contour of the object to be inspected based on segmentation processing of the thermal image; c) generating a 3D contour of the object to be inspected based on the 2D contour of the object to be inspected and the depth image of the object to be inspected; and d) predicting whether the object to be inspected will collide with the imaging device scanning the object to be inspected on its moving path based on the 3D contour of the object to be inspected.

2. The method according to claim 1, wherein, the step b) includes: performing segmentation processing on the thermal image based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and extracting the 2D contour of the object to be inspected from the thermal contour image that best conforms to the contour of the object to be inspected among the plurality of thermal contour images.

3. The method according to claim 1, wherein, the image packet of the object to be inspected is obtained in real time through the multi-modal camera system, and the step b) includes: performing segmentation processing on the current thermal image based on a preselected temperature threshold to obtain a thermal contour image; and extracting the 2D contour of the object to be inspected from the thermal contour image.

4. The method according to claim 3, wherein, the preselected temperature threshold is obtained through the following steps: performing segmentation processing on the thermal image obtained at a previous moment based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and selecting the thermal contour image that best conforms to the contour of the object to be inspected from the plurality of thermal contour images, and determining the corresponding temperature threshold as the preselected temperature threshold.

5. The method according to claim 2 or 4, wherein, the thermal contour image that best conforms to the contour of the object to be inspected is selected by comparing with a pre-obtained prior template image, and the prior template image is obtained through the following steps: collecting a plurality of thermal images of different objects to be inspected under different conditions in advance; performing segmentation processing on each of the plurality of thermal images based on a plurality of predetermined temperature thresholds to obtain a plurality of prior thermal contour images, and selecting the best thermal contour image that best conforms to the contour of the object to be inspected from the plurality of prior thermal contour images; and extracting features from the best thermal contour images corresponding to all the plurality of thermal images, and establishing the prior template image based on the extracted features.

6. The method according to claim 5, wherein, the thermal contour image that best conforms to the contour of the object to be inspected is obtained by comparing the features in the plurality of thermal contour images with the features in the prior template image.

7. The method according to claim 1, wherein, the step c) includes: calculating 3D coordinate values of each point on the object to be inspected based on the depth information in the depth image and the pixel distance information in the 2D contour, and obtaining the 3D contour based on all the 3D coordinate values.

8. The method according to claim 7, wherein, the depth information includes the vertical depth of each point on the inspection object to the focus of the depth camera module or the thermal camera module, and the pixel distance information includes the pixel distance of each pixel in the 2D contour to the focus of the depth camera module or the thermal camera module, wherein each pixel in the 2D contour corresponds to each point on the inspection object.

9. The method according to claim 7 or 8, wherein, in step c), the thermal image or the 2D contour is converted into the depth camera coordinate system or the depth image is converted into the thermal camera coordinate system.

10. The method according to claim 9, wherein, in step c), the thermal image or the 2D contour is converted into the depth camera coordinate system or the depth image is converted into the thermal camera coordinate system through a thermal image conversion matrix, wherein the thermal image conversion matrix is obtained through the following steps: Place a calibration tool so that the calibration tool is simultaneously located in the field of view of the depth camera and the field of view of the thermal camera; Image the calibration tool through the depth camera and the thermal camera respectively, and calculate the depth image inner angle coordinate values of the inner angles on the calibration tool in the depth camera coordinate system and the thermal image inner angle coordinate values in the thermal camera coordinate system, wherein the calibration tool is heated to form a thermal difference from its original temperature; and Calculate the thermal image conversion matrix based on the depth image inner angle coordinate values and the thermal image inner angle coordinate values.

11. The method according to claim 1, wherein, step d) includes: Calculating the 3D contour coordinate values of the 3D contour of the inspection object in the frame coordinate system of the imaging device, and the 3D contour coordinate values include the 3D contour coordinate values of the inspection object at various positions during the scanning process; and When the 3D contour coordinate values overlap with the coordinate values of the frame holes of the imaging device, it is determined that the inspection object will collide with the frame holes on its moving path.

12. An imaging device, comprising: A frame including frame holes for accommodating an inspection object; A multimodal camera system including a depth camera module and a thermal camera module, the multimodal camera system being configured to obtain an image packet of the inspection object, the image packet including a depth image and a thermal image of the inspection object, and the multimodal camera system including a depth camera module and a thermal camera module; and A processing unit configured to: Obtain the 2D contour of the inspection object based on the segmentation processing of the thermal image; Generate the 3D contour of the inspection object based on the 2D contour of the inspection object and the depth image of the inspection object; and Estimate whether the inspection object will collide with the imaging device scanning the inspection object on its moving path based on the 3D contour of the inspection object.

13. The imaging device according to claim 12, wherein, the processing unit is further configured to: Perform segmentation processing on the thermal image based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; Extract the 2D contour of the object under inspection from the thermal contour image that best matches the contour of the object under inspection among the multiple thermal contour images.

14. The imaging device according to claim 12, wherein, the multimodal camera system obtains an image packet of the object under inspection in real time, and the processing unit is further configured to: perform segmentation processing on the current thermal image based on a preselected temperature threshold to obtain a thermal contour image; and extract the 2D contour of the object under inspection from the thermal contour image.

15. The imaging device according to claim 14, wherein, the processing unit is further configured to: perform segmentation processing on the thermal image obtained at a previous moment based on a plurality of predetermined temperature thresholds to obtain a plurality of thermal contour images; and select the thermal contour image that best matches the contour of the object under inspection from the plurality of thermal contour images, and determine the corresponding temperature threshold as the preselected temperature threshold.

16. The imaging device according to claim 13 or 15, wherein, select the thermal contour image that best matches the contour of the object under inspection by comparing it with a pre-obtained prior template image, and the processing unit is further configured to: collect a plurality of thermal images of different objects under inspection under different conditions in advance; perform segmentation processing on each of the plurality of thermal images based on a plurality of predetermined temperature thresholds to obtain a plurality of prior thermal contour images, and select the best thermal contour image that best matches the contour of the object under inspection from the plurality of prior thermal contour images; and extract features from the best thermal contour images corresponding to all the plurality of thermal images, and establish the prior template image based on the extracted features.

17. The imaging device according to claim 16, wherein, the processing unit is further configured to: obtain the thermal contour image that best matches the contour of the object under inspection by comparing the features in the plurality of thermal contour images with the features in the prior template image.

18. The imaging device according to claim 12, wherein, the processing unit is further configured to: calculate the 3D coordinate values of each point on the object under inspection based on the depth information in the depth image and the pixel distance information in the 2D contour, and obtain the 3D contour based on all the 3D coordinate values.

19. The imaging device according to claim 12, wherein, the processing unit is further configured to: calculate the 3D contour coordinate values of the 3D contour of the object under inspection in the gantry coordinate system of the imaging device, and the 3D contour coordinate values include the 3D contour coordinate values of the object under inspection moving to various positions during the scanning process; and when the 3D contour coordinate values overlap with the coordinate values of the gantry holes of the imaging device, determine that the object under inspection will collide with the gantry holes on its moving path.

20. A computer-readable storage medium, on which encoded instructions are recorded, and when the instructions are executed, the method according to any one of claims 1-11 is implemented.

Images