A three-dimensional image fusion method and device based on computer tomography

Abstract

The present invention discloses a three-dimensional image fusion method and device based on computer tomography, the method comprising: obtaining at least one two-dimensional CT slice image, and reconstructing a three-dimensional geometric structure according to at least one CT slice image; rotating the three-dimensional geometric structure at different angles, and projecting the three-dimensional geometric structure under the current viewing angle to form a projection image; identifying a certain target object in the projection images at different angles, and marking the certain target object based on a marking frame; judging whether the difference index between the marking frame in the projection image at the orthographic projection angle and the marking frame in the projection image at a certain angle is less than a preset threshold; if it is less than the preset threshold, then fusing the marking frame in the projection image at a certain angle with the marking frame in the projection image at the orthographic projection angle. Intelligent recognition and automatic marking of three-dimensional contraband can be realized, which greatly reduces the workload of security personnel and greatly improves security inspection efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer tomography, and in particular to a three-dimensional image fusion method and device based on computer tomography. Background Art

[0002] X-ray imaging technology is currently widely used in customs, airports, subways and other places to scan large freight containers and small pedestrian bags to check whether there are any contraband items inside.

[0003] Since X-ray images are two-dimensional images generated by superimposing various layers of projection, they cannot well display objects concealed in luggage, especially when there are obstructions, which makes it very difficult for security personnel to review the images. Summary of the invention

[0004] The present invention provides a three-dimensional image fusion method and device based on computer tomography, which are used to solve at least one of the above technical problems.

[0005] In a first aspect, the present invention provides a three-dimensional image fusion method based on computer tomography, comprising: acquiring at least one two-dimensional CT slice image, and reconstructing a three-dimensional geometric structure according to the at least one CT slice image; rotating the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and projecting the three-dimensional geometric structure at the current viewing angle to form a projection image; identifying a certain target object in the projection images at different angles according to an image recognition method, and annotating the certain target object based on a labeling box; judging whether a difference index between a labeling box in the projection image at an orthographic projection angle and a labeling box in the projection image at a certain angle is less than a preset threshold based on a matching algorithm; if a difference index between a source labeling box in the projection image at an orthographic projection angle and a target labeling box in the projection image at a certain angle is less than a preset threshold, then fusing the labeling box in the projection image at a certain angle with the labeling box in the projection image at an orthographic projection angle to obtain a three-dimensional labeling box of the certain target object.

[0006] In a second aspect, the present invention provides a three-dimensional image fusion device based on computer tomography, comprising: a reconstruction module, configured to obtain at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image; a projection module, configured to rotate the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and project the three-dimensional geometric structure at the current viewing angle to form a projection image; a recognition module, configured to recognize a certain target object in the projection images at different angles according to an image recognition method, and mark the certain target object based on a marking box; a judgment module, configured to judge whether a difference index between a marking box in the projection image at an orthographic projection angle and a marking box in the projection image at a certain angle is less than a preset threshold based on a matching algorithm; and a fusion module, configured to fuse the marking box in the projection image at a certain angle with the marking box in the projection image at an orthographic projection angle if the difference index between the source marking box in the projection image at an orthographic projection angle and the target marking box in the projection image at a certain angle is less than a preset threshold, so as to obtain a three-dimensional marking box of the certain target object.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the three-dimensional image fusion method based on computed tomography of any embodiment of the present invention.

[0008] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the three-dimensional image fusion method based on computed tomography of any embodiment of the present invention.

[0009] The computer tomography-based three-dimensional image fusion method and device of the present application utilizes the two-dimensional slice images acquired by the CT device to reconstruct three-dimensional data through a fusion algorithm, and realizes intelligent recognition and automatic labeling of three-dimensional contraband, which greatly reduces the workload of security personnel and significantly improves security inspection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flowchart of a three-dimensional image fusion method based on computer tomography provided by an embodiment of the present invention;

[0012] Figure 2 A projection diagram obtained by rotating a three-dimensional geometric structure around the Z axis provided by an embodiment of the present invention;

[0013] Figure 3 Annotated projection diagrams of all angles of a three-dimensional geometric structure provided by an embodiment of the present invention;

[0014] Figure 4 A projection diagram obtained by rotating a three-dimensional geometric structure around the X-axis by 0 degrees provided by an embodiment of the present invention;

[0015] Figure 5 A stereogram of a three-dimensional geometric structure provided by an embodiment of the present invention undergoing a first cut in three-dimensional space;

[0016] Figure 6 A projection diagram of a three-dimensional geometric structure provided by an embodiment of the present invention obtained by rotating 30 degrees around the X-axis;

[0017] Figure 7 A three-dimensional diagram of a three-dimensional geometric structure provided by an embodiment of the present invention undergoing a second cut in three-dimensional space;

[0018] Figure 8 A structural block diagram of a three-dimensional image fusion device based on computer tomography provided by an embodiment of the present invention;

[0019] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 , which shows a flow chart of a three-dimensional image fusion method based on computer tomography of the present application.

[0022] like Figure 1 As shown, in step S101, at least one two-dimensional CT slice image is acquired, and a three-dimensional geometric structure is reconstructed according to the at least one CT slice image;

[0023] In step S102, the three-dimensional geometric structure is rotated at different angles based on the X-axis, the Y-axis or the Z-axis, and the three-dimensional geometric structure at the current viewing angle is projected to form a projection image;

[0024] In step S103, a target object in the projection images at different angles is identified according to an image recognition method, and the target object is marked based on a marking frame;

[0025] In step S104, it is determined based on a matching algorithm whether a difference index between a marked box in the projection image at the orthographic projection angle and a marked box in the projection image at a certain angle is less than a preset threshold;

[0026] In step S105, if the difference index between the source annotation box in the projection image at the orthographic projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, the annotation box in the projection image at a certain angle is fused with the annotation box in the projection image at the orthographic projection angle to obtain a three-dimensional annotation box of the certain target object.

[0027] The method of this embodiment reads a two-dimensional CT slice image, generates three-dimensional point data, and rotates it around the X-axis, Y-axis or Z-axis at different angles to generate a screenshot at the current viewing angle, generates a projection image file, uses AI to recognize the projection image, and automatically annotates the coordinates, type, and confidence of the annotation box of the prohibited items. Finally, each annotation box in the projection image at different angles is matched to obtain a multi-angle annotation box for each prohibited item, which is then fused to obtain a three-dimensional annotation box of the prohibited item, thereby realizing intelligent recognition and automatic annotation of three-dimensional prohibited items, greatly reducing the workload of security personnel and significantly improving security inspection efficiency.

[0028] In a specific embodiment, the matching algorithm is a matching algorithm of single rotation axis projection, specifically:

[0029] Figure 1 The dotted label box in the figure is the source label box, and the black label box is the target label box. We need to find the one that best matches the dotted label box among the three black label boxes.

[0030] The coordinates of the source annotation box are (313 221 502 283), the minimum value of the horizontal coordinate min=313, and the maximum value max=502.

[0031] The coordinates of the three black target annotation boxes are

[0032] Box1 coordinates = (338 290 577 356) min1 = 338, max1 = 577

[0033] Box2 coordinates = (315 190 502 253) min2 = 315, max2 = 502

[0034] Box3 coordinates = (354 241 604 304) min3 = 354, max3 = 604

[0035] Because it is rotating up and down, the horizontal coordinate of the object will remain unchanged. Define the difference index V:

[0036] V = (minSrc - minDst) 2 +(maxSrc-maxDst) 2 ,

[0037] Wherein, V is the difference index between the annotation box in the projection image at the positive projection angle and the annotation box in the projection image at a certain angle, minSrc is the minimum horizontal coordinate of the source annotation box, minDst is the minimum horizontal coordinate of the target annotation box, maxSrc is the maximum horizontal coordinate of the source annotation box, and maxDst is the maximum horizontal coordinate of the target annotation box.

[0038] get

[0039] V1=(313-338)^2+(502-577)^2=6250

[0040] V2=(313-315)^2+(502-502)^2=4

[0041] V3=(313-354)^2+(502-604)^2=12085

[0042] It can be seen that V2 is the smallest, so Figure 1 The dashed box (313 221 502 283) is best matched by Figure 1 Medium mark box (315 190 502 253).

[0043] If the minimum V is still greater than a threshold (set to 500 in this program), it is considered Figure 1 There is no box matching the dashed box.

[0044] Find the projections from other angles that can be matched with Figure 1 The dotted box matches the annotation box to get the list of annotation boxes of all angles of a contraband: 3 0 313 221 502 283 3 20 313 188 502 252 3 40 313 177 503 242 3 60 315 190 502 253 3 80 315 225 500 288 3 100 317 276 497 339 3 120 315 339 503 402 3 140 313 404 503 469 3 160 316 466 502 529

[0054] The first field is the rotation axis, 1 represents the x-axis, the second field is the rotation angle, and the last four fields are the four coordinates of the two-dimensional annotation box.

[0055] It can be seen that the horizontal coordinates of all matched coordinate frames are approximately equal.

[0056] In another specific embodiment, the matching algorithm is a matching algorithm of multi-rotation axis projection, specifically:

[0057] To match the projection images between different rotation axes, it is necessary to use the projection images at four angles, x90, y0, z0, and z90, as intermediaries and perform multiple matches.

[0058] By swapping the horizontal and vertical coordinates of the z0 label box, we can get x90.

[0059] For example, if the source annotation box is z60 (338 290 577 356) and the target annotation box is x30, first look for the matching box box1 (338 423 576 482) of the source annotation box in z0, then swap the horizontal and vertical coordinates and look for the matching box box2 (423 338 481 575) of box1 in x90, and finally look for the matching box box3 (422 242 482 401) of box2 in x30, that is, match them in the order of z60->z0->x90->x30.

[0060] Finally, the (422 242 482 401) box in the x30 projection image is the box that best matches the (338 290 577356) box in the z60 projection image.

[0061] Get the matching list of all angles of this contraband in turn, as follows: 1 0 422 252 483 313 1 30 422 242 482 401 1 60 422 273 481 501 1 90 423 338 481 575 3 0 338 423 576 482 3 30 339 356 579 417 3 60 338 290 577 356 3 90 343 252 584 313

[0070] After matching, fusion begins. Taking the matching list of the 2D annotation box of a contraband item as an example, the detailed process of the fusion algorithm is introduced. 1 0 422 252 483 313 1 30 422 242 482 401 1 60 422 273 481 501 1 90 423 338 481 575 3 0 338 423 576 482 3 30 339 356 579 417 3 60 338 290 577 356 3 90 343 252 584 313

[0079] x Annotation box in the projection at 0 degrees

[0080] Process the first angle first: 1 0 422 252 483 313

[0081] The annotation box in the x 0 degree projection (422 252 483 313) is Figure 4 The black frame in the

[0082] If we cut in three-dimensional space, Figure 5 shown.

[0083] The annotation box in the x 30 degree projection (422 242 482 401) is Figure 6 The black frame in the

[0084] The second cut is performed in three-dimensional space, as Figure 7 shown.

[0085] After all the matched angles are cut, the coordinate range of all cutting points is calculated to obtain the three-dimensional coordinate frame: 160 356 16 228 428 76.

[0086] See also Figure 8 , which shows a structural block diagram of a three-dimensional image fusion device based on computer tomography of the present application.

[0087] like Figure 8 As shown, the three-dimensional image fusion device 200 includes a reconstruction module 210 , a projection module 220 , a recognition module 230 , a judgment module 240 and a fusion module 250 .

[0088] Among them, the reconstruction module 210 is configured to obtain at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image; the projection module 220 is configured to rotate the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and project the three-dimensional geometric structure at the current viewing angle to form a projection image; the recognition module 230 is configured to recognize a certain target object in the projection images at different angles according to an image recognition method, and mark the certain target object based on the annotation box; the judgment module 240 is configured to judge whether the difference index between the annotation box in the projection image at the positive projection angle and the annotation box in the projection image at a certain angle is less than a preset threshold based on a matching algorithm; the fusion module 250 is configured to fuse the annotation box in the projection image at a certain angle with the annotation box in the projection image at the positive projection angle if the difference index between the source annotation box in the projection image at the positive projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, so as to obtain the three-dimensional annotation box of the certain target object.

[0089] It should be understood that Figure 8 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 8 The modules in it will not be described in detail here.

[0090] In some other embodiments, the embodiments of the present invention further provide a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions can execute the three-dimensional image fusion method based on computer tomography in any of the above method embodiments;

[0091] As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0092] Acquire at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image;

[0093] Rotating the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and projecting the three-dimensional geometric structure at a current viewing angle to form a projection image;

[0094] Identify a target object in the projection images at different angles according to an image recognition method, and mark the target object based on a marking frame;

[0095] Determine based on a matching algorithm whether a difference index between a marked box in the projection image at an orthographic projection angle and a marked box in the projection image at a certain angle is less than a preset threshold;

[0096] If the difference index between the source annotation box in the projection image at the orthographic projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, the annotation box in the projection image at a certain angle is fused with the annotation box in the projection image at the orthographic projection angle to obtain a three-dimensional annotation box of the certain target object.

[0097] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the three-dimensional image fusion device based on computer tomography, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the three-dimensional image fusion device based on computer tomography via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] Fig. 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Fig. 9 In the example, the bus connection is used. The memory 320 is the above-mentioned computer-readable storage medium. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the three-dimensional image fusion method based on computer tomography in the above-mentioned method embodiment is realized. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the three-dimensional image fusion device based on computer tomography. The output device 340 may include display devices such as display screens.

[0099] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0100] As an implementation mode, the electronic device is applied to a three-dimensional image fusion device based on computer tomography, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:

[0101] Acquire at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image;

[0102] Rotating the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and projecting the three-dimensional geometric structure at a current viewing angle to form a projection image;

[0103] Identify a target object in the projection images at different angles according to an image recognition method, and mark the target object based on a marking frame;

[0104] Determine based on a matching algorithm whether a difference index between a marked box in the projection image at an orthographic projection angle and a marked box in the projection image at a certain angle is less than a preset threshold;

[0105] If the difference index between the source annotation box in the projection image at the orthographic projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, the annotation box in the projection image at a certain angle is fused with the annotation box in the projection image at the orthographic projection angle to obtain a three-dimensional annotation box of the certain target object.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional image fusion method based on computer tomography, characterized in that: include: Acquire at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image; Rotating the three-dimensional geometric structure at different angles based on the X-axis, the Y-axis or the Z-axis, and projecting the three-dimensional geometric structure at a current viewing angle to form a projection image; Identify a target object in the projection images at different angles according to an image recognition method, and mark the target object based on a marking frame; Determine based on a matching algorithm whether a difference index between a marked box in the projection image at an orthographic projection angle and a marked box in the projection image at a certain angle is less than a preset threshold; If the difference index between the source annotation box in the projection image at the orthographic projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, the annotation box in the projection image at a certain angle is fused with the annotation box in the projection image at the orthographic projection angle to obtain a three-dimensional annotation box of the target object, wherein the expression for calculating the difference index is: ， In the formula, is a difference index between the annotation box in the projection image at the normal projection angle and the annotation box in the projection image at a certain angle, is the minimum horizontal coordinate of the source annotation box, is the minimum horizontal coordinate of the target annotation box, is the maximum horizontal coordinate of the source annotation box, The maximum horizontal coordinate of the target label box.

2. The three-dimensional image fusion method based on computer tomography according to claim 1, characterized in that: The matching algorithm is a single rotation axis projection matching algorithm.

3. The three-dimensional image fusion method based on computer tomography according to claim 1, characterized in that: The matching algorithm is a matching algorithm of multi-rotation axis projection.

4. A three-dimensional image fusion device based on computer tomography, characterized in that: include: A reconstruction module, configured to acquire at least one two-dimensional CT slice image, and reconstruct a three-dimensional geometric structure according to the at least one CT slice image; A projection module is configured to rotate the three-dimensional geometric structure at different angles based on an X-axis, a Y-axis or a Z-axis, and project the three-dimensional geometric structure at a current viewing angle to form a projection image; A recognition module, configured to recognize a target object in the projection images at different angles according to an image recognition method, and mark the target object based on a marking frame; A judgment module is configured to judge whether a difference index between a marked box in the projection image at an orthographic projection angle and a marked box in the projection image at a certain angle is less than a preset threshold based on a matching algorithm; The fusion module is configured to fuse the annotation box in the projection image at a certain angle with the annotation box in the projection image at the orthographic projection angle to obtain a three-dimensional annotation box of the certain target object if the difference index between the source annotation box in the projection image at the orthographic projection angle and the target annotation box in the projection image at a certain angle is less than a preset threshold, wherein the expression for calculating the difference index is: ， In the formula, is a difference index between the annotation box in the projection image at the normal projection angle and the annotation box in the projection image at a certain angle, is the minimum horizontal coordinate of the source annotation box, is the minimum horizontal coordinate of the target annotation box, is the maximum horizontal coordinate of the source annotation box, The maximum horizontal coordinate of the target label box.

5. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images