Target recognition methods, devices, and X-ray scanning detection systems based on 3D CT data

Abstract

Provided is a target recognition method for three-dimensional CT data, which can be used in the field of image processing, including: acquiring three-dimensional CT data; initially recognizing the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets; extracting three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; using the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as inputs of the three-dimensional target recognition method; and re-recognizing the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets. The present invention effectively utilizes the three-dimensional information of CT data to improve the accuracy of target recognition. In addition, a target recognition device, electronic device, ray scanning detection system, computer-readable storage medium and program product for three-dimensional CT data are also provided.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and / or target detection, and in particular to a target recognition method and device for three-dimensional CT data, an electronic device, a ray scanning detection system, a computer-readable storage medium and a program product. Background Art

[0002] Among related technologies, security inspection CT (Computed Tomography) equipment can provide three-dimensional images with richer information and is widely used in luggage security inspections at customs, airports, railway stations, subway stations and other places.

[0003] With the development of science and technology, the security inspection field has higher and higher requirements for the accuracy and real-time performance of target recognition. On the one hand, 3D CT data contains richer information. On the other hand, 3D CT data usually contains a huge number of voxels, which is sparse and contains certain noise and redundant information. Recognizing 3D CT targets will face the problem of huge computational complexity. In security inspection scenarios, it is more difficult to identify targets with strong 3D shape features but complex material composition and physical properties. At the same time, security inspection scenarios also have higher requirements for the real-time and accuracy of 3D CT data target recognition.

[0004] Therefore, how to effectively use 3D CT data to identify targets is one of the key issues to improve algorithm accuracy and real-time performance.

[0005] It should be noted that the above information disclosed in this section is only used for understanding the background of the inventive concept of the present invention, and therefore, the above information may contain information that does not constitute the prior art. Summary of the invention

[0006] In view of at least one aspect of the above technical problems, a method and device for target recognition of three-dimensional CT data, an electronic device, a ray scanning detection system, a computer-readable storage medium and a program product are proposed.

[0007] In one aspect, a method for target recognition of three-dimensional CT data is provided, the method comprising: acquiring three-dimensional CT data; performing initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; extracting three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; using the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as inputs of a three-dimensional target recognition method; and re-recognizing the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0008] According to some exemplary embodiments, the first three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the first three-dimensional image semantic description set is in units of three-dimensional regions of interest and / or three-dimensional CT images, and includes: category information, confidence, and at least one of target location information.

[0009] According to some exemplary embodiments, the three-dimensional region of interest includes volume data corresponding to a local area where the target is located, extracted from the three-dimensional CT data.

[0010] According to some exemplary embodiments, a first three-dimensional image semantic description set of at least one of the N targets and a three-dimensional region of interest of at least one of the N targets are used as inputs of a three-dimensional target recognition method, specifically including: using the first three-dimensional image semantic description set of all the N targets and the three-dimensional regions of interest of all the N targets as inputs of the three-dimensional target recognition method.

[0011] According to some exemplary embodiments, the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets are used as input of the three-dimensional target recognition method, specifically including: screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets; and using the first three-dimensional image semantic description set of n targets that meet preset conditions and / or the three-dimensional regions of interest of the n targets as input of the three-dimensional target recognition method, wherein n is a positive integer greater than or equal to 1 and less than N.

[0012] According to some exemplary embodiments, the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets are screened, specifically including: based on the category information, screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets that are judged to belong to a preset category; and / or, based on the confidence level, screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets whose confidence levels are within a preset interval.

[0013] According to some exemplary embodiments, the initial recognition of the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets specifically includes: performing dimensionality reduction processing on the three-dimensional CT data to generate multiple two-dimensional views; using a two-dimensional image recognition method to perform target recognition on the multiple two-dimensional views to obtain a two-dimensional image semantic description set of N targets; and performing dimensionality increase on the two-dimensional image semantic description geometry to obtain a first three-dimensional image semantic description set of the N targets.

[0014] According to some exemplary embodiments, the three-dimensional target recognition method is used to re-recognize the three-dimensional region of interest to obtain a three-dimensional recognition result of at least one of the N targets, specifically including: using a pre-trained deep learning network for three-dimensional CT data to perform feature extraction to obtain a feature map.

[0015] According to some exemplary embodiments, the extracted features include at least one of shape, texture, and color in the three-dimensional CT data.

[0016] According to some exemplary embodiments, the feature extraction using a pre-trained deep learning network for three-dimensional CT data further specifically includes: generating a feature vector using the three-dimensional mask, category information and confidence included in the first three-dimensional image semantic description set.

[0017] According to some exemplary embodiments, the re-identification of the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: using the deep learning network to operate the feature map, wherein the operation includes at least one of feature map fusion and feature map dimensionality reduction.

[0018] According to some exemplary embodiments, the three-dimensional target recognition method is used to re-recognize the three-dimensional region of interest to obtain a three-dimensional recognition result of at least one of the N targets, and specifically includes: applying the feature map to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

[0019] According to some exemplary embodiments, the re-identifying the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: directly using the second three-dimensional image semantic description set as the three-dimensional recognition result; or, integrating the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, and using the integrated result as the three-dimensional recognition result.

[0020] According to some exemplary embodiments, the second three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the second three-dimensional image semantic description set is in units of three-dimensional regions of interest and / or three-dimensional CT images, and includes: category information, confidence, and at least one of target location information.

[0021] According to some exemplary embodiments, pre-training the deep learning network for three-dimensional CT data includes: acquiring training data; and augmenting the training data using a data augmentation operation, wherein the data augmentation operation includes at least one of rotation, translation, scaling, flipping, and data fusion.

[0022] According to some exemplary embodiments, the training data includes a three-dimensional region of interest, and the data fusion includes fusing the three-dimensional region of interest with a safety product or a safety package.

[0023] According to some exemplary embodiments, the three-dimensional region of interest is acquired by a first CT device, and the safety product or safety package is acquired by a second CT device, and the first CT device and the second CT device are the same or different devices.

[0024] On the other hand, a target recognition method for three-dimensional CT data is provided, the method comprising: acquiring three-dimensional CT data; performing initial recognition on the three-dimensional CT data; in response to the three-dimensional region of interest of the target not being included in the result of the initial recognition, using the entire or a partial region of the three-dimensional CT data as an input of a three-dimensional target recognition method; and re-recognizing the entire or a partial region of the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, wherein N is a positive integer greater than or equal to 1.

[0025] According to some exemplary embodiments, using the entirety or a portion of the three-dimensional CT data as input to a three-dimensional target recognition method specifically includes: extracting multiple candidate regions from the three-dimensional CT data; using at least one of the multiple candidate regions as input to the three-dimensional target recognition method; or, using the entirety or a portion of the three-dimensional CT data as input to the three-dimensional target recognition method specifically includes: using the entirety of the three-dimensional CT data as input to the three-dimensional target recognition method.

[0026] In another aspect, a target recognition device for three-dimensional CT data is provided, the device comprising: an acquisition module, used to acquire three-dimensional CT data; an initial recognition module, used to perform initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; a first extraction module, used to extract three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; an input module, used to use the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as inputs of a three-dimensional target recognition method; and a re-recognition module, used to re-recognize the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0027] In another aspect, a target recognition device for three-dimensional CT data is provided, the device comprising: an acquisition module for acquiring three-dimensional CT data; an initial recognition module for performing initial recognition on the three-dimensional CT data; an input module for, in response to the result of the initial recognition not including the three-dimensional region of interest of the target, using the entire or a partial area of ​​the three-dimensional CT data as an input of a three-dimensional target recognition method; and a re-recognition module for re-recognizing the entire or a partial area of ​​the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, wherein N is a positive integer greater than or equal to 1.

[0028] On yet another aspect, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0029] On the other hand, a ray scanning detection system is provided, comprising: a ray source for emitting rays; a detector for receiving rays emitted from the ray source and passing through a detection object; and a data processing device, the data processing device being communicatively connected to the detector, wherein the data processing device comprises: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method as described above.

[0030] In yet another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0031] In yet another aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described above are implemented.

[0032] In an embodiment of the present invention, after the initial recognition method is used to obtain a set of semantic description information of the three-dimensional image and a three-dimensional region of interest, a cascaded three-dimensional target recognition method is used for further recognition, which effectively utilizes the three-dimensional information of the CT data, can further distinguish between contraband and safe items, and improve the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to better understand the present invention, the present invention will be described in detail according to the following drawings:

[0034] Figure 1 is a schematic structural diagram of a ray scanning detection system according to some exemplary embodiments of the present invention.

[0035] Figure 2 is a schematic diagram of a CT scanning device implementing a reconstruction scanning mode according to some exemplary embodiments of the present invention.

[0036] Figure 3 An example of attenuated signal data obtained according to an embodiment of the present invention is shown.

[0037] Figure 4 is a schematic flow chart of a method for object recognition in three-dimensional CT data according to an embodiment of the present invention.

[0038] Figure 5 It is a schematic flow chart of the initial recognition operation in the target recognition method for three-dimensional CT data according to an embodiment of the present invention.

[0039] Figure 6 is a schematic flow chart of a re-identification operation in a target recognition method for three-dimensional CT data according to an embodiment of the present invention.

[0040] Figure 7 is an architecture diagram of a cascaded deep learning network used in a target recognition method according to an exemplary embodiment of the present invention.

[0041] Figure 8 is a schematic flow chart of a target recognition method for three-dimensional CT data according to some other embodiments of the present invention.

[0042] Fig. 9 is a structural block diagram of an object recognition apparatus for three-dimensional CT data according to an exemplary embodiment of the present invention.

[0043] Fig.10 is a structural block diagram of an object recognition device for three-dimensional CT data according to some other exemplary embodiments of the present invention.

[0044] Fig.11 A structural block diagram of an electronic device suitable for implementing the target recognition method according to an exemplary embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0045] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not have to be adopted to implement the present invention. In other examples, in order to avoid confusing the present invention, known structures, materials or methods are not specifically described.

[0046] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or subcombination. In addition, it should be understood by those of ordinary skill in the art that the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0047] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0048] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0049] Below, taking CT scanning detection as an example, the embodiment of the present invention is described in detail. It should be understood that the embodiment of the present invention is not limited to the CT scanning detection scene, and it can be applied to various scanning detection scenes. For example, it can be applied to scanning detection scenes including various different inspection objects, including but not limited to vehicle scanning detection, luggage / package scanning detection, human body scanning detection, etc. It should be noted that the description of the scanning detection scene here is not an exhaustive description, and the description here should not be understood as limiting the scope of protection of the present invention.

[0050] It should be noted that the embodiments of the present invention are not limited to target detection tasks, but can also be applied to target classification, image segmentation and other tasks. Thus, in this article, unless otherwise specified, the expression "target recognition" includes but is not limited to various tasks such as target detection, target classification, image segmentation and the like.

[0051] It should also be noted that, in this article, the expression "target" may include but is not limited to, for example, a target object of interest, an item of interest, etc. in a scanning detection scene.

[0052] Figure 1 is a schematic diagram of the structure of a ray scanning detection system according to some exemplary embodiments of the present invention. Figure 1 In the figure, a CT scanning device is schematically shown as an example of the ray scanning detection system. Figure 1As shown, the CT scanning device according to this embodiment includes: a rack 20, a carrying mechanism 40, a controller 50, a data processing device 60 (such as a computer), etc. The rack 20 includes a ray source 10 that emits X-rays for inspection, such as an X-ray machine, and a detection and acquisition device 30. The carrying mechanism 40 carries the detection object (such as checked luggage) 70 to pass through the scanning area between the ray source 10 of the rack 20 and the detection and acquisition device 30, and the rack 20 rotates around the forward direction of the detection object 70, so that the rays emitted by the ray source 10 can pass through the detection object 70, and the detection object 70 is CT scanned. The detection and acquisition device 30, for example, includes a detector and a data acquisition device with an integral module structure, such as a flat panel detector, which is used to detect the rays that pass through the detection object 70 (such as checked luggage), obtain analog signals, and convert the analog signals into digital signals, thereby outputting the projection data of the detection object 70 for the X-rays. The controller 50 is used to control the various parts of the entire system to work synchronously. The data processing device 60 is used to process the data collected by the data collector, process and reconstruct the data, and output the results.

[0053] like Figure 1 As shown, the ray source 10 is placed on one side where the object to be inspected can be placed, and the detection and acquisition device 30 is placed on the other side of the inspection object 70, including a detector and a data collector, which is used to obtain the transmission data and / or multi-angle projection data of the inspection object 70. The data collector includes a data amplification and shaping circuit, which can work in a (current) integration mode or a pulse (counting) mode. The data output cable of the detection and acquisition device 30 is connected to the controller 50 and the data processing device 60, and the collected data is stored in the data processing device 60 according to the trigger command.

[0054] In an embodiment of the present invention, a ray scanning detection module may be deployed in the data processing device 60, and an image recognition model or a target recognition model may be deployed in the ray scanning detection module. The ray scanning detection module may use the image recognition model or the target recognition model to identify the collected field data (e.g., ray scanning images), and the identification may include target detection, for example, detecting a target object or an object of interest in the ray scanning image. For example, the target object or the object of interest may be various types of contraband. It should be understood that the specific category of the target object or the object of interest depends on the scanning detection site arranged by the ray scanning detection system, or in other words, the specific category of the target object or the object of interest is determined by the user of the ray scanning detection system (also referred to as a user in the embodiment of the present invention) according to the specific scanning detection needs, and it can be dynamically adjusted according to the scanning detection needs.

[0055] It should be noted that, for the convenience of description, in this article, the target object or object of interest here can be collectively referred to as a target.

[0056] In the embodiment of the present invention, the ray source 10 may be, for example, an X-ray machine, and a suitable X-ray machine focal spot size may be selected according to the imaging resolution. In other embodiments, an X-ray machine may not be used, but a linear accelerator may be used to generate an X-ray beam.

[0057] The detection and acquisition device 30 includes an X-ray detector and a data acquisition circuit. The X-ray detector may be a solid detector, a gas detector or other detectors, but the embodiments of the present invention are not limited thereto. The data acquisition circuit includes a readout circuit, an acquisition trigger circuit, and a data transmission circuit.

[0058] The combination of the controller 50 and the data processing device 60 includes, for example, a computer device installed with a control program and a data processing program, which is responsible for completing the control of the operation process of the CT scanning device, including mechanical rotation, electrical control, safety interlock control, etc., reconstructing CT images from projection data, training image recognition models or target recognition models, and using the trained image recognition models or target recognition models to recognize X-ray scanning images, etc.

[0059] Figure 2 Schematic diagram of a CT scanning device implementing a reconstruction scanning mode according to some exemplary embodiments of the present invention. Figure 2 As shown, after the X-rays emitted from the ray source 10 pass through a portion of the detection object 70 in the field of view 45, they are received by the detection and acquisition device 30, converted into electrical signals and then converted into digital signals representing attenuation values, which are used as projection data and reconstructed by a computer to obtain a higher quality image.

[0060] According to an embodiment of the present invention, the CT scanning device is used to perform CT scanning on the object to be detected to obtain the original attenuation signal. The attenuation signal data corresponding to a row of detectors can also be displayed in the form of a two-dimensional image. Figure 3 FIG. 2 shows an example of attenuation signal data obtained according to an embodiment of the present invention. Figure 3 The horizontal axis of the original attenuation signal shown represents the pixel sequence in the detector column direction (for example, from 1 to 256), and the vertical axis represents the angle (for example, from 1 degree to 360 degrees). The original attenuation signal is preprocessed to become projection data. For example, the projection data can be preprocessed by a CT scanning device such as negative logarithmic transformation to obtain the projection data. Then, the data processing device 60 executes a reconstruction program, for example, a mainstream analytical reconstruction method in the relevant technical field can be used to obtain a reconstructed image, which is the ray scanning image mentioned in the embodiment of the present invention.

[0061] In an embodiment of the present invention, the data processing device 60 may perform target recognition (e.g., target detection) on the reconstructed image (i.e., the ray scanning image) to identify the target (e.g., contraband) in the ray scanning image. Specifically, a ray scanning detection module may be deployed in the data processing device 60, and an image recognition model or a target recognition model may be deployed in the ray scanning detection module. The ray scanning detection module may use the image recognition model or the target recognition model to recognize the ray scanning image.

[0062] The applicant has found through research that 3D CT data usually contains a huge number of voxels, which is sparse, and also contains certain noise and redundant information. Identifying 3D CT targets will face the problem of huge computational complexity. In security inspection scenarios, it is difficult to identify targets with strong 3D shape features but complex material composition and physical properties. At the same time, security inspection scenarios also have high requirements for the real-time and accuracy of 3D CT data target identification.

[0063] In view of the above problems, the applicant has proposed a method for identifying security inspection CT targets (see the published patent of the same applicant as the present invention, the authorization announcement number of the patent is CN113792623B, the invention name is "security inspection CT target identification method and device", hereinafter, the patent may be referred to as patent document 1). In the security inspection CT target identification method, the method of generating multiple two-dimensional views by reducing the dimension of three-dimensional CT data, then performing target recognition on the multiple two-dimensional views to obtain a two-dimensional semantic description set of the target, and then upgrading the obtained two-dimensional semantic description set to finally obtain a three-dimensional recognition result of the target, thereby realizing effective recognition of security inspection CT targets. In the security inspection CT target identification method, a method of first reducing the dimension of three-dimensional data to two-dimensional for recognition and then upgrading the dimension to generate a three-dimensional result is proposed, which can not only reduce the computational complexity during recognition, but also use the current two-dimensional target recognition technology with high accuracy and good performance to identify the target, and can effectively identify targets with complex material composition and physical properties and shape characteristics, while providing a three-dimensional recognition result with rich information.

[0064] The applicant has found through further research that in the above method, the use of a limited number of two-dimensional views for target recognition will result in insufficient expression of three-dimensional features, and may lose some three-dimensional shape features, spatial position information, etc. At the same time, since the three-dimensional probability map is the result of effective simplification and abstraction, the method of extracting features from the three-dimensional probability map to obtain three-dimensional target recognition results in the above method cannot be applied to the original three-dimensional CT data. Compared with traditional two-dimensional images, three-dimensional CT data provides more comprehensive and three-dimensional information. If the original three-dimensional CT data and the three-dimensional image semantic description set obtained by the prior art can be effectively used to further identify the target, it is possible to further distinguish between contraband and safe items and improve the recognition accuracy.

[0065] To this end, an embodiment of the present invention proposes a target recognition method for three-dimensional CT data, which can effectively utilize the feature expression of three-dimensional objects to further improve the recognition accuracy. In the method provided by the embodiment of the present invention, the three-dimensional CT data is first recognized to obtain a three-dimensional semantic description set of the target, and then a cascade three-dimensional target recognition method is used, that is, the acquired three-dimensional semantic description set and its corresponding three-dimensional region of interest are used as input to further recognize the three-dimensional region of interest. In addition, if the three-dimensional recognition result obtained in advance does not contain the three-dimensional region of interest of the target, the method provided by the embodiment of the present invention can also be processed, and the entire or partial area of ​​the three-dimensional CT data can be used as the input of the cascaded three-dimensional target recognition method for recognition.

[0066] For example, some exemplary embodiments of the present invention provide a target recognition method for three-dimensional CT data, the method comprising: acquiring three-dimensional CT data; performing initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; extracting three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data based on the first three-dimensional image semantic description set of the targets; using the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as inputs of a three-dimensional target recognition method; and re-recognizing the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0067] In this embodiment, after the initial recognition method is used to obtain the set of semantic description information of the three-dimensional image and the three-dimensional region of interest, the cascaded three-dimensional target recognition method is used for further recognition, which effectively utilizes the three-dimensional information of the CT data, can further distinguish between contraband and safe items, and improve the accuracy of target recognition.

[0068] For another example, some exemplary embodiments of the present invention also provide a target recognition method for three-dimensional CT data, the method comprising: acquiring three-dimensional CT data; performing initial recognition on the three-dimensional CT data; in response to the three-dimensional region of interest of the target not being included in the result of the initial recognition, using the entire or partial region of the three-dimensional CT data as input to a three-dimensional target recognition method; and re-recognizing the entire or partial region of the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, wherein N is a positive integer greater than or equal to 1.

[0069] In this embodiment, if the three-dimensional recognition result obtained by the initial recognition method does not contain the three-dimensional region of interest of the target, the method can still be processed, and the entire or partial area of ​​the three-dimensional CT data can be used as the input of the cascaded three-dimensional target recognition method for recognition, and the result is used as the basis for further judgment, which is conducive to expanding the application scenarios of the target recognition method.

[0070] Figure 4 is a schematic flow chart of a method for object recognition in three-dimensional CT data according to an embodiment of the present invention. Figure 4 , the target recognition method for three-dimensional CT data may include the following operations or steps S410~S450.

[0071] Reference Figure 4 , in operation or step S410, three-dimensional CT data is acquired.

[0072] Combined with reference Figures 1 to 3 After the X-rays emitted from the ray source 10 pass through a portion of the detection object 70 in the field of view 45, they are received by the detection and acquisition device 30, converted into an electrical signal and then converted into a digital signal representing the attenuation value as projection data; then, the mainstream analytical reconstruction method in the relevant technical field can be used to obtain a three-dimensional reconstructed image.

[0073] In an embodiment of the present invention, the three-dimensional CT data may include the three-dimensional reconstructed image.

[0074] Reference Figure 4 In operation or step S420, the three-dimensional CT data is initially identified to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1.

[0075] Figure 5 is a schematic flow chart of the initial recognition operation in the target recognition method for three-dimensional CT data according to an embodiment of the present invention. Figure 5 , the initial identification operation may include the following sub-operations or sub-steps S510~S530.

[0076] Reference Figure 5 In the sub-operation or sub-step S510, the three-dimensional CT data is subjected to dimensionality reduction processing to generate a plurality of two-dimensional views.

[0077] Specifically, first, multiple directions can be set for the three-dimensional CT data. Here, the multiple directions are arbitrary directions, not limited to specific directions such as the orthogonal directions of the object's traveling direction during the detection process. In addition, while setting multiple directions, or before and after setting multiple directions, it is optional to perform certain preprocessing operations on the three-dimensional volume data, for example, filtering invalid voxels, and precalculating the geometric parameters required for projection or rendering, thereby improving the subsequent processing speed. Secondly, projection or rendering can be performed in multiple directions to obtain multiple two-dimensional views. As an example, ray or light projection can be performed based on a CT image slice sequence, and a ray or light is emitted from each pixel of the image along a specific direction. The ray or light passes through the entire image sequence, and in this process, the image sequence is sampled to obtain attribute or color information, and the attribute or color value is accumulated according to a certain model until the ray or light passes through the entire image sequence, and the attribute or color value finally obtained is used as the two-dimensional view after dimensionality reduction.

[0078] In this embodiment, by projecting in an arbitrary direction to obtain a two-dimensional reduced-dimensionality view, it is possible to avoid performing dimensionality reduction only along a specific direction. For example, dimensionality reduction is performed along a direction orthogonal to the direction of travel of the object during the detection process. This can solve the following problems that exist when dimensionality reduction is performed only along a specific direction: (1) Under certain placement postures of the object, the area of ​​the object after dimensionality reduction is too small and the shape information is incomplete, making it impossible to accurately identify the object; (2) The shape information of the object is lost due to being blocked by other objects, making it impossible to accurately identify the target.

[0079] In the sub-operation or sub-step S520, a two-dimensional image recognition method is used to perform object recognition on the multiple two-dimensional views to obtain a set of two-dimensional image semantic descriptions of N objects.

[0080] Here, the multiple two-dimensional views may include the multiple two-dimensional views obtained in the above sub-operation S510.

[0081] Specifically, as a method for object recognition of a two-dimensional view, at least one of an image processing method for two-dimensional images, a classical machine learning method, and a deep learning method, or a combination thereof, may be used. For example, a two-dimensional view is used as an input and input into a neural network model, and a two-dimensional semantic description set is obtained as an output.

[0082] Specifically, a target detection neural network based on deep learning can be used to detect the two-dimensional position of the target. The convolutional neural network used in the target detection task is a typical structure of deep learning in computer vision tasks. Such a convolutional neural network has the characteristics of local connection, weight sharing, and spatial resampling. These characteristics make the convolutional neural network have a certain degree of translation and scaling invariance. Here, the two-dimensional semantic description set is in units of one or more of pixels, regions of interest, and two-dimensional images, and contains category information and / or confidence, or the two-dimensional semantic description set is in units of regions of interest and / or two-dimensional images, and contains at least one of category information, confidence, and target location information. Among them, the category information indicates the category to which the target belongs, such as a gun, a knife, etc. The location information may include center coordinates, a bounding box, etc. The confidence indicates the possibility of the existence of the target and can be a normalized scalar or vector.

[0083] In other words, the two-dimensional semantic description set may include: category information, confidence, etc. of a pixel belonging to an object; category information, location information, confidence, etc. of an object contained in a region of interest; at least one of category information, location information, confidence, etc. of an object contained in a two-dimensional image. The at least one information may be information contained in one group or may be information contained in different groups.

[0084] In addition, the description set of two-dimensional semantic information may also include other semantic information such as the posture of the target and the number of targets in addition to category information, confidence, and location information.

[0085] It should be noted that, in the embodiments of the present invention, there is no particular limitation on the two-dimensional image recognition method for the two-dimensional view, as long as it is a method that can obtain the above-mentioned two-dimensional semantic description set based on the two-dimensional view.

[0086] In the sub-operation or sub-step S530, the 2D image semantic description geometry is upgraded to obtain a first 3D image semantic description set of the N objects.

[0087] In an embodiment of the present invention, in the sub-operation or sub-step S530, first, the two-dimensional semantic description set is mapped to a three-dimensional space using a back-projection method to obtain a three-dimensional probability map.

[0088] It should be understood that back projection can be considered as the inverse process of projection. Optionally, the back projection process can be implemented by voxel driving or pixel driving. Specifically, a semantic feature matrix can be obtained by voxel driving or pixel driving, and the semantic feature matrix can be compressed into a three-dimensional probability map. Here, voxel driving includes: corresponding each voxel in the three-dimensional CT data to a pixel in each two-dimensional view, querying and accumulating the two-dimensional semantic description information corresponding to the pixel, and generating a semantic feature matrix. The correspondence between voxels and pixels can establish a mapping function or a lookup table to increase the calculation speed. As described above, according to voxel driving, each voxel in the three-dimensional CT data is traversed, and its semantic feature matrix is ​​obtained in turn, and finally the semantic feature matrix is ​​compressed to obtain a three-dimensional probability map.

[0089] In the sub-operation or sub-step S530, feature extraction is then performed on the three-dimensional probability map to obtain a first three-dimensional image semantic description set of the target.

[0090] For example, for a three-dimensional probability map, at least one of an image processing method, a classical machine learning method, and a deep learning method, or a combination thereof, is used to extract features, thereby obtaining a set of three-dimensional image semantic descriptions.

[0091] As an example, the three-dimensional probability map is used as input and input into the deep learning model, and the three-dimensional recognition results such as confidence and three-dimensional bounding box are output. The deep learning model used here can adopt technologies such as classification neural network or target detection network with fewer layers. By adopting such technology, the amount of information contained in the original three-dimensional CT data is effectively simplified and abstracted after the above steps, which is closer to the ultimate goal of contraband identification. A simple feature extraction method can be used to quickly and accurately extract a set of three-dimensional semantic descriptions.

[0092] In this article, the 3D image semantic description set may be in units of one or more of voxels, 3D regions of interest, and 3D CT images, and include: category information and / or confidence; or, the 3D image semantic description set may be in units of 3D regions of interest and / or 3D CT images, and include: category information, target location information, and confidence. The target location information in the 3D CT image may include a 3D bounding box.

[0093] In other words, the 3D image semantic description set includes: the category information and confidence level of a certain voxel belonging to a target; the category information, location information and confidence level of a certain 3D region of interest (VOI); and the category information, location information and confidence level of a certain 3D CT image. The at least one information may be information included in one group or information included in different groups.

[0094] Since the three-dimensional image semantic description set is generated from a three-dimensional probability map generated based on a two-dimensional semantic description set, the types of semantic information contained in the three-dimensional image semantic description set and the types of semantic information contained in the two-dimensional semantic description set are consistent or mutually convertible.

[0095] In this embodiment, the two-dimensional semantic description set is upgraded through the above-mentioned dimensionality increase processing to obtain a three-dimensional recognition result of the target, thereby solving the problem of significant reduction in the amount of information when only two-dimensional recognition is performed through dimensionality reduction. It is possible to reduce the loss of information while adopting two-dimensional recognition, while taking into account the real-time and accuracy of security inspection.

[0096] It should be noted that the specific implementation of the initial recognition operation (i.e., the above-mentioned operation S420) or step in the target recognition method for three-dimensional CT data according to the embodiment of the present invention can refer to the method described in patent document 1, that is, in the absence of conflict, the method described in patent document 1 is fully or partially combined here as the initial recognition method in the embodiment of the present invention.

[0097] Return to reference Figure 4 In operation or step S430, three-dimensional regions of interest corresponding to the N targets are extracted from the three-dimensional CT data according to a first three-dimensional image semantic description set of the target.

[0098] In an embodiment of the present invention, the three-dimensional region of interest may include volume data corresponding to a local area where the target is located, extracted from the three-dimensional CT data.

[0099] Reference Figure 4 In operation or step S440, a first three-dimensional image semantic description set of at least one of the N targets and a three-dimensional region of interest of at least one of the N targets are used as inputs of a three-dimensional target recognition method.

[0100] In some exemplary embodiments of the present invention, operation or step S440 may specifically include: taking the first three-dimensional image semantic description set of all the N objects and the three-dimensional regions of interest of all the N objects as inputs of the three-dimensional object recognition method.

[0101] In some other exemplary embodiments of the present invention, operation or step S440 may specifically include: screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets; and using the first three-dimensional image semantic description set of n targets that meet preset conditions and / or the three-dimensional regions of interest of the n targets as input of the three-dimensional target recognition method, where n is a positive integer greater than or equal to 1 and less than N.

[0102] Exemplarily, in operation or step S440, screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets may specifically include: screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets that are judged to belong to a preset category based on the category information; and / or, screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets whose confidence is within a preset interval based on the confidence.

[0103] That is, in some exemplary embodiments of the present invention, the first three-dimensional image semantic description set of n targets that meet the preset conditions may include: the first three-dimensional image semantic description set of n targets belonging to one or some preset categories; and / or, the first three-dimensional image semantic description set of n targets whose confidence is within a preset interval. Additionally or optionally, the three-dimensional region of interest that meets the preset conditions may include: the three-dimensional region of interest of n targets belonging to one or some preset categories; and / or, the three-dimensional region of interest of n targets whose confidence is within a preset interval.

[0104] In an embodiment of the present invention, it supports taking all three-dimensional regions of interest and their three-dimensional image semantic description sets obtained in the previous step as input for the next stage, and also supports screening the three-dimensional regions of interest or three-dimensional image semantic description sets and taking partial data or partial semantic description sets that meet certain conditions as input for the next stage.

[0105] Reference Figure 4 In operation or step S450, the three-dimensional target recognition method is used to re-recognize the three-dimensional region of interest to obtain a three-dimensional recognition result of at least one of the N targets.

[0106] Figure 6 is a schematic flow chart of a re-identification operation in a target recognition method for three-dimensional CT data according to an embodiment of the present invention. Figure 6 , the re-identification operation may include the following sub-operations or sub-steps S610 to S650. It should be noted that, Figure 6 In the present invention, some of the sub-operations or sub-steps S610 to S650 are optional operations or steps.

[0107] In some exemplary embodiments of the present invention, in the sub-operation or sub-step S620, a pre-trained deep learning network for three-dimensional CT data is used to perform feature extraction to obtain a feature map. That is, the cascaded three-dimensional object recognition method can be a three-dimensional object recognition method based on a deep learning network.

[0108] In an embodiment of the present invention, the cascaded three-dimensional target recognition method can identify the three-dimensional region of interest, support the input of multiple three-dimensional data such as single channel and multi-channel, and is compatible with multiple information including original three-dimensional CT data, the three-dimensional image semantic description set, etc.

[0109] Optionally, in the sub-operation or sub-step S610, the 3D region of interest may be preprocessed before the feature extraction operation is performed. The preprocessing method includes but is not limited to operations such as denoising, morphological transformation, geometric transformation, image type conversion, etc., and supports converting 3D volume data into point cloud data. For example, the preprocessing may include format conversion, size adjustment, etc. For another example, the 3D region of interest may be preprocessed such as rotation correction and denoising to ensure image quality.

[0110] Rotation correction can help correct the tilt of an image. For example, a rotation correction method based on the image edge contour can be used to perform rotation correction on at least one of the first image, the second image, and the third image. Specifically, the rotation correction method based on the image edge contour may include the following steps: image grayscale: converting a color image into a grayscale image for subsequent processing; threshold binarization: converting a grayscale image into a binary image so that the target and background in the image are clearly distinguished; detecting contours: detecting the edge contours of the target in the binary image; extracting the contour's enclosing matrix: finding the minimum rectangular area that encloses the target contour, and the edge of this rectangular area can be used to estimate the image's tilt angle; obtaining the deflection angle through the extracted enclosing matrix: calculating the angle between the long side of the enclosing matrix and the horizontal line, and this angle is the image's tilt angle; using affine transformation to deflect the image: using affine transformation to rotate the image to a normal angle based on the calculated tilt angle.

[0111] For example, a rotation correction method based on Fourier transform and Hough line detection can be used to perform rotation correction on at least one of the first image, the second image, and the third image. Specifically, the rotation correction method based on Hough line detection can include the following steps: image preprocessing and edge detection: first, the image is preprocessed, such as grayscale and binarization, and then an edge detection algorithm (such as Sobel operator) is used to detect the edge in the image; Hough transform: Hough transform is a technology for detecting straight lines and curves in an image. Here, it is used to detect straight lines in an image. The basic principle of Hough transform is to use the duality of points and lines to transform the lines in the image space into clustering points in the parameter space, so as to detect whether there is a curve of a given property in a given image; calculate the tilt angle: through Hough transform, the polar coordinate representation (ρ, θ) of each straight line in the image can be obtained. Among them, θ represents the tilt angle of the straight line. The tilt angle of the entire image can be obtained by counting the tilt angles of all detected straight lines and then taking the average value; Image rotation: According to the calculated tilt angle, the image can be rotated by an image rotation algorithm (such as affine transformation) to correct it; the corrected image is output.

[0112] In some exemplary embodiments, the three-dimensional region of interest may be denoised using a filter-based method, a model-based method, or a learning-based method. Denoising is an operation that reduces noise interference in an image or data. In the preprocessing of the three-dimensional region of interest, the goal of denoising is to eliminate or reduce irrelevant information introduced by the acquisition process, sensor noise, or other interference factors. For example, denoising methods may include median filtering, Gaussian filtering, mean filtering, etc., which can smooth data and remove noise points to improve the accuracy and stability of subsequent processing.

[0113] In some exemplary embodiments, the three-dimensional region of interest may be preprocessed using morphological transformation. Morphological transformation is a type of operation based on image shape and structure, used to change the geometric structure of an image or extract shape features of interest. In the preprocessing of the three-dimensional region of interest, morphological transformation may be used to remove small spots or isolated noise, fill holes, connect adjacent regions, etc. For example, morphological transformation includes dilation, erosion, opening operation, closing operation, etc.

[0114] In some exemplary embodiments, the three-dimensional region of interest may be preprocessed using a geometric transformation. A geometric transformation refers to an operation on the position, scale, rotation, or affine transformation of an object in three-dimensional space to better match or adjust an image or data. In the three-dimensional region of interest preprocessing, a geometric transformation may be used to correct or align data to conform to a specific reference coordinate system or geometric shape. For example, transformation operations such as translation, rotation, scaling, and mirroring may be performed.

[0115] In some exemplary embodiments, the three-dimensional region of interest may be preprocessed using image type conversion. For example, image type conversion may be an operation of converting raw three-dimensional volume data into point cloud data. In three-dimensional region of interest preprocessing, such conversion may help convert volume data representation into a series of discrete three-dimensional point coordinates for subsequent point cloud processing and analysis. For example, image type conversion methods may include voxelization, surface reconstruction, and sampling.

[0116] Through the preprocessing method, data quality can be improved, noise interference can be reduced, geometric structure can be adjusted, or three-dimensional volume data can be converted into point cloud data, providing more accurate and reliable input for subsequent three-dimensional recognition and analysis.

[0117] It should be noted that three-dimensional volume data refers to data represented in three-dimensional space with voxels as the basic unit. Voxels can be understood as pixels in three-dimensional space, and each voxel contains attribute or numerical information at a specific position in space. Three-dimensional volume data can be constructed by dividing a three-dimensional object or scene into a series of voxels and assigning attribute values ​​to each voxel. Three-dimensional volume data can be represented in the form of a cube or a cuboid, providing detailed information on the internal and external structure of the object.

[0118] Point cloud data is a data set consisting of a set of discrete three-dimensional point coordinates. Each point contains the location information in three-dimensional space, and usually has other attributes such as color, normal vector or intensity. Point cloud data can be obtained in different ways, such as lidar scanning, stereo vision or three-dimensional reconstruction. Point cloud data can represent the surface geometry of a three-dimensional object or the distribution of scattered points in a scene. It provides a description of the shape, surface details and spatial position of the object. Point cloud data has a wide range of applications in computer vision, computer graphics, robotics, etc., such as target detection and recognition, three-dimensional reconstruction, environmental perception, etc.

[0119] In general, three-dimensional volume data is three-dimensional data with volume information represented by voxels, while point cloud data is a data set composed of discrete three-dimensional point coordinates, which is used to represent the surface geometry of an object or the scattered point distribution in a scene.

[0120] It should be noted that there are some differences in the representation and data structure between 3D volume data and point cloud data: (1) Representation: 3D volume data uses voxels as the basic unit, dividing the 3D space into regular cubes or cuboids. Each voxel contains the attribute or numerical information of a specific position in the space. Point cloud data is a data set consisting of a set of discrete 3D point coordinates, and each point contains the position information in the 3D space. (2) Data structure: 3D volume data is usually stored in the form of voxel grids, represented by data structures such as multidimensional arrays or voxel trees, where the attribute value of each voxel can represent the density, color, material and other information of the object. Point cloud data is stored as a collection of points, and each point can contain additional attributes, such as color, normal vector or intensity. (3) Data density: Since 3D volume data is represented by regular voxel grids, it can provide detailed information about the internal and external structure of the object, including density distribution, shape, etc. Point cloud data is a collection of discrete point coordinates, which usually only provides geometric information on the surface of an object. The density and distribution of the points depend on the method and parameters used to collect or generate the point cloud. (4) Data scale: Since point cloud data only stores discrete point coordinates, the storage space of point cloud data is more compact than that of 3D volume data. Therefore, for large-scale 3D scenes or complex objects, point cloud data may be more suitable for storage and processing.

[0121] Reference Figure 6 In the sub-operation or sub-step S620, a deep learning network for three-dimensional CT data is used, specifically, a specially designed feature extraction network is used to extract features from the pre-processed three-dimensional data.

[0122] In the embodiment of the present invention, the extracted features include but are not limited to the shape, texture, color, etc. of the three-dimensional data. The shape of the three-dimensional data corresponds to the large structure of the target, the texture corresponds to the small structure or detailed structure of the target, and the color corresponds to the physical properties of the target such as the Z value and density.

[0123] In an embodiment of the present invention, useful feature representations are extracted from input data by neural networks or other methods. These features may include information such as the shape, texture, and color of an object, which are used to describe and express key characteristics of the input data. That is, in an embodiment of the present invention, various information of three-dimensional data can be comprehensively utilized, which is conducive to further improving the accuracy of recognition. In a deep learning network of three-dimensional CT data, by using a three-dimensional convolutional neural network (3D CNN) or a feature extraction method based on an attention mechanism, local and global features in the input data can be learned, and feature maps with distinguishing and expressive capabilities can be extracted.

[0124] In some exemplary embodiments, in the sub-operation or sub-step S620, feature extraction may use a three-dimensional convolutional neural network or a feature extraction method based on an attention mechanism, etc. These methods may be used alone or in combination to extract feature maps of three-dimensional CT data.

[0125] If the target 3D image semantic description set includes 3D mask, category information and confidence information of voxels, accurate generation of feature vectors is also supported. That is, in the sub-operation or sub-step S620, the feature vector can be generated using the 3D mask, category information and confidence information included in the first 3D image semantic description set.

[0126] In the process of feature extraction network training, in order to increase the diversity of data, a variety of data augmentation techniques can be applied, including but not limited to rotation, translation, scaling, flipping, data fusion, etc. Data augmentation helps to improve the robustness and generalization of the network.

[0127] In actual applications, security inspection scenarios may be very complex, including packages and items of various shapes, sizes, and internal structures. By fusing the 3D region of interest with the security items or security packages, the data richness is increased by fusing security packages or security items of different shapes, sizes, and internal structures, allowing the network to learn a wider range of feature representations. This allows the network to maintain a high recognition accuracy when processing unseen data, improving the generalization ability of the model.

[0128] When performing data fusion, the 3D region of interest can be fused with the security product or security package of the same CT device, or with the CT data of different device models. Because in actual applications, you may encounter multiple security inspection CT devices of different models or brands, and fusing data from different devices can improve data compatibility, reduce the limitations and shortcomings that may exist in a single device, and enable algorithms and systems to adapt to inputs from different devices.

[0129] Due to the physical principle of X-ray imaging of CT equipment, it is not a simple operation to fuse two 3D CT data. Especially for the fusion of data from different CT equipment, the selection of a suitable fusion method also needs to consider the differences between equipment and the consistency requirements of data. In order to ensure the correctness of the fusion result, it may be necessary to perform some processing on the 3D CT data, which includes but is not limited to one or more of the following.

[0130] (1) Data calibration and registration: Scanning with different CT devices or the same CT device at different time points or with different imaging technologies may cause inconsistencies in the data, including differences in scanning parameters, resolution, and imaging mode, which may lead to data mismatches in space, time, or physical properties. For example, different CT devices may use different spatial resolution parameters, which may include pixel size, slice thickness, etc. Therefore, when performing data fusion, data calibration or registration may be required to ensure the correctness of the fused image. For example, methods for data calibration or registration may include registration algorithms, resampling, etc.

[0131] (2) Artifact correction, noise processing, etc.: CT data may contain artifacts, noise or other image distortions during the imaging process, which will affect the quality and accuracy of the data. When fusion is performed, it is necessary to consider how to reduce or correct these artifacts and noise to maintain the reliability of the data. For example, methods to reduce or correct these artifacts and noise may include image correction technology, filtering algorithms, etc.

[0132] (3) Screening the area to be fused: For different targets, when performing data fusion, it is sometimes necessary to avoid fusing the target with objects that seriously affect its physical properties. For example, when fusing organic substances such as drugs and explosives, it is necessary to avoid fusing with strong metal interference areas. Therefore, when performing data fusion, it may be necessary to judge one or some areas of the security product or security package selected for fusion with the three-dimensional area of ​​interest. Judgment methods include methods based on physical properties (setting different thresholds through prior knowledge to screen suitable fusion areas) and methods based on machine learning methods.

[0133] Before fusing the 3D ROI with the safety product or safety package, data augmentation techniques such as random rotation, translation, scaling, etc. can be used to increase data diversity before fusion. During fusion, the 3D ROI can be fused with the safety product obtained from the safety package, or the 3D ROI can be directly fused with one or some areas of the safety package, and then the fused new 3D ROI can be extracted based on the 3D semantic description set (such as 3D bounding box, 3D mask, etc.) corresponding to the 3D ROI. For example, 3D CT data fusion methods can include the following:

[0134] The first is a voxel-based fusion method, which includes: (1) Fusion based on the physical principle of X-ray imaging of CT equipment to generate fused three-dimensional data. (2) Weighted average method: assign a weight to each voxel in the two CT data, and then perform weighted average of the values ​​of these voxels according to the weight to generate fused data. The weight can be adjusted according to factors such as data quality and credibility; (3) Maximum / minimum method: select the maximum or minimum value of multiple CT values ​​at each voxel position as the new fused value to generate fused data.

[0135] The second is a feature-based fusion method, which includes: extracting representative features from the original CT data, such as edges, textures, shapes, etc. Matching the extracted features to find the correspondence between different CT data. Based on the matched features, the original CT data is fused. This method can retain more detailed information and has a certain degree of robustness to noise and artifacts.

[0136] The third is a fusion method based on deep learning, which includes: using a deep learning model (such as a convolutional neural network) to extract and encode features from the original CT data; fusing the extracted features to obtain a fused feature representation, and finally, restoring the fused features to a three-dimensional CT image through a decoder. This method can automatically learn complex features in the data.

[0137] The choice of fusion method should be determined according to the specific application scenario and task requirements. Different fusion methods have different advantages and applicability, and it is necessary to weigh and select them by comprehensively considering factors such as data characteristics, system requirements, and algorithm models.

[0138] Optionally, in the sub-operation or sub-step S630, the extracted feature graphs may be operated, including but not limited to feature graph fusion, feature graph dimensionality reduction, etc. These operations may be selected and combined according to specific tasks and network architectures. The purpose is to improve the performance and generalization ability of the model by performing appropriate operations and processing on the feature graphs, so as to better adapt to various complex data and application scenarios.

[0139] Specifically, if the network has multiple branches or multiple levels of feature maps, feature map fusion operations can be performed to obtain a more comprehensive and multi-scale feature representation. The fusion operation can use weighted summation of feature maps, feature map concatenation, feature pyramid, feature fusion based on Transformer encoding and decoding, etc.

[0140] For example, in the feature map weighted sum fusion method, feature maps from different levels or sources are weighted and summed according to certain weights. Each feature map is multiplied by the corresponding weight, and they are added together to obtain the fused feature map. The weights can be learned or set manually.

[0141] For example, in the fusion method of feature map splicing, feature maps of different levels or sources are spliced ​​in the channel dimension. For example, if the number of channels of two feature maps is C1 and C2 respectively, the number of channels of the spliced ​​feature map is C1 + C2. This method can increase the channel dimension of the feature map, allowing the network to use information from multiple levels or sources at the same time.

[0142] For example, in the feature pyramid fusion method, the feature pyramid is realized by extracting feature maps at different scales and fusing them. Information at different scales is captured by using convolution kernels of different sizes or feature maps of different levels. Then, these feature maps are fused together to obtain multi-scale feature representations.

[0143] For example, in the feature fusion method based on Transformer encoding and decoding, Transformer is a neural network model based on the self-attention mechanism, which can be used to encode and decode sequence data. In feature fusion, the encoder-decoder structure of Transformer can be used to fuse feature maps. The encoder encodes the input feature map into context-aware feature representations, and the decoder decodes these feature representations into the final fused feature map.

[0144] It should be noted that these feature fusion methods can be selected and applied according to specific tasks and network structures. Their goal is to improve the network's expressiveness and performance by effectively combining feature information from different sources.

[0145] Specifically, in order to improve computing efficiency or reduce feature dimensions, a feature map dimensionality reduction operation can be performed on the feature map to reduce the resolution or reduce the spatial dimension. For example, the operation method may include but is not limited to global average pooling, downsampling, convolution dimensionality reduction (such as 1×1 convolution, depth-wise separable convolution, etc.), etc.

[0146] In an embodiment of the present invention, the deep learning network for three-dimensional CT data may include but is not limited to a three-dimensional convolutional neural network, for example, 3D U-Net, V-Net, 3D DenseNet, 3D ResNet, etc. It should be noted that 3D U-Net is a three-dimensional convolutional neural network based on the U-Net architecture. It has an encoder-decoder structure, extracts and restores features through stacked convolution, pooling and upsampling operations, and can effectively capture contextual information of different scales. V-Net is a volume segmentation network based on 3D CNN, which is specifically used for medical image segmentation tasks. It introduces residual connection and batch normalization techniques, learns the feature representation of the image through multiple 3D convolution and deconvolution layers, and generates pixel-level segmentation results. 3D DenseNet is a network structure extended to three dimensions based on 2D DenseNet. It connects the feature map of each layer with the feature maps of all previous layers by dense connection, promotes the transmission of features and the flow of information, thereby improving the performance and stability of the model. 3D ResNet is a three-dimensional convolutional neural network extended on the basis of 2D ResNet. It uses the idea of ​​residual connection and deep network to build a network by stacking multiple residual blocks. It can effectively solve the problems of gradient disappearance and model degradation and improve the representation and learning ability of features.

[0147] Figure 7 is an architecture diagram of a cascaded deep learning network used in a target recognition method according to an exemplary embodiment of the present invention.

[0148] Reference Figure 7, the cascaded deep learning network is a three-dimensional convolutional neural network based on the U-Net architecture. Its structure consists of an encoder and a decoder, and features are extracted by stacking convolution, pooling and upsampling operations. The cascaded deep learning network includes an encoder part and a decoder part. The encoder part is a typical convolutional network, including repeated applications of convolutions, each followed by a ReLU activation function and a maximum pooling operation. For example, in this network, each layer in the encoder part may include two 3×3×3 convolutions, each followed by a ReLU layer, and then a 2×2×2 maximum pooling layer with a stride of 2 in each direction. The decoder part mainly includes upsampling operations and convolution operations. The upsampling operation is used to restore the resolution of the image, while the convolution operation is used to combine into an accurate output based on this information. In the decoder part, each layer contains a 2×2×2 deconvolution layer with a stride of 2, followed by two 3×3×3 convolution layers, each followed by a ReLU layer. The cascaded deep learning network also includes a jump connection for connecting the corresponding layer feature map of the encoder part with the corresponding layer feature map of the decoder part to provide richer context information. In the output layer of the network, a 1×1×1 convolution layer and an appropriate activation function (such as Sigmoid or Softmax) are used to generate the final result.

[0149] The cascaded deep learning network is characterized by a symmetrical structure between the encoder and the decoder, and the low-level features can be fused with the high-level features through the jump connection, thereby improving the accuracy of segmentation and the ability to retain details. In addition, the depth and width of the network can be adjusted to adapt to different tasks and data sets to improve performance and generalization capabilities.

[0150] In some exemplary embodiments of the present invention, in sub-operation or sub-step S640, the feature map is applied to a classification, segmentation or detection task to obtain a second set of three-dimensional image semantic descriptions. The "feature map" here can be a feature map directly extracted by sub-operation or sub-step S620, or a feature map extracted by sub-operation or sub-step S620 and processed by S630 of sub-operation or sub-step S620.

[0151] Specifically, a multi-layer perceptron or other classifier can be used to map the feature map to a category probability distribution to achieve the classification task. Additionally or alternatively, deconvolution, bilinear interpolation, and other methods can be used to restore the feature map to the original image size, and a semantic label can be assigned to each voxel to achieve voxel-level segmentation. Additionally or alternatively, a target detection head can be used to perform classification and regression operations on the feature map, predict the category and location of the target, generate category information, confidence, and location information, etc., to achieve the target detection task.

[0152] In an embodiment of the present invention, the second three-dimensional image semantic description set is based on one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the second three-dimensional image semantic description set is based on three-dimensional regions of interest and / or three-dimensional CT images, and includes: at least one of category information, confidence, and location information of the target. For example, the location information of the target in the three-dimensional CT image may include a three-dimensional bounding box.

[0153] Research has found that compared with the results of out-recognition, the set of semantic descriptions of the three-dimensional image obtained after re-recognition is more accurate, which is a correction and improvement of the semantic description set in the previous stage.

[0154] In some exemplary embodiments of the present invention, in sub-operation or sub-step S650, the second three-dimensional image semantic description set is directly used as the three-dimensional recognition result; or, the first three-dimensional image semantic description set and the second three-dimensional image semantic description set are integrated, and the integrated result is used as the three-dimensional recognition result.

[0155] Exemplarily, in the step of integrating the first three-dimensional image semantic description set with the second three-dimensional image semantic description set, for category information, for the same target, if the category information in the first three-dimensional image semantic description set is inconsistent with the category information in the second three-dimensional image semantic description set, the category information in the first three-dimensional image semantic description set can be directly replaced by the category information in the second three-dimensional image semantic description set, that is, the newly identified category information is used as the final result; for position information, if the three-dimensional bounding box in the first three-dimensional image semantic description set is larger than the range of the three-dimensional bounding box in the second three-dimensional image semantic description set, the three-dimensional bounding box in the second three-dimensional image semantic description set with a more accurate position can be used. The bounding box is used as the final location information; for confidence, if the confidence in the first three-dimensional image semantic description set is inconsistent with the confidence in the second three-dimensional image semantic description set, the confidence in the first three-dimensional image semantic description set can be directly replaced by the confidence in the second three-dimensional image semantic description set, that is, the newly identified confidence is used; or, different weights can be assigned to the confidence in the first three-dimensional image semantic description set and the confidence in the second three-dimensional image semantic description set, respectively, and then a weighted sum is performed, and the confidence obtained after the weighted sum is used as the final confidence. For example, the weight of the confidence in the first three-dimensional image semantic description set can be smaller than the weight of the confidence in the second three-dimensional image semantic description set.

[0156] Figure 8 is a schematic flow chart of a method for object recognition in three-dimensional CT data according to some other embodiments of the present invention. Figure 8The target recognition method for three-dimensional CT data may include the following operations or steps S810 to S840. It should be noted that, in the following, the focus is on Figure 8 The embodiments are different from the parts of the above embodiments. In the absence of conflict, the steps or operations described in the above embodiments may be used in the embodiments described below.

[0157] Reference Figure 8 , in operation or step S810, three-dimensional CT data is acquired.

[0158] In operation or step S820, the three-dimensional CT data is initially identified.

[0159] In operation or step S830, in response to the initial recognition result not including the 3D region of interest of the target, the entire or partial region of the 3D CT data is used as input of a 3D target recognition method.

[0160] In some exemplary embodiments, taking the entirety or a partial area of ​​the three-dimensional CT data as an input of a three-dimensional target recognition method specifically includes: taking the entirety of the three-dimensional CT data as an input of the three-dimensional target recognition method.

[0161] In some exemplary embodiments, the whole or a part of the three-dimensional CT data is used as the input of the three-dimensional target recognition method, which specifically includes: extracting multiple candidate regions from the three-dimensional CT data; and using at least one of the multiple candidate regions as the input of the three-dimensional target recognition method.

[0162] In this embodiment, a three-dimensional region extraction method may be used to extract candidate regions from the entire three-dimensional CT data, and then these candidate regions may be identified one by one. The three-dimensional region extraction method includes but is not limited to 3D Region Proposal Network (3D RPN), sliding window (e.g., sliding window) and other methods. Alternatively, the three-dimensional CT data may be directly identified as a whole, that is, without extracting the three-dimensional candidate regions, the entire three-dimensional CT data may be directly identified using a cascaded three-dimensional target recognition method.

[0163] In operation or step S840, the three-dimensional target recognition method is used to re-recognize the entire or partial area of ​​the three-dimensional CT data to obtain a three-dimensional recognition result of at least one of N targets, where N is a positive integer greater than or equal to 1.

[0164] In some embodiments of the present invention, if the three-dimensional recognition result obtained by the initial recognition method does not contain the three-dimensional region of interest of the target, the method provided by the embodiment of the present invention can still be used for subsequent processing. In this case, the entire or partial area of ​​the three-dimensional CT data can be used as the input of the cascaded deep learning network used in the re-recognition method, and the result is used as the basis for further discrimination.

[0165] It should be noted that in the embodiment of the present invention, in the re-identification method, in addition to using the above-mentioned deep learning-based network, other image processing and machine learning methods can also be used to further identify the three-dimensional region of interest. For example, three-dimensional edge detection, texture analysis, feature descriptors and other methods can be used to extract the features of the three-dimensional region of interest, and then the extracted features can be input into a machine learning classifier (such as a support vector machine, random forest, etc.) for identification.

[0166] Based on the above method, an embodiment of the present invention further provides a target recognition device for three-dimensional CT data. Fig. 9 The device is described in detail.

[0167] Fig. 9 is a block diagram of a target recognition device for three-dimensional CT data according to an exemplary embodiment of the present invention. Fig. 9 The device 900 includes an acquisition module 910, an initial recognition module 920, a first extraction module 930, an input module 940 and a re-recognition module 950.

[0168] The acquisition module 910 is used to acquire three-dimensional CT data. In some exemplary embodiments, the acquisition module 910 may be used to perform the operation S410 and its sub-operations described above, which will not be described in detail herein.

[0169] The initial recognition module 920 is used to perform initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1. In some exemplary embodiments, the initial recognition module 920 can be used to perform the operation S420 and its sub-operations described above, which will not be repeated here.

[0170] Exemplarily, the initial recognition module 920 is used to: perform dimensionality reduction processing on the three-dimensional CT data to generate multiple two-dimensional views; use a two-dimensional image recognition method to perform target recognition on the multiple two-dimensional views to obtain a set of two-dimensional image semantic descriptions of N targets; and perform dimensionality increase on the two-dimensional image semantic description geometry to obtain a first three-dimensional image semantic description set of the N targets.

[0171] The first extraction module 930 is used to extract the three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the target. In some exemplary embodiments, the first extraction module 930 can be used to perform the operation S430 and its sub-operations described above, which will not be repeated here.

[0172] The input module 940 is used to use the first 3D image semantic description set of at least one of the N targets and the 3D region of interest of at least one of the N targets as inputs of the 3D target recognition method. In some exemplary embodiments, the input module 940 can be used to perform the operation S440 and its sub-operations described above, which will not be repeated here.

[0173] In some embodiments, the input module 940 is used to use the first three-dimensional image semantic description set of all the N objects and the three-dimensional regions of interest of all the N objects as inputs of the three-dimensional object recognition method.

[0174] In other embodiments, the input module 940 is used to: screen the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets; and use the first three-dimensional image semantic description set of n targets that meet preset conditions and / or the three-dimensional regions of interest of the n targets as input to the three-dimensional target recognition method, where n is a positive integer greater than or equal to 1 and less than N.

[0175] Exemplarily, the first three-dimensional image semantic description set of the N targets and / or the three-dimensional regions of interest of the N targets are screened, specifically including: based on the category information, screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets that are judged to belong to a preset category; and / or, based on the confidence level, screening out the first three-dimensional image semantic description set and / or the three-dimensional regions of interest of the n targets whose confidence levels are within a preset interval.

[0176] The re-identification module 950 is used to re-identify the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets. In some exemplary embodiments, the re-identification module 950 can be used to perform the operation S450 and its sub-operations described above, which will not be repeated here.

[0177] In some embodiments, the re-identification module 950 is used to: use a pre-trained deep learning network for three-dimensional CT data to perform feature extraction to obtain a feature map. In particular, the three-dimensional mask, category information and confidence included in the first three-dimensional image semantic description set can be used to generate a feature vector.

[0178] Exemplarily, the extracted features include at least one of shape, texture, and color in the three-dimensional CT data.

[0179] Optionally, the re-identification module 950 is further used to: use the deep learning network to operate the feature map, wherein the operation includes at least one of feature map fusion and feature map dimensionality reduction.

[0180] Exemplarily, the re-identification module 950 is further used to: apply the feature map to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

[0181] In some embodiments, the re-recognition module 950 is configured to use the second three-dimensional image semantic description set directly as the three-dimensional recognition result.

[0182] In some other embodiments, the re-recognition module 950 is used to: integrate the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, and use the integrated result as the three-dimensional recognition result.

[0183] In an embodiment of the present invention, the second three-dimensional image semantic description set is based on one or more of voxels, three-dimensional regions of interest and three-dimensional CT images, and includes: category information and / or confidence; or, the second three-dimensional image semantic description set is based on three-dimensional regions of interest and / or three-dimensional CT images, and includes: at least one of category information, confidence and target location information.

[0184] Based on the above method, an embodiment of the present invention further provides a target recognition device for three-dimensional CT data. Fig.10 The device is described in detail.

[0185] Fig.10 is a structural block diagram of a target recognition device for three-dimensional CT data according to some other exemplary embodiments of the present invention. Fig.10 The device 1000 includes an acquisition module 1010, an initial recognition module 1020, an input module 1030 and a re-recognition module 1040.

[0186] The acquisition module 1010 is used to acquire three-dimensional CT data. In some exemplary embodiments, the acquisition module 1010 may be used to perform the above-described operation S410 or S810 and its sub-operations, which will not be described in detail herein.

[0187] The initial recognition module 1020 is used to perform initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1. In some exemplary embodiments, the initial recognition module 1020 can be used to perform the operation S420 or S820 and its sub-operations described above, which will not be repeated here.

[0188] The input module 1030 is used to respond to the initial recognition result not including the 3D region of interest of the target, using the whole or part of the 3D CT data as input of the 3D target recognition method. In some exemplary embodiments, the input module 1030 can be used to perform the operation S830 and its sub-operations described above, which will not be repeated here.

[0189] The re-identification module 1040 is used to re-identify the entire or partial area of ​​the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets, where N is a positive integer greater than or equal to 1. In some exemplary embodiments, the re-identification module 1040 can be used to perform the operation S840 and its sub-operations described above, which will not be repeated here.

[0190] According to an embodiment of the present invention, the acquisition module 910, the initial recognition module 920, the first extraction module 930, the input module 940 and the re-recognition module 950 included in the device 900, and any multiple modules of the acquisition module 1010, the initial recognition module 1020, the input module 1030 and the re-recognition module 1040 and their submodules included in the device 1000 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, the acquisition module 910, the initial recognition module 920, the first extraction module 930, the input module 940 and the re-recognition module 950 included in the device 900, and at least one of the acquisition module 1010, the initial recognition module 1020, the input module 1030 and the re-recognition module 1040 and their sub-modules included in the device 1000 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, the acquisition module 910, initial recognition module 920, first extraction module 930, input module 940 and re-recognition module 950 included in the device 900, and at least one of the acquisition module 1010, initial recognition module 1020, input module 1030 and re-recognition module 1040 and their sub-modules included in the device 1000 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0191] Fig.11 A structural block diagram of an electronic device suitable for implementing the target recognition method according to an exemplary embodiment of the present invention is schematically shown.

[0192] like Fig.11As shown, the electronic device 1100 according to an embodiment of the present invention includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0193] For example, the electronic device may be the data processing device 60 of the ray scanning detection system.

[0194] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other via bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0195] According to an embodiment of the present invention, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the I / O interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0196] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0197] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0198] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0199] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 1101. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0200] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0201] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0202] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0203] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0204] A target recognition method and device, electronic device, ray scanning detection system, computer-readable storage medium, and program product provided by the embodiments of the present invention have at least one of the following beneficial effects:

[0205] After acquiring the 3D CT data, the initial recognition is first performed, and then the cascaded 3D target recognition method is used. The 3D image semantic description set obtained through the initial recognition and the corresponding 3D region of interest extracted based on the information in the 3D image semantic description set are supported as the input of the cascaded 3D target recognition method. If the 3D recognition result obtained through the initial recognition does not contain the 3D region of interest of the target, the entire or partial area of ​​the 3D CT data can be used as the input of the cascaded 3D target recognition method, and the result is used as the basis for further judgment. Through the cascaded 3D target recognition method, the feature expression of the 3D object can be enhanced, and the recognition accuracy can be further improved.

[0206] In particular, in the field of security inspection CT, three-dimensional CT data provides more comprehensive, accurate and visual information. Effectively using three-dimensional CT data to identify contraband is conducive to improving the accuracy of the recognition algorithm, so that potential threats and hidden objects can be more accurately identified. In an embodiment of the present invention, after using the recognition to obtain a set of three-dimensional image semantic description information and a three-dimensional region of interest, a cascade three-dimensional target recognition method is used to effectively utilize the three-dimensional information of the CT data, which can further distinguish between contraband and safe items, improve recognition accuracy, and ensure the efficiency of the recognition algorithm.

[0207] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for object recognition in three-dimensional CT data, characterized in that: The method comprises: Acquire three-dimensional CT data; Performing preliminary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; the first three-dimensional image semantic description set is based on one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the first three-dimensional image semantic description set is based on the three-dimensional regions of interest and / or the three-dimensional CT images, and includes: at least one of the category information, the confidence, and location information of the target; extracting three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to a first three-dimensional image semantic description set of the targets; Using a first 3D image semantic description set of at least one of the N objects and a 3D region of interest of at least one of the N objects as inputs of a 3D object recognition method; and The three-dimensional target recognition method is used to re-recognize the three-dimensional region of interest to obtain a three-dimensional recognition result of at least one of the N targets.

2. The method according to claim 1, characterized in that: The three-dimensional region of interest includes volume data corresponding to a local area where the target is located and extracted from the three-dimensional CT data.

3. The method according to claim 1, characterized in that Using a first three-dimensional image semantic description set of at least one of the N objects and a three-dimensional region of interest of at least one of the N objects as inputs of a three-dimensional object recognition method, specifically comprising: The first three-dimensional image semantic description set of all the N objects and the three-dimensional regions of interest of all the N objects are used as inputs of the three-dimensional object recognition method.

4. The method according to claim 1, characterized in that: Using a first three-dimensional image semantic description set of at least one of the N objects and a three-dimensional region of interest of at least one of the N objects as inputs of a three-dimensional object recognition method, specifically comprising: Screening the first three-dimensional image semantic description set of the N objects and / or the three-dimensional regions of interest of the N objects; and A first three-dimensional image semantic description set of n targets satisfying preset conditions and / or the three-dimensional regions of interest of the n targets are used as inputs of a three-dimensional target recognition method, wherein n is a positive integer greater than or equal to 1 and less than N.

5. The method according to claim 4, characterized in that Screening the first three-dimensional image semantic description set of the N objects and / or the three-dimensional regions of interest of the N objects specifically includes: According to the category information, a first three-dimensional image semantic description set and / or a three-dimensional region of interest of n objects that are judged to belong to a preset category are screened out; and / or, According to the confidence level, a first three-dimensional image semantic description set and / or a three-dimensional region of interest of n targets whose confidence levels are within a preset interval are screened out.

6. The method according to any one of claims 1 and 5, characterized in that The performing initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets specifically includes: Performing dimensionality reduction processing on the three-dimensional CT data to generate a plurality of two-dimensional views; Using a two-dimensional image recognition method, performing target recognition on the multiple two-dimensional views to obtain a set of two-dimensional image semantic descriptions of N targets; and The two-dimensional image semantic description geometry is upgraded to obtain a first three-dimensional image semantic description set of the N targets.

7. The method according to any one of claims 1 and 5, characterized in that The re-identifying the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets specifically includes: A pre-trained deep learning network for three-dimensional CT data is used to extract features to obtain a feature map.

8. The method according to claim 7, characterized in that The extracted features include at least one of shape, texture, and color in the three-dimensional CT data.

9. The method according to claim 7, characterized in that: The method of extracting features using a pre-trained deep learning network for three-dimensional CT data further specifically includes: A feature vector is generated using the three-dimensional mask, category information and confidence included in the first three-dimensional image semantic description set.

10. The method according to claim 7, characterized in that The re-identifying the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets also specifically includes: using the deep learning network to operate the feature map, wherein the operation includes at least one of feature map fusion and feature map dimensionality reduction.

11. The method according to claim 7, characterized in that The re-identifying the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: The feature map is applied to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

12. The method according to claim 11, characterized in that The re-identifying the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: directly using the second three-dimensional image semantic description set as the three-dimensional recognition result; or, The first three-dimensional image semantic description set and the second three-dimensional image semantic description set are integrated, and the integrated result is used as the three-dimensional recognition result.

13. The method according to claim 11 or 12, characterized in that: The second three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, The second three-dimensional image semantic description set is based on a three-dimensional region of interest and / or a three-dimensional CT image, and includes at least one of category information, confidence, and target location information.

14. The method according to claim 7, characterized in that Pre-training the deep learning network for three-dimensional CT data includes: Obtaining training data; and Using a data augmentation operation, the training data is augmented, The data augmentation operation includes at least one of rotation, translation, scaling, flipping and data fusion.

15. The method according to claim 14, characterized in that The training data includes a three-dimensional region of interest, and the data fusion includes fusing the three-dimensional region of interest with a safety product or a safety package.

16. The method according to claim 15, characterized in that The three-dimensional region of interest is acquired by a first CT device, and the safety product or safety package is acquired by a second CT device, and the first CT device and the second CT device are the same or different devices.

17. A method for object recognition in three-dimensional CT data, characterized in that: The method comprises: Acquire three-dimensional CT data; Performing preliminary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets as a result of preliminary recognition, wherein N is a positive integer greater than or equal to 1; the first three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the first three-dimensional image semantic description set is in units of the three-dimensional regions of interest and / or the three-dimensional CT images, and includes: at least one of the category information, the confidence, and location information of the target; In response to the initial recognition result not including the 3D region of interest of the target, using the entire or a partial region of the 3D CT data as an input of a 3D target recognition method; and The three-dimensional target recognition method is used to re-recognize the entire or partial area of ​​the three-dimensional CT data to obtain a three-dimensional recognition result of at least one of the N targets, where N is a positive integer greater than or equal to 1.

18. The method according to claim 17, characterized in that The whole or a part of the three-dimensional CT data is used as an input of a three-dimensional object recognition method, specifically including: extracting a plurality of candidate regions from the three-dimensional CT data; Using at least one of the plurality of candidate regions as input to a three-dimensional object recognition method, or, The whole or a part of the three-dimensional CT data is used as an input of a three-dimensional object recognition method, specifically including: The entirety of the three-dimensional CT data is used as input to a three-dimensional object recognition method.

19. A target recognition device for three-dimensional CT data, characterized in that: The device comprises: An acquisition module, used for acquiring three-dimensional CT data; an initial recognition module, configured to perform initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; the first three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the first three-dimensional image semantic description set is in units of the three-dimensional regions of interest and / or the three-dimensional CT images, and includes: at least one of the category information, the confidence, and location information of the target; A first extraction module, configured to extract three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to a first three-dimensional image semantic description set of the targets; an input module, configured to use a first three-dimensional image semantic description set of at least one of the N objects and a three-dimensional region of interest of at least one of the N objects as inputs of a three-dimensional object recognition method; and A re-identification module is used to re-identify the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

20. A target recognition device for three-dimensional CT data, characterized in that: The device comprises: An acquisition module, used for acquiring three-dimensional CT data; an initial recognition module, configured to perform initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets as a result of the initial recognition, wherein N is a positive integer greater than or equal to 1; the first three-dimensional image semantic description set is in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images, and includes: category information and / or confidence; or, the first three-dimensional image semantic description set is in units of the three-dimensional regions of interest and / or the three-dimensional CT images, and includes: at least one of the category information, the confidence, and target location information; an input module, configured to use the entire or a partial area of ​​the three-dimensional CT data as an input of a three-dimensional target recognition method in response to the result of the initial recognition not including the three-dimensional region of interest of the target; and A re-identification module is used to re-identify the entire or partial area of ​​the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, where N is a positive integer greater than or equal to 1.

21. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 18.

22. A ray scanning detection system, comprising: A ray source, used for emitting rays; A detector, used to receive the radiation emitted from the radiation source and passing through the detection object; as well as a data processing device, the data processing device being communicatively connected to the detector, It is characterized in that the data processing device comprises: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 18.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

Images