Object recognition method and apparatus for three-dimensional CT data, and ray scanning detection system

Abstract

Provided is an object recognition method for three-dimensional CT data, which method can be used in the field of image processing. The method comprises: acquiring three-dimensional CT data; performing initial recognition on the three-dimensional CT data, so as to acquire first three-dimensional image semantic description sets of N objects; on the basis of the first three-dimensional image semantic description sets of the objects, extracting, from the three-dimensional CT data, three-dimensional regions-of-interest corresponding to the N objects; taking a first three-dimensional image semantic description set of at least one of the N objects and a three-dimensional region-of-interest of at least one of the N objects as inputs for a three-dimensional object recognition method; and using the three-dimensional object recognition method to perform re-recognition on the three-dimensional regions-of-interest, so as to acquire a three-dimensional recognition result of at least one of the N objects. In the present disclosure, three-dimensional information of CT data is effectively used, thereby improving the recognition accuracy of objects. In addition, further provided are an object recognition apparatus for three-dimensional CT data, and an electronic device, a ray scanning detection system, a computer-readable storage medium and a program product.

Description

Target recognition method and device for three-dimensional CT data, and ray scanning detection system

[0001] The present application claims priority from Chinese Patent Application No. 202411226561.3, filed on September 3, 2024, the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of image processing and / or target detection, and in particular to a target recognition method and device for three-dimensional CT data, an electronic device, a ray scanning detection system, a computer readable storage medium and a program product. BACKGROUND

[0003] In the related art, a security CT (Computed Tomography) device can provide three-dimensional images with more abundant information and is widely used in luggage security inspection in customs, airports, railway stations, subway stations and other places.

[0004] With the development of science and technology, the accuracy and real-time requirement of target recognition in the security field is becoming higher and higher. On the one hand, three-dimensional CT data contains more abundant information, on the other hand, three-dimensional CT data usually contains a large number of voxels, which is sparse, and also contains certain noise and redundant information. Identifying three-dimensional CT targets will face the problem of huge amount of calculation. In the security scene, it is difficult to identify targets with strong three-dimensional shape features but complex material composition and physical properties, and the real-time and accuracy requirements of three-dimensional CT data target recognition in the security scene are also higher.

[0005] Therefore, how to effectively use three-dimensional CT data to identify targets is one of the key issues to improve the accuracy and real-time of the algorithm.

[0006] It should be noted that the above information disclosed in this part is only for understanding the background of the inventive concept of the present disclosure, and therefore, the above information can contain information that does not constitute prior art. SUMMARY

[0007] The present disclosure proposes a target recognition method and device for three-dimensional CT data, an electronic device, a ray scanning detection system, a computer readable storage medium and a program product.

[0008] In one aspect, a method for target recognition of three-dimensional CT data is provided, the method comprising: obtaining three-dimensional CT data; performing primary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; extracting a three-dimensional region of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; taking the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as input of a three-dimensional target recognition method; and performing re-recognition on the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0009] According to some exemplary embodiments, the first three-dimensional image semantic description set contains class information and / or confidence in units of one or more of voxels, three-dimensional regions of interest and three-dimensional CT images; or the first three-dimensional image semantic description set contains at least one of class information, confidence and position information of the target in units of three-dimensional regions of interest and / or three-dimensional CT images.

[0010] According to some exemplary embodiments, the three-dimensional region of interest includes volume data corresponding to a local region where the target is located extracted from the three-dimensional CT data.

[0011] According to some exemplary embodiments, taking the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as input of the three-dimensional target recognition method specifically comprises: taking the first three-dimensional image semantic description set of all the N targets and the three-dimensional region of interest of all the N targets as input of the three-dimensional target recognition method.

[0012] According to some exemplary embodiments, taking the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as input of the three-dimensional target recognition method specifically comprises: screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets; and taking the first three-dimensional image semantic description set of n targets meeting a preset condition and / or the three-dimensional region of interest of the n targets as input of the three-dimensional target recognition method, wherein n is a positive integer greater than or equal to 1 and less than N.

[0013] According to some exemplary embodiments, the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets are screened, specifically including: screening the first three-dimensional image semantic description set of the n targets and / or the three-dimensional region of interest of the n targets which are judged to belong to the preset category according to the category information; and / or, screening the first three-dimensional image semantic description set of the n targets and / or the three-dimensional region of interest of the n targets whose confidence degrees are in the preset interval according to the confidence degrees.

[0014] According to some exemplary embodiments, the three-dimensional CT data is preliminarily identified to obtain the first three-dimensional image semantic description set of the N targets, specifically including: performing dimension reduction processing on the three-dimensional CT data to generate a plurality of two-dimensional views; performing target identification on the plurality of two-dimensional views by using a two-dimensional image recognition method to obtain a two-dimensional image semantic description set of the N targets; and performing dimension increase on the two-dimensional image semantic description geometry to obtain the first three-dimensional image semantic description set of the N targets.

[0015] According to some exemplary embodiments, the three-dimensional region of interest is re-identified by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets, specifically including: performing feature extraction by using a deep learning network for three-dimensional CT data which is trained in advance to obtain a feature map.

[0016] According to some exemplary embodiments, the extracted features include at least one of shape, texture and color in the three-dimensional CT data.

[0017] According to some exemplary embodiments, the feature extraction by using the deep learning network for three-dimensional CT data which is trained in advance further specifically includes: generating a feature vector by using the three-dimensional mask, the category information and the confidence degree contained in the first three-dimensional image semantic description set.

[0018] According to some exemplary embodiments, the three-dimensional region of interest is re-identified by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: operating the feature map by using the deep learning network, wherein the operation includes at least one of feature map fusion and feature map dimension reduction.

[0019] According to some exemplary embodiments, the three-dimensional region of interest is re-identified by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically includes: applying the feature map to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

[0020] According to some exemplary embodiments, the re-identifying the three-dimensional region of interest by using the three-dimensional target identification method to obtain the three-dimensional identification result of at least one of the N targets further specifically comprises: directly taking the second three-dimensional image semantic description set as the three-dimensional identification result; or integrating the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, and taking the integrated result as the three-dimensional identification result.

[0021] According to some exemplary embodiments, the second three-dimensional image semantic description set contains category information and / or confidence in units of one or more of voxels, three-dimensional regions of interest and three-dimensional CT images; or the second three-dimensional image semantic description set contains at least one of category information, confidence and position information of a target in units of three-dimensional regions of interest and / or three-dimensional CT images.

[0022] According to some exemplary embodiments, the pre-training of the deep learning network for three-dimensional CT data comprises: obtaining training data; and augmenting the training data by using a data augmentation operation, wherein the data augmentation operation comprises at least one of rotation, translation, scaling, flipping and data fusion.

[0023] According to some exemplary embodiments, the training data comprises a three-dimensional region of interest, and the data fusion comprises fusing the three-dimensional region of interest with a security product or a security package.

[0024] According to some exemplary embodiments, the three-dimensional region of interest is obtained by a first CT device, and the security product or the security package is obtained by a second CT device, wherein the first CT device and the second CT device are the same or different devices.

[0025] In another aspect, a target identification method for three-dimensional CT data is provided, which comprises: obtaining three-dimensional CT data; performing primary identification on the three-dimensional CT data; in response to a three-dimensional region of interest not containing the target in a result of the primary identification, taking the whole or part of the three-dimensional CT data as an input of a three-dimensional target identification method; and re-identifying the whole or part of the three-dimensional CT data by using the three-dimensional target identification method to obtain a three-dimensional identification result of at least one of N targets, wherein N is a positive integer greater than or equal to 1.

[0026] According to some exemplary embodiments, the whole or a partial region of the three-dimensional CT data is taken as an input of a three-dimensional target recognition method, specifically including: extracting a plurality of candidate regions from the three-dimensional CT data; taking at least one of the plurality of candidate regions as an input of a three-dimensional target recognition method, or taking the whole or a partial region of the three-dimensional CT data as an input of a three-dimensional target recognition method, specifically including: taking the whole of the three-dimensional CT data as an input of a three-dimensional target recognition method.

[0027] In yet another aspect, a device for target recognition of three-dimensional CT data is provided, the device comprising: an acquisition module configured to acquire three-dimensional CT data; a preliminary recognition module configured to perform preliminary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1; a first extraction module configured to extract a three-dimensional region of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; an input module configured to take the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as an input of a three-dimensional target recognition method; and a re-recognition module configured to perform re-recognition on the three-dimensional region of interest using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0028] In yet another aspect, a device for target recognition of three-dimensional CT data is provided, the device comprising: an acquisition module configured to acquire three-dimensional CT data; a preliminary recognition module configured to perform preliminary recognition on the three-dimensional CT data; an input module configured to take the whole or a partial region of the three-dimensional CT data as an input of a three-dimensional target recognition method in response to the result of the preliminary recognition not including a three-dimensional region of interest of the target; and a re-recognition module configured to perform re-recognition on the whole or the partial region of the three-dimensional CT data using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, where N is a positive integer greater than or equal to 1.

[0029] In yet another aspect, an electronic device is provided, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method as described above.

[0030] In yet another aspect, there is provided a radiation scanning detection system, comprising: a radiation source configured to emit radiation; a detector configured to receive radiation emitted from the radiation source and passing through a detection object; and a data processing device communicatively connected with the detector, wherein the data processing device comprises: one or more processors; and a memory storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method as described above.

[0031] In yet another aspect, there is provided a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method as described above.

[0032] In yet another aspect, there is provided a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0033] For a better understanding of the present disclosure, the present disclosure will be described in detail with reference to the following drawings:

[0034] Fig. 1 is a structural schematic diagram of a radiation scanning detection system according to some exemplary embodiments of the present disclosure.

[0035] Fig. 2 is a schematic diagram of a CT scanning device implementing a reconstruction scanning mode according to some exemplary embodiments of the present disclosure.

[0036] Fig. 3 shows an example of attenuation signal data obtained according to an embodiment of the present disclosure.

[0037] Fig. 4 is a schematic flowchart of a target identification method for three-dimensional CT data according to an embodiment of the present disclosure.

[0038] Fig. 5 is a schematic flowchart of a preliminary identification operation in a target identification method for three-dimensional CT data according to an embodiment of the present disclosure.

[0039] Fig. 6 is a schematic flowchart of a re-identification operation in a target identification method for three-dimensional CT data according to an embodiment of the present disclosure.

[0040] Fig. 7 is an architecture diagram of a cascaded deep learning network used in a target identification method according to an exemplary embodiment of the present disclosure.

[0041] Fig. 8 is a schematic flowchart of a target identification method for three-dimensional CT data according to another embodiment of the present disclosure.

[0042] Fig. 9 is a structural block diagram of a target identification device for three-dimensional CT data according to an exemplary embodiment of the present disclosure.

[0043] FIG. 10 is a structural block diagram of a target recognition apparatus for three-dimensional CT data according to further exemplary embodiments of the present disclosure.

[0044] FIG. 11 schematically illustrates a structural block diagram of an electronic device suitable for implementing the target recognition method according to exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] Specific embodiments of the present disclosure will be described in detail below, it should be noted that the embodiments described herein are only for illustration and do not limit the present disclosure. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent to one of ordinary skill in the art that the present disclosure can be practiced without necessarily employing these specific details. In other instances, well-known structures, materials or methods have not been specifically described in detail in order to avoid obscuring the present disclosure.

[0046] Throughout the specification, reference to "one embodiment", "an embodiment", "one example" or "an example" means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the disclosure. Therefore, the appearance of the phrases "in one embodiment", "in an embodiment", "one example" or "an example" in various places throughout the specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics can be combined in any suitable

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The use of the terms "including", "comprising", and the like, are meant to encompass the items listed thereafter, but do not preclude the addition or

[0048] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal way.

[0049] In the following, embodiments of the present disclosure are described in detail taking CT scanning detection as an example. It should be understood that embodiments of the present disclosure are not limited to the CT scanning detection scenario, and can be applied to various scanning detection scenarios. For example, it can be applied to scanning detection scenarios including various different inspection objects, including but not limited to vehicle scanning detection, luggage / packaging scanning detection, human body scanning detection, etc. It should be noted that the description of the scanning detection scenario here is not an exhaustive description, and the description here should not be understood as limiting the protection scope of the present disclosure.

[0050] It should be noted that embodiments of the present disclosure are not limited to the target detection task, and can also be applied to target classification, image segmentation, etc. Thus, in this paper, unless otherwise stated, the expression "target recognition" includes but is not limited to various tasks such as target detection, target classification, image segmentation, etc.

[0051] It should also be noted that in this paper, the expression "target" can include but is not limited to, for example, target objects of interest, items of interest, etc. in a scanning detection scenario.

[0052] FIG. 1 is a structural schematic diagram of a ray scanning detection system according to some exemplary embodiments of the present disclosure. In FIG. 1, a CT scanning device is schematically shown as an example of the ray scanning detection system. As shown in FIG. 1, the CT scanning device according to the present embodiment includes a gantry 20, a carrying mechanism 40, a controller 50, a data processing device 60 (such as a computer), etc. The gantry 20 includes a ray source 10, such as an X-ray machine, which emits X-rays for inspection, and a detection and acquisition device 30. The carrying mechanism 40 carries a detection object (such as inspected luggage) 70 to pass through a scanning area between the ray source 10 and the detection and acquisition device 30 of the gantry 20, while the gantry 20 rotates around the advancing direction of the detection object 70, so that the rays emitted by the ray source 10 can penetrate the detection object 70, and the detection object 70 is subjected to CT scanning. The detection and acquisition device 30, for example, includes a detector and a data acquisition device having a monolithic module structure, such as a flat panel detector, for detecting rays transmitted through the detection object 70 (such as inspected luggage), obtaining analog signals, and converting the analog signals into digital signals, thereby outputting projection data of the detection object 70 against X-rays. The controller 50 is used to control the synchronous operation of various parts of the entire system. The data processing device 60 is used to process the data collected by the data acquisition device, process and reconstruct the data, and output the results.

[0053] As shown in FIG. 1, a ray source 10 is disposed on one side where an object to be detected can be placed, and a detection and acquisition device 30 is disposed on the other side of the detection object 70, which includes a detector and a data acquisition device, and is used to acquire transmission data and / or multi-angle projection data of the detection object 70. The data acquisition device includes a data amplification and shaping circuit therein, which can work in an (current) integration mode or a pulse (counting) mode. A data output cable of the detection and acquisition device 30 is connected with a controller 50 and a data processing device 60, and the collected data is stored in the data processing device 60 according to a trigger command.

[0054] In an embodiment of the present disclosure, a ray scanning detection module can be deployed in the data processing device 60, and an image recognition model or a target recognition model is deployed in the ray scanning detection module. The ray scanning detection module can use the image recognition model or the target recognition model to recognize the collected field data (for example, a ray scanning image), and the recognition can include target detection, for example, detecting a target object or an object of interest in the ray scanning image. For example, the target object or the object of interest can be various kinds of contraband. It should be understood that the specific category of the target object or the object of interest depends on the scanning detection field where the ray scanning detection system is arranged, or in other words, the specific category of the target object or the object of interest is determined by a user of the ray scanning detection system (also referred to as a user in an embodiment of the present disclosure) according to specific scanning detection needs, which can be dynamically adjusted according to scanning detection needs.

[0055] It should be noted that, for the convenience of description, the target object or the object of interest herein can be collectively referred to as a target.

[0056] In an embodiment of the present disclosure, the ray source 10 can be an X-ray machine, and a suitable X-ray machine focal point size can be selected according to the resolution of imaging. In other embodiments, an X-ray machine can not be used, and an X-ray beam can be generated by using a linear accelerator or the like.

[0057] The detection and acquisition device 30 includes an X-ray detector and a data acquisition circuit, etc. The X-ray detector can use a solid detector, or can use a gas detector or other detectors, and embodiments of the present disclosure are not limited thereto. The data acquisition circuit includes a readout circuit, an acquisition trigger circuit, a data transmission circuit, etc.

[0058] The combination of the controller 50 and the data processing device 60, for example, includes a computer device installed with a control program and a data processing program, which is responsible for completing the control of the operation process of the CT scanning device, including mechanical rotation, electrical control, safety interlocking control, etc., reconstructing a CT image from projection data, training an image recognition model or a target recognition model, and recognizing a ray scanning image by using the trained image recognition model or target recognition model, etc.

[0059] FIG. 2 is a schematic diagram of a CT scanning device implementing a reconstruction scanning mode, according to some example embodiments of the present disclosure. As shown in FIG. 2, a portion of a detection object 70 in a transmission field of view 45 emitted from a radiation source 10 is detected and collected by a detection device 30, converted into an electrical signal and further converted into a digital signal representing an attenuation value as projection data, and reconstructed by a computer to obtain a high-quality image.

[0060] According to an embodiment of the present disclosure, a CT scan is performed on a detection object by using the CT scanning device described above to obtain raw attenuation signals. The attenuation signal data corresponding to a row of detectors can also be displayed in the form of a two-dimensional image. FIG. 3 shows an example of attenuation signal data obtained according to an embodiment of the present disclosure. As shown in FIG. 3, the horizontal axis direction of the raw attenuation signal represents a sequence of detector column directions (e.g., from 1 to 256), and the vertical axis represents an angle (e.g., from 1 degree to 360 degrees). The raw attenuation signal is pre-processed to become projection data. For example, the projection data can be pre-processed by the CT scanning device, such as by performing a negative logarithmic transformation, to obtain the projection data. Then, the data processing device 60 executes a reconstruction program, for example, a mainstream analytical reconstruction method in the related art, to obtain a reconstructed image, which is the radiation scanning image mentioned in the embodiments of the present disclosure.

[0061] In an embodiment of the present disclosure, the data processing device 60 can perform target recognition (e.g., target detection) on the reconstructed image (i.e., the radiation scanning image) described above to identify a target (e.g., contraband) in the radiation scanning image. Specifically, a radiation scanning detection module can be deployed in the data processing device 60, and an image recognition model or a target recognition model can be deployed in the radiation scanning detection module. The radiation scanning detection module can use the image recognition model or the target recognition model to recognize the radiation scanning image.

[0062] The applicant has found through research that three-dimensional CT data usually contains a large number of voxels, is sparse, and also contains certain noise and redundant information. Identifying a three-dimensional CT target can face the problem of a huge amount of computation. In a security check scenario, it is difficult to identify a target that has a strong three-dimensional shape feature but has a complex material composition and physical properties. At the same time, the real-time performance and accuracy of three-dimensional CT data target recognition in a security check scenario are also relatively high.

[0063] To solve the above problems, the applicant proposes a security CT target recognition method (see the published patent of the same applicant as this disclosure, the authorized publication number of which is CN113792623B, and the invention name is "security CT target recognition method and device", hereinafter, the patent can be referred to as patent document 1). In this security CT target recognition method, a plurality of two-dimensional views are generated by reducing the dimensionality of three-dimensional CT data, and then target recognition is performed on the plurality of two-dimensional views to obtain a two-dimensional semantic description set of the target, and the obtained two-dimensional semantic description set is upgraded, and finally the three-dimensional recognition result of the target is obtained. The method realizes effective recognition of the security CT target. In this security CT target recognition method, the method of reducing dimensionality from three-dimensional data to two-dimensional recognition and then upgrading to generate three-dimensional results is proposed, which can not only reduce the computational complexity of recognition, but also use the two-dimensional target recognition technology with higher accuracy and better performance to recognize the target. It can effectively identify the target with complex material composition and physical properties and shape features, and also give three-dimensional recognition results with rich information.

[0064] The applicant has further found that in the above method, using a limited number of two-dimensional views for target recognition can result in insufficient three-dimensional feature expression, and some three-dimensional shape features, spatial position information, etc. may be lost. At the same time, since the three-dimensional probability map is the result of effective simplification and abstraction, the method of extracting features from the three-dimensional probability map to obtain the three-dimensional target recognition result in the above method cannot be applied to the original three-dimensional CT data. Compared with traditional two-dimensional images, three-dimensional CT data provides more comprehensive and three-dimensional information. If the original three-dimensional CT data and the three-dimensional image semantic description set obtained by related technologies can be effectively used to further recognize the target, the identification accuracy can be further improved.

[0065] Therefore, the embodiments of the present disclosure propose a target recognition method for three-dimensional CT data, which can effectively utilize the feature expression of three-dimensional objects and further improve the recognition accuracy. In the method provided by the embodiments of the present disclosure, the three-dimensional CT data is first recognized to obtain a three-dimensional semantic description set of the target, and then a three-dimensional target recognition method is cascaded, i.e. the obtained three-dimensional semantic description set and the corresponding three-dimensional region of interest are taken as input to further recognize the three-dimensional region of interest. In addition, if the three-dimensional region of interest of the target is not included in the pre-obtained three-dimensional recognition result, the method provided by the embodiments of the present disclosure can also be processed, and the whole or part of the three-dimensional CT data can be taken as input of the cascaded three-dimensional target recognition method for recognition.

[0066] For example, some example embodiments of the present disclosure provide a target recognition method for three-dimensional CT data, the method comprising: obtaining three-dimensional CT data; performing primary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1; extracting a three-dimensional region of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; taking the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as input of a three-dimensional target recognition method; and performing re-recognition on the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets.

[0067] In this embodiment, after obtaining the three-dimensional image semantic description set and the three-dimensional region of interest by using the primary recognition method, the cascaded three-dimensional target recognition method is used for further recognition, the three-dimensional information of the CT data is effectively utilized, the contraband and the safety goods can be further distinguished, and the target recognition accuracy is improved.

[0068] For another example, some example embodiments of the present disclosure also provide a target recognition method for three-dimensional CT data, the method comprising: obtaining three-dimensional CT data; performing primary recognition on the three-dimensional CT data; taking the whole or part of the three-dimensional CT data as input of a three-dimensional target recognition method in response to the three-dimensional region of interest of the target not being included in the result of the primary recognition; and performing re-recognition on the whole or part of the three-dimensional CT data by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of N targets, where N is a positive integer greater than or equal to 1.

[0069] In this embodiment, if the three-dimensional region of interest of the target is not included in the three-dimensional recognition result obtained by using the primary recognition method, the method can still be processed, the whole or part of the three-dimensional CT data can be taken as input of the cascaded three-dimensional target recognition method for recognition, and the result is used as a basis for further judgment, which is beneficial to expand the application scenarios of the target recognition method.

[0070] FIG. 4 is a schematic flowchart of a target recognition method for three-dimensional CT data according to an embodiment of the present disclosure. For example, referring to FIG. 4, the target recognition method for three-dimensional CT data can comprise the following operations or steps S410-S450.

[0071] Referring to FIG. 4, in operation or step S410, three-dimensional CT data is obtained.

[0072] With reference to Figs. 1-3, a portion of the detection object 70 in the X-ray transmission field of view 45 emitted from the X-ray source 10 is received by the detection and acquisition device 30, converted into an electrical signal and further converted into a digital signal representing an attenuation value as projection data; then, a three-dimensional reconstruction image can be obtained by using a mainstream analytical reconstruction method in the related technical field.

[0073] In an embodiment of the present disclosure, the three-dimensional CT data can include the three-dimensional reconstruction image.

[0074] With reference to Fig. 4, in operation or step S420, primary identification is performed on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1.

[0075] Fig. 5 is a schematic flowchart of a primary identification operation in a target identification method for three-dimensional CT data according to an embodiment of the present disclosure. By way of example, with reference to Fig. 5, the primary identification operation can include the following sub-operations or sub-steps S510-S530.

[0076] With reference to Fig. 5, in sub-operation or sub-step S510, dimension reduction processing is performed on the three-dimensional CT data to generate a plurality of two-dimensional views.

[0077] Specifically, first, a plurality of directions can be set for the three-dimensional CT data. Here, the plurality of directions are arbitrary directions, and are not limited to specific directions such as the orthogonal direction of the object travel direction in the detection process. In addition, while or before or after the plurality of directions are set, optionally, a certain preprocessing operation of the three-dimensional body data can be performed, for example, filtering of invalid voxels, pre-computation of geometric parameters required for projection or rendering, so as to improve the subsequent processing speed. Second, projection or rendering can be performed according to the plurality of directions to obtain a plurality of two-dimensional views. As an example, ray or light ray casting can be performed based on a CT image slice sequence, a ray or light ray is emitted from each pixel of the image along a certain direction, the ray or light ray passes through the entire image sequence, and in this process, the image sequence is sampled to obtain attribute or color information, and the attribute or color value is accumulated according to a certain model, until the ray or light ray passes through the entire image sequence, and finally the attribute or color value is obtained as the two-dimensional view after dimension reduction.

[0078] In this embodiment, by obtaining the two-dimensional reduced dimension views through projection in any direction, the problem of reducing dimension along a specific direction, for example, along the orthogonal direction of the running direction of the object in the detection process, can be solved, thereby solving the following problems caused by reducing dimension along a specific direction: (1) in some object poses, the area of the object after dimension reduction is too small, and the shape information is not complete, so that accurate recognition cannot be performed; (2) the object loses the shape information due to being blocked by other objects, so that the target cannot be accurately recognized.

[0079] In the sub-operation or sub-step S520, a two-dimensional image recognition method is used to perform target recognition on the plurality of two-dimensional views to obtain a two-dimensional image semantic description set of N targets.

[0080] Here, the plurality of two-dimensional views can include the plurality of two-dimensional views obtained in the above-mentioned sub-operation S510.

[0081] Specifically, as the target recognition method of the two-dimensional view, at least one of the image processing method for two-dimensional images, the classical machine learning method, and the deep learning method or a combination thereof can be used. For example, the two-dimensional view is input into a neural network model as input, and a two-dimensional semantic description set is obtained as output.

[0082] Specifically, a target two-dimensional position detection can be performed using a target detection neural network based on deep learning. The convolutional neural network used in the target detection task is a typical structure of deep learning in computer vision tasks. Such a convolutional neural network has the characteristics of local connection, weight sharing, and spatial resampling. These characteristics make the convolutional neural network have a certain degree of translation and scaling invariance. Here, the two-dimensional semantic description set includes class information and / or confidence in units of pixels, regions of interest, and / or two-dimensional images. Alternatively, the two-dimensional semantic description set includes at least one of class information, confidence, and target position information in units of regions of interest and / or two-dimensional images. The class information represents the class to which the target belongs, such as a gun, a knife, etc. The position information can include center coordinates, a bounding box, etc. The confidence represents the possibility of the existence of the target, which can be a normalized scalar or vector.

[0083] In other words, the two-dimensional semantic description set can include at least one of the following information: class information, confidence, etc. of a certain pixel; class information, position information, and confidence, etc. of a certain region of interest; and class information, position information, and confidence, etc. of a certain two-dimensional image. The at least one information can be information included in a group, or information included in different groups respectively.

[0084] In addition, the two-dimensional semantic information description set can further include other semantic information such as a pose of the target and a number of the target in addition to the category information, the confidence, and the position information.

[0085] It should be noted that in the embodiments of the present disclosure, the two-dimensional image recognition method for the two-dimensional view is not particularly limited, as long as the method can obtain the two-dimensional semantic description set based on the two-dimensional view.

[0086] In the sub-operation or sub-step S530, the two-dimensional image semantic description geometry is upgraded to obtain a first three-dimensional image semantic description set of the N targets.

[0087] In the embodiments of the present disclosure, in the sub-operation or sub-step S530, first, the two-dimensional semantic description set is mapped to a three-dimensional space by using a back projection method to obtain a three-dimensional probability map.

[0088] It should be understood that the back projection can be considered as an inverse process of the projection. Alternatively, the back projection process can be implemented by voxel driving or pixel driving, etc. Specifically, the semantic feature matrix can be obtained by voxel driving or pixel driving, and the semantic feature matrix is compressed to obtain the three-dimensional probability map. Here, the voxel driving includes: corresponding each voxel in the three-dimensional CT data to a pixel in each two-dimensional view, querying and accumulating the two-dimensional semantic description information corresponding to the pixel to generate the semantic feature matrix. The voxel-to-pixel correspondence can establish a mapping function or a lookup table to improve the calculation speed. As described above, according to the voxel driving, each voxel in the three-dimensional CT data is traversed to sequentially obtain the semantic feature matrix thereof, and finally the semantic feature matrix is compressed to obtain the three-dimensional probability map.

[0089] In the sub-operation or sub-step S530, then, feature extraction is performed on the three-dimensional probability map to obtain the first three-dimensional image semantic description set of the target.

[0090] For example, the three-dimensional probability map is subjected to feature extraction by using at least one of an image processing method, a classical machine learning method, a deep learning method, or a combination thereof, so as to obtain the three-dimensional image semantic description set.

[0091] As an example, the three-dimensional probability map is input into a deep learning model as an input, and a three-dimensional recognition result including the confidence and a three-dimensional bounding box is output. The deep learning model used herein can adopt a classification neural network with a small number of layers or a target detection network, etc. By using such a technology, the information amount contained in the original three-dimensional CT data is effectively simplified and abstracted after the above steps, which is closer to the final goal of the contraband identification, and a simple feature extraction method can quickly and accurately extract the three-dimensional semantic description set.

[0092] In this embodiment, the three-dimensional image semantic description set can contain, in units of one or more of voxels, three-dimensional regions of interest, and three-dimensional CT images: category information and / or confidence; or the three-dimensional image semantic description set contains, in units of three-dimensional regions of interest and / or three-dimensional CT images: at least one of category information, location information of the target, and confidence. The location information of the target in the three-dimensional CT image can contain a three-dimensional bounding box.

[0093] In other words, the three-dimensional image semantic description set contains: category information of a certain voxel belonging to a target, confidence, etc.; a certain three-dimensional region of interest (VOI) contains category information of a target, location information of a target, confidence, etc.; and a certain three-dimensional CT image contains at least one of category information of a target, location information of a target, and confidence. The at least one information can be information contained in a group, or information contained in different groups respectively.

[0094] Since the three-dimensional image semantic description set is generated from the three-dimensional probability map generated based on the two-dimensional semantic description set, the types of semantic information contained in the three-dimensional image semantic description set and the types of semantic information contained in the two-dimensional semantic description set are consistent or convertible.

[0095] In this embodiment, by the dimensionality increasing processing described above, the two-dimensional semantic description set is dimensionality increased to obtain the three-dimensional recognition result of the target, thereby solving the problem of significant reduction of information amount when only two-dimensional recognition is performed by dimensionality reduction, and being able to reduce the loss of information amount while adopting two-dimensional recognition, and taking into account the real-time performance and accuracy of security inspection.

[0096] It should be noted that the specific implementation of the preliminary identification operation (i.e., the operation S420 described above) or step in the target recognition method for three-dimensional CT data according to the embodiments of the present disclosure can refer to the method described in the patent document 1, that is, the method described in the patent document 1 is wholly or partially incorporated herein as the preliminary identification method in the embodiments of the present disclosure without conflict.

[0097] Referring back to FIG. 4, in operation or step S430, according to the first three-dimensional image semantic description set of the target, a three-dimensional region of interest corresponding to the N targets is extracted from the three-dimensional CT data.

[0098] In the embodiments of the present disclosure, the three-dimensional region of interest can include body data corresponding to a local region where the target is located extracted from the three-dimensional CT data.

[0099] Referring to FIG. 4, in operation or step S440, the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets are taken as inputs of the three-dimensional target recognition method.

[0100] In some example embodiments of the present disclosure, operation or step S440 can specifically include: taking the first three-dimensional image semantic description set of all the N targets and the three-dimensional region of interest of all the N targets as inputs of the three-dimensional target recognition method.

[0101] In some other example embodiments of the present disclosure, operation or step S440 can specifically include: screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets; and taking the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of the n targets satisfying a preset condition as inputs of the three-dimensional target recognition method, where n is a positive integer greater than or equal to 1 and less than N.

[0102] Exemplarily, in operation or step S440, screening the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets can specifically include: according to the category information, screening out the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of n targets judged to belong to a preset category; and / or, according to the confidence, screening out the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of n targets with confidence in a preset interval.

[0103] That is, in some example embodiments of the present disclosure, the first three-dimensional image semantic description set of n targets satisfying a preset condition can include: the first three-dimensional image semantic description set of n targets belonging to a preset category or categories; and / or, the first three-dimensional image semantic description set of n targets with confidence in a preset interval. Additionally or alternatively, the three-dimensional region of interest satisfying a preset condition can include: the three-dimensional region of interest of n targets belonging to a preset category or categories; and / or, the three-dimensional region of interest of n targets with confidence in a preset interval.

[0104] In embodiments of the present disclosure, both supporting taking all the three-dimensional regions of interest and their three-dimensional image semantic description sets obtained in the previous step as inputs of the next stage, and supporting screening the three-dimensional regions of interest or the three-dimensional image semantic description sets, taking part of the data or part of the semantic description sets satisfying certain conditions as inputs of the next stage.

[0105] Referring to FIG. 4, in operation or step S450, the three-dimensional target recognition method is used to re-identify the three-dimensional region of interest to obtain a three-dimensional recognition result of at least one of the N targets.

[0106] FIG. 6 is a schematic flow chart of a re-identification operation in a target recognition method for three-dimensional CT data, according to an embodiment of the present disclosure. Illustratively, referring to FIG. 6, the re-identification operation can include the following sub-operations or sub-steps S610-S650. It should be noted that in FIG. 6, some of the sub-operations or sub-steps S610-S650 are optional operations or steps.

[0107] In some exemplary embodiments of the present disclosure, in the sub-operation or sub-step S620, a pre-trained deep learning network for three-dimensional CT data is utilized for feature extraction to obtain a feature map. That is, the cascaded three-dimensional target recognition method can be a three-dimensional target recognition method based on a deep learning network.

[0108] In an embodiment of the present disclosure, the cascaded three-dimensional target recognition method can recognize the three-dimensional region of interest, support input of various three-dimensional data such as single-channel, multi-channel, and the like, and be compatible with various information including original three-dimensional CT data, the three-dimensional image semantic description set, and the like.

[0109] Optionally, in the sub-operation or sub-step S610, the three-dimensional region of interest can be pre-processed before the operation of feature extraction. The pre-processing method includes but is not limited to operations such as denoising, morphological transformation, geometric transformation, image type conversion, and the like, and supports conversion of three-dimensional volume data into point cloud data. For example, the pre-processing can include format conversion, size adjustment, and the like. For another example, the three-dimensional region of interest can be pre-processed by rotation correction, denoising, and the like to ensure image quality.

[0110] Rotation correction can help correct the tilt of the image. For example, rotation correction of at least one of the first image, the second image, and the third image can be performed by using an image edge contour-based rotation correction method. Specifically, the image edge contour-based rotation correction method can include the following steps: image graying: converting a color image into a gray image for subsequent processing; threshold binarization: converting the gray image into a binary image so that the target and the background in the image have obvious differences; detecting the contour: detecting the edge contour of the target in the binary image; extracting the enclosing matrix of the contour: finding the smallest rectangular region enclosing the contour of the target, and the edge of this rectangular region can be used to estimate the tilt angle of the image; obtaining the deflection angle through the extracted enclosing matrix: calculating the angle between the long side of the enclosing matrix and the horizontal line, and this angle is the tilt angle of the image; and deflection of the image by using affine transformation: rotating the image to a normal angle by using affine transformation according to the calculated tilt angle.

[0111] For example, rotation correction of at least one of the first image, the second image, and the third image can be performed using a rotation correction method based on Fourier transform and Hough line detection. Specifically, the rotation correction method based on Hough line detection can include the following steps: image preprocessing and edge detection: first, the image is preprocessed, such as grayscale and binarization, and then an edge detection algorithm (such as Sobel operator) is used to detect the edges in the image; Hough transform: Hough transform is a technique for detecting lines and curves in an image. Here, it is used to detect straight lines in the image. The basic principle of Hough transform is to use the duality of points and lines to convert the lines in the image space into clustered points in the parameter space, so as to detect whether there is a curve with a given property in the given image; calculate the tilt angle: through Hough transform, the polar coordinate representation (p, q) of each straight line in the image can be obtained. Wherein, q represents the tilt angle of the straight line. The tilt angle of the entire image can be obtained by counting the tilt angles of all detected straight lines and then taking the average; image rotation: according to the calculated tilt angle, the image can be rotated by an image rotation algorithm (such as affine transformation) to correct it; output the corrected image.

[0112] In some exemplary embodiments, the three-dimensional region of interest can be denoised using a filter-based method, a model-based method, or a learning-based method. Denoising is an operation to reduce noise interference in an image or data. In three-dimensional region of interest preprocessing, the goal of denoising is to eliminate or reduce irrelevant information introduced by the acquisition process, sensor noise, or other interference factors. For example, denoising methods can include median filtering, Gaussian filtering, mean filtering, etc., which can smooth the data and remove noise points to improve the accuracy and stability of subsequent processing.

[0113] In some exemplary embodiments, the three-dimensional region of interest can be preprocessed using morphological transformations. Morphological transformations are a class of operations based on image shape and structure, used to change the geometric structure of the image or extract the shape features of interest. In three-dimensional region of interest preprocessing, morphological transformations can be used to remove small spots or isolated noise, fill holes, connect adjacent regions, etc. For example, morphological transformations include dilation, erosion, opening operation, and closing operation, etc.

[0114] In some exemplary embodiments, the three-dimensional region of interest can be preprocessed using geometric transformations. Geometric transformations refer to operations on the position, scale, rotation, or affine transformation of objects in three-dimensional space in order to better match or adjust the image or data. In three-dimensional region of interest preprocessing, geometric transformations can be used to correct or align the data to conform to a specific reference coordinate system or geometric shape. For example, translation, rotation, scaling, mirroring, etc. transformation operations can be performed.

[0115] In some exemplary embodiments, the three-dimensional region of interest can be preprocessed by image type conversion. For example, image type conversion can convert the original three-dimensional volume data into point cloud data. In the preprocessing of the three-dimensional region of interest, such conversion can help to transform the volume data representation into a series of discrete three-dimensional point coordinates for subsequent point cloud processing and analysis. For example, the image type conversion method can include voxelization, surface reconstruction, and sampling, etc.

[0116] Through the preprocessing method, the data quality can be improved, the noise interference can be reduced, the geometric structure can be adjusted, or the three-dimensional volume data can be converted into point cloud data, providing more accurate and reliable input for subsequent three-dimensional recognition and analysis.

[0117] It should be noted that three-dimensional volume data refers to data represented in three-dimensional space with voxels as the basic unit. Voxel can be understood as a pixel in three-dimensional space, each voxel contains attribute or numerical information at a specific location in space. By dividing a three-dimensional object or scene into a series of voxels and assigning attribute values to each voxel, three-dimensional volume data can be constructed. Three-dimensional volume data can be represented in the form of a cube or a cuboid, providing detailed information about the internal and external structure of the object.

[0118] Point cloud data is a dataset composed of a set of discrete three-dimensional point coordinates. Each point contains position information in three-dimensional space, and can also be accompanied by other attributes such as color, normal vector or intensity, etc. Point cloud data can be obtained in different ways, such as laser radar scanning, stereo vision or three-dimensional reconstruction technology. Point cloud data can represent the surface geometry of a three-dimensional object or the distribution of scattered points in a scene, providing a description of the object shape, surface details and spatial position. Point cloud data has wide applications in computer vision, computer graphics, robotics, etc., such as target detection and recognition, three-dimensional reconstruction, environment perception, etc.

[0119] In summary, three-dimensional volume data is three-dimensional data with volume information represented by voxels, while point cloud data is a dataset composed of discrete three-dimensional point coordinates, used to represent the surface geometry of an object or the distribution of scattered points in a scene.

[0120] It should be noted that there are some differences between three-dimensional body data and point cloud data in terms of representation and data structure: (1) representation: three-dimensional body data takes voxel as the basic unit, divides three-dimensional space into regular cubes or cuboids, and each voxel contains attribute or numerical information at a specific location in space. Point cloud data is a data set composed of a set of discrete three-dimensional point coordinates, and each point contains location information in three-dimensional space. (2) Data structure: three-dimensional body data is usually stored in the form of voxel grid, using multi-dimensional array or voxel tree data structure to represent, where the attribute value of each voxel can represent the density, color, material and other information of the object. Point cloud data is stored in the form of point set, and each point can contain additional attributes such as color, normal vector or intensity. (3) Data density: because three-dimensional body data uses regular voxel grid representation, it can provide detailed information about the internal and external structure of the object, including density distribution, shape, etc. Point cloud data is a set of discrete point coordinates, which usually only provides geometric information of the object surface, and the density and distribution of points depend on the method and parameters of collecting or generating point cloud. (4) Data size: because point cloud data only stores discrete point coordinates, the storage space of point cloud data is more compact compared with three-dimensional body data. Therefore, for large-scale three-dimensional scenes or complex objects, point cloud data may be more suitable for storage and processing.

[0121] Referring to FIG. 6, in sub-operation or sub-step S620, the preprocessed three-dimensional data is subjected to feature extraction by using a deep learning network for three-dimensional CT data, specifically, by using a specially designed feature extraction network.

[0122] In embodiments of the present disclosure, the extracted features include but are not limited to shape, texture, color and the like of three-dimensional data. Among them, the shape of three-dimensional data corresponds to the large structure of the target, the texture corresponds to the small structure or detailed structure of the target, and the color corresponds to the physical attributes such as Z value and density of the target.

[0123] In embodiments of the present disclosure, useful feature representations are extracted from input data by neural networks or other methods. These features can include shape, texture, color and other information of objects, which are used to describe and express the key characteristics of input data. That is, in embodiments of the present disclosure, various information of three-dimensional data can be comprehensively utilized, which is conducive to further improving the accuracy of recognition. In the deep learning network for three-dimensional CT data, by using three-dimensional convolutional neural network (3D CNN) or feature extraction method based on attention mechanism, local and global features in input data can be learned, and feature maps with distinguishing and expressive ability can be extracted.

[0124] In some exemplary embodiments, in the sub-operation or sub-step S620, feature extraction can use a three-dimensional convolutional neural network or an attention mechanism-based feature extraction method, etc., which can be used alone or in combination, to extract a feature map of the three-dimensional CT data.

[0125] If the target three-dimensional image semantic description set contains three-dimensional masks, class information and confidence information of voxels, it also supports accurate generation of feature vectors. That is, in the sub-operation or sub-step S620, the three-dimensional masks, class information and confidence contained in the first three-dimensional image semantic description set can be used to generate a feature vector.

[0126] During the training of the feature extraction network, in order to increase the diversity of the data, a variety of data augmentation techniques can be applied, including but not limited to rotation, translation, scaling, flipping, data fusion, etc. Data augmentation helps to improve the robustness and generalization of the network.

[0127] In actual application, the security inspection scene can be very complex, containing packages and items of various shapes, sizes and internal structures. Fusing the three-dimensional region of interest with the security product or security package, by fusing security packages or security products of different shapes, sizes and internal structures, increases the richness of the data, so that the network can learn more extensive feature representations. This enables the network to maintain a high recognition accuracy when processing unseen data, improving the generalization ability of the model.

[0128] When performing data fusion, the three-dimensional region of interest can be fused with the security product or security package of the same CT device, or it can be fused with CT data of different device models. Because in actual application, multiple security inspection CT devices of different models or brands can be encountered, fusing data of different devices can improve the compatibility of the data, reduce the limitations and deficiencies that can exist in a single device, and enable the algorithm and system to adapt to the input of different devices.

[0129] Due to the physical principles of CT device X-ray imaging, the fusion of two three-dimensional CT data is not a simple operation, especially for the fusion of data from different CT devices. Selecting an appropriate fusion method also needs to consider the differences between devices and the consistency requirements of data. To ensure the correctness of the fusion result, some processing of the three-dimensional CT data can be required, including but not limited to one or more of the following.

[0130] (1) Data calibration and registration: Different CT devices or the same CT device at different time points or different imaging techniques may cause inconsistencies in the data, including differences in scanning parameters, resolution, and imaging modes, which may lead to mismatches in space, time, or physical properties. For example, different CT devices may use different spatial resolution parameters, which may include pixel size, slice thickness, etc. Therefore, data calibration or registration may be required when performing data fusion to ensure the correctness of the fused image. For example, methods for data calibration or registration may include registration algorithms, resampling, etc.

[0131] (2) Artifact correction, noise processing, etc.: CT data may have artifacts, noise, or other image distortions during imaging, which can affect the quality and accuracy of the data. When data fusion is performed, it is necessary to consider how to reduce or correct these artifacts and noise to maintain the reliability of the data. For example, methods for reducing or correcting these artifacts and noise may include image correction techniques, filtering algorithms, etc.

[0132] (3) Screening of regions to be fused: For different targets, when performing data fusion, it is sometimes necessary to consider avoiding fusion of the target and objects that severely affect its physical properties. For example, organic substances such as drugs and explosives should be considered for fusion with strong metal interference regions. Therefore, when performing data fusion, it may be necessary to judge certain region or regions of the selected security product or security package to be fused with the three-dimensional region of interest. Judgment methods include methods based on physical properties (setting different thresholds based on prior knowledge to select appropriate fusion regions), methods based on machine learning, etc.

[0133] Before fusing the three-dimensional region of interest with the security product or security package, data augmentation techniques such as random rotation, translation, scaling, etc. can be used to increase the diversity of the data, and then fusion can be performed. During fusion, the three-dimensional region of interest can be fused with the security product obtained from the security package, or the three-dimensional region of interest can be directly fused with certain region or regions of the security package, and then a new three-dimensional region of interest can be extracted based on the corresponding three-dimensional semantic description set (such as three-dimensional bounding box, three-dimensional mask, etc.) of the three-dimensional region of interest. For example, the three-dimensional CT data fusion method can include the following:

[0134] The first method is a voxel-based fusion method, which includes: (1) Fusion according to the physical principle of CT device X-ray imaging to generate fused three-dimensional data. (2) Weighted average method: assign a weight to each voxel in the two CT data, then weight the values of these voxels according to the weight to generate the fused data. The weight can be adjusted according to the quality, reliability and other factors of the data; (3) Maximum / minimum method: select the maximum or minimum value of multiple CT values at each voxel position as the new value after fusion to generate the fused data.

[0135] The second method is a feature-based fusion method, which includes: extracting representative features such as edges, textures, shapes, etc. from the original CT data. Match the extracted features to find the correspondence between different CT data. According to the matched features, the original CT data is fused. This method can preserve more detailed information and has certain robustness to noise and artifacts.

[0136] The third method is a deep learning-based fusion method, which includes: using a deep learning model (such as a convolutional neural network) to extract and encode features from the original CT data; fuse the extracted features to get the fused feature representation, and finally restore the fused features to a three-dimensional CT image through a decoder. This method can automatically learn complex features in the data.

[0137] The choice of fusion method should be determined according to the specific application scenario and task requirements. Different fusion methods have different advantages and applicability, and need to consider factors such as data characteristics, system requirements and algorithm models for trade-off and selection.

[0138] Optionally, in sub-operation or sub-step S630, the extracted feature map can be operated, including but not limited to feature map fusion, feature map dimension reduction, etc. These operations can be selected and combined according to specific tasks and network architectures. The purpose is to improve the performance and generalization ability of the model through appropriate operations and processing of the feature map, so as to better adapt to various complex data and application scenarios.

[0139] Specifically, if the network has multiple branches or multiple levels of feature maps, feature map fusion operations can be performed to obtain more comprehensive and multi-scale feature representations. Fusion operations can use feature map weighted summation, feature map splicing, feature pyramid, Transformer-based feature fusion, etc.

[0140] For example, in the feature map weighted summation fusion method, different levels or sources of feature maps are weighted and summed according to certain weights. Each feature map is multiplied by the corresponding weight and added to obtain the fused feature map. The weight can be learned or manually set.

[0141] For example, in the fusion mode of feature map splicing, feature maps of different levels or sources are spliced in the channel dimension. For example, if the channel numbers of two feature maps are C1 and C2 respectively, the channel number of the spliced feature map is C1+C2. This way can increase the channel dimension of the feature map, so that the network can utilize information of multiple levels or sources at the same time.

[0142] For example, in the fusion mode of feature pyramid, the feature pyramid is realized by extracting feature maps at different scales and fusing them. Different sizes of convolution kernels or different levels of feature maps are used to capture information at different scales. Then these feature maps are fused together to obtain multi-scale feature representation.

[0143] For example, in the feature fusion mode based on Transformer coding and decoding, Transformer is a neural network model based on self-attention mechanism, which can be used to encode and decode sequence data. In feature fusion, the encoder-decoder structure of Transformer can be used to fuse feature maps. The encoder encodes the input feature map into context-aware feature representation, and the decoder decodes these feature representations into the final fused feature map.

[0144] It should be noted that these feature fusion modes can be selected and applied according to specific tasks and network structures. Their goal is to improve the expression ability and performance of the network by effectively combining feature information from different sources.

[0145] Specifically, to improve computational efficiency or reduce feature dimension, feature map dimension reduction operations such as reducing resolution or reducing spatial dimension can be performed on the feature map, for example, the operation mode can include but is not limited to global average pooling, down-sampling, convolution dimension reduction (such as 1x1 convolution, depth separable convolution, etc.) and the like.

[0146] In embodiments of the present disclosure, the deep learning network for three-dimensional CT data can include, but is not limited to, a three-dimensional convolutional neural network, such as 3D U-Net, V-Net, 3D DenseNet, 3D ResNet, etc. It should be noted that 3D U-Net is a three-dimensional convolutional neural network based on U-Net architecture. It has an encoder-decoder structure, extracts and recovers features through stacking convolution, pooling and upsampling operations, and can effectively capture context information at different scales. V-Net is a volumetric segmentation network based on 3D CNN, which is specifically used for medical image segmentation tasks. It introduces techniques such as residual connection and batch normalization, learns feature representations of images through multiple 3D convolution and deconvolution layers, and generates pixel-level segmentation results. 3D DenseNet is a network structure extended to three dimensions on the basis of 2D DenseNet. It connects the feature maps of each layer with all previous layers through dense connection, promotes the transmission of features and the flow of information, and thus improves the performance and stability of the model. 3D ResNet is a three-dimensional convolutional neural network extended from 2D ResNet. It uses the idea of residual connection and deep network, constructs the network by stacking multiple residual blocks, effectively solves the problems of gradient vanishing and model degradation, and improves the feature representation and learning ability.

[0147] FIG. 7 is an architecture diagram of a cascaded deep learning network used in a target recognition method according to an exemplary embodiment of the present disclosure.

[0148] Referring to FIG. 7, the cascaded deep learning network is a three-dimensional convolutional neural network based on a U-Net architecture. Its structure is composed of an encoder and a decoder, which extracts features through the stacking of convolution, pooling and upsampling operations. The cascaded deep learning network includes an encoder part and a decoder part. The encoder part is a typical convolutional network, which includes repeated application of convolution, followed by a ReLU activation function and a max-pooling operation after each convolution. For example, in this network, each layer in the encoder part can include two 3x3x3 convolutions, each followed by a ReLU layer, and then a 2x2x2 max-pooling layer with a stride of 2 in each direction. The decoder part mainly includes upsampling operations and convolution operations. The upsampling operation is used to restore the resolution of the image, while the convolution operation is used to combine the information to generate an accurate output. In the decoder part, each layer contains a 2x2x2 deconvolution layer with a stride of 2, followed by two 3x3x3 convolution layers, each followed by a ReLU layer. The cascaded deep learning network also includes a skip connection for connecting the feature maps of the corresponding layers of the encoder part and the decoder part to provide more rich contextual information. In the output layer of the network, a 1x1x1 convolution layer and an appropriate activation function (such as Sigmoid or Softmax) are used to generate the final result.

[0149] The cascaded deep learning network is characterized by a symmetrical structure between the encoder and the decoder, which can fuse low-level features with high-level features through skip connections, thereby improving the accuracy and detail preservation ability of segmentation. In addition, the depth and width of the network can be adjusted to adapt to different tasks and data sets to improve performance and generalization ability.

[0150] In some exemplary embodiments of the present disclosure, in sub-operation or sub-step S640, the feature map is applied to a classification, segmentation or detection task to obtain a second set of three-dimensional image semantic descriptions. The "feature map" here can be the feature map directly extracted by sub-operation or sub-step S620, or the feature map after being extracted by sub-operation or sub-step S620 and processed by S630 of sub-operation or sub-step S620.

[0151] Specifically, a multi-layer perceptron or other classifier can be used to map the feature map to a class probability distribution to implement a classification task. Additionally or alternatively, methods such as deconvolution, bilinear interpolation, etc. can be used to restore the feature map to the original image size and assign semantic labels to each voxel, thereby implementing voxel-level segmentation. Additionally or alternatively, a target detection head can be used to perform classification and regression operations on the feature map to predict the class and location of the target, generate class information, confidence and location information, etc. to implement a target detection task.

[0152] In some example embodiments of the present disclosure, the second three-dimensional image semantic description set is directly taken as the three-dimensional recognition result in the sub-operation or sub-step S650; or, the first three-dimensional image semantic description set and the second three-dimensional image semantic description set are integrated, and the integrated result is taken as the three-dimensional recognition result.

[0153] It is found through research that the three-dimensional image semantic description set obtained after the re-identification is more accurate compared with the recognition result, which is a modification and improvement of the previous stage semantic description set.

[0154] In some example embodiments of the present disclosure, the second three-dimensional image semantic description set is directly taken as the three-dimensional recognition result in the sub-operation or sub-step S650; or, the first three-dimensional image semantic description set and the second three-dimensional image semantic description set are integrated, and the integrated result is taken as the three-dimensional recognition result.

[0155] For example, in the step of integrating the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, for the category information, for the same target, if the category information in the first three-dimensional image semantic description set is inconsistent with the category information in the second three-dimensional image semantic description set, the category information in the second three-dimensional image semantic description set can be directly used to replace the category information in the first three-dimensional image semantic description set, that is, the newly identified category information is used as the final result; for the position information, if the three-dimensional bounding box in the first three-dimensional image semantic description set is larger than the range of the three-dimensional bounding box in the second three-dimensional image semantic description set, the three-dimensional bounding box in the second three-dimensional image semantic description set with more accurate position can be used as the final position information; for the confidence, if the confidence in the first three-dimensional image semantic description set is inconsistent with the confidence in the second three-dimensional image semantic description set, the confidence in the second three-dimensional image semantic description set can be directly used to replace the confidence in the first three-dimensional image semantic description set, that is, the newly identified confidence is used; or, the confidence in the first three-dimensional image semantic description set and the confidence in the second three-dimensional image semantic description set can be respectively given different weights, and then weighted summation is performed, and the confidence obtained after the weighted summation is taken as the final confidence, for example, the weight of the confidence in the first three-dimensional image semantic description set can be smaller than the weight of the confidence in the second three-dimensional image semantic description set.

[0156] FIG. 8 is a schematic flowchart of a target recognition method for three-dimensional CT data according to some embodiments of the present disclosure. Referring to FIG. 8, the target recognition method for three-dimensional CT data can include the following operations or steps S810-S840. It should be noted that in the following, the embodiments described with reference to FIG. 8 differ from the above-described embodiments in the parts emphasized below, and the steps or operations described in the above-described embodiments can be combined with the embodiments described in the following in the case of no conflict.

[0157] Referring to FIG. 8, in operation or step S810, three-dimensional CT data is acquired.

[0158] In operation or step S820, the three-dimensional CT data is preliminarily recognized.

[0159] In operation or step S830, in response to the three-dimensional region of interest not including the target in the result of the preliminary recognition, the entire or partial region of the three-dimensional CT data is input into the three-dimensional target recognition method.

[0160] In some exemplary embodiments, inputting the entire or partial region of the three-dimensional CT data into the three-dimensional target recognition method specifically includes inputting the entire three-dimensional CT data into the three-dimensional target recognition method.

[0161] In some exemplary embodiments, inputting the entire or partial region of the three-dimensional CT data into the three-dimensional target recognition method specifically includes extracting a plurality of candidate regions from the three-dimensional CT data, and inputting at least one of the plurality of candidate regions into the three-dimensional target recognition method.

[0162] In this embodiment, the three-dimensional region extraction method can be used to extract candidate regions from the entire three-dimensional CT data, and then the candidate regions are recognized one by one. The three-dimensional region extraction method includes but is not limited to 3D Region Proposal Network (3D RPN), sliding window (such as sliding window), etc. Alternatively, the entire three-dimensional CT data can be directly recognized, i.e., without extracting three-dimensional candidate regions, the entire three-dimensional CT data is directly recognized using a cascaded three-dimensional target recognition method.

[0163] In operation or step S840, the entire or partial region of the three-dimensional CT data is re-recognized using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets, wherein N is a positive integer greater than or equal to 1.

[0164] In some embodiments of the present disclosure, if the three-dimensional region of interest of the target is not included in the three-dimensional recognition result obtained by the preliminary recognition method, the method provided by the embodiments of the present disclosure can still be processed. At this time, the whole or part of the three-dimensional CT data can be taken as the input of the cascaded deep learning network used in the re-recognition method, and the result thereof can be taken as the basis for further discrimination.

[0165] It should be noted that, in the embodiments of the present disclosure, in the re-recognition method, in addition to using the above deep learning-based network, other image processing and machine learning methods can also be used for further recognition of the three-dimensional region of interest. For example, the features of the three-dimensional region of interest can be extracted by using three-dimensional edge detection, texture analysis, feature descriptor and the like, and then the extracted features are input into a machine learning classifier (such as a support vector machine, a random forest, etc.) for recognition.

[0166] Based on the above method, the embodiments of the present disclosure further provide a target recognition device for three-dimensional CT data. The device will be described in detail below in combination with FIG. 9.

[0167] FIG. 9 is a structural block diagram of a target recognition device for three-dimensional CT data according to an exemplary embodiment of the present disclosure. Referring to FIG. 9, the device 900 includes an acquisition module 910, a preliminary recognition module 920, a first extraction module 930, an input module 940 and a re-recognition module 950.

[0168] The acquisition module 910 is configured to acquire three-dimensional CT data. In some exemplary embodiments, the acquisition module 910 can be configured to perform the operation S410 and its sub-operations described above, which will not be repeated here.

[0169] The preliminary recognition module 920 is configured to perform preliminary recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, where N is a positive integer greater than or equal to 1. In some exemplary embodiments, the preliminary recognition module 920 can be configured to perform the operation S420 and its sub-operations described above, which will not be repeated here.

[0170] Exemplarily, the preliminary recognition module 920 is configured to: perform dimension reduction processing on the three-dimensional CT data to generate a plurality of two-dimensional views; perform target recognition on the plurality of two-dimensional views by using a two-dimensional image recognition method to obtain a two-dimensional image semantic description set of N targets; and perform dimension increase on the two-dimensional image semantic description to obtain the first three-dimensional image semantic description set of the N targets.

[0171] The first extraction module 930 is configured to extract, from the three-dimensional CT data, a three-dimensional region of interest corresponding to each of the N targets according to the first three-dimensional image semantic description set of the targets. In some exemplary embodiments, the first extraction module 930 can be configured to perform the operation S430 and sub-operations thereof as described above, and details are not repeated here.

[0172] The input module 940 is configured to input, as inputs of the three-dimensional target recognition method, the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets. In some exemplary embodiments, the input module 940 can be configured to perform the operation S440 and sub-operations thereof as described above, and details are not repeated here.

[0173] In some embodiments, the input module 940 is configured to input, as inputs of the three-dimensional target recognition method, the first three-dimensional image semantic description set of all the N targets and the three-dimensional region of interest of all the N targets.

[0174] In other embodiments, the input module 940 is configured to: filter the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets; and input, as inputs of the three-dimensional target recognition method, the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of the n targets that satisfy a preset condition, where n is a positive integer greater than or equal to 1 and less than N.

[0175] For example, filtering the first three-dimensional image semantic description set of the N targets and / or the three-dimensional region of interest of the N targets specifically includes: according to the category information, filtering out the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of the n targets that are judged to belong to a preset category; and / or, according to the confidence, filtering out the first three-dimensional image semantic description set of n targets and / or the three-dimensional region of interest of the n targets that have a confidence within a preset interval.

[0176] The re-identification module 950 is configured to re-identify the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets. In some exemplary embodiments, the re-identification module 950 can be configured to perform the operation S450 and sub-operations thereof as described above, and details are not repeated here.

[0177] In some embodiments, the re-identification module 950 is configured to perform feature extraction by using a deep learning network pre-trained for three-dimensional CT data to obtain a feature map. In particular, a feature vector can be generated by using the three-dimensional mask, the category information and the confidence included in the first three-dimensional image semantic description set.

[0178] Exemplarily, the extracted features include at least one of shape, texture and color in the three-dimensional CT data.

[0179] Optionally, the re-identification module 950 is further configured to operate the feature map by using the deep learning network, wherein the operation includes at least one of feature map fusion and feature map dimension reduction.

[0180] Exemplarily, the re-identification module 950 is further configured to apply the feature map to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

[0181] In some embodiments, the re-identification module 950 is configured to directly use the second three-dimensional image semantic description set as the three-dimensional identification result.

[0182] In some other embodiments, the re-identification module 950 is configured to integrate the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, and use the integrated result as the three-dimensional identification result.

[0183] In embodiments of the present disclosure, the second three-dimensional image semantic description set contains class information and / or confidence, in units of one or more of voxels, three-dimensional regions of interest and three-dimensional CT images; or the second three-dimensional image semantic description set contains at least one of class information, confidence and position information of the target, in units of three-dimensional regions of interest and / or three-dimensional CT images.

[0184] Based on the above method, embodiments of the present disclosure further provide a device for target identification of three-dimensional CT data. The device will be described in detail below in combination with FIG. 10.

[0185] FIG. 10 is a structural block diagram of a device for target identification of three-dimensional CT data according to some other exemplary embodiments of the present disclosure. Referring to FIG. 10, the device 1000 includes an acquisition module 1010, a primary identification module 1020, an input module 1030 and a re-identification module 1040.

[0186] The acquisition module 1010 is configured to acquire three-dimensional CT data. In some exemplary embodiments, the acquisition module 1010 can be configured to perform the operation S410 or S810 and the sub-operations thereof described above, which will not be repeated here.

[0187] The primary identification module 1020 is configured to perform primary identification on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1. In some exemplary embodiments, the primary identification module 1020 can be configured to perform the operation S420 or S820 and the sub-operations thereof described above, which will not be repeated here.

[0188] The input module 1030 is configured to input the whole or a partial region of the three-dimensional CT data as an input of the three-dimensional target identification method, in response to the three-dimensional region of interest not including the target in the result of the initial identification. In some example embodiments, the input module 1030 can be configured to perform the operation S830 and sub-operations thereof described above, and details are not repeated here.

[0189] The re-identification module 1040 is configured to re-identify the whole or a partial region of the three-dimensional CT data by using the three-dimensional target identification method, to obtain a three-dimensional identification result of at least one of the N targets, where N is a positive integer greater than or equal to 1. In some example embodiments, the re-identification module 1040 can be configured to perform the operation S840 and sub-operations thereof described above, and details are not repeated here.

[0190] According to embodiments of the present disclosure, any one or more of the acquisition module 910, the initial identification module 920, the first extraction module 930, the input module 940, and the re-identification module 950 included in the apparatus 900, and any one or more of the acquisition module 1010, the initial identification module 1020, the input module 1030, and the re-identification module 1040 included in the apparatus 1000, and sub-modules thereof, can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to embodiments of the present disclosure, at least one of the acquisition module 910, the initial identification module 920, the first extraction module 930, the input module 940, and the re-identification module 950 included in the apparatus 900, and at least one of the acquisition module 1010, the initial identification module 1020, the input module 1030, and the re-identification module 1040 included in the apparatus 1000, and sub-modules thereof, can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging circuits, etc. hardware or firmware, or any one of software, hardware, and firmware or any appropriate combination of any of them. Alternatively, at least one of the acquisition module 910, the initial identification module 920, the first extraction module 930, the input module 940, and the re-identification module 950 included in the apparatus 900, and at least one of the acquisition module 1010, the initial identification module 1020, the input module 1030, and the re-identification module 1040 included in the apparatus 1000, and sub-modules thereof, can be at least partially implemented as computer program modules that can perform corresponding functions when the computer program modules are run.

[0191] FIG. 11 schematically illustrates a structural block diagram of an electronic device suitable for implementing the object recognition method according to exemplary embodiments of the present disclosure.

[0192] As shown in FIG. 11, the electronic device 1100 according to embodiments of the present disclosure includes a processor 1101 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1102 or loaded into a random access memory (RAM) 1103 from a storage section 1108. The processor 1101 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and the like. The processor 1101 can also include on-board memory for cache use. The processor 1101 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0193] For example, the electronic device can be the data processing device 60 of the ray scanning detection system.

[0194] In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are stored. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other by a bus 1104. The processor 1101 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1102 and / or the RAM 1103. It should be noted that the programs can also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0195] According to an embodiment of the present disclosure, the electronic device 1100 can further include an input / output (I / O) interface 1105 that is also connected to the bus 1104. The electronic device 1100 can further include one or more of the following components connected to the I / O interface 1105: an input part 1106 including, for example, a keyboard and a mouse; an output part 1107 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage part 1108 including, for example, a hard disk; and a communication part 1109 including, for example, a LAN card, a modem, and the like. The communication part 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as necessary. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 1110 as necessary, so that a computer program read therefrom is installed in the storage part 1108 as necessary.

[0196] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0197] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, for example, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories such as the ROM 1102 and / or the RAM 1103 described above and / or one or more memory other than the ROM 1102 and the RAM 1103.

[0198] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the present disclosure.

[0199] The above-described functions of the system / device defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiments of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by the computer program modules.

[0200] In one embodiment, the computer program can be stored in a tangible storage medium, such as an optical, magnetic, or other memory on a hard disk drive, solid-state drive, or other storage device. In another embodiment, the computer program can be transmitted over a network, using a wireless or wired communication protocol, and be downloaded and installed by the communication portion 1109 and / or installed from the removable medium 1111. The program code embodied in the computer program can be transmitted using any suitable network medium, including, but not limited to, wireless, wired, or any suitable combination of the above.

[0201] In such an embodiment, the computer program can be downloaded and installed from the network through the communication portion 1109 and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-described functions of the system defined in the embodiments of the present disclosure are performed. According to the embodiments of the present disclosure, the system, apparatus, device, module, unit, etc. described above can be implemented by the computer program modules.

[0202] According to the embodiments of the present disclosure, the program code for execution of the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. The programming language includes, but is not limited to, a programming language such as Java, C++, Python, "C" language, or a similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet using an Internet service provider).

[0203] The computer program product of the present disclosure can include a computer readable storage medium. The computer readable storage medium can include instructions. When executed, the instructions can cause a processor to perform functions of the present disclosure. The computer program product of the present disclosure can include a computer readable storage medium. The computer readable storage medium can include instructions. When executed, the instructions can cause a processor to perform functions of the present disclosure.

[0204] The target identification method and device, the electronic device, the ray scanning detection system, the computer readable storage medium and the program product provided by the embodiments of the present disclosure have at least one of the following beneficial effects:

[0205] After obtaining the three-dimensional CT data, first, preliminary identification is performed, and then a three-dimensional target identification method is cascaded. The three-dimensional image semantic description set obtained through preliminary identification and the corresponding three-dimensional region of interest extracted based on information in the three-dimensional image semantic description set are supported as inputs of the cascaded three-dimensional target identification method. If the three-dimensional region of interest of the target is not included in the three-dimensional identification result obtained through preliminary identification, the whole or part of the three-dimensional CT data is supported as an input of the cascaded three-dimensional target identification method, and the result is used as a basis for further judgment. Through the cascaded three-dimensional target identification method, the feature expression of the three-dimensional object can be enhanced, and the recognition accuracy can be further improved.

[0206] In particular, in the field of security CT, three-dimensional CT data provides more comprehensive, accurate and visualized information, and effective use of three-dimensional CT data for identification of contraband is conducive to improving the accuracy of the identification algorithm, so that potential threats and hidden items can be more accurately identified. In the embodiments of the present disclosure, after obtaining the three-dimensional image semantic description set and the three-dimensional region of interest through preliminary identification, the three-dimensional target identification method is cascaded, the three-dimensional information of the CT data is effectively used, the contraband and safe items can be further distinguished, the recognition accuracy is improved, and the efficiency of the identification algorithm is ensured.

[0207] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A method for object recognition of three-dimensional CT data, comprising: obtaining three-dimensional CT data; performing initial recognition on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N objects, wherein N is a positive integer greater than or equal to 1; extracting a three-dimensional region of interest corresponding to the N objects from the three-dimensional CT data according to the first three-dimensional image semantic description set of the objects; taking the first three-dimensional image semantic description set of at least one of the N objects and the three-dimensional region of interest of at least one of the N objects as input of a three-dimensional object recognition method; and performing re-recognition on the three-dimensional region of interest by using the three-dimensional object recognition method to obtain a three-dimensional recognition result of at least one of the N objects.

2. The method of claim 1, wherein, The first three-dimensional image semantic description set contains class information and / or confidence in units of one or more of voxels, three-dimensional regions of interest and three-dimensional CT images; or The first three-dimensional image semantic description set contains at least one of the class information, the confidence and position information of the object in units of the three-dimensional region of interest and / or the three-dimensional CT image.

3. The method of claim 1 or 2, wherein, The three-dimensional region of interest includes body data corresponding to a local region where the object is located extracted from the three-dimensional CT data.

4. The method of claim 1 or 2, wherein, Taking the first three-dimensional image semantic description set of at least one of the N objects and the three-dimensional region of interest of at least one of the N objects as input of the three-dimensional object recognition method specifically comprises: Taking the first three-dimensional image semantic description set of all the N objects and the three-dimensional region of interest of all the N objects as input of the three-dimensional object recognition method.

5. The method of claim 1 or 2, wherein, Taking the first three-dimensional image semantic description set of at least one of the N objects and the three-dimensional region of interest of at least one of the N objects as input of the three-dimensional object recognition method specifically comprises: screening the first three-dimensional image semantic description set of the N objects and / or the three-dimensional region of interest of the N objects; and taking the first three-dimensional image semantic description set of n objects and / or the three-dimensional region of interest of the n objects meeting a preset condition as input of the three-dimensional object recognition method, wherein n is a positive integer greater than or equal to 1 and less than N.

6. The method of claim 5, wherein, Screening the first three-dimensional image semantic description set of the N objects and / or the three-dimensional region of interest of the N objects specifically comprises: screening the first three-dimensional image semantic description set of n objects and / or the three-dimensional region of interest of the n objects judged to belong to a preset class according to the class information; and / or screening the first three-dimensional image semantic description set of n objects and / or the three-dimensional region of interest of the n objects with a confidence in a preset interval according to the confidence.

7. The method of any one of claims 1, 2, and 6, wherein, The initial recognition on the three-dimensional CT data to obtain the first three-dimensional image semantic description set of the N objects specifically comprises: performing dimension reduction processing on the three-dimensional CT data to generate a plurality of two-dimensional views; performing object recognition on the plurality of two-dimensional views by using a two-dimensional image recognition method to obtain a two-dimensional image semantic description set of N objects; and dimensionalize geometric semantic description of the two-dimensional image to obtain a first three-dimensional image semantic description set of the N targets.

8. The method of any one of claims 1, 2, and 6, wherein, The re-identification of the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets, specifically comprising: Performing feature extraction by using a deep learning network for three-dimensional CT data obtained by pre-training to obtain a feature map.

9. The method of claim 8, wherein, The extracted features include at least one of shape, texture and color in the three-dimensional CT data.

10. The method of claim 8, wherein, The feature extraction by using the deep learning network for three-dimensional CT data obtained by pre-training further specifically comprises: Generating a feature vector by using the three-dimensional mask, class information and confidence contained in the first three-dimensional image semantic description set.

11. The method of claim 8, wherein, The re-identification of the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically comprises: operating the feature map by using the deep learning network, wherein the operation includes at least one of feature map fusion and feature map dimension reduction.

12. The method of claim 8, wherein, The re-identification of the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically comprises: Applying the feature map to a classification, segmentation or detection task to obtain a second three-dimensional image semantic description set.

13. The method of claim 12, wherein, The re-identification of the three-dimensional region of interest by using the three-dimensional target recognition method to obtain a three-dimensional recognition result of at least one of the N targets further specifically comprises: Directly taking the second three-dimensional image semantic description set as the three-dimensional recognition result; or, Integrating the first three-dimensional image semantic description set and the second three-dimensional image semantic description set, and taking the integrated result as the three-dimensional recognition result.

14. The method of claim 12 or 13, wherein, The second three-dimensional image semantic description set contains class information and / or confidence in units of one or more of voxels, three-dimensional regions of interest and three-dimensional CT images; or The second three-dimensional image semantic description set contains at least one of class information, confidence and position information of the target in units of three-dimensional regions of interest and / or three-dimensional CT images.

15. The method of claim 8, wherein, The pre-training of the deep learning network for three-dimensional CT data includes: Obtaining training data; and Augmenting the training data by using a data augmentation operation, Wherein the data augmentation operation includes at least one of rotation, translation, scaling, flipping and data fusion.

16. The method of claim 15, wherein, The training data includes three-dimensional regions of interest, and the data fusion includes fusion of the three-dimensional regions of interest with security products or security packages.

17. The method of claim 16, wherein, The three-dimensional regions of interest are obtained by a first CT device, and the security products or security packages are obtained by a second CT device, and the first CT device and the second CT device are the same or different devices.

18. A target recognition method for three-dimensional CT data, comprising: Obtaining three-dimensional CT data; Performing preliminary identification on the three-dimensional CT data; in response to the three-dimensional region of interest of the target not being included in the result of the initial identification, inputting the whole or a partial region of the three-dimensional CT data as input of the three-dimensional target identification method; and re-identifying the whole or the partial region of the three-dimensional CT data by using the three-dimensional target identification method to obtain the three-dimensional identification result of at least one of the N targets, wherein N is a positive integer greater than or equal to 1.

19. The method of claim 18, wherein, inputting the whole or a partial region of the three-dimensional CT data as input of the three-dimensional target identification method, specifically comprising: extracting a plurality of candidate regions from the three-dimensional CT data; inputting at least one of the plurality of candidate regions as input of the three-dimensional target identification method, or inputting the whole or a partial region of the three-dimensional CT data as input of the three-dimensional target identification method, specifically comprising: inputting the whole of the three-dimensional CT data as input of the three-dimensional target identification method.

20. An apparatus for target identification of three-dimensional CT data, comprising: an acquisition module configured to acquire three-dimensional CT data; an initial identification module configured to perform initial identification on the three-dimensional CT data to obtain a first three-dimensional image semantic description set of N targets, wherein N is a positive integer greater than or equal to 1; a first extraction module configured to extract three-dimensional regions of interest corresponding to the N targets from the three-dimensional CT data according to the first three-dimensional image semantic description set of the targets; an input module configured to input the first three-dimensional image semantic description set of at least one of the N targets and the three-dimensional region of interest of at least one of the N targets as input of a three-dimensional target identification method; and a re-identification module configured to re-identify the three-dimensional region of interest by using the three-dimensional target identification method to obtain a three-dimensional identification result of at least one of the N targets.

21. An apparatus for target identification of three-dimensional CT data, comprising: an acquisition module configured to acquire three-dimensional CT data; an initial identification module configured to perform initial identification on the three-dimensional CT data; an input module configured to, in response to the three-dimensional region of interest of the target not being included in the result of the initial identification, input the whole or a partial region of the three-dimensional CT data as input of the three-dimensional target identification method; and a re-identification module configured to re-identify the whole or the partial region of the three-dimensional CT data by using the three-dimensional target identification method to obtain the three-dimensional identification result of at least one of the N targets, wherein N is a positive integer greater than or equal to 1.

22. An electronic device, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-19.

23. A ray scanning detection system, comprising: a ray source configured to emit rays; a detector configured to receive rays emitted from the ray source and passing through a detection object; and a data processing apparatus in communication connection with the detector, ​ The data processing apparatus comprises one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-19.

24. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-19.

25. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-19.

Images