Liquid identification method based on security check X-ray machine multi-mode fusion image

Abstract

The invention provides a liquid identification method based on a security check X-ray machine multi-mode fusion image, and the method comprises the steps: scanning a container containing to-be-detected liquid, and obtaining a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; determining a first type of cross region corresponding to the container in the pseudo-color X-ray image; determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; determining a second type of cross region in the high-energy X-ray image, a third type of cross region in the low-energy X-ray image and a fourth type of cross region in the effective atomic number image based on the first type of cross region; determining a multi-modal fusion image based on the second type of cross region, the third type of cross region and the fourth type of cross region; and identifying the liquid type of the to-be-detected liquid based on the multi-modal fusion image. The accuracy of the liquid type identification result can be improved.

Description

Technical Field

[0001] The present invention relates to the field of security inspection technology, and in particular to a liquid identification method based on multi-modal fusion images of a security inspection X-ray machine. Background Art

[0002] In the field of modern safety inspection, liquid detection is an important task, especially in public places such as airports and stations, where rapid and accurate identification of liquid dangerous goods is particularly important.

[0003] In the prior art, high-energy images and low-energy images generated by X-ray irradiation of the target object are obtained, and an atomic number image is generated through the high-energy image and the low-energy image. Then, a liquid target area mask is obtained through a segmentation algorithm, and the high- and low-energy images and atomic number images of the mask area are input into a pre-trained liquid model, thereby achieving the purpose of liquid classification.

[0004] However, since liquids are usually contained in containers, and the shapes of different containers vary greatly, even if bottles of different shapes contain the same type of liquid, the high- and low-energy images and atomic number images obtained will be different. This difference will affect the results of liquid type recognition, thereby causing inaccurate liquid type recognition results. Summary of the invention

[0005] The present invention provides a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine, which is used to solve the defect of inaccurate liquid type identification results in the prior art and achieve the purpose of improving the accuracy of liquid type identification results.

[0006] The present invention provides a liquid identification method based on a multi-modal fusion image of a security inspection X-ray machine, comprising: Scan the container containing the liquid to be tested to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; Determine a first type of cross region corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross region includes a region where a center line of the container is located and a region where a perpendicular bisector of the center line is located; Determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; Based on the first type of cross region, determining a second type of cross region in the high energy X-ray image, a third type of cross region in the low energy X-ray image, and a fourth type of cross region in the effective atomic number image; Determine a multimodal fusion image based on the second-type cross region, the third-type cross region, and the fourth-type cross region; The liquid type of the liquid to be detected is identified based on the multimodal fusion image.

[0007] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the step of determining a first type of cross region corresponding to the container in the pseudo-color X-ray image comprises: Inputting the pseudo-color X-ray image into an image segmentation model to obtain a mask region representing the container in the pseudo-color X-ray image output by the image segmentation model; determining a centerline of the container in the mask area; Extending the center line by a first preset number of pixels in a direction perpendicular to the center line to obtain an area where the center line is located; Extend the perpendicular bisector by a second preset number of pixels in a direction perpendicular to the perpendicular bisector to obtain an area where the perpendicular bisector is located; The first type of cross area is determined based on the area where the center line is located and the area where the perpendicular bisector is located.

[0008] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the pseudo-color X-ray image is input into an image segmentation model to obtain a mask area representing the container in the pseudo-color X-ray image output by the image segmentation model, including: Inputting the pseudo-color X-ray image into the backbone network of the image segmentation model to obtain at least two image features output by the backbone network; Inputting each of the image features into the fusion network of the image segmentation model to obtain fusion features output by the fusion network; Inputting the fused features into the target detection network and the mask extraction network of the image segmentation model respectively, to obtain the container features output by the target detection network and the mask features corresponding to the container output by the mask extraction network; The mask region is obtained based on the container feature and the mask feature.

[0009] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the determining of the center line of the container in the mask area includes: Performing distance transformation on the pixel values ​​of each first pixel point in the mask area to obtain a transformation area, wherein the pixel value of each second pixel point in the transformation area is used to represent the minimum distance between the corresponding first pixel point in the mask area and a target boundary of the mask area, wherein the target boundary is the boundary closest to the corresponding pixel point; The center line is determined based on a second pixel point corresponding to a maximum pixel value in the transformed area.

[0010] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the multimodal fusion image is determined based on the second type of cross area, the third type of cross area and the fourth type of cross area, including: For each third pixel in the second-type cross region, determine a normalized high and low energy coefficient based on a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the third pixel in the third-type cross region; Determine the pixel value of each pixel in the first channel based on the normalized high and low energy coefficients and the number of image data bits corresponding to the effective atomic number image; Determine the pixel value of the corresponding pixel point in the second channel based on the pixel value of each pixel point in the first channel and the number of image data bits corresponding to the effective atomic number image; The multimodal fusion image is determined based on the pixel value of each pixel in the first channel, the pixel value of the corresponding pixel in the second channel, and the pixel value of the corresponding pixel in the effective atomic number image.

[0011] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the normalized high and low energy coefficients are determined based on the pixel value of the third pixel point and the pixel value of the fourth pixel point corresponding to the third pixel point in the third type cross area, including: Determine a first maximum value between a preset value and a pixel value of the third pixel point, wherein the preset value is used to represent a maximum value corresponding to normalization; Determine a second maximum value between the preset value and the pixel value of the fourth pixel point; The normalized high and low energy coefficients are determined based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high energy X-ray image, and the number of image data bits corresponding to the low energy X-ray image.

[0012] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the normalized high and low energy coefficients are determined based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high energy X-ray image, and the number of image data bits corresponding to the low energy X-ray image, including: The normalized high and low energy coefficients are determined based on the following formula (1): (1) in, represents the normalized high and low energy coefficients, A represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the high energy X-ray image, B represents the first maximum value, C represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the low energy X-ray image, and D represents the second maximum value.

[0013] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the liquid type of the liquid to be detected is identified based on the multimodal fusion image, comprising: Inputting the multimodal fusion image into a feature extraction module of an image classification model, dividing the multimodal fusion image into blocks through the feature extraction module, encoding each obtained image block, and performing feature extraction on each encoded image block to obtain each multimodal feature; Each of the multimodal features is input into a classification module of the image classification model to obtain a recognition result of the liquid type output by the classification module.

[0014] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, a container containing a liquid to be inspected is scanned to obtain a pseudo-color X-ray image, comprising: Scan the container containing the liquid to be tested to obtain an original X-ray image; The original X-ray image is colored by an image coloring algorithm to obtain the pseudo-color X-ray image.

[0015] According to a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided by the present invention, the multimodal fusion image is determined based on the second type of cross area, the third type of cross area and the fourth type of cross area, including: Rotating the second-type cross region, the third-type cross region, and the fourth-type cross region respectively to obtain a rotated second-type cross region, a rotated third-type cross region, and a rotated fourth-type cross region; Based on a preset size, the rotated second-type cross region, the rotated third-type cross region, and the rotated fourth-type cross region are scaled to obtain a scaled second-type cross region, a scaled third-type cross region, and a scaled fourth-type cross region; The scaled second-type cross region, the scaled third-type cross region, and the scaled fourth-type cross region are fused to obtain the multimodal fusion image.

[0016] The present invention also provides a liquid identification device based on a multi-modal fusion image of a security inspection X-ray machine, comprising: A scanning module, used to scan the container containing the liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; A determination module, used to determine a first type of cross region corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross region includes a region where a center line of the container is located and a region where a perpendicular bisector of the center line is located; The determination module is further used to determine an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; The determination module is further used to determine, based on the first type of cross region, a second type of cross region in the high energy X-ray image, a third type of cross region in the low energy X-ray image, and a fourth type of cross region in the effective atomic number image; The determination module is further used to determine a multimodal fusion image based on the second type of cross region, the third type of cross region and the fourth type of cross region; An identification module is used to identify the liquid type of the liquid to be detected based on the multimodal fusion image.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine as described in any one of the above methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine as described in any one of the above is implemented.

[0019] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned liquid identification methods based on multimodal fusion images of a security inspection X-ray machine.

[0020] The liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided in the embodiment of the present invention scans the container containing the liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image, and determines the first type of cross area corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross area includes the area where the center line of the container is located and the area where the perpendicular bisector of the center line is located, and determines the effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image, and determines the second type of cross area in the high-energy X-ray image, the third type of cross area in the low-energy X-ray image, and the fourth type of cross area in the effective atomic number image based on the first type of cross area, and after determining the multimodal fusion image based on the second type of cross area, the third type of cross area and the fourth type of cross area, the liquid type of the liquid to be detected is identified based on the multimodal fusion image. Since the multimodal fusion image is obtained based on the fusion of cross-like regions extracted from the high-energy X-ray image, the low-energy X-ray image and the effective atomic number image, the multimodal fusion image is independent of the shape of the container. When the liquid type of the liquid to be detected is identified by the multimodal fusion image, the identification result will no longer be affected by the shape of the container, thereby improving the accuracy of liquid type identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic flow chart of a liquid identification method based on multimodal fusion images of a security inspection X-ray machine provided in an embodiment of the present invention.

[0023] Figure 2 A schematic diagram of a high-energy X-ray image provided by an embodiment of the present invention.

[0024] Figure 3 A schematic diagram of a low-energy X-ray image provided by an embodiment of the present invention.

[0025] Figure 4 A schematic diagram of a pseudo-color X-ray image provided by an embodiment of the present invention.

[0026] Figure 5 A schematic diagram of the centerline of a container provided in an embodiment of the present invention.

[0027] Figure 6 A schematic diagram of a first type of cross region corresponding to a container provided in an embodiment of the present invention.

[0028] Figure 7 A schematic diagram of an effective atomic number image provided by an embodiment of the present invention.

[0029] Figure 8 A schematic diagram of a first type of cross region provided in an embodiment of the present invention.

[0030] Fig. 9 It is a schematic diagram of the second type of cross area, the third type of cross area and the fourth type of cross area determined based on the first type of cross area.

[0031] Fig.10 A schematic diagram of a scaled second-type cross region, a scaled third-type cross region, and a scaled fourth-type cross region provided in an embodiment of the present invention.

[0032] Fig.11 A schematic diagram of mask region segmentation provided by an embodiment of the present invention.

[0033] Fig.12 A schematic diagram of mask area distance transformation provided by an embodiment of the present invention.

[0034] Fig.13 A flowchart of a liquid identification method based on multimodal fusion images of a security inspection X-ray machine provided in an embodiment of the present invention.

[0035] Fig.14 A schematic diagram of the structure of a liquid identification device based on multimodal fusion images of a security inspection X-ray machine provided in an embodiment of the present invention.

[0036] Fig.15 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In the prior art, a liquid recognition model is pre-trained, and after collecting high-energy and low-energy images of the container containing the liquid to be detected, an atomic number map is generated through the high-energy and low-energy images, and a liquid target area mask in the high-energy, low-energy and atomic number images is obtained through a segmentation algorithm. The high-energy, low-energy and atomic number images of the liquid target area mask are input into the pre-trained liquid recognition model to obtain the liquid type of the liquid to be detected output by the liquid recognition model. However, in actual applications, the shapes of containers are various, and the collected high-energy and low-energy images, as well as the atomic number images obtained based on the high-energy and low-energy images are related to the shape of the container. Even if the same type of liquid is contained, the atomic number map obtained will be different when the shape of the container is different. Therefore, the pre-trained liquid recognition model cannot be applied to containers of all shapes, and the liquid type determined in the above manner is not accurate enough.

[0039] In response to the above problems, an embodiment of the present invention provides a liquid identification method based on the multimodal fusion image of a security inspection X-ray machine. Considering that no matter what shape the container is, it is usually cylindrical, and the high-energy image, low-energy image and effective atomic number image obtained for the cylinder all have the situation that the features of the area where the center line of the container and the area where the perpendicular bisector of the center line are located gradually change, therefore, no matter what shape the container is, the area where the center line and the area where the perpendicular bisector of the center line are located can be used to replace the overall liquid area. In this way, it is equivalent to normalizing containers of different shapes. When liquid type identification is performed based on the area where the center line and the area where the perpendicular bisector of the center line are located, the identification result will no longer be affected by the shape of the bottle, thereby improving the accuracy of liquid type identification.

[0040] Combine the following Figures 1 to 13 The liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided in an embodiment of the present invention is described. The execution subject of the method can be an electronic device such as a security inspection machine, a computer or a server, or a specially designed intelligent device, or a liquid identification device based on the multimodal fusion image of the security inspection X-ray machine set in the electronic device or the intelligent device. The liquid identification device based on the multimodal fusion image of the security inspection X-ray machine can be implemented by software, hardware or a combination of the two. The method can be applied to various scenes such as bus stations, railway stations, subway stations or high-speed railway stations where security inspections are required to distinguish whether the liquid to be detected is a safe liquid such as water and beverages, or a flammable liquid such as alcohol and gasoline, or a chemical liquid such as sulfuric acid.

[0041] Figure 1 A schematic diagram of a flow chart of a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided in an embodiment of the present invention, such as Figure 1 As shown, the method includes: Step 101: Scan a container containing a liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image, and a pseudo-color X-ray image.

[0042] In this step, Figure 2 A schematic diagram of a high-energy X-ray image provided by an embodiment of the present invention, Figure 3 A schematic diagram of a low-energy X-ray image provided by an embodiment of the present invention, Figure 4 A schematic diagram of a pseudo-color X-ray image provided by an embodiment of the present invention, such as Figure 2-4 As shown in the figure, when the container containing the liquid to be inspected passes through the security inspection X-ray machine, the security inspection X-ray machine uses high energy to scan the container and obtains the following Figure 2 The high-energy X-ray image Hi shown in the figure, after the security inspection X-ray machine scans the container with low energy, the following will be obtained Figure 3 The low-energy X-ray image Lo shown in FIG. 1 can be colored by using an artificial coloring algorithm to obtain the following: Figure 4 Pseudo-color X-ray image RGB is shown.

[0043] Combination Figure 2-Figure 4 It can be seen that the color of the image data near the center axis of the container is darker, and the farther the image data deviates from the center axis, the lighter the color. The darker the image color, the fewer X-rays pass through, and the lighter the image color, the more X-rays pass through.

[0044] Step 102: determining a first type of cross region corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross region includes a region where the center line of the container is located and a region where the perpendicular bisector of the center line is located.

[0045] In this step, Figure 5 A schematic diagram of the center line of a container provided by an embodiment of the present invention, such as Figure 5 As shown, by performing image recognition on the pseudo-color X-ray image, the liquid container mask area (mask) can be identified, and based on the position of each pixel point in the liquid container mask area (mask), the center line of the container can be determined, such as Figure 5 The center line of the container can also be understood as the central axis of the container.

[0046] Figure 6 A schematic diagram of a first type of cross region corresponding to a container provided in an embodiment of the present invention, such as Figure 6As shown, the center line can be used as a reference to expand a preset number of pixels in the left and right directions of the container to obtain the area where the center line is located. Further, the perpendicular bisector of the center line can be determined, and the perpendicular bisector is used as a reference to expand a preset number of pixels in the vertical direction of the container to obtain the area where the perpendicular bisector is located. The preset number can be set according to actual conditions or experience, such as according to the size of the container. The larger the volume of the container, the larger the value of the preset number.

[0047] Step 103: Determine an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image.

[0048] In this step, due to the different atomic numbers of different elements, the attenuation characteristics of X-rays are also different. Elements with higher atomic numbers have a stronger attenuation effect on X-rays, while elements with lower atomic numbers have a weaker attenuation effect on X-rays. Therefore, the high-energy X-ray image and low-energy X-ray image obtained by X-rays passing through a container containing the liquid to be tested can be obtained, that is, the attenuation information of X-rays of different energies can be obtained. Specifically, Figure 7 A schematic diagram of an effective atomic number image provided by an embodiment of the present invention, such as Figure 7 As shown, the relationship between the known attenuation coefficient and the atomic number, as well as the ratio of the attenuation coefficients under high-energy X-ray images and low-energy X-ray images, can be used to determine the effective atomic number of each material area, and the effective atomic number of each material area can be presented in the form of an image to obtain an effective atomic number image.

[0049] Step 104: Based on the first type of cross region, determine the second type of cross region in the high energy X-ray image, the third type of cross region in the low energy X-ray image, and the fourth type of cross region in the effective atomic number image.

[0050] In this step, Figure 8 A schematic diagram of a first type of cross region provided in an embodiment of the present invention, Fig. 9 is a schematic diagram of the second type of cross area, the third type of cross area and the fourth type of cross area determined based on the first type of cross area, such as Figure 8-Figure 9 As shown, Figure 8 The outline of the first type of cross area in the middle of the pseudo-color X-ray image RGB in is used as a template to extract the second type of cross area at the same position in the high-energy X-ray image Hi, the third type of cross area at the same position in the low-energy X-ray image Lo, and the fourth type of cross area at the same position in the effective atomic number image. Fig. 9 From left to right, the second type of cross region, the third type of cross region, and the fourth type of cross region are shown in sequence.

[0051] Step 105: Determine a multimodal fusion image based on the second type of cross region, the third type of cross region, and the fourth type of cross region.

[0052] In this step, after obtaining the image blocks corresponding to the second type of cross area, the image blocks corresponding to the third type of cross area, and the image blocks corresponding to the fourth type of cross area, the image data of these three image blocks are combined to obtain a three-channel multimodal fusion image. It should be understood that the multimodal fusion image contains key information in the high-energy X-ray image, the low-energy X-ray image, and the effective atomic number image that can be used to determine the type of liquid to be detected. The multimodal fusion image is equivalent to unifying or normalizing the image features corresponding to containers of different shapes.

[0053] Exemplarily, based on the above embodiments, due to the various placement positions of the containers, the shapes or angles of the second type of cross area, the third type of cross area, and the fourth type of cross area used to characterize the container features are not uniform. In order to prevent errors in the liquid type recognition results due to angle differences or position differences, in this embodiment, the second type of cross area, the third type of cross area, and the fourth type of cross area need to be corrected. When determining a multimodal fusion image based on the second type of cross area, the third type of cross area, and the fourth type of cross area, the second type of cross area, the third type of cross area, and the fourth type of cross area can be rotated respectively to obtain the rotated second type of cross area, the rotated third type of cross area, and the rotated fourth type of cross area, and the rotated second type of cross area, the rotated third type of cross area, and the rotated fourth type of cross area are scaled based on the preset size to obtain the scaled second type of cross area, the scaled third type of cross area, and the scaled fourth type of cross area, and the scaled second type of cross area, the scaled third type of cross area, and the scaled fourth type of cross area are fused to obtain a multimodal fusion image.

[0054] Specifically, Fig.10 A schematic diagram of a scaled second type cross region, a scaled third type cross region, and a scaled fourth type cross region provided by an embodiment of the present invention is shown in FIG. Fig.10As shown, the second type of cross area, the third type of cross area and the fourth type of cross area are rotated respectively to ensure that the orientations of the rotated second type of cross area, the rotated third type of cross area and the rotated fourth type of cross area are consistent. The rotation process can align the features in each cross area with the features used in the training of the image classification model. Furthermore, the rotated second type of cross area, the rotated third type of cross area and the rotated fourth type of cross area are scaled based on the preset size to uniformly scale different cross-like areas to the same size. For example, after the above-mentioned rotation and scaling process, the scaled second type of cross area, the scaled third type of cross area and the scaled fourth type of cross area of ​​the same width and direction will be obtained, wherein the standard width can be, for example, 224×224 pixels. The scaled second type of cross area, the scaled third type of cross area and the scaled fourth type of cross area are then fused to obtain a three-channel multimodal fusion image. Among them, Fig.10 From left to right, the second type of cross area after scaling, the third type of cross area after scaling, and the fourth type of cross area after scaling are shown in sequence. In this embodiment, by performing correction processing such as rotation and scaling on the second type cross area, the third type cross area and the fourth type cross area, the orientation and size of different cross-like areas can be unified. When performing image fusion, only the features of pixel points at the same position need to be fused, and there will be no pixel confusion, thereby improving the accuracy of the multimodal fusion image.

[0055] Step 106: Identify the liquid type of the liquid to be detected based on the multimodal fusion image.

[0056] In this step, the multimodal fusion image is input into a pre-trained liquid classification model to obtain a liquid type recognition result output by the liquid classification model, wherein the liquid classification model is obtained by training the initial liquid classification model based on the sample multimodal fusion image and the liquid type label.

[0057] The liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided in the embodiment of the present invention scans the container containing the liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image, and determines the first type of cross area corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross area includes the area where the center line of the container is located and the area where the perpendicular bisector of the center line is located, and determines the effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image, and determines the second type of cross area in the high-energy X-ray image, the third type of cross area in the low-energy X-ray image, and the fourth type of cross area in the effective atomic number image based on the first type of cross area, and after determining the multimodal fusion image based on the second type of cross area, the third type of cross area and the fourth type of cross area, the liquid type of the liquid to be detected is identified based on the multimodal fusion image. Since the multimodal fusion image is obtained based on the fusion of cross-like regions extracted from the high-energy X-ray image, the low-energy X-ray image and the effective atomic number image, the multimodal fusion image is independent of the shape of the container. When the liquid type of the liquid to be detected is identified by the multimodal fusion image, the identification result will no longer be affected by the shape of the container, thereby improving the accuracy of liquid type identification.

[0058] Exemplarily, based on the above embodiments, when determining the first type of cross area corresponding to the container in the pseudo-color X-ray image, the pseudo-color X-ray image can be input into the image segmentation model to obtain a mask area representing the container in the pseudo-color X-ray image output by the image segmentation model, and the center line of the container in the mask area is determined, and the center line is extended by a first preset number of pixels in a vertical direction of the center line to obtain the area where the center line is located, and then the perpendicular bisector is extended by a second preset number of pixels in a vertical direction of the perpendicular bisector to obtain the area where the perpendicular bisector is located, and then the first type of cross area is determined based on the area where the center line is located and the area where the perpendicular bisector is located.

[0059] Specifically, the image segmentation model is a neural network model or any other model capable of image segmentation. After the pseudo-color X-ray image is input into the image segmentation model, the image segmentation model can generate a mask region of the liquid container in the pseudo-color X-ray image. Figure 5 The mask region representing the container is shown in the green polygonal area.

[0060] In a possible implementation, when a pseudo-color X-ray image is input into an image segmentation model to obtain a mask region representing a container in the pseudo-color X-ray image output by the image segmentation model, the pseudo-color X-ray image can be input into a backbone network of the image segmentation model to obtain at least two image features output by the backbone network, and each image feature is input into a fusion network of the image segmentation model to obtain a fusion feature output by the fusion network, and the fusion feature is respectively input into a target detection network and a mask extraction network of the image segmentation model to obtain a container feature output by the target detection network and a mask feature corresponding to the container output by the mask extraction network, and then a mask region is obtained based on the container feature and the mask feature.

[0061] Specifically, Fig.11 A schematic diagram of mask region segmentation provided by an embodiment of the present invention is shown in FIG. Fig.11 As shown, when performing image segmentation on a pseudo-color X-ray image, an instance segmentation algorithm combining a segmentation head and a detection head is used in this embodiment. After the pseudo-color X-ray image is input into the image segmentation model, feature extraction is performed through the backbone network to obtain at least two image features, and then the at least two image features are input into the fusion network of the image segmentation model, and feature maps of different scales are fused through the Feature Pyramid Networks (FPN) to obtain fused features. The fused features are input into the target detection network and the mask extraction network of the image segmentation model. While the target detection network outputs the detection frame corresponding to the container, the mask extraction network can also output the mask features corresponding to the container. When extracting the mask area of ​​the container, the container features corresponding to the detection frame of the container and the mask features corresponding to the container are merged, and then passed through a three-layer dynamic convolutional neural network to finally generate a segmented image and obtain the mask area of ​​the container.

[0062] In this embodiment, multi-scale image features can be extracted through the backbone network, and the accuracy of recognition can be improved when liquid recognition is performed based on these image features. In addition, by fusing multi-scale image features, a fusion feature with richer content can be obtained. Based on the fusion feature, the container feature and the mask feature can be extracted in parallel, which improves the efficiency of determining the mask area.

[0063] After determining the mask area representing the container in the pseudo-color X-ray image, the skeleton of the container can be extracted through the image thinning algorithm to form a Figure 5 The centerline of the container in the mask area shown by the medium blue line.

[0064] Furthermore, if Figure 6As shown, the generated center line is extended by a first preset number of pixels on both sides along the direction perpendicular to the center line to form a center line strip mask, that is, a strip area where the center line is located. The first preset number can be set according to experience or the size of the container, for example, set to 32. At the same time, a perpendicular midline strip mask can also be constructed on the perpendicular midline of the center line of the container. For example, the perpendicular midline of the center line is extended by a second preset number of pixels on both sides along the direction perpendicular to the perpendicular midline to form a strip area where the perpendicular midline is located. The value of the second preset number can be the same as or different from the value of the first preset number. For example, it can also be set to 32. It should be understood that when the perpendicular midline of the center line is extended, there is a gradient change in the liquid characteristics in the area of ​​the perpendicular midline of the container, which can characterize the characteristics of the liquid in the entire container.

[0065] By combining the long strip area where the center line is located and the long strip area where the perpendicular bisector is located, the first type of cross area in the pseudo-color X-ray image can be obtained.

[0066] In this embodiment, after the center line of the container in the mask area is determined, the center line is extended by a first preset number of pixels in a direction perpendicular to the center line, and the perpendicular bisector of the center is extended by a second preset number of pixels in a direction perpendicular to the perpendicular bisector, to obtain a first type of cross area. The features in the first type of cross area can characterize the overall characteristics of the liquid and are independent of the shape of the container. Therefore, the liquid type of the liquid can be subsequently determined based on the features in the first type of cross area, which not only eliminates the interference of the container shape, but also reduces the amount of calculated data, thereby improving the efficiency of liquid recognition.

[0067] Exemplarily, based on the above embodiments, when determining the center line of the container in the mask area, it is necessary to perform a distance transformation on the pixel values ​​of each first pixel point in the mask area to obtain a transformation area, and the pixel values ​​of each second pixel point in the transformation area are used to characterize the minimum distance between the corresponding first pixel point in the mask area and the target boundary of the mask area, the target boundary is the boundary closest to the corresponding pixel point, and the center line is determined based on the second pixel point corresponding to the maximum pixel value in the transformation area.

[0068] Specifically, the mask area representing the container is determined from the pseudo-color X-ray image, which is usually a binary image. Then, an image thinning algorithm is used to extract the center line or central axis of the mask area, that is, a skeleton extraction algorithm. The skeleton removes some points in the original container image and still maintains the original structural information of the container. Fig.12 A schematic diagram of mask area distance transformation provided by an embodiment of the present invention is shown in FIG. Fig.12As shown, firstly, the pixel value of each first pixel point in the mask area is distance transformed, and the pixel value of each second pixel point in the obtained transformed area image is used to represent the minimum distance between the corresponding first pixel point in the original mask area and the target boundary of the mask area, wherein the target boundary is the boundary closest to the corresponding pixel point, and the target boundary can also be understood as the background. The above distance can be Euclidean distance, Manhattan distance or chessboard distance, etc.

[0069] After obtaining the transformation area, based on the pixel values ​​of each second pixel point in the transformation area, the second pixel point farthest from the target boundary, that is, the second pixel point corresponding to the maximum pixel value in the transformation area, is retained, and the skeleton formed by combining these second pixel points is the center line of the container. Of course, the second pixel point farthest and second farthest from the target boundary, that is, the second pixel point corresponding to the maximum pixel value and the second largest pixel value in the transformation area, can also be retained, and these points can be combined to obtain the center line of the container.

[0070] In this embodiment, by performing distance transformation on the pixel values ​​of each first pixel point in the mask area, and determining the center line of the container based on the second pixel point corresponding to the maximum pixel value in the transformed area, the center line can be quickly and accurately determined based on the above-mentioned skeleton extraction algorithm.

[0071] Exemplarily, based on the above embodiments, when determining the multimodal fusion image based on the second type of cross region, the third type of cross region and the fourth type of cross region, it can be performed in the following manner: For each third pixel in the second type of cross area, the normalized high and low energy coefficients are determined based on the pixel value of the third pixel and the pixel value of the fourth pixel corresponding to the third pixel in the third type of cross area, the pixel value of each pixel in the first channel is determined based on the normalized high and low energy coefficients and the number of image data bits corresponding to the effective atomic number image, and after the pixel value of the corresponding pixel in the second channel is determined based on the pixel value of each pixel in the first channel and the number of image data bits corresponding to the effective atomic number image, a multimodal fusion image is determined based on the pixel value of each pixel in the first channel, the pixel value of the corresponding pixel in the second channel, and the pixel value of the corresponding pixel in the effective atomic number image.

[0072] Specifically, since the number of image data bits of the collected high-energy X-ray images and low-energy X-ray images is different from that of the commonly used image channels, for example, the number of image data bits of the high-energy X-ray images and low-energy X-ray images are both 16 bits, and the pixel value range of each pixel is 0~65535, while the image data of commonly used image channels (such as RGB images and effective atomic number images) is 8 bits, and the pixel value range of the pixel is 0~255. Therefore, when performing multimodal data fusion on the second type of cross area, the third type of cross area and the fourth type of cross area, normalization processing is required.

[0073] Among them, for each third pixel point in the second type of cross area of ​​the high-energy X-ray image, the normalized high and low energy coefficients can be determined based on the pixel value of the third pixel point and the pixel value of the fourth pixel point corresponding to the third pixel point in the third type of cross area of ​​the low-energy X-ray image. In a possible implementation, when determining the normalized high and low energy coefficients, a first maximum value of a preset value and the pixel value of the third pixel point can be determined, the preset value is used to characterize the maximum value corresponding to normalization, and a second maximum value of a preset value and the pixel value of the fourth pixel point is determined, and the normalized high and low energy coefficients are determined based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high-energy X-ray image, and the number of image data bits corresponding to the low-energy X-ray image.

[0074] Among them, the preset value can be 1, for example. Based on the number of image data bits corresponding to the high-energy X-ray image, the maximum value of the pixel value range of the pixel point in the high-energy X-ray image can be determined. Based on the number of image data bits corresponding to the low-energy X-ray image, the maximum value of the pixel value range of the pixel point in the low-energy X-ray image can be determined. Therefore, based on the first maximum value, the second maximum value, the maximum value of the pixel value range of the pixel point in the high-energy X-ray image, and the maximum value of the pixel value range of the pixel point in the low-energy X-ray image, the normalized high and low energy coefficients are determined.

[0075] Specifically, the normalized high and low energy coefficients can be determined based on the following formula (1): (1) in, represents the normalized high and low energy coefficients, A represents the maximum pixel value of the pixel under the number of image data bits corresponding to the high energy X-ray image, B represents the first maximum value, C represents the maximum pixel value of the pixel under the number of image data bits corresponding to the low energy X-ray image, and D represents the second maximum value.

[0076] Taking the preset value as 1 and the image data bits of both high-energy X-ray images and low-energy X-ray images as 16 bits as an example, the above normalized high and low energy coefficients can be obtained by the following formula (2): (2) in, represents the pixel value of the third pixel (i, j), Represents the pixel value of the fourth pixel (i, j).

[0077] In this embodiment, the normalized high and low energy coefficients are determined by the pixel value of the third pixel and the pixel value of the fourth pixel corresponding to the third pixel in the third type of cross area, so that the number of image data bits of the second type of cross area and the third type of cross area is normalized to the same number of image data bits as the commonly used image channel, eliminating the differences between different images, facilitating the subsequent fusion of multimodal data, and improving the accuracy of data fusion.

[0078] After determining the normalized high and low energy coefficients, the pixel value of each pixel in the first channel is determined by the normalized high and low energy coefficients and the number of bits of image data corresponding to the effective atomic number image. Taking the image data bit corresponding to the effective atomic number image as 8 bits and the maximum pixel value of the corresponding pixel as 255 as an example, the pixel value of the pixel point (i, j) in the first channel can be determined by the following formula (3): : (3) The pixel value of the pixel point (i, j) in the second channel is determined by the following formula (4): : (4) in, The value is 10. The value is 1.1.

[0079] In addition, the pixel value of the third channel of the multimodal fusion image is the pixel value of the corresponding pixel point in the effective atomic number image.

[0080] In this embodiment, by fusing the second type of cross area, the third type of cross area and the fourth type of cross area, determining the multimodal fusion image through the pixel values ​​of different channels, and integrating the pixel values ​​of multiple channels, more comprehensive image information can be provided, which helps to improve the accuracy of liquid type identification.

[0081] Exemplarily, on the basis of the above-mentioned embodiments, when the liquid type of the liquid to be detected is identified based on the multimodal fusion image, the multimodal fusion image can be input into the feature extraction module of the image classification model, the multimodal fusion image can be divided into blocks by the feature extraction module, the obtained image blocks are encoded, and features are extracted from each encoded image block to obtain each multimodal feature, and each multimodal feature is input into the classification module of the image classification model to obtain the recognition result of the liquid type output by the classification module.

[0082] Specifically, the image classification model uses the ViT (Vision Transformer) model. The entire Vision Transformer can be divided into two parts, one is the feature extraction module, and the other is the classification module. In the feature extraction module, ViT is used for feature extraction. The corresponding areas of the feature extraction module in the multimodal image are block plus position embedding and Transformer Encoder. The main function of block plus position embedding is to block the input multimodal fusion image. The image blocks are divided every certain area size. Usually a convolution layer is introduced and implemented by convolution through a sliding window, such as 16×16. Then the divided image blocks are combined into an image sequence. At the same time, each image block needs to be position encoded and a class token (class token). The class token is associated with the labeled image category during the training process. After obtaining the image sequence, it is passed to the Transformer Encoder for feature extraction to obtain various multimodal features. This is the Transformer's unique multi-head self-attention structure. Through the self-attention mechanism, the importance of each image block is paid attention to.

[0083] In the classification module, the multimodal features extracted by the Transformer Encoder are input into the Multilayer Perceptron (MLP) classifier for classification to obtain the liquid type recognition results. The liquid type recognition results include safe liquid drinks such as water and beverages, flammable and explosive liquids such as alcohol and gasoline, or dangerous chemical liquids such as sulfuric acid.

[0084] In this embodiment, the multimodal fusion image is divided into blocks and encoded through the feature extraction module of the image classification model, which can capture the key information in the multimodal fusion image in more detail, and the local features in the multimodal fusion image can be identified through block processing, and the expression ability of the features can be enhanced through encoding, so that when the liquid type is classified by each multimodal feature, the accuracy of the classification result can be improved.

[0085] Exemplarily, on the basis of the above-mentioned embodiments, when scanning a container containing the liquid to be detected to obtain a pseudo-color X-ray image, the container containing the liquid to be detected can be scanned to obtain an original X-ray image, and the original X-ray image can be colored by an image colorization algorithm to obtain a pseudo-color X-ray image.

[0086] Specifically, when an X-ray security inspection machine is used to scan a container containing a liquid to be inspected, the resulting original X-ray image is usually displayed in grayscale, where different grayscale values ​​represent different material densities. When the original X-ray image is colored using an image coloring algorithm, colors can be assigned to different areas in the original X-ray image based on the grayscale information in the original X-ray image. This process can enhance the visual effect of the image, making it easier to distinguish different materials and structures in the pseudo-color X-ray image.

[0087] In this embodiment, the original X-ray image is colored by an image colorization algorithm to obtain a pseudo-color X-ray image, which retains the density information of the original X-ray image and makes the image more intuitive through color changes, thereby improving the accuracy of the detected first-type cross area.

[0088] Fig.13 A flowchart of a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine provided in an embodiment of the present invention is shown in FIG. Fig.13 As shown, after obtaining the high-energy X-ray image Hi, the low-energy X-ray image Lo and the pseudo-color X-ray image RGB of the container containing the liquid to be detected, the pseudo-color X-ray image RGB is segmented to obtain the mask area to which the liquid container belongs in the pseudo-color X-ray image RGB, and the center line of the container is extracted from the mask area, and then the first type of cross area in the pseudo-color X-ray image RGB is determined through the skeleton extraction algorithm.

[0089] In addition, the effective atomic number image Zeff can be determined based on the high-energy X-ray image Hi and the low-energy X-ray image Lo, and based on the first-type cross area, the second-type cross area in the high-energy X-ray image Hi, the third-type cross area in the low-energy X-ray image Lo, and the fourth-type cross area in the effective atomic number image Zeff can be determined.

[0090] Multi-channel data fusion is performed on the second type of cross area, the third type of cross area and the fourth type of cross area to obtain a multimodal fusion image, and the multimodal fusion image is input into the image classification model to obtain the liquid classification result.

[0091] The liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided in the embodiment of the present invention scans the container containing the liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image, and determines the first type of cross area corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross area includes the area where the center line of the container is located and the area where the perpendicular bisector of the center line is located, and determines the effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image, and determines the second type of cross area in the high-energy X-ray image, the third type of cross area in the low-energy X-ray image, and the fourth type of cross area in the effective atomic number image based on the first type of cross area, and after determining the multimodal fusion image based on the second type of cross area, the third type of cross area and the fourth type of cross area, the liquid type of the liquid to be detected is identified based on the multimodal fusion image. Since the multimodal fusion image is obtained based on the fusion of cross-like regions extracted from the high-energy X-ray image, the low-energy X-ray image and the effective atomic number image, the multimodal fusion image is independent of the shape of the container. When the liquid type of the liquid to be detected is identified by the multimodal fusion image, the identification result will no longer be affected by the shape of the container, thereby improving the accuracy of liquid type identification.

[0092] The following is a description of the liquid identification device based on the multimodal fusion image of the security inspection X-ray machine provided by the present invention. The liquid identification device based on the multimodal fusion image of the security inspection X-ray machine described below and the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine described above can be referenced to each other.

[0093] Fig.14 A schematic diagram of a liquid identification device based on a multimodal fusion image of a security inspection X-ray machine provided in an embodiment of the present invention, referring to Fig.14 As shown, the liquid identification device 1400 based on the multimodal fusion image of the security inspection X-ray machine includes: A scanning module 11 is used to scan a container containing a liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; A determination module 12 is used to determine a first type of cross region corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross region includes a region where a center line of the container is located and a region where a perpendicular bisector of the center line is located; The determination module 12 is further used to determine an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; The determination module 12 is further configured to determine, based on the first type of cross region, a second type of cross region in the high energy X-ray image, a third type of cross region in the low energy X-ray image, and a fourth type of cross region in the effective atomic number image; The determination module 12 is further configured to determine a multimodal fusion image based on the second type of cross region, the third type of cross region, and the fourth type of cross region; The identification module 13 is used to identify the liquid type of the liquid to be detected based on the multimodal fusion image.

[0094] In an exemplary embodiment, the determining module 12 is specifically configured to: Inputting the pseudo-color X-ray image into an image segmentation model to obtain a mask region representing the container in the pseudo-color X-ray image output by the image segmentation model; determining a centerline of the container in the mask area; Extending the center line by a first preset number of pixels in a direction perpendicular to the center line to obtain an area where the center line is located; Extend the perpendicular bisector by a second preset number of pixels in a direction perpendicular to the perpendicular bisector to obtain an area where the perpendicular bisector is located; The first type of cross area is determined based on the area where the center line is located and the area where the perpendicular bisector is located.

[0095] In an exemplary embodiment, the determining module 12 is specifically configured to: Inputting the pseudo-color X-ray image into the backbone network of the image segmentation model to obtain at least two image features output by the backbone network; Inputting each of the image features into the fusion network of the image segmentation model to obtain fusion features output by the fusion network; Inputting the fused features into the target detection network and the mask extraction network of the image segmentation model respectively, to obtain the container features output by the target detection network and the mask features corresponding to the container output by the mask extraction network; The mask region is obtained based on the container feature and the mask feature.

[0096] In an exemplary embodiment, the determining module 12 is specifically configured to: Performing distance transformation on the pixel values ​​of each first pixel point in the mask area to obtain a transformation area, wherein the pixel value of each second pixel point in the transformation area is used to represent the minimum distance between the corresponding first pixel point in the mask area and a target boundary of the mask area, wherein the target boundary is the boundary closest to the corresponding pixel point; The center line is determined based on a second pixel point corresponding to a maximum pixel value in the transformed area.

[0097] In an exemplary embodiment, the determining module 12 is specifically configured to: For each third pixel in the second-type cross region, determine a normalized high and low energy coefficient based on a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the third pixel in the third-type cross region; Determine the pixel value of each pixel in the first channel based on the normalized high and low energy coefficients and the number of image data bits corresponding to the effective atomic number image; Determine the pixel value of the corresponding pixel point in the second channel based on the pixel value of each pixel point in the first channel and the number of image data bits corresponding to the effective atomic number image; The multimodal fusion image is determined based on the pixel value of each pixel in the first channel, the pixel value of the corresponding pixel in the second channel, and the pixel value of the corresponding pixel in the effective atomic number image.

[0098] In an exemplary embodiment, the determining module 12 is specifically configured to: Determine a first maximum value between a preset value and a pixel value of the third pixel point, wherein the preset value is used to represent a maximum value corresponding to normalization; Determine a second maximum value between the preset value and the pixel value of the fourth pixel point; The normalized high and low energy coefficients are determined based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high energy X-ray image, and the number of image data bits corresponding to the low energy X-ray image.

[0099] In an exemplary embodiment, the determining module 12 is specifically configured to: The normalized high and low energy coefficients are determined based on the following formula (1): (1) in, represents the normalized high and low energy coefficients, A represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the high energy X-ray image, B represents the first maximum value, C represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the low energy X-ray image, and D represents the second maximum value.

[0100] In an exemplary embodiment, the identification module 13 is specifically configured to: Inputting the multimodal fusion image into a feature extraction module of an image classification model, dividing the multimodal fusion image into blocks through the feature extraction module, encoding each obtained image block, and performing feature extraction on each encoded image block to obtain each multimodal feature; Each of the multimodal features is input into a classification module of the image classification model to obtain a recognition result of the liquid type output by the classification module.

[0101] In an exemplary embodiment, the scanning module 11 is specifically configured to: Scan the container containing the liquid to be tested to obtain an original X-ray image; The original X-ray image is colored by an image coloring algorithm to obtain the pseudo-color X-ray image.

[0102] In an exemplary embodiment, the determination module 12 is specifically configured to: Rotating the second-type cross region, the third-type cross region, and the fourth-type cross region respectively to obtain a rotated second-type cross region, a rotated third-type cross region, and a rotated fourth-type cross region; Based on a preset size, the rotated second-type cross region, the rotated third-type cross region, and the rotated fourth-type cross region are scaled to obtain a scaled second-type cross region, a scaled third-type cross region, and a scaled fourth-type cross region; The scaled second-type cross region, the scaled third-type cross region, and the scaled fourth-type cross region are fused to obtain the multimodal fusion image.

[0103] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine. Its specific implementation process and technical effects are similar to those in the embodiment of the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine. For details, please refer to the detailed introduction in the embodiment of the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine, which will not be repeated here.

[0104] Fig.15 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.15As shown, the electronic device may include: a processor (processor) 1510 , a communication interface (Communications Interface) 1520 , a memory (memory) 1530 and a communication bus 1540 , wherein the processor 1510 , the communication interface 1520 , and the memory 1530 communicate with each other through the communication bus 1540 . The processor 1510 can call the logic instructions in the memory 1530 to execute a liquid identification method based on a multimodal fusion image of a security inspection X-ray machine, the method comprising: scanning a container containing a liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image, and a pseudo-color X-ray image; determining a first type of cross area corresponding to the container in the pseudo-color X-ray image, the first type of cross area including an area where the center line of the container is located and an area where the perpendicular bisector of the center line is located; determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; determining a second type of cross area in the high-energy X-ray image, a third type of cross area in the low-energy X-ray image, and a fourth type of cross area in the effective atomic number image based on the first type of cross area; determining a multimodal fusion image based on the second type of cross area, the third type of cross area, and the fourth type of cross area; and identifying the liquid type of the liquid to be detected based on the multimodal fusion image.

[0105] In addition, the logic instructions in the above-mentioned memory 1530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided by the above methods, and the method includes: scanning a container containing a liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; determining a first type of cross area corresponding to the container in the pseudo-color X-ray image, the first type of cross area including an area where the center line of the container is located and an area where the perpendicular bisector of the center line is located; determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; determining a second type of cross area in the high-energy X-ray image, a third type of cross area in the low-energy X-ray image and a fourth type of cross area in the effective atomic number image based on the first type of cross area; determining a multimodal fusion image based on the second type of cross area, the third type of cross area and the fourth type of cross area; and identifying the liquid type of the liquid to be detected based on the multimodal fusion image.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the liquid identification method based on the multimodal fusion image of the security inspection X-ray machine provided by the above methods, the method comprising: scanning a container containing a liquid to be detected to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; determining a first type of cross area corresponding to the container in the pseudo-color X-ray image, the first type of cross area including an area where the center line of the container is located and an area where the perpendicular bisector of the center line is located; determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; determining a second type of cross area in the high-energy X-ray image, a third type of cross area in the low-energy X-ray image and a fourth type of cross area in the effective atomic number image based on the first type of cross area; determining a multimodal fusion image based on the second type of cross area, the third type of cross area and the fourth type of cross area; and identifying the liquid type of the liquid to be detected based on the multimodal fusion image.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A liquid identification method based on multimodal fusion images of security inspection X-ray machines, characterized in that: include: Scan the container containing the liquid to be tested to obtain a high-energy X-ray image, a low-energy X-ray image and a pseudo-color X-ray image; Determine a first type of cross region corresponding to the container in the pseudo-color X-ray image, wherein the first type of cross region includes a region where a center line of the container is located and a region where a perpendicular bisector of the center line is located; Determining an effective atomic number image based on the high-energy X-ray image and the low-energy X-ray image; Based on the first type of cross region, determining a second type of cross region in the high energy X-ray image, a third type of cross region in the low energy X-ray image, and a fourth type of cross region in the effective atomic number image; Determine a multimodal fusion image based on the second-type cross region, the third-type cross region, and the fourth-type cross region; The liquid type of the liquid to be detected is identified based on the multimodal fusion image.

2. The liquid identification method based on the multimodal fusion image of the security inspection X-ray machine according to claim 1 is characterized in that: The determining of the first type of cross region corresponding to the container in the pseudo-color X-ray image includes: Inputting the pseudo-color X-ray image into an image segmentation model to obtain a mask region representing the container in the pseudo-color X-ray image output by the image segmentation model; determining a centerline of the container in the mask area; Extending the center line by a first preset number of pixels in a direction perpendicular to the center line to obtain an area where the center line is located; Extend the perpendicular bisector by a second preset number of pixels in a direction perpendicular to the perpendicular bisector to obtain an area where the perpendicular bisector is located; The first type of cross area is determined based on the area where the center line is located and the area where the perpendicular bisector is located.

3. The liquid identification method based on multimodal fusion images of security inspection X-ray machine according to claim 2 is characterized in that: The step of inputting the pseudo-color X-ray image into an image segmentation model to obtain a mask region representing the container in the pseudo-color X-ray image output by the image segmentation model comprises: Inputting the pseudo-color X-ray image into the backbone network of the image segmentation model to obtain at least two image features output by the backbone network; Inputting each of the image features into the fusion network of the image segmentation model to obtain fusion features output by the fusion network; Inputting the fused features into the target detection network and the mask extraction network of the image segmentation model respectively, to obtain the container features output by the target detection network and the mask features corresponding to the container output by the mask extraction network; The mask region is obtained based on the container feature and the mask feature.

4. The liquid identification method based on multimodal fusion images of security inspection X-ray machine according to claim 2 is characterized in that: The determining the center line of the container in the mask area includes: Performing distance transformation on the pixel values ​​of each first pixel point in the mask area to obtain a transformation area, wherein the pixel value of each second pixel point in the transformation area is used to represent the minimum distance between the corresponding first pixel point in the mask area and a target boundary of the mask area, wherein the target boundary is the boundary closest to the corresponding pixel point; The center line is determined based on a second pixel point corresponding to a maximum pixel value in the transformed area.

5. The liquid identification method based on multimodal fusion images of security inspection X-ray machine according to claim 1 is characterized in that: The determining of the multimodal fusion image based on the second type of cross region, the third type of cross region and the fourth type of cross region includes: For each third pixel in the second-type cross region, determine a normalized high and low energy coefficient based on a pixel value of the third pixel and a pixel value of a fourth pixel corresponding to the third pixel in the third-type cross region; Determine the pixel value of each pixel in the first channel based on the normalized high and low energy coefficients and the number of image data bits corresponding to the effective atomic number image; Determine the pixel value of the corresponding pixel point in the second channel based on the pixel value of each pixel point in the first channel and the number of image data bits corresponding to the effective atomic number image; The multimodal fusion image is determined based on the pixel value of each pixel in the first channel, the pixel value of the corresponding pixel in the second channel, and the pixel value of the corresponding pixel in the effective atomic number image.

6. The liquid identification method based on multimodal fusion images of security inspection X-ray machine according to claim 5 is characterized in that: The determining of the normalized high and low energy coefficients based on the pixel value of the third pixel and the pixel value of a fourth pixel corresponding to the third pixel in the third type cross region includes: Determine a first maximum value between a preset value and a pixel value of the third pixel point, wherein the preset value is used to represent a maximum value corresponding to normalization; Determine a second maximum value between the preset value and the pixel value of the fourth pixel point; The normalized high and low energy coefficients are determined based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high energy X-ray image, and the number of image data bits corresponding to the low energy X-ray image.

7. The liquid identification method based on multimodal fusion images of security inspection X-ray machine according to claim 6 is characterized in that: The determining the normalized high and low energy coefficients based on the first maximum value, the second maximum value, the number of image data bits corresponding to the high energy X-ray image, and the number of image data bits corresponding to the low energy X-ray image includes: The normalized high and low energy coefficients are determined based on the following formula (1): （1） in, represents the normalized high and low energy coefficients, A represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the high energy X-ray image, B represents the first maximum value, C represents the maximum pixel value of the pixel point under the number of image data bits corresponding to the low energy X-ray image, and D represents the second maximum value.

8. The liquid identification method based on multi-modal fusion images of security inspection X-ray machine according to claim 1 is characterized in that: The identifying the liquid type of the liquid to be detected based on the multimodal fusion image includes: Inputting the multimodal fusion image into a feature extraction module of an image classification model, dividing the multimodal fusion image into blocks through the feature extraction module, encoding each obtained image block, and performing feature extraction on each encoded image block to obtain each multimodal feature; Each of the multimodal features is input into a classification module of the image classification model to obtain a recognition result of the liquid type output by the classification module.

9. The liquid identification method based on multi-modal fusion images of security inspection X-ray machine according to claim 1, characterized in that: Scan the container containing the liquid to be tested to obtain a pseudo-color X-ray image, including: Scan the container containing the liquid to be tested to obtain an original X-ray image; The original X-ray image is colored by an image coloring algorithm to obtain the pseudo-color X-ray image.

10. The liquid identification method based on multi-modal fusion images of security inspection X-ray machine according to any one of claims 1 to 9, characterized in that: The determining of the multimodal fusion image based on the second type of cross region, the third type of cross region and the fourth type of cross region includes: Rotating the second-type cross region, the third-type cross region, and the fourth-type cross region respectively to obtain a rotated second-type cross region, a rotated third-type cross region, and a rotated fourth-type cross region; Based on a preset size, the rotated second-type cross region, the rotated third-type cross region, and the rotated fourth-type cross region are scaled to obtain a scaled second-type cross region, a scaled third-type cross region, and a scaled fourth-type cross region; The scaled second-type cross region, the scaled third-type cross region, and the scaled fourth-type cross region are fused to obtain the multimodal fusion image.

Images