Security check image fusion method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a security check image fusion method and device, electronic equipment and a storage medium, and is applied to the technical field of image processing. By applying the method provided by the embodiment of the invention, the first outer contour and the second outer contour of the to-be-detected object in the first X-ray image frame and the second X-ray image frame can be detected by acquiring the first X-ray image frame and the second X-ray image frame, and the minimum geometric bounding box of the first outer contour and the minimum geometric bounding box of the second outer contour are calculated; matching is carried out according to the first outer contour, the second outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box, and pixel fusion is carried out on the first X-ray image frame and the second X-ray image frame under the condition of successful matching to obtain a fused image, so that the quality of the security check image is improved; therefore, the judgment of identifying the prohibited goods by security check personnel according to the fused image is more accurate.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a security inspection image fusion method, device, electronic device and storage medium. Background Art

[0002] Security inspection machine is an electronic device that uses X-ray scanning imaging technology to perform security inspection on objects to be inspected. It is composed of X-ray generator, X-ray detector, image processing system and other parts. It judges the safety of objects to be inspected based on scanning imaging. It is widely used in public places such as airports, railway stations, subway stations, bus stations, large event venues, etc. Its main principle is to present the objects to be inspected in different colors according to the degree of absorption of X-ray electromagnetic waves by the objects to be inspected, so that security personnel can observe the outer contour image of the objects to be inspected, and thus judge the safety of the objects to be inspected based on the outer contour.

[0003] However, in the related art, the security inspection machine performs object recognition on the collected single-frame X-ray images respectively. However, due to the different absorption degrees of X-ray electromagnetic waves by the objects to be detected, the quality of the collected and generated security inspection images is poor, which in turn affects the accuracy of image judgment by security inspectors. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a security inspection image fusion method, device, electronic device and storage medium to improve the quality of security inspection images. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a security inspection image fusion method is provided, the method comprising:

[0006] Acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected;

[0007] Performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame;

[0008] Calculating the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes;

[0009] Matching the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box;

[0010] When the first X-ray image frame and the second X-ray image frame are matched successfully, the first X-ray image frame and the second X-ray image frame are pixel-fused according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box to obtain a fused image, and the fused image is used to determine whether the object to be detected is a prohibited item.

[0011] In a possible implementation, matching the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box includes:

[0012] Calculating the similarity between the first outer contour and the second outer contour;

[0013] Calculating a difference between geometric parameters of the first minimum geometric bounding box and geometric parameters of the second minimum geometric bounding box; wherein the geometric parameters include a frame size and / or a rotation angle of the bounding box in the image;

[0014] When the similarity is greater than a preset similarity threshold and the difference value of the geometric parameter is less than a preset parameter threshold, it is determined that the first X-ray image frame matches the second X-ray image frame successfully.

[0015] In a possible implementation, performing pixel fusion on the first X-ray image frame and the second X-ray image frame to obtain a fused image according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box includes:

[0016] Determine, according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, a first pixel row corresponding to a fusion start row of the second X-ray image frame in the first X-ray image frame, and a second pixel row corresponding to a fusion end row of the second X-ray image frame in the first X-ray image frame;

[0017] Each pixel row from the fusion start row to the fusion end row in the second X-ray image frame is pixel-fused with each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image.

[0018] In a possible implementation, determining, according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, a first pixel row corresponding to a fusion start row of the second X-ray image frame in the first X-ray image frame, and a second pixel row corresponding to a fusion end row of the second X-ray image frame in the first X-ray image frame; comprising:

[0019] Select the minimum row where the pixels in the second minimum geometric bounding box are located to obtain a fusion start row; select the maximum row where the pixels in the second minimum geometric bounding box are located to obtain a fusion end row;

[0020] Obtaining each preset offset value within a preset pixel row offset range; for each preset offset value, aligning the second X-ray image frame with the first X-ray image frame after offsetting it by the preset offset value, calculating the difference between each aligned pixel within a specified range between the aligned second X-ray image frame and the first X-ray image frame, and calculating the average of each difference to obtain the image difference under the preset offset value; wherein the specified range includes at least a portion of the first minimum geometric bounding box and the second minimum geometric bounding box;

[0021] Select a target preset offset value when the image difference is minimal; after the second X-ray image frame is offset by the target preset offset value, determine the first pixel row corresponding to the fusion start row in the first X-ray image frame, and the second pixel row corresponding to the fusion end row in the first X-ray image frame.

[0022] In a possible implementation, the first X-ray image frame and the second X-ray image frame are two X-ray image frames acquired one after the other, or the first X-ray image frame is an image frame synthesized from multiple X-ray image frames acquired before the second X-ray image frame;

[0023] The step of performing pixel fusion on each pixel row from the fusion start row to the fusion end row in the second X-ray image frame and each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image comprises:

[0024] The fused image is obtained by calculating the following fusion function;

[0025] The fusion function is:

[0026]

[0027] Among them, G(x i ,y j ) indicates that the coordinates in the fused image are (x i ,y j) pixel value, N represents the number of times the first X-ray image frame has been fused, G1(x i ,y j+d ) indicates that the coordinates in the first X-ray image frame are (x i ,y j+d ) pixel value, G2(x i ,y j ) indicates that the middle coordinate of the second X-ray image frame is (x i ,y j ) pixel value, wherein i∈[1, a], j∈[b, c], a is the total number of pixel columns in the first X-ray image frame, b is the number of fusion start rows in the second X-ray image frame, c is the number of fusion end rows in the second X-ray image frame, and d is the difference between the number of fusion start rows and the number of the first pixel rows.

[0028] In a possible implementation manner, the first minimum geometric bounding box is a first minimum rotated rectangular box, and the second minimum geometric bounding box is a second minimum rotated rectangular box;

[0029] The step of respectively calculating the minimum geometric bounding boxes of the first outer contour and the second outer contour to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour comprises:

[0030] Extracting convex edge vertices of the first outer contour and the second outer contour respectively to obtain a plurality of first convex edge vertices and a plurality of second convex edge vertices;

[0031] Taking the straight line determined by any two adjacent first convex edge vertices as the straight line where the side of the circumscribed rectangle is located, determining each first minimum circumscribed rectangle including the first outer contour inside, and selecting the first minimum circumscribed rectangle with the smallest area to obtain a first minimum geometric bounding box;

[0032] The straight line determined by any two adjacent second convex edge vertices is used as the straight line where the side of the circumscribed rectangle is located, and each second minimum circumscribed rectangle including the second outer contour is determined, and the second minimum circumscribed rectangle with the smallest area is selected to obtain the second minimum geometric bounding box.

[0033] In a possible implementation, performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame includes:

[0034] Binarizing the first X-ray image frame and the second X-ray image frame respectively to obtain a first binarized image and a second binarized image;

[0035] For each first pixel point of the first binary image, determine whether the pixel values ​​of each pixel point adjacent to the first pixel point are all the first pixel value, if not, consider the first pixel point to be a first outer contour point; wherein the value of the pixel point representing the object to be detected is the first pixel value;

[0036] For each second pixel point of the second binary image, determine whether pixel values ​​of each pixel point adjacent to the second pixel point are all the first pixel value; if not, consider the second pixel point to be a second outer contour point;

[0037] For each first outer contour point, calculate a first number of first outer contour points that belong to the same connected domain as the first outer contour point, and if the first number is less than a first preset threshold, remove the first outer contour point to obtain the first outer contour of the object to be detected in the first X-ray image frame;

[0038] For each second outer contour point, calculate the second number of second outer contour points that belong to the same connected domain as the second outer contour point; if the second number is less than the first preset threshold, remove the second outer contour point to obtain the second outer contour of the object to be detected in the second X-ray image frame.

[0039] In a second aspect of the implementation of the present application, a security inspection image fusion device is provided, the device comprising:

[0040] An image frame acquisition module, used to acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected;

[0041] a contour detection module, configured to perform contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame, and obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame;

[0042] a minimum geometric bounding box calculation module, used to calculate the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes;

[0043] an image frame matching module, configured to match the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box;

[0044] An image fusion module is used to perform pixel fusion on the first X-ray image frame and the second X-ray image frame to obtain a fused image according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box when the first X-ray image frame and the second X-ray image frame are successfully matched, and the fused image is used to determine whether the object to be detected is a prohibited item.

[0045] In a possible implementation, the image frame matching module includes:

[0046] A similarity calculation submodule, specifically used to calculate the similarity between the first outer contour and the second outer contour;

[0047] a difference value calculation submodule, specifically used to calculate the difference value between the geometric parameters of the first minimum geometric bounding box and the geometric parameters of the second minimum geometric bounding box; wherein the geometric parameters include the border size and / or rotation angle of the bounding box in the image;

[0048] The matching judgment submodule is specifically used to determine that the first X-ray image frame and the second X-ray image frame are successfully matched when the similarity is greater than a preset similarity threshold and the difference value of the geometric parameter is less than a preset parameter threshold.

[0049] In a possible implementation, the image fusion module includes:

[0050] a pixel row determination submodule, specifically used to determine, according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, a first pixel row corresponding to a fusion start row of the second X-ray image frame in the first X-ray image frame, and a second pixel row corresponding to a fusion end row of the second X-ray image frame in the first X-ray image frame;

[0051] The pixel fusion submodule is specifically used to perform pixel fusion on each pixel row from the fusion start row to the fusion end row in the second X-ray image frame and each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image.

[0052] In a possible implementation manner, the pixel row determination submodule includes:

[0053] The fusion row determination unit is specifically used to select the minimum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion start row; select the maximum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion end row;

[0054] The image difference calculation unit is specifically used to obtain each preset offset value within a preset pixel row offset range; for each preset offset value, the second X-ray image frame is offset by the preset offset value and then aligned with the first X-ray image frame, and the difference between each aligned pixel within a specified range between the aligned second X-ray image frame and the first X-ray image frame is calculated, and the average of each difference is calculated to obtain the image difference under the preset offset value; wherein the specified range includes at least a part of the first minimum geometric bounding box and the second minimum geometric bounding box;

[0055] The fusion pixel row determination unit is specifically used to select a target preset offset value when the image difference is minimized; after the second X-ray image frame is offset by the target preset offset value, determine the first pixel row corresponding to the fusion start row in the first X-ray image frame, and the second pixel row corresponding to the fusion end row in the first X-ray image frame.

[0056] In a possible implementation, the first X-ray image frame and the second X-ray image frame are two X-ray image frames acquired one after the other, or the first X-ray image frame is an image frame synthesized from multiple X-ray image frames acquired before the second X-ray image frame;

[0057] The pixel fusion submodule includes:

[0058] The pixel fusion calculation unit is specifically used to calculate and obtain a fused image through the following fusion function;

[0059] The fusion function is:

[0060]

[0061] Among them, G(x i ,y j ) indicates that the coordinates in the fused image are (x i ,y j ) pixel value, N represents the number of times the first X-ray image frame has been fused, G1(x i ,y j+d ) indicates that the coordinates in the first X-ray image frame are (x i ,y j+d ) pixel value, G2(x i ,y j ) indicates that the middle coordinate of the second X-ray image frame is (xi ,y j ) pixel value, wherein i∈[1, a], j∈[b, c], a is the total number of pixel columns in the first X-ray image frame, b is the number of fusion start rows in the second X-ray image frame, c is the number of fusion end rows in the second X-ray image frame, and d is the difference between the number of fusion start rows and the number of the first pixel rows.

[0062] In a possible implementation, the first minimum geometric bounding box is a first minimum rotated rectangular box, and the second minimum geometric bounding box is a second minimum rotated rectangular box; the minimum geometric bounding box calculation module includes:

[0063] A convex edge vertex extraction submodule is specifically used to extract the convex edge vertices of the first outer contour and the second outer contour respectively, to obtain a plurality of first convex edge vertices and a plurality of second convex edge vertices;

[0064] A first minimum geometric bounding box determination submodule is specifically used to determine each first minimum circumscribed rectangle including the first outer contour by taking the straight line determined by any two adjacent first convex edge vertices as the straight line where the side of the circumscribed rectangle is located, and select the first minimum circumscribed rectangle with the smallest area to obtain the first minimum geometric bounding box;

[0065] The second minimum geometric bounding box determination submodule is specifically used to determine the second minimum circumscribed rectangles including the second outer contour inside, using the straight line determined by any two adjacent second convex edge vertices as the straight line where the side of the circumscribed rectangle is located, and select the second minimum circumscribed rectangle with the smallest area to obtain the second minimum geometric bounding box.

[0066] In a possible implementation, the contour detection module includes:

[0067] An image binarization submodule is specifically used to binarize the first X-ray image frame and the second X-ray image frame respectively to obtain a first binarized image and a second binarized image;

[0068] A first pixel point judgment submodule is specifically used to judge, for each first pixel point of the first binary image, whether the pixel values ​​of each pixel point adjacent to the first pixel point are all the first pixel value, and if not, the first pixel point is considered to be a first outer contour point; wherein the value of the pixel point representing the object to be detected is the first pixel value;

[0069] A second pixel point determination submodule is specifically used to determine, for each second pixel point of the second binary image, whether the pixel values ​​of each pixel point adjacent to the second pixel point are all the first pixel value, and if not, the second pixel point is considered to be a second outer contour point;

[0070] A first image erosion submodule is specifically configured to calculate, for each first outer contour point, a first number of first outer contour points belonging to the same connected domain as the first pixel point, and if the first number is less than a first preset threshold, remove the first outer contour point to obtain the first outer contour of the object to be detected in the first X-ray image frame;

[0071] The second image corrosion submodule is specifically used to calculate, for each second outer contour point, a second number of second outer contour points belonging to the same connected domain as the second outer contour point; if the second number is less than the first preset threshold, the second outer contour point is removed to obtain the second outer contour of the object to be detected in the second X-ray image frame.

[0072] Another aspect of the embodiments of the present application further provides an electronic device, including:

[0073] Memory, used to store computer programs;

[0074] The processor is used to implement the security inspection image fusion method described in any of the embodiments of the present application when executing the program stored in the memory.

[0075] In another aspect of the implementation of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned security inspection image fusion methods is implemented.

[0076] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the security inspection image fusion methods described above.

[0077] Beneficial effects of the embodiments of the present application:

[0078] The embodiments of the present application provide a method, device, electronic device and storage medium for fusion of security inspection images. The method can detect the first outer contour and the second outer contour of the object to be detected in the first X-ray image frame and the second X-ray image frame by acquiring the first X-ray image frame and the second X-ray image frame, and calculate the minimum geometric bounding box of the first outer contour and the second outer contour; match according to the first outer contour, the second outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box, and fuse the first X-ray image frame and the second X-ray image frame pixel by pixel to obtain a fused image when the match is successful, wherein the fused image is used to determine whether the object to be detected is a contraband; by fusing the first X-ray image frame and the second X-ray image frame that have been successfully matched, the fused image includes more image information, thereby improving the quality of the security inspection image, so that the security inspector can make a more accurate judgment on the identification of contraband according to the fused image. In particular, for the scenario of scanning and detecting the object to be detected by linear array scanning, the quality of the security inspection image of the object to be detected is low due to the limitation of the hardware structure, and the image details are not clearly displayed. The scheme provided by the embodiments of the present application can improve the quality of the security inspection image, thereby improving the accuracy of the security inspection judgment image.

[0079] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0081] Figure 1 A schematic diagram of the working principle of the security inspection machine provided in the embodiment of the present application;

[0082] Figure 2 A schematic diagram of a flow chart of a security inspection image fusion method provided in an embodiment of the present application;

[0083] Figure 3 A schematic diagram of a process for image processing provided in an embodiment of the present application;

[0084] Figure 4 A schematic diagram of a second minimum geometric bounding box corresponding to an object to be detected in a second X-ray image frame provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of a first minimum geometric bounding box corresponding to an object to be detected in a first X-ray image frame provided in an embodiment of the present application;

[0086] Figure 6 A schematic diagram of generating a first X-ray image frame or a second X-ray image frame provided in an embodiment of the present application;

[0087] Figure 7 A schematic diagram of determining the convex edge vertex provided in an embodiment of the present application;

[0088] Figure 8 A schematic diagram of determining a minimum geometric bounding box provided in an embodiment of the present application;

[0089] Fig. 9 A schematic diagram of a process for calculating a minimum geometric bounding box provided in an embodiment of the present application;

[0090] Fig.10 A schematic diagram of determining contour points provided in an embodiment of the present application;

[0091] Fig.11 Another schematic diagram of the security inspection image fusion method provided in the embodiment of the present application;

[0092] Fig.12 A schematic diagram of the structure of a security inspection image fusion device provided in an embodiment of the present application;

[0093] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0095] As an electronic device for security inspection, the working principle of the security inspection machine is as follows: Figure 1 As shown. The object to be inspected is placed on the conveyor belt of the security inspection machine for transportation. An X-ray source is provided below the conveyor belt for emitting X-rays, and an X-ray detector is provided above the conveyor belt for receiving X-rays. When the object to be inspected is transported above the X-ray source, the X-rays emitted by the X-ray source penetrate the object to be inspected, and the X-ray detector transmits the detected X-rays to the image processing module. The image processing module distinguishes different components with different colors according to the X-ray absorption degree of the object to be inspected, and displays them on the display. The security inspector can make a safety judgment on the object to be inspected by observing the outer contours of each component of the object to be inspected on the display.

[0096] In the related art, the security inspection machine obtains a single frame image for processing through linear scanning, and the detail processing of the image is prone to errors, which is not conducive to the security inspector's image recognition. Linear scanning refers to the process of using a row of continuously arranged sensor elements (such as photodiodes, CCD (Charge Coupled Device) or CMOS sensors to scan the target area line by line (or point by point) to collect image information. The movement of the object to be inspected on the conveyor belt may affect the image acquisition. Therefore, the generated outer contour image will have poor image quality, which in turn affects the accuracy of the security inspector's image recognition.

[0097] In order to solve at least one of the above problems, a first aspect of an embodiment of the present application provides a security inspection image fusion method, the method comprising: Figure 2 Steps shown:

[0098] Step S201: Acquire a first X-ray image frame and a second X-ray image frame.

[0099] Among them, the first X-ray image frame and the second X-ray image frame are security inspection images for the same object to be inspected, and in order to facilitate subsequent image fusion, the resolution of the first X-ray image frame and the second X-ray image frame needs to be the same. The first X-ray image frame and the second X-ray image frame can be obtained by scanning the object to be inspected moving on the conveyor belt using a linear array scanning method, or can be obtained by other scanning methods. In an example, the industrial computer of the security inspection machine can control the movement of the object to be inspected on the conveyor belt, so that it passes over the X-ray source for multiple times for linear array scanning, thereby obtaining two frames of images of the object to be inspected.

[0100] In another example, two X-ray sources may be provided so that the object to be inspected can pass through the two X-ray sources for linear scanning after moving once on the conveyor belt, thereby obtaining two frames of security inspection images of the object to be inspected.

[0101] In a possible implementation manner, the first X-ray image frame and the second X-ray image frame of the object to be detected may be preprocessed before executing step S202 to perform contour detection. Figure 3As shown, first obtain the image data of the object to be detected when it moves on the transmission belt, that is, the first X-ray image frame and the second X-ray image frame. In practical applications, its image data may be stored in a database, and the server calls the data at the corresponding position in the database through a pointer to obtain the image data. Therefore, after obtaining the image data, parameter verification is required. For example, a pointer address verification can be performed. If the image data is not obtained according to the pointer address, that is, the pointer information of the image is a null pointer, it is considered that the verification has not passed. For another example, in practical applications, it is necessary to have a size limit on the image data. If the image is too large, the global information of the image cannot be obtained. If the image is too small, the local detail information of the image cannot be obtained. Therefore, if the height or width of the image data exceeds the upper and lower limits, it is still considered that the image data has not passed the parameter verification. If the image data fails to pass the parameter verification, the image data is discarded and no image processing is performed on it.

[0102] After the image data passes the parameter verification, the image data can also be preliminarily downsampled. In one example, a reasonable downsampling factor can be set to selectively merge or delete pixels of the image data that passes the parameter verification, thereby ensuring that the amount of image data is greatly reduced while maintaining key image data information, thereby reducing the complexity of image processing.

[0103] Step S202: performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame.

[0104] Among them, there are many methods for contour detection, for example, contour detection can be performed through methods such as Canny edge detection algorithm, threshold method, Sobel edge detection operator, Prewitt (differential operator edge detection) edge detection operator, Roberts edge detection operator, and Laplace transform method; in addition, deep learning models can also be used to determine the outer contour of the object to be detected.

[0105] In one example, the first X-ray image frame and the second X-ray image frame may be subjected to smoothing filtering to remove image noise, and then edge detection may be performed to obtain the first outer contour and the second outer contour of the object to be detected. Specifically, the first X-ray image frame and the second X-ray image frame may be subjected to grayscale processing, and grayscale histograms may be statistically calculated to quantitatively represent the frequency distribution of each grayscale pixel in the image data. Figure 3As shown, the downsampled image is grayed, and the grayscale histogram is counted. The peaks representing different grayscale areas of the image data are identified through the grayscale histogram, and the maximum grayscale value is selected from each peak as the binarization threshold. Among them, the peak of the grayscale area is the grayscale value with the largest number of corresponding pixels in the grayscale area. For example, there are four grayscale values ​​of 14, 35, 95, and 105 in a grayscale area of ​​the image data. Among them, there are 24 pixels with a grayscale value of 14, 83 pixels with a grayscale value of 35, 42 pixels with a grayscale value of 95, and 8 pixels with a grayscale value of 35. The number of pixels with a grayscale value of 35 is the largest, and 35 is taken as the peak of the grayscale area. The maximum grayscale value is selected from each peak as the binarization threshold, and the pixel value greater than the binarization threshold is represented as 1, and the pixel value less than the binarization threshold is represented as 0. The image data is binarized to obtain the binarized image data. Image binarization can transform image data into a binary image containing only black and white, significantly enhancing the contrast between the object to be detected and the background, making the edge of the object to be detected clearly discernible. After the image is binarized, the image data can be eroded. For example, each pixel in the image data can be traversed, and isolated or unstable boundary pixels can be removed based on its relationship with the surrounding pixels to ensure the accuracy of the final contour line. Finally, contour detection is performed on the binarized image data to obtain the outer contour of the object to be detected.

[0106] Step S203: Calculate the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, and obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour.

[0107] The first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes, for example, both can be triangles, rectangles, pentagons or other convex polygons, etc. Generally, the first minimum geometric bounding box is a geometric shape with the smallest area that can enclose the first outer contour, and the second minimum geometric bounding box is a geometric shape with the smallest area that can enclose the second outer contour.

[0108] In an example, taking a rectangle as an example, the first minimum geometric bounding box can be the minimum circumscribed rectangle of the first outer contour, and the second minimum geometric bounding box can be the minimum circumscribed rectangle of the second outer contour; in this case, when calculating the first minimum geometric bounding box and the second minimum geometric bounding box, the minimum circumscribed rectangles corresponding to the first outer contour and the second outer contour of the object to be detected can be calculated first, the first outer contour and the second outer contour can be rotated, and the minimum circumscribed rectangles corresponding to the rotated first outer contour and the second outer contour can be calculated, and the minimum circumscribed rectangles with the smallest area can be selected as the first minimum geometric bounding box and the second minimum geometric bounding box.

[0109] Step S204: matching the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box.

[0110] When matching the first X-ray image frame and the second X-ray image frame, matching is performed according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box, respectively. The matching process is used to determine whether the objects in the two image frames are the same object, that is, whether the first outer contour and the second outer contour represent the same object. When the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box indicate that the objects contained in the two image frames are the same object, it is considered that the first X-ray image frame and the second X-ray image frame are matched successfully.

[0111] In a possible implementation, the first X-ray image frame and the second X-ray image frame may be matched by the following steps:

[0112] Step I: Calculate the similarity between the first outer contour and the second outer contour.

[0113] Step II: Calculate the difference between the geometric parameters of the first minimum geometric bounding box and the geometric parameters of the second minimum geometric bounding box.

[0114] Step III: When the similarity is greater than a preset similarity threshold and the difference value of the geometric parameter is less than a preset parameter threshold, it is determined that the first X-ray image frame and the second X-ray image frame are successfully matched. The geometric parameter includes the frame size and / or rotation angle of the bounding box in the image.

[0115] The similarity between the first outer contour and the second outer contour can be calculated using a classic algorithm or a deep learning model in the relevant technology. For example, the similarity between the first outer contour and the second outer contour can be calculated using a Hash algorithm, a SIFT (Scale-invariant feature transform) algorithm, a cosine similarity algorithm, or the like.

[0116] The geometric parameters may include parameters such as the border size and rotation angle of the bounding box (the first minimum geometric bounding box and the second minimum geometric bounding box) in the image. In an example, when the first minimum geometric bounding box is the first minimum rotated rectangle and the second minimum geometric bounding box is the second minimum rotated rectangle, its geometric parameters may be parameters such as the width, height and rotation angle of the first minimum rotated rectangle and the second minimum rotated rectangle; the difference values ​​of the geometric parameters include the width difference, height difference and rotation angle difference.

[0117] In an example, taking the first minimum geometric bounding box as the first minimum rotated rectangular box and the second minimum geometric bounding box as the second minimum rotated rectangular box as an example, the similarity between the first outer contour and the second outer contour is calculated. When the similarity is greater than the preset threshold of the match, it is considered that the first outer contour and the second outer contour are matched successfully. The difference in width, height and rotation angle between the first minimum geometric bounding box and the second minimum geometric bounding box is calculated. When the difference is less than the preset threshold of the match, it is considered that the first minimum geometric bounding box and the second minimum geometric bounding box are matched successfully. If the first outer contour and the second outer contour are matched successfully, and the first minimum geometric bounding box and the second minimum geometric bounding box are matched successfully, it is considered that the first X-ray image frame and the second X-ray image frame are matched successfully.

[0118] By using the method of the embodiment of the present application, in addition to matching the first outer contour with the second outer contour, matching the first minimum geometric bounding box with the second minimum geometric bounding box is also added during the matching process, which can significantly improve the accuracy of the matching. In addition, matching the first minimum geometric bounding box with the second minimum geometric bounding box also provides a prerequisite for subsequent image fusion. Only when the two match can the two be used for image fusion, which can improve the credibility of the fused image.

[0119] Step S205: When the first X-ray image frame and the second X-ray image frame are matched successfully, pixel fusion is performed on the first X-ray image frame and the second X-ray image frame according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box to obtain a fused image.

[0120] Among them, the fused image is used to determine whether the object to be detected is a contraband, and the contraband can be determined according to the actual usage scenario. When fusing images, the pixels of the first X-ray image frame and the second X-ray image frame can be superimposed according to their weights to obtain a fused image. The fusion of the first X-ray image frame and the second X-ray image frame is mainly to fuse the objects represented by the first outer contour and the second outer contour. Therefore, when determining the fusion start line and the fusion end line, the first outer contour and the second outer contour should be included in the fusion range. For example, the start line of the second minimum geometric bounding box is selected as the second fusion start line of the second X-ray image frame, the end line of the second minimum geometric bounding box is selected as the second fusion end line of the second X-ray image frame, and the start line of the first minimum geometric bounding box is selected as the first fusion start line of the first X-ray image frame. Align the first fusion start line with the second fusion start line and start fusion line by line until the fusion of the second fusion end line is completed, and the pixel lines starting from the second fusion start line to the second fusion end line in the second X-ray image frame are fused into the first X-ray image frame to obtain a fused image.

[0121] In other possible examples, in order to improve the accuracy of the fused image, the second fusion starting row can be respectively matched with each pixel row near the first fusion starting row (for example, the front and back A rows, A is a positive integer), and the grayscale difference between the first X-ray image frame and the second X-ray image frame when aligned with each pixel row is calculated respectively, so as to select the pixel row with the smallest grayscale difference as the first fusion starting row of the first X-ray image frame.

[0122] In the subsequent image fusion process, the fused image is used as the first X-ray image frame, and the newly acquired image is used as the second X-ray image frame to continue the next fusion.

[0123] By applying the method of the embodiment of the present application, the first outer contour and the second outer contour of the object to be detected in the first X-ray image frame and the second X-ray image frame can be detected by acquiring the first X-ray image frame and the second X-ray image frame, and the minimum geometric bounding box of the first outer contour and the second outer contour can be calculated; matching is performed according to the first outer contour, the second outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box, and when the matching is successful, the first X-ray image frame and the second X-ray image frame are pixel-fused to obtain a fused image, so that the fused image obtains more image information, thereby performing a security judgment based on the outer contour image of the object to be detected generated by the fused image. Compared with processing using a single frame image, the method of the embodiment of the present application improves the accuracy of the security inspection machine in acquiring the outer contour image, and thus the security inspectors are more accurate in judging whether the object to be detected is a prohibited item based on the fused image.

[0124] In a possible implementation, step S205 may be implemented by the following steps:

[0125] Step 1: According to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, determine the first pixel row corresponding to the fusion start row of the second X-ray image frame in the first X-ray image frame, and the second pixel row corresponding to the fusion end row of the second X-ray image frame in the first X-ray image frame.

[0126] In practical applications, the fusion start row and the fusion end row can be determined according to the offset values ​​of the first minimum geometric bounding box and the second minimum geometric bounding box. After determining the fusion start row and the fusion end row, the image difference between the first X-ray image frame and the second X-ray image frame is calculated to determine the first pixel row and the second pixel row corresponding to the fusion start row and the fusion end row in the first X-ray image frame. In a possible implementation, step one can be implemented by the following steps:

[0127] Step A: Select the minimum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion start row; select the maximum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion end row.

[0128] like Figure 4 The second minimum geometric bounding box corresponding to the object to be detected in the second X-ray image frame is shown. In which, x represents the number of columns of the image frame, y represents the number of rows of the image frame, and the second outer contour is mainly contained in the second minimum geometric bounding box. Therefore, the maximum row and the minimum row of the second minimum geometric bounding box can be determined as the fusion start row and the fusion end row, that is, the 5th row and the 18th row of the second X-ray image frame are the fusion start row and the fusion end row.

[0129] In practical applications, the row direction of the second X-ray image frame is parallel to the moving direction of the object to be detected. Figure 6 As shown, the moving direction of the object to be detected is the moving direction of the conveyor belt, and the position of the X-ray source is fixed. Therefore, when the object to be detected passes through the X-ray source, the row direction of the image frame obtained is parallel to the conveyor belt direction, and the row direction of the first X-ray image frame and the second X-ray image frame is parallel to the moving direction of the object to be detected. When the object to be detected moves on the conveyor belt, point P passes through the X-ray source first. Therefore, when the linear array scans and images, point P is also imaged first.

[0130] Step B: Obtain each preset offset value within a preset pixel row offset range; for each preset offset value, align the second X-ray image frame with the first X-ray image frame after offsetting it by the preset offset value, calculate the difference between each aligned pixel within a specified range between the aligned second X-ray image frame and the first X-ray image frame, and calculate the average of each difference to obtain the image difference under the preset offset value.

[0131] The specified range includes at least a portion of the first minimum geometric bounding box and the second minimum geometric bounding box, and the portion of the region must include the content of the first outer contour and the second outer contour. The specified range may also include the entire region of the first minimum geometric bounding box and the second minimum geometric bounding box.

[0132] The offset range can be set manually, for example, according to actual conditions, the 8 rows before and after the preset pixel row are determined as the offset range. The preset pixel row is the fusion start row or the fusion end row in the second X-ray image frame. In an example, the preset pixel row is the 5th row of the second X-ray image frame, then in the first X-ray image frame, the offset range of the 8 rows before and after the preset pixel row is from the -3rd row to the 13th row. Taking the 8th row of the first X-ray image frame as an example, its preset offset value relative to the 5th row of the second X-ray image frame is 3 rows, then the first X-ray image frame is shifted down by 3 rows to align the 8th row of pixels of the first X-ray image frame with the 5th row of the second X-ray image frame, and the difference between each aligned pixel of the first X-ray image frame and the second X-ray image frame is calculated to obtain the image difference when the 8th row of the first X-ray image frame is aligned with the 5th row of the second X-ray image frame.

[0133] Step C: Select the target preset offset value when the image difference is minimal; after the second X-ray image frame is offset by the target preset offset value, determine the first pixel row corresponding to the fusion start row in the first X-ray image frame, and the second pixel row corresponding to the fusion end row in the first X-ray image frame.

[0134] In one example, calculating the image difference may be calculating the average of the image differences between the first X-ray image frame and the second X-ray image frame, and selecting the pixel row with the smallest image difference as the first pixel row corresponding to the first X-ray image frame as the fusion start row. Figure 4 The 5th row of the second X-ray image frame is the fusion start row. Figure 5 After the image difference between each pixel row from row -3 to row 13 in the first X-ray image frame and the 5th row of the second X-ray image frame is aligned, if the image difference between the 8th row of the first X-ray image frame and the 5th row of the second X-ray image frame is the smallest, then the 8th row of the first X-ray image frame is determined to be the first pixel row corresponding to the fusion start row in the first X-ray image frame; calculated in the same way, the 21st row of the first X-ray image frame is obtained as the second pixel row corresponding to the fusion end row in the first X-ray image frame.

[0135] By applying the method of the embodiment of the present application, the image difference between the first X-ray image frame and the second X-ray image frame within the offset range can be calculated, and the first pixel row and the second pixel row corresponding to the fusion start row and the fusion end row in the first X-ray image frame can be determined according to the image difference, thereby facilitating subsequent image fusion calculations and improving the accuracy of image fusion.

[0136] Step 2: Perform pixel fusion on each pixel row from the fusion start row to the fusion end row in the second X-ray image frame and each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image.

[0137] In an example, starting from the fusion start row of the second X-ray image frame and the first pixel row of the first X-ray image frame, and ending with the fusion end row of the second X-ray image frame and the second pixel row of the first X-ray image frame, the pixel values ​​of the aligned corresponding pixel rows are superimposed according to a certain weight to obtain the fusion data of the fused image.

[0138] By applying the method of the embodiment of the present application, the first pixel row and the second pixel row corresponding to the fusion start row and the fusion end row in the second X-ray image frame in the first X-ray image frame are determined through the first minimum geometric bounding box and the second minimum geometric bounding box, so that image fusion is performed on the two image frames starting from the fusion start row of the second X-ray image frame and the first pixel row of the first X-ray image frame to the fusion end row of the second X-ray image frame and the second pixel row of the first X-ray image frame, so that the image range to be fused can be determined more accurately, thereby improving the accuracy of the fused image.

[0139] In a possible implementation, the first X-ray image frame and the second X-ray image frame are two X-ray image frames collected one after the other, or the first X-ray image frame is an image frame synthesized from multiple X-ray image frames collected before the second X-ray image frame. When the first X-ray image frame is an image frame synthesized from multiple X-ray image frames collected before the second X-ray image frame, that is, the first X-ray image frame already contains fusion information of the previous frames, then the pixel value of the first X-ray image frame has a greater weight than that of the second X-ray image frame. The fused image can be calculated using the following fusion function;

[0140] The fusion function is:

[0141]

[0142] Among them, G(x i ,y j ) indicates that the coordinates in the fused image are (x i ,y j ) pixel value, N represents the number of times the first X-ray image frame has been fused, G1(x i ,y j+d ) indicates that the coordinates in the first X-ray image frame are (x i ,y j+d ) pixel value, G2(x i ,y j ) indicates that the middle coordinate of the second X-ray image frame is (x i ,y j ) pixel value, where i∈[1, a], j∈[b, c], a is the total number of pixel columns in the first X-ray image frame, b is the number of the fusion start row in the second X-ray image frame, c is the number of the fusion end row in the second X-ray image frame, and d is the difference between the number of the fusion start row and the number of the first pixel row.

[0143] By applying the method of the embodiment of the present application, the image data of the first X-ray image frame and the second X-ray image frame can be fused through a fusion function, and the weight of the first X-ray image frame is larger, so that the fused image can obtain more fusion information, thereby improving the accuracy of the fused image.

[0144] In a possible implementation, the first minimum geometric bounding box is a first minimum rotated rectangular box, and the second minimum geometric bounding box is a second minimum rotated rectangular box; step S203 may be implemented by the following steps:

[0145] Step 1: Extract the convex edge vertices of the first outer contour and the second outer contour respectively to obtain a plurality of first convex edge vertices and a plurality of second convex edge vertices.

[0146] The convex edge vertex refers to the vertex of the convex hull of the first outer contour and the second outer contour. In an example, Figure 7 As shown in the figure, the outer contour point set of the object to be detected is from point a to point i, and the convex hull formed by them is a polygon connected by solid lines in the figure. The convex edge vertices of the outer contour are point a, point c, point g, point h and point i. When identifying the convex edge vertices, the contour points can be arranged in an orderly manner so that they can be traversed in a certain order to determine whether each contour point is a convex edge vertex.

[0147] Step 2: Take the straight line determined by any two adjacent first convex edge vertices as the straight line where the side of the circumscribed rectangle is located, determine the first minimum circumscribed rectangles that include the first outer contour, and select the first minimum circumscribed rectangle with the smallest area to obtain the first minimum geometric bounding box.

[0148] In an example, Figure 8 As shown, for Figure 7 The convex hull composed of the convex edge vertices shown in the figure has a total of 5 convex edges. Then, a minimum circumscribed rectangle can be determined for the straight line determined by every two adjacent convex edge vertices. Figure 8 In the convex hull shown, the minimum circumscribed rectangle corresponding to the ac contour is rectangle aDHK, the cg contour and the hi contour are perpendicular to each other, so their corresponding minimum circumscribed rectangles are the same, which is rectangle AEiL, the minimum circumscribed rectangle corresponding to the gh contour is rectangle CGJa, and the minimum circumscribed rectangle corresponding to the ai contour is rectangle aBFi. Compare the areas of each rectangle and select the rectangle with the smallest area as the minimum geometric bounding box.

[0149] Step 3: Take the straight line determined by any two adjacent second convex edge vertices as the straight line where the side of the circumscribed rectangle is located, determine each second minimum circumscribed rectangle that includes the second outer contour, and select the second minimum circumscribed rectangle with the smallest area to obtain the second minimum geometric bounding box.

[0150] In practical applications, there is no order between step 2 and step 3, and they can be performed sequentially or simultaneously. After calculating the first minimum enclosing rectangle and the second minimum enclosing rectangle, a bubble sort algorithm can be used to obtain the first minimum geometric bounding box and the second minimum geometric bounding box. In an example, Fig. 9 The process shown determines the minimum geometric bounding box. First, obtain the outer contour point set (that is, obtain the first outer contour or the second outer contour), and sort the outer contour point set (that is, sort the first outer contour or the second outer contour); according to the order of the outer contour point set, calculate the convex edge points of the outer contour point set to obtain the first convex edge vertex or the second convex edge vertex, and for the straight line determined by every two adjacent convex edge vertices, calculate the minimum circumscribed rectangle area of ​​the current index convex edge, where index refers to the straight line index determined by the current convex edge vertex. Determine whether the calculated minimum circumscribed rectangle is the rotated rectangle with the smallest area. If so, determine that the minimum circumscribed rectangle is the minimum geometric bounding box; if not, continue to calculate and determine the convex edge straight line index+1.

[0151] By applying the method of the embodiment of the present application, the convex hull vertices of the first outer contour and the second outer contour are calculated through the convex hull algorithm, and each minimum circumscribed rectangle is calculated through the convex edge vertices, so as to determine the minimum geometric bounding box through the area of ​​the minimum circumscribed rectangle. This can make the determined first minimum geometric bounding box and the second minimum geometric bounding box more accurate, thereby improving the accuracy of subsequent image matching and image fusion.

[0152] In a possible implementation, step S201 may be implemented by the following steps:

[0153] Step (1): binarizing the first X-ray image frame and the second X-ray image frame respectively to obtain a first binarized image and a second binarized image.

[0154] The image binarization process may first grayscale the image, perform grayscale histogram statistics, determine the binarization threshold according to the distribution of grayscale values, and then binarize each pixel of the first X-ray image frame and the second X-ray image frame according to the binarization threshold. Specifically, the image binarization may be performed with reference to the above embodiment of step S201.

[0155] Step (2): for each first pixel point of the first binary image, determine whether the pixel values ​​of each pixel point adjacent to the first pixel point are all the first pixel values; if not, consider the first pixel point to be a first outer contour point.

[0156] The value of the pixel representing the object to be detected is the first pixel value; the pixels adjacent to the first pixel are eight adjacent pixels in the up, down, left, right and diagonal directions of the first pixel.

[0157] Step (3): for each second pixel point of the second binary image, determine whether the pixel values ​​of each pixel point adjacent to the second pixel point are all the first pixel values; if not, consider the second pixel point to be a second outer contour point.

[0158] Among them, the pixel points adjacent to the second pixel point are eight adjacent pixel points in the up, down, left, right and diagonal directions of the second pixel point. In practical applications, there is no order between step (2) and step (3), and they can be executed simultaneously or sequentially. After the first X-ray image frame and the second X-ray image frame are binarized, the value of each pixel point is "1" or "0" to indicate that each pixel point is a pixel point corresponding to the object to be detected or a pixel point corresponding to the object not to be detected.

[0159] In one example, in the first binary image or the second binary image, the pixel corresponding to the object to be detected is represented by "1", and "1" is the first pixel value. Fig.10 As shown, black represents the pixel value of "1" and white represents the pixel value of "0". For the first pixel point or the second pixel point, there are three adjacent pixel points with opposite values, and their pixel values ​​are "0". Therefore, the first pixel point or the second pixel point is considered to be the first outer contour point or the second outer contour point.

[0160] For the first binary image or the second binary image, traverse each pixel point and execute the operation of step (2) or step (3) to obtain the first outer contour point set or the second outer contour point set, and the outer contour point set constitutes the first outer contour or the second outer contour.

[0161] By applying the method of the embodiment of the present application, each pixel point can be judged through the eight-neighborhood line patrol algorithm, and whether the pixel point is a contour point can be judged through the pixel values ​​of each pixel point adjacent to the pixel point, thereby obtaining the first outer contour and the second outer contour, making the first outer contour and the second outer contour more complete and accurate.

[0162] In a possible implementation manner, it is further necessary to perform an erosion operation on the first binary image or the second binary image. For example, the erosion operation may be performed by the following steps:

[0163] Step a: For each first outer contour point, calculate the first number of first outer contour points that belong to the same connected domain as the first outer contour point. If the first number is less than a first preset threshold, remove the first outer contour point to obtain the first outer contour of the object to be detected in the first X-ray image frame.

[0164] In one example, in order to obtain a more accurate first outer contour, the first outer contour points can also be filtered. A first outer contour point that has not been selected is selected from each first outer contour point, and the first number of first outer contour points that belong to the same connected domain as the current first outer contour point is calculated. If the first number is less than a first preset threshold, the current first outer contour point is removed, otherwise the current first outer contour point is retained; the above process is repeated until there are no first outer contour points that have not been selected, and the contour composed of the remaining first outer contour points is used as the first outer contour.

[0165] Step b: For each second outer contour point, calculate the second number of second outer contour points that belong to the same connected domain as the second outer contour point. If the second number is less than the first preset threshold, remove the second outer contour point to obtain the second outer contour of the object to be detected in the second X-ray image frame.

[0166] The filtering method of the second outer contour points is similar to the filtering method of the first outer contour points, which will not be repeated here.

[0167] Similarly, step a and step b have no order of precedence, and can be executed simultaneously or sequentially. Among them, the connected domain can be determined according to the actual application, which is the influence range of the corrosion operation determined artificially. The connected domain can be of any shape and size. For example, the connected domain can be a rectangle, an ellipse, a cross, etc. The relevant technicians can control the fineness of the corrosion operation by adjusting the size and shape of the connected domain. The first preset value can be determined according to the actual use requirements. For example, the first preset threshold can be set to 9, 8, 6, etc. In an example, the first preset value is 9. For a certain outer contour point, since the outer contour point represents the object to be detected, its pixel value is the first pixel value, for example, "1". In the connected domain of the pixel point, if there are 12 points with a pixel value of "0", and there are only 3 outer contour points with a pixel value of "1", then the outer contour point may be "surrounded" by pixel points representing the background or non-objects to be detected, forming an isolated point, so the outer contour point should be removed.

[0168] By applying the method of the embodiment of the present application, the isolated or unstable boundary pixels can be removed by performing an erosion operation on the first X-ray image frame and the second X-ray image frame after binarization, so that the contour line of the binarized image is more accurate and can reflect the actual edge structure of the target object.

[0169] The following example is used to illustrate the security inspection image fusion method in the embodiment of the present application. Fig.11As shown, for a current frame image, i.e., the second X-ray image frame, the outer contour geometric features of the current frame are calculated, i.e., contour detection is performed on the second X-ray image frame to obtain the second outer contour; the outer contour area and position of the current frame image are matched with those of the previous frame image; wherein the previous frame image is the first X-ray image frame, and contour detection is also performed on the first X-ray image frame to obtain the first outer contour.

[0170] According to the area and shape of the first outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box are calculated, and matching is performed based on the first minimum geometric bounding box and the second minimum geometric bounding box. If the match is successful, the fusion start row and the fusion end row are calculated. If the match fails, the image is not processed and is directly output.

[0171] When calculating the fusion start line and the fusion end line, the pixel rows corresponding to the fusion start line and the fusion end line on the two frames are determined according to the offset range between the current frame image and the previous frame image, and then the previous and next frame images are fused and the fused image is output.

[0172] By applying the method of the embodiment of the present application, two frames of images of the object to be detected are fused to obtain more image information of the object to be detected, which can improve the image signal-to-noise ratio, enhance the image quality, and improve the accuracy of the outer contour image generated by the security inspection machine. In addition, for old security inspection equipment whose hardware cannot be upgraded, the method of the embodiment of the present application is to upgrade the software, which can improve the accuracy of security inspection while saving the cost of hardware upgrade.

[0173] In a second aspect of the present application, a security inspection image fusion device is provided, the device comprising: Fig.12 The structure shown:

[0174] The image frame acquisition module 1201 is used to acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected.

[0175] The contour detection module 1202 is used to perform contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame.

[0176] The minimum geometric bounding box calculation module 1203 is used to calculate the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, and obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes.

[0177] The image frame matching module 1204 is used to match the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box and the second minimum geometric bounding box.

[0178] The image fusion module 1205 is used to perform pixel fusion on the first X-ray image frame and the second X-ray image frame to obtain a fused image according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box when the first X-ray image frame and the second X-ray image frame are successfully matched. The fused image is used to determine whether the object to be detected is a prohibited item.

[0179] In a possible implementation, the image frame matching module includes:

[0180] The similarity calculation submodule is specifically used to calculate the similarity between the first outer contour and the second outer contour.

[0181] The difference value calculation submodule is specifically used to calculate the difference value between the geometric parameters of the first minimum geometric bounding box and the geometric parameters of the second minimum geometric bounding box; wherein the geometric parameters include the frame size and / or rotation angle of the bounding box in the image.

[0182] The matching judgment submodule is specifically used to determine that the first X-ray image frame and the second X-ray image frame are successfully matched when the similarity is greater than a preset similarity threshold and the difference value of the geometric parameter is less than a preset parameter threshold.

[0183] In a possible implementation, the image fusion module includes:

[0184] The pixel row determination submodule is specifically used to determine the first pixel row corresponding to the fusion start row of the second X-ray image frame in the first X-ray image frame, and the second pixel row corresponding to the fusion end row of the second X-ray image frame in the first X-ray image frame according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box.

[0185] The pixel fusion submodule is specifically used to perform pixel fusion on each pixel row from the fusion start row to the fusion end row in the second X-ray image frame and each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image.

[0186] In a possible implementation manner, the pixel row determination submodule includes:

[0187] The fusion row determination unit is specifically used to select the minimum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion start row; select the maximum row where the pixels in the second minimum geometric bounding box are located to obtain the fusion end row.

[0188] The image difference calculation unit is specifically used to obtain each preset offset value within a preset pixel row offset range; for each preset offset value, the second X-ray image frame is offset by the preset offset value and then aligned with the first X-ray image frame, the difference between each aligned pixel within a specified range between the aligned second X-ray image frame and the first X-ray image frame is calculated, and the average of each difference is calculated to obtain the image difference under the preset offset value; wherein the specified range includes at least a portion of the first minimum geometric bounding box and the second minimum geometric bounding box.

[0189] The fusion pixel row determination unit is specifically used to select a target preset offset value when the image difference is minimized; after the second X-ray image frame is offset by the target preset offset value, determine the first pixel row corresponding to the fusion start row in the first X-ray image frame, and the second pixel row corresponding to the fusion end row in the first X-ray image frame.

[0190] In a possible implementation, the first X-ray image frame and the second X-ray image frame are two X-ray image frames collected one after the other, or the first X-ray image frame is an image frame synthesized from multiple X-ray image frames collected before the second X-ray image frame; the pixel fusion submodule includes:

[0191] The pixel fusion calculation unit is specifically used to calculate and obtain a fused image through the following fusion function.

[0192] The fusion function is:

[0193]

[0194] Among them, G(x i ,y j ) indicates that the coordinates in the fused image are (x i ,y j ) pixel value, N represents the number of times the first X-ray image frame has been fused, G1(x i ,y j+d ) indicates that the coordinates in the first X-ray image frame are (x i ,y j+d ) pixel value, G2(x i ,y j ) indicates that the middle coordinate of the second X-ray image frame is (x i ,y j ) pixel value, where i∈[1, a], j∈[b, c], a is the total number of pixel columns in the first X-ray image frame, b is the number of the fusion start row in the second X-ray image frame, c is the number of the fusion end row in the second X-ray image frame, and d is the difference between the number of the fusion start row and the number of the first pixel row.

[0195] In a possible implementation, the first minimum geometric bounding box is a first minimum rotated rectangular box, and the second minimum geometric bounding box is a second minimum rotated rectangular box; the minimum geometric bounding box calculation module includes:

[0196] The convex edge vertex extraction submodule is specifically used to extract the convex edge vertices of the first outer contour and the second outer contour respectively, and obtain a plurality of first convex edge vertices and a plurality of second convex edge vertices.

[0197] The first minimum geometric bounding box determination submodule is specifically used to determine the first minimum circumscribed rectangles including the first outer contour inside by taking the straight line determined by any two adjacent first convex edge vertices as the straight line where the side of the circumscribed rectangle is located, and select the first minimum circumscribed rectangle with the smallest area to obtain the first minimum geometric bounding box.

[0198] The second minimum geometric bounding box determination submodule is specifically used to determine the second minimum circumscribed rectangles including the second outer contour inside, using the straight line determined by any two adjacent second convex edge vertices as the straight line where the side of the circumscribed rectangle is located, and select the second minimum circumscribed rectangle with the smallest area to obtain the second minimum geometric bounding box.

[0199] In a possible implementation, the contour detection module includes:

[0200] The image binarization submodule is specifically used to binarize the first X-ray image frame and the second X-ray image frame respectively to obtain a first binarized image and a second binarized image.

[0201] The first pixel point judgment submodule is specifically used to judge, for each first pixel point of the first binary image, whether the pixel values ​​of each pixel point adjacent to the first pixel point are all the first pixel values; if not, the first pixel point is considered to be the first outer contour point; wherein the value of the pixel point representing the object to be detected is the first pixel value.

[0202] The second pixel point judgment submodule is specifically used to judge, for each second pixel point of the second binary image, whether the pixel values ​​of each pixel point adjacent to the second pixel point are all the first pixel values; if not, the second pixel point is considered to be a second outer contour point.

[0203] The first image corrosion submodule is specifically used to calculate, for each first outer contour point, the first number of first outer contour points that belong to the same connected domain as the first outer contour point. If the first number is less than a first preset threshold, the first outer contour point is removed to obtain the first outer contour of the object to be detected in the first X-ray image frame.

[0204] The second image corrosion submodule is specifically used to calculate, for each second outer contour point, a second number of second outer contour points belonging to the same connected domain as the second outer contour point; if the second number is less than a first preset threshold, the second outer contour point is removed to obtain the second outer contour of the object to be detected in the second X-ray image frame.

[0205] By using the device of the embodiment of the present application, two frames of images of the object to be detected are fused to obtain more image information of the object to be detected, which can improve the image signal-to-noise ratio, enhance the image quality, and improve the accuracy of the outer contour image generated by the security inspection machine. In addition, for old security inspection equipment whose hardware cannot be upgraded, the method of the embodiment of the present application is to upgrade the software, which can improve the accuracy of security inspection while saving the cost of hardware upgrade.

[0206] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0207] Memory 1301, used for storing computer programs;

[0208] The processor 1302 is used to implement the following steps when executing the program stored in the memory 1301:

[0209] Acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected;

[0210] Performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame;

[0211] Calculating the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes;

[0212] Matching the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box;

[0213] When the first X-ray image frame and the second X-ray image frame are matched successfully, the first X-ray image frame and the second X-ray image frame are pixel-fused according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box to obtain a fused image, and the fused image is used to determine whether the object to be detected is a prohibited item.

[0214] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0215] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0216] The communication interface is used for communication between the above electronic device and other devices.

[0217] Among them, the electronic device of the embodiment of the present application can be a device that performs imaging by penetrating rays through the object to be detected, such as an X-ray security inspection machine, a baggage security inspection machine, a millimeter wave security inspection machine and other equipment. The electronic device includes a ray source, a ray receiver, a transmission device for the object to be detected, an image imaging device and other components.

[0218] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0219] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0220] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned security inspection image fusion methods are implemented.

[0221] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any security inspection image fusion method in the above embodiments.

[0222] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0223] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0224] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0225] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A security inspection image fusion method, characterized in that: The method comprises: Acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected; Performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame; Calculating the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes; Matching the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box; When the first X-ray image frame and the second X-ray image frame are matched successfully, the first X-ray image frame and the second X-ray image frame are pixel-fused according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box to obtain a fused image, and the fused image is used to determine whether the object to be detected is a prohibited item.

2. The method according to claim 1, characterized in that The matching of the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box includes: Calculating the similarity between the first outer contour and the second outer contour; Calculating a difference between geometric parameters of the first minimum geometric bounding box and geometric parameters of the second minimum geometric bounding box; wherein the geometric parameters include a frame size and / or a rotation angle of the bounding box in the image; When the similarity is greater than a preset similarity threshold and the difference value of the geometric parameter is less than a preset parameter threshold, it is determined that the first X-ray image frame matches the second X-ray image frame successfully.

3. The method according to claim 1, characterized in that The step of performing pixel fusion on the first X-ray image frame and the second X-ray image frame to obtain a fused image according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box includes: Determine, according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, a first pixel row corresponding to a fusion start row of the second X-ray image frame in the first X-ray image frame, and a second pixel row corresponding to a fusion end row of the second X-ray image frame in the first X-ray image frame; Each pixel row from the fusion start row to the fusion end row in the second X-ray image frame is pixel-fused with each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image.

4. The method according to claim 3, characterized in that Determining, according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box, a first pixel row corresponding to a fusion start row of the second X-ray image frame in the first X-ray image frame, and a second pixel row corresponding to a fusion end row of the second X-ray image frame in the first X-ray image frame, comprising: Select the minimum row where the pixels in the second minimum geometric bounding box are located to obtain a fusion start row; select the maximum row where the pixels in the second minimum geometric bounding box are located to obtain a fusion end row; Obtaining each preset offset value within a preset pixel row offset range; for each preset offset value, aligning the second X-ray image frame with the first X-ray image frame after offsetting it by the preset offset value, calculating the difference between each aligned pixel within a specified range between the aligned second X-ray image frame and the first X-ray image frame, and calculating the average of each difference to obtain the image difference under the preset offset value; wherein the specified range includes at least a portion of the first minimum geometric bounding box and the second minimum geometric bounding box; Select a target preset offset value when the image difference is minimal; after the second X-ray image frame is offset by the target preset offset value, determine the first pixel row corresponding to the fusion start row in the first X-ray image frame, and the second pixel row corresponding to the fusion end row in the first X-ray image frame.

5. The method according to claim 3, characterized in that: The first X-ray image frame and the second X-ray image frame are two X-ray image frames acquired one after the other, or the first X-ray image frame is an image frame synthesized from multiple X-ray image frames acquired before the second X-ray image frame; The step of performing pixel fusion on each pixel row from the fusion start row to the fusion end row in the second X-ray image frame and each pixel row from the first pixel row to the second pixel row in the first X-ray image frame to obtain a fused image comprises: The fused image is obtained by calculating the following fusion function; The fusion function is: Among them, G(x i ,y j ) indicates that the coordinates in the fused image are (x i ,y j ) pixel value, N represents the number of times the first X-ray image frame has been fused, G1(x i ,y j+d ) indicates that the coordinates in the first X-ray image frame are (x i ,y j+d ) pixel value, G2(x i ,y j ) indicates that the middle coordinate of the second X-ray image frame is (x i ,y j ) pixel value, wherein i∈[1, a], j∈[b, c], a is the total number of pixel columns in the first X-ray image frame, b is the number of fusion start rows in the second X-ray image frame, c is the number of fusion end rows in the second X-ray image frame, and d is the difference between the number of fusion start rows and the number of the first pixel rows.

6. The method according to claim 1, characterized in that The first minimum geometric bounding box is a first minimum rotated rectangular box, and the second minimum geometric bounding box is a second minimum rotated rectangular box; The step of respectively calculating the minimum geometric bounding boxes of the first outer contour and the second outer contour to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour includes: Extracting convex edge vertices of the first outer contour and the second outer contour respectively to obtain a plurality of first convex edge vertices and a plurality of second convex edge vertices; Taking the straight line determined by any two adjacent first convex edge vertices as the straight line where the side of the circumscribed rectangle is located, determining each first minimum circumscribed rectangle including the first outer contour inside, and selecting the first minimum circumscribed rectangle with the smallest area to obtain a first minimum geometric bounding box; The straight line determined by any two adjacent second convex edge vertices is used as the straight line where the side of the circumscribed rectangle is located, and each second minimum circumscribed rectangle including the second outer contour is determined, and the second minimum circumscribed rectangle with the smallest area is selected to obtain the second minimum geometric bounding box.

7. The method according to claim 1, characterized in that The step of performing contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame to obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame includes: Binarizing the first X-ray image frame and the second X-ray image frame respectively to obtain a first binarized image and a second binarized image; For each first pixel point of the first binary image, determine whether the pixel values ​​of each pixel point adjacent to the first pixel point are all the first pixel value, if not, consider the first pixel point to be a first outer contour point; wherein the value of the pixel point representing the object to be detected is the first pixel value; For each second pixel point of the second binary image, determine whether pixel values ​​of each pixel point adjacent to the second pixel point are all the first pixel value; if not, consider the second pixel point to be a second outer contour point; For each first outer contour point, calculate a first number of first outer contour points that belong to the same connected domain as the first outer contour point, and if the first number is less than a first preset threshold, remove the first outer contour point to obtain the first outer contour of the object to be detected in the first X-ray image frame; For each second outer contour point, calculate the second number of second outer contour points that belong to the same connected domain as the second outer contour point; if the second number is less than the first preset threshold, remove the second outer contour point to obtain the second outer contour of the object to be detected in the second X-ray image frame.

8. A security inspection image fusion device, characterized in that: The device comprises: An image frame acquisition module, used to acquire a first X-ray image frame and a second X-ray image frame; wherein the first X-ray image frame and the second X-ray image frame are security inspection images of the same object to be detected; a contour detection module, configured to perform contour detection on the object to be detected in the first X-ray image frame and the second X-ray image frame, and obtain a first outer contour of the object to be detected in the first X-ray image frame and a second outer contour of the object to be detected in the second X-ray image frame; a minimum geometric bounding box calculation module, used to calculate the minimum geometric bounding boxes of the first outer contour and the second outer contour respectively, to obtain a first minimum geometric bounding box for enclosing the first outer contour and a second minimum geometric bounding box for enclosing the second outer contour; wherein the first minimum geometric bounding box and the second minimum geometric bounding box have the same or similar geometric shapes; an image frame matching module, configured to match the first X-ray image frame and the second X-ray image frame according to the first outer contour, the second outer contour, the first minimum geometric bounding box, and the second minimum geometric bounding box; An image fusion module is used to perform pixel fusion on the first X-ray image frame and the second X-ray image frame to obtain a fused image according to the positions of the first minimum geometric bounding box and the second minimum geometric bounding box when the first X-ray image frame and the second X-ray image frame are successfully matched, and the fused image is used to determine whether the object to be detected is a prohibited item.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images