Package matching method based on visible light image and X-ray image

Abstract

The present invention provides a package matching method based on visible light images and X-ray images, comprising: obtaining a visible light image and an X-ray image; determining a first package detection frame in the visible light image and a second package detection frame in the X-ray image; based on the first center point coordinates of the nth first package detection frame and the second center point coordinates of the nth second package detection frame, in the case where it is determined that the nth first package detection frame and the nth second package detection frame fail to match, determining a target image with a missing package detection frame; in the case where the target image is a visible light image and the nth second package detection frame includes contraband, based on the second center point coordinates of the nth second package detection frame, adding a target first package detection frame. The embodiment of the present invention can correspond the package detection frame in the visible light image to the package detection frame in the X-ray image, quickly and accurately locate the target package in the visible light image, and improve the efficiency of security inspection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a package matching method based on visible light images and X-ray images. Background Art

[0002] During the security inspection, the security inspection process of the package by the security inspection machine usually takes a period of time, that is, the security inspection machine continuously passes the package. In addition, the visible light imaging device will also output a continuous video image of the security inspection area. In the video image, the package will continue to move with the conveyor belt of the security inspection machine, and the X-ray image output by the security inspection machine after scanning the package is also a continuous long image with the same output speed as the conveyor belt movement speed of the security inspection machine.

[0003] Usually, at any time t, a visible light image of the security inspection area obtained by the visible light imaging device and an X-ray image corresponding to the visible light image can be obtained, and the X-ray image is equivalent to a portion intercepted from a continuous long image. When performing security inspections, the target detection model can be used to perform image recognition on the visible light image and the X-ray image to perform package target detection, and the portions detected as packages in the visible light image and the X-ray image are output respectively, where these portions are selected by the package target detection frame, and the two form a pair of initial image groups. In this way, security inspections can be performed based on the initial image group.

[0004] Since the target detection model will inevitably miss detections, this will cause the package in the visible light image to not be selected by the detection box, that is, missed detection, which will cause the subsequent visible light image and multiple packages in the X-ray image to be unable to match one-to-one. When a suspicious package is found in the X-ray image, the operator will not be able to quickly and accurately find the physical target package and open it for inspection in time, resulting in low security inspection efficiency. Summary of the invention

[0005] The present invention provides a package matching method based on visible light images and X-ray images, which is used to solve the defect in the prior art that multiple packages in visible light images and X-ray images cannot be matched one-to-one, resulting in low security inspection efficiency, and achieves the purpose of matching multiple packages in visible light images and X-ray images one-to-one to improve security inspection efficiency.

[0006] The present invention provides a package matching method based on visible light images and X-ray images, comprising:

[0007] Obtain visible light images and X-ray images taken at the same time;

[0008] respectively determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image;

[0009] Based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, determining whether the nth first package detection frame and the nth second package detection frame match, wherein the matching result is used to indicate whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer;

[0010] In the case where the nth first package detection frame and the nth second package detection frame fail to match, determining a target image in which the package detection frame is missing;

[0011] When the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, a target first package detection frame is added to the visible light image based on the second center point coordinates of the nth second package detection frame, and the target first package detection frame is used to select the target package.

[0012] According to a package matching method based on visible light images and X-ray images provided by the present invention, adding a target first package detection frame in the visible light image based on the second center point coordinates of the nth second package detection frame includes:

[0013] Determine, based on the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image, a mapping coefficient for converting a second coordinate system where the X-ray image is located to a first coordinate system where the visible light image is located;

[0014] Determining the target center point coordinates of the target first package detection frame in the visible light image based on the second center point coordinates of the nth second package detection frame and the mapping coefficient;

[0015] Based on the size of the nth second package detection frame and the target center point coordinates, the target first package detection frame is added to the visible light image.

[0016] According to a package matching method based on a visible light image and an X-ray image provided by the present invention, the method determines whether the nth first package detection frame matches the nth second package detection frame based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, including:

[0017] A first coordinate system is constructed with the lower left corner of the visible light image as a first origin, and a second coordinate system is constructed with the lower left corner of the X-ray image as a second origin;

[0018] When n is 1, a first distance is determined based on the coordinates of the first center point of the nth first parcel detection frame and the coordinates of the first origin, and a second distance is determined based on the coordinates of the second center point of the nth second parcel detection frame and the coordinates of the second origin; the first first parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the visible light image, and the first second parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the X-ray image;

[0019] determining a first ratio of the first distance to the second distance;

[0020] When the first ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match;

[0021] When the first ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

[0022] According to a package matching method based on visible light images and X-ray images provided by the present invention, the step of determining a target image with a missing package detection frame includes:

[0023] When the first ratio is less than a lower limit of the threshold range, determining that the visible light image is a target image with a missing package detection frame;

[0024] When the first ratio is greater than the upper limit value of the threshold range, the X-ray image is determined to be a target image with a missing package detection frame.

[0025] According to a package matching method based on visible light images and X-ray images provided by the present invention, the method further includes:

[0026] When the first ratio is less than the lower limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

[0027] According to a package matching method based on visible light images and X-ray images provided by the present invention, the method further includes:

[0028] When the first ratio is greater than an upper limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

[0029] According to a package matching method based on a visible light image and an X-ray image provided by the present invention, the method determines whether the nth first package detection frame matches the nth second package detection frame based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, including:

[0030] When n is greater than 1 and the n-1th first package detection frame and the n-1th second package detection frame are matched successfully, a third distance is determined based on the first center point coordinates of the n-1th first package detection frame and the first center point coordinates of the n-1th first package detection frame, and a fourth distance is determined based on the second center point coordinates of the n-2nd package detection frame and the second center point coordinates of the n-1th second package detection frame; the n-1th first package detection frame is on the left side of the n-1th first package detection frame; the n-2nd package detection frame is on the left side of the n-1th second package detection frame;

[0031] determining a second ratio of the third distance to the fourth distance;

[0032] When the second ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match;

[0033] When the second ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

[0034] According to a package matching method based on visible light images and X-ray images provided by the present invention, the step of determining a target image with a missing package detection frame includes:

[0035] When the second ratio is greater than an upper limit value of the threshold range, determining that the visible light image is a target image with a missing package detection frame;

[0036] When the second ratio is less than the lower limit of the threshold range, the X-ray image is determined to be a target image with a missing package detection frame.

[0037] According to a package matching method based on visible light images and X-ray images provided by the present invention, the method further includes:

[0038] When the second ratio is smaller than a lower limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

[0039] According to a package matching method based on visible light images and X-ray images provided by the present invention, the method further includes:

[0040] When the second ratio is greater than the upper limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

[0041] The present invention also provides a package matching device based on visible light images and X-ray images, comprising:

[0042] An acquisition module, used for acquiring visible light images and X-ray images taken at the same time;

[0043] a determination module, configured to respectively determine at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image;

[0044] The determination module is further used to determine whether the nth first package detection frame and the nth second package detection frame match based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, wherein the matching result is used to indicate whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer;

[0045] The determination module is further used to determine the target image with the missing package detection frame when the nth first package detection frame and the nth second package detection frame fail to match;

[0046] An adding module is used to add a target first package detection frame in the visible light image based on the second center point coordinates of the nth second package detection frame when the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, and the target first package detection frame is used to select the target package.

[0047] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the package matching method based on visible light images and X-ray images as described above is implemented.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the package matching method based on visible light images and X-ray images as described in any one of the above is implemented.

[0049] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned package matching methods based on visible light images and X-ray images.

[0050] The package matching method based on visible light images and X-ray images provided by the present invention obtains a visible light image and an X-ray image taken at the same time, and respectively determines at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image. When it is determined that the nth first package detection frame and the nth second package detection frame do not correspond to the same package based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, a target image with a missing package detection frame is determined. When the target image is a visible light image and the target package represented by the nth second package detection frame includes contraband, a target first package detection frame is added to the visible light image based on the second center point coordinates of the nth second package detection frame, and the target first package detection frame is used to select the target package. When a package is missed in the visible light image, the target first package detection frame corresponding to the same package as the nth second package detection frame in the X-ray image can be determined based on the second center point coordinates of the nth second package detection frame in the X-ray image, so that the target package can be quickly and accurately located in the visible light image, which makes it convenient for security inspectors to accurately find the target package and open it for inspection in time, thereby improving security inspection efficiency. In addition, since the target package can be accurately located from the visible light image, the phenomenon of missing suspicious packages from security inspection can be avoided, thereby improving the safety of transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 This is a schematic diagram of security inspection equipment;

[0053] Figure 2 A schematic diagram of a process flow of a package matching method based on visible light images and X-ray images provided in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a first package detection frame in a visible light image provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a second package detection frame in an X-ray image provided by an embodiment of the present invention;

[0056] Figure 5 One of the schematic diagrams of adding a target first package detection frame in a visible light image provided by an embodiment of the present invention;

[0057] Figure 6 A schematic diagram of deleting the first second package detection frame from an X-ray image provided by an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of deleting the first first package detection frame from a visible light image provided by an embodiment of the present invention;

[0059] Figure 8 A schematic diagram of determining a third distance in a visible light image provided by an embodiment of the present invention;

[0060] Fig. 9 A schematic diagram of determining a fourth distance in an X-ray image provided by an embodiment of the present invention;

[0061] Fig.10 A schematic diagram of a package detection frame that is missed in a visible light image provided by an embodiment of the present invention;

[0062] Fig.11 A schematic diagram of a package detection frame that is missed in an X-ray image provided by an embodiment of the present invention;

[0063] Fig.12 A second schematic diagram of adding a target first package detection frame in a visible light image provided by an embodiment of the present invention;

[0064] Fig.13 A schematic diagram showing that a package detection frame is missed in a visible light image and a target package in an X-ray image does not include contraband provided by an embodiment of the present invention;

[0065] Fig.14 A schematic diagram of deleting the nth second package detection frame from an X-ray image provided by an embodiment of the present invention;

[0066] Fig.15 A schematic diagram of deleting the nth first package detection frame from a visible light image provided by an embodiment of the present invention;

[0067] Fig.16 A schematic diagram of the structure of a package matching device based on visible light images and X-ray images provided by an embodiment of the present invention;

[0068] Fig.17 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Figure 1 It is a schematic diagram of security inspection equipment, such as Figure 1 As shown, the security inspection equipment generally includes a visible light imaging device, an X-ray security inspection machine and a conveyor belt arranged on the security inspection table. In addition, the visible light imaging device can be, for example, a high-definition camera. Currently, the package can be scanned by the X-ray security inspection machine to obtain an X-ray image, and the visible light image can be obtained by the visible light imaging device.

[0071] After the object detection model identifies the packages in the X-ray image and the visible light image, the detection frame is used to select the identified packages, so that the packages in the X-ray image and the packages in the visible light image are matched one by one. During the security inspection process, when a suspicious package is found in the X-ray image, the security inspector can accurately find the physical target package through the visible light image through the above correspondence, so as to open the package for inspection.

[0072] However, since the target detection model will inevitably miss some packages in the visible light image or X-ray image, some packages will not be selected by the detection frame, which will cause multiple packages in the visible light image and X-ray image to be unable to match one to one. When a suspicious package is found in the X-ray image, the security inspectors will not be able to quickly and accurately find the physical target package and open it for inspection in time, resulting in low security inspection efficiency; it is even possible that the suspicious package will be missed in the security inspection, causing safety hazards for subsequent transportation.

[0073] In consideration of the above problems, an embodiment of the present invention proposes a package matching method based on a visible light image and an X-ray image. In the method, after determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image, it is possible to determine whether the nth first package detection frame and the nth second package detection frame correspond to the same package based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image. If they do not correspond to the same package, the target image with the package detection frame missing is determined. If the target image is a visible light image, and it is detected that the target package represented by the nth second package detection frame in the X-ray image includes contraband, the target first package detection frame can be added to the visible light image based on the second center point coordinates of the nth second package detection frame to select the target package. Through the above method, even if a package is missed in the visible light image, the corresponding package detection frame in the visible light image can be determined based on the package detection frame in the X-ray image, so that the target package can be quickly and accurately located in the visible light image, so that the security inspector can accurately find the target package and open it for inspection in time, thereby improving the security inspection efficiency. In addition, since the target package can be accurately located from the visible light image, the phenomenon of missing suspicious packages from security inspection can be avoided, thereby improving the safety of transportation.

[0074] Combine the following Figures 2 to 15 The package matching method based on visible light images and X-ray images provided in an embodiment of the present invention is described. The execution subject of the method can be an electronic device such as a security inspection machine, an X-ray machine, a computer or a server, or a specially designed intelligent device, or a package matching device set in the electronic device or the intelligent device, and the package matching device can be implemented by software, hardware or a combination of the two. The package matching method based on visible light images and X-ray images provided in an embodiment of the present invention can be applied to the scenario of matching packages in visible light images and X-ray images in the field of security inspection.

[0075] The nth first package detection frame described in the embodiment of the present invention can also be understood as the first package detection frame corresponding to the nth package in the visible light image, and the nth second package detection frame can also be understood as the second package detection frame corresponding to the nth package in the X-ray image.

[0076] Figure 2 A flowchart of a package matching method based on visible light images and X-ray images provided in an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0077] Step 201: Acquire a visible light image and an X-ray image taken at the same time.

[0078] In this step, since the security inspection process of the package by the security inspection machine usually takes a period of time, after the security inspection machine scans the package, it will output a continuous long image with the same speed as the conveyor belt movement speed of the security inspection machine. At the initial time T, an X-ray image can be captured from the continuous long image. The internal information of the package can be obtained through the X-ray image, such as the shape or category of the items carried in the package. In addition, the visible light imaging device will also output a continuous video image of the security inspection area. Therefore, at the initial time T, a visible light image of the security inspection area can also be obtained. Among them, the visible light imaging device can be, for example, a device that can capture visible light images, such as a camera or a video camera. The external information of the package, such as shape, color or other appearance information, can be obtained through the visible light image.

[0079] By acquiring visible light images and X-ray images taken at the same time, it can be ensured that the visible light image and the X-ray image capture the same package, thereby providing a basis for subsequent package matching.

[0080] Step 202: Determine at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image respectively.

[0081] In this step, the acquired visible light image can be input into a pre-trained target detection model to obtain a target visible light image output by the target detection model, wherein the target visible light image includes at least one first package detection frame, and each first package detection frame is used to select a package in the visible light image. Similarly, the acquired X-ray image can be input into a pre-trained target detection model to obtain a target X-ray image output by the target detection model, wherein the target X-ray image includes at least one second package detection frame, and each second package detection frame is used to select a package in the X-ray image.

[0082] Among them, the target detection model can be obtained by training the initial target detection model based on the sample image and label information, and the label information is a sample package detection box in the pre-labeled sample image, and the sample package detection box is used to select the package in the sample image.

[0083] It should be understood that the first package detection frame included in the target visible light image may or may not select all the packages in the visible light image, that is, missed detection. Similarly, the second package detection frame included in the X-ray image may or may not select all the packages in the X-ray image, that is, missed detection.

[0084] Step 203: Determine whether the nth first package detection frame matches the nth second package detection frame based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image.

[0085] The matching result is used to indicate whether the nth first package detection frame and the nth second package detection frame correspond to the same package, where n is a positive integer.

[0086] In this step, a first coordinate system can be constructed with the lower left corner of the visible light image as the first origin, and the width and height of the visible light image as the horizontal axis and the vertical axis, respectively, so that the first center point coordinates of the nth first package detection frame in the visible light image in the first coordinate system can be determined. In addition, a second coordinate system can be constructed with the lower left corner of the X-ray image as the second origin, and the width and height of the X-ray image as the horizontal axis and the vertical axis, respectively, so that the second center point coordinates of the nth first package detection frame in the X-ray image in the second coordinate system can be determined.

[0087] Furthermore, it is possible to determine whether the nth first package detection frame and the nth second package detection frame match based on the first center point coordinates of the nth first package detection frame and the second center point coordinates of the nth second package detection frame, and the matching result obtained after matching indicates whether the nth first package detection frame and the nth second package detection frame correspond to the same package. For example, if the nth first package detection frame and the nth second package detection frame match successfully, it means that the packages selected by the nth first package detection frame and the nth second package detection frame are the same, and if the nth first package detection frame and the nth second package detection frame fail to match, it means that the packages selected by the nth first package detection frame and the nth second package detection frame are different.

[0088] Step 204: when the nth first package detection frame and the nth second package detection frame fail to match, determine the target image in which the package detection frame is missing.

[0089] In this step, since the visible light image and the X-ray image are taken at the same time, the packages included in the visible light image and the X-ray image are the same. If the nth first package detection frame and the nth second package detection frame fail to match, it can be said that the package is missed in one of the images, that is, the package detection frame is missing. Therefore, it is necessary to further determine the target image of the missing package detection frame.

[0090] Step 205: When the target image is a visible light image and the target package represented by the nth second package detection frame includes contraband, a target first package detection frame is added to the visible light image based on the second center point coordinates of the nth second package detection frame, and the target first package detection frame is used to select the target package.

[0091] In this step, contraband can be understood as items that are prohibited from being carried in the rail transit industry. The contraband recognition model can be used to determine whether the package included in the X-ray image contains contraband. If it is determined that the target image with a missing package detection frame is a visible light image, it is determined that the package included in the X-ray image contains contraband, and the package is the target package represented by the nth second package detection frame, since security inspectors usually need to accurately locate the actual package through the visible light image, it is necessary to add a target first package detection frame to the visible light image based on the second center point coordinates of the nth second package detection frame in the X-ray image, so as to select the target package through the added target first package detection frame. After adding the target first package detection frame, the target first package detection frame will correspond to the nth second package detection frame, that is, the two correspond to the same package. In this way, security inspectors can quickly locate the target package containing contraband through the target first package detection frame to open it for security inspection.

[0092] The package matching method based on visible light images and X-ray images provided by an embodiment of the present invention obtains a visible light image and an X-ray image taken at the same time, and respectively determines at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image. When it is determined that the nth first package detection frame and the nth second package detection frame do not correspond to the same package based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, a target image with a missing package detection frame is determined. When the target image is a visible light image and the target package represented by the nth second package detection frame includes contraband, a target first package detection frame is added to the visible light image based on the second center point coordinates of the nth second package detection frame, and the target first package detection frame is used to select the target package. When a package is missed in the visible light image, the target first package detection frame corresponding to the same package as the nth second package detection frame in the X-ray image can be determined based on the second center point coordinates of the nth second package detection frame in the X-ray image, so that the target package can be quickly and accurately located in the visible light image, which makes it convenient for security inspectors to accurately find the target package and open it for inspection in time, thereby improving security inspection efficiency. In addition, since the target package can be accurately located from the visible light image, the phenomenon of missing suspicious packages from security inspection can be avoided, thereby improving the safety of transportation.

[0093] Below, the specific process of how to determine whether the nth first package detection frame and the nth second package detection frame match, how to determine the target image of the missing package detection frame, and how to add the target first package detection frame in the visible light image will be described in detail when n is 1 and n is greater than 1.

[0094] Exemplarily, when n is 1, when determining whether the nth first package detection frame and the nth second package detection frame match based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, a first coordinate system can be constructed with the lower left corner of the visible light image as the first origin, and a second coordinate system can be constructed with the lower left corner of the X-ray image as the second origin; a first distance is determined based on the first center point coordinates of the nth first package detection frame and the coordinates of the first origin, and a second coordinate system can be constructed based on the lower left corner of the X-ray image as the second origin. The coordinates of the second center point and the coordinates of the second origin are used to determine the second distance; the first first package detection frame is the package detection frame with the largest horizontal coordinate of the center point in the visible light image, and the first second package detection frame is the package detection frame with the largest horizontal coordinate of the center point in the X-ray image; a first ratio of the first distance to the second distance is determined; when the first ratio is not within the threshold range, it is determined that the nth first package detection frame and the nth second package detection frame fail to match; when the first ratio is within the threshold range, it is determined that the nth first package detection frame and the nth second package detection frame successfully match.

[0095] Specifically, Figure 3 A schematic diagram of a first package detection frame in a visible light image provided by an embodiment of the present invention, Figure 4 A schematic diagram of a second package detection frame in an X-ray image provided by an embodiment of the present invention, such as Figure 3 and Figure 4 As shown, a first coordinate system can be constructed with the lower left corner of the visible light image as the first origin and the width and height of the visible light image as the horizontal axis and vertical axis respectively, and a second coordinate system can be constructed with the lower left corner of the X-ray image as the second origin and the width and height of the X-ray image as the horizontal axis and vertical axis respectively.

[0096] After target detection is performed on the visible light image, a plurality of first package detection frames distributed along the width direction of the image will be formed on the visible light image, forming a visible light package sequence. In addition, after target detection is performed on the X-ray image, a plurality of second package detection frames distributed along the width direction of the image will be formed on the X-ray image, forming an X-ray package sequence. The rightmost first package detection frame in the visible light package sequence is selected as the first first package detection frame, and the rightmost second package detection frame in the X-ray package sequence is selected as the first second package detection frame. Figure 3 As shown in , the first first package detection frame can also be understood as the package detection frame with the largest horizontal coordinate of the center point in the visible light image, which is the first package detection frame corresponding to the first package, such as Figure 4 As shown, the first second package detection frame can also be understood as the package detection frame with the largest horizontal coordinate of the center point in the X-ray image, which is the second package detection frame corresponding to the first package.

[0097] After determining the first center point coordinates of the first first package detection frame and the second center point coordinates of the first second package detection frame, the first distance between the first center point coordinates of the first first package detection frame and the first origin in the first coordinate system can be determined based on the first center point coordinates of the first first package detection frame and the coordinates of the first origin, and the second distance between the second center point coordinates of the first second package detection frame and the coordinates of the second origin in the second coordinate system can be determined based on the second center point coordinates of the first second package detection frame and the coordinates of the second origin. The first distance and the second distance can be Euclidean distances.

[0098] Furthermore, a first ratio of the first distance to the second distance may be determined based on the following formula (1):

[0099]

[0100] Among them, R 基准 represents the first ratio, which can also be understood as the reference package matching coefficient, (x 1 ,y 1 ) represents the coordinates of the first center point of the first package detection frame in the visible light image; (x' 1 ,y' 1 ) represents the coordinates of the second center point of the first second package detection box in the X-ray image.

[0101] When the first ratio is within the threshold range, it means that the first distance from the first center point of the first first package detection frame to the first origin and the second distance from the second center point of the first second package detection frame to the second origin are approximately equal. Therefore, it can be determined that the first first package detection frame and the first second package detection frame are matched successfully, that is, the first first package detection frame and the first second package detection frame correspond to the same package.

[0102] It should be noted that the first first package detection frame and the first second package detection frame can be used as an initial reference package detection frame pair as a basis for subsequent detection of other package detection frames, and the specific content will be introduced below.

[0103] When the first ratio is not within the threshold range, it means that the first distance from the first center point of the first first package detection frame to the first origin and the second distance from the second center point of the first second package detection frame to the second origin are significantly different, which may be caused by missed detection or missing package detection frame in a certain image. Therefore, it can be determined that the first first package detection frame and the first second package detection frame fail to match, that is, the first first package detection frame and the first second package detection frame do not correspond to the same package.

[0104] The above threshold range may be set according to actual conditions or experience, for example, may be set to [1, 1.3].

[0105] In this embodiment, whether the first package detection frame and the first second package detection frame match can be determined based only on a first ratio of a first distance between a first center point and a first origin of the first first package detection frame and a second distance between a second center point and a second origin of the first second package detection frame, without resorting to other parameters, thereby making the package matching method simple and efficient.

[0106] Exemplarily, based on the above embodiment, when it is determined that the first package detection frame and the first second package detection frame fail to match, it is also necessary to determine the target image of the missing package detection frame. In the embodiment of the present invention, when the first ratio is less than the lower limit of the threshold range, the visible light image is determined to be the target image of the missing package detection frame; when the first ratio is greater than the upper limit of the threshold range, the X-ray image is determined to be the target image of the missing package detection frame.

[0107] Specifically, when the first ratio is less than the lower limit of the threshold range, for example, less than 1, it means that the first distance between the first first package detection frame in the visible light image and the first origin is less than the second distance between the first second package detection frame in the X-ray image and the second origin. Therefore, it can be determined that the package detection frame is missing in the visible light image, that is, the package detection frame corresponding to the first second package detection frame in the X-ray image is missed in the visible light image.

[0108] When the first ratio is greater than the upper limit of the threshold range, for example, greater than 1.3, it means that the first distance between the first first package detection frame in the visible light image and the first origin is greater than the second distance between the first second package detection frame in the X-ray image and the second origin. Therefore, it can be determined that the package detection frame is missing in the X-ray image, that is, the package detection frame corresponding to the first first package detection frame in the visible light image is missed in the X-ray image.

[0109] In this embodiment, the target image with the missing package detection frame can be determined based on the comparison result of the first ratio with the upper limit and the lower limit of the threshold range, so that the method of determining the target image is relatively simple and the efficiency of determining the target image is improved.

[0110] Further, when it is determined that the first ratio is less than the lower limit of the threshold range, that is, when it is determined that the target image is a visible light image, it will be further determined whether the target package represented by the first second package detection frame includes contraband. When it is determined that the target package represented by the first second package detection frame includes contraband, a target first package detection frame will be added to the visible light image based on the second center point coordinates of the first second package detection frame to select the target package. Exemplarily, the mapping coefficient of the first coordinate system where the visible light image is located to the second coordinate system where the X-ray image is located can be determined based on the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image, and the target center point coordinates of the target first package detection frame in the visible light image are determined based on the second center point coordinates and the mapping coefficients of the first second package detection frame, thereby adding the target first package detection frame to the visible light image based on the size of the first second package detection frame and the target center point coordinates.

[0111] Specifically, Figure 5 One of the schematic diagrams of adding a target first package detection frame in a visible light image provided by an embodiment of the present invention, such as Figure 5 As shown in the figure, when the package detection frame is missing in the visible light image and the target package corresponding to the first second package detection frame contains contraband, the target center point coordinates of the supplementary target first package detection frame can be determined as (αx' 1 ,βαy' 1 ), the size of the target first package detection frame is the same as the size of the first second package detection frame in the X-ray image, that is, the width and height of the target first package detection frame are the same as the pixel width of the first second package detection frame, such as Figure 5 Wherein, α and β are mapping coefficients of converting the second coordinate system where the X-ray image is located to the first coordinate system where the visible light image is located; optionally, the mapping coefficient can be determined according to the ratio of the coordinate of the upper right corner of the visible light image to the coordinate of the upper right corner of the X-ray image, for example, based on Figure 5 The coordinate value ratio of the boundary vertex A of the visible light image and the coordinate value ratio of the boundary vertex A' of the X-ray image shown in is determined.

[0112] It should be noted that after adding the target first package detection frame to the visible light image, the target first package detection frame and the first second package detection frame can be used as an initial reference package detection frame pair as a basis for subsequent detection of other package detection frames.

[0113] In this embodiment, the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image can be selected as a reference to determine the mapping coefficient, so as to determine the target center point coordinates of the target first package detection frame in the visible light image based on the mapping coefficient, and then add the target first package detection frame to the visible light image according to the target center point coordinates and the size of the first second package detection frame, so that the target package containing contraband can be quickly and accurately located in the visible light image.

[0114] Exemplarily, based on the above embodiment, when the first ratio is less than the lower limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

[0115] Specifically, Figure 6 A schematic diagram of deleting the first second package detection frame from an X-ray image provided by an embodiment of the present invention, such as Figure 6 It is shown that when the first ratio is less than the lower limit of the threshold range, it can be determined that the first first package detection frame is missing in the visible light image. At this time, after the X-ray image is detected for contraband, when it is determined that the target package represented by the first second package detection frame does not include contraband, since there is no need to locate the target package in the visible light image, the first second package detection frame in the X-ray image can be removed.

[0116] It should be noted that at this time, the next-to-left parcel detection frame adjacent to the first second parcel detection frame in the X-ray image needs to be used as the new first second parcel detection frame of the X-ray image, and according to the methods in the aforementioned embodiments, the reference parcel matching coefficient between the new first second parcel detection frame and the first first parcel detection frame is determined to determine whether the new first second parcel detection frame matches the first first parcel detection frame, and the above process is repeated until the initial reference parcel detection frame pair is detected.

[0117] In this embodiment, when the package detection frame is missing in the visible light image and the target package represented by the first second package detection frame does not include contraband, the first second package detection frame can be deleted from the X-ray image, thereby saving system resources.

[0118] Exemplarily, based on the above embodiments, when the first ratio is greater than the upper limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

[0119] Specifically, Figure 7 A schematic diagram of deleting the first package detection frame from a visible light image provided by an embodiment of the present invention, such as Figure 7It is shown that when the first ratio is greater than the upper limit of the threshold range, it can be determined that the first second package detection frame is missing in the X-ray image, that is, the package corresponding to the first first package detection frame is missed. Since it is impossible to detect whether the package contains contraband based on the visible light image, the first first package detection frame in the visible light image can be removed.

[0120] In addition, at this time, it is necessary to use the next-to-left parcel detection frame adjacent to the first first parcel detection frame in the visible light image as the new first first parcel detection frame of the visible light image, and determine the reference parcel matching coefficient between the new first first parcel detection frame and the first second parcel detection frame according to the methods in the aforementioned embodiments, so as to determine whether the new first first parcel detection frame and the first second parcel detection frame match, and repeat the above process until the initial reference parcel detection frame pair is detected.

[0121] It should be noted that in this case, since the first second package detection frame in the X-ray image is lost, it is impossible to determine whether the package corresponding to the first second package detection frame contains prohibited items based on the X-ray image by using the contraband detection model. Therefore, in order to ensure the reliability of security inspection, it is necessary to manually open the package corresponding to the first first package detection frame in the visible light image for inspection.

[0122] In this embodiment, when the package detection frame is missing in the X-ray image, since it is impossible to detect contraband based on the visible light image, the first package detection frame can be directly deleted from the visible light image, thereby saving system resources.

[0123] Exemplarily, when n is greater than 1 and the n-1th first package detection frame and the n-1th second package detection frame are matched successfully, when determining whether the nth first package detection frame and the nth second package detection frame match based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, the third distance may be determined based on the first center point coordinates of the nth first package detection frame and the first center point coordinates of the n-1th first package detection frame, and the third distance may be determined based on the second center point coordinates of the nth second package detection frame. The fourth distance is determined by the center point coordinates and the second center point coordinates of the n-1th second package detection frame; the nth first package detection frame is on the left side of the n-1th first package detection frame; the nth second package detection frame is on the left side of the n-1th second package detection frame, and a second ratio of the third distance to the fourth distance is determined; when the second ratio is not within the threshold range, it is determined that the nth first package detection frame and the nth second package detection frame fail to match; when the second ratio is within the threshold range, it is determined that the nth first package detection frame and the nth second package detection frame successfully match.

[0124] Specifically, based on the description in the foregoing embodiments, an initial reference parcel detection frame pair can be determined, and the initial reference parcel detection frame pair can be understood as the successful matching of the n-1th first parcel detection frame and the n-1th second parcel detection frame. The nth first parcel detection frame is the parcel detection frame that is located to the left of the n-1th first parcel detection frame in the visible light image, and the nth second parcel detection frame is the parcel detection frame that is located to the left of the n-1th second parcel detection frame in the X-ray image.

[0125] Further, Figure 8 A schematic diagram of determining a third distance in a visible light image provided by an embodiment of the present invention, Fig. 9 A schematic diagram of determining a fourth distance in an X-ray image provided by an embodiment of the present invention, such as Figure 8 and Fig. 9 As shown, the second ratio can be determined according to the following formula (2):

[0126]

[0127] Among them, R n Represents the second ratio, which can also be understood as the matching coefficient. n represents the nth package detection frame from right to left in the visible light image or X-ray image, (x n ,y n ) represents the coordinates of the first center point of the nth first package detection frame in the visible light image; (x n-1 ,y n-1 ) represents the coordinates of the first center point of the n-1th first package detection box in the visible light image, Δd n represents the third distance, (x' n ,y' n ) represents the coordinates of the second center point of the nth second package detection frame in the X-ray image, (x' n-1 ,y' n-1 ) represents the coordinates of the second center point of the n-1th second package detection frame in the X-ray image, Δd' n represents the fourth distance. The third distance and the fourth distance can be understood as Euclidean distance.

[0128] When the second ratio is within the threshold range, it means that the third distance and the fourth distance are approximately equal, and it can be determined that there is no missed detection in the visible light image and the X-ray image. Therefore, the nth first package detection frame and the nth second package detection frame are matched successfully, and the nth first package detection frame and the nth second package detection frame are used as new reference package detection frames as the basis for the detection of subsequent other package detection frames. Among them, the nth first package detection frame can be understood as the first package detection frame corresponding to the nth package, and the nth second package detection frame can be understood as the second package detection frame corresponding to the nth package.

[0129] When the second ratio is not within the threshold range, it means that the third distance and the fourth distance are significantly different, which is usually caused by the missing package detection frame in the visible light image or the X-ray image. Therefore, it can be determined that the nth first package detection frame and the nth second package detection frame fail to match.

[0130] For example, assuming that the first first package detection frame and the first second package detection frame are matched successfully, the matching coefficient R between the second first package detection frame and the second second package detection frame can be calculated based on formula (2): 2 , according to the matching coefficient R 2 The second first package detection frame in the visible light image and the second second package detection frame in the X-ray image are compared with the threshold range to determine whether the second first package detection frame in the visible light image and the second second package detection frame in the X-ray image match. If they match, the second first package detection frame and the second second package detection frame are further determined as new reference package detection frames, so as to continue to match the remaining package detection frames in the visible light image and the X-ray image in turn; until the nth package detection frame that matches in the visible light image and the X-ray image is completed.

[0131] It should be noted that, when the distance between two adjacent package detection frames in a visible light image or an X-ray image is greater than the preset spacing L, the reference package detection frame in the visible light image or the X-ray image that is successfully matched may not appear within the display width range of the same frame image as the next package detection frame to be matched, resulting in the inability to calculate the Euclidean distance between two adjacent package detection frames in the visible light image or the X-ray image. In order to solve this problem, in an embodiment of the present invention, when it is determined that the distance between two adjacent package detection frames in a visible light image or an X-ray image is greater than the preset spacing L, it is necessary to re-determine the distance between the package detection frame and the origin based on the coordinates of the center point of the package detection frame to determine the initial reference package detection frame; wherein the preset spacing L is determined by the product of the speed v of the conveyor belt of the security inspection machine and the time t.

[0132] In this embodiment, the third distance can be determined based on the first center point coordinates of the nth first package detection frame and the first center point coordinates of the n-1th first package detection frame, and the fourth distance can be determined based on the second center point coordinates of the nth second package detection frame and the second center point coordinates of the n-1th second package detection frame. Therefore, whether the nth first package detection frame and the nth second package detection frame match can be determined based only on the second ratio of the third distance to the fourth distance, without the aid of other parameters, so that the package matching method is simple and efficient.

[0133] Exemplarily, based on the above embodiment, when it is determined that the nth first package detection frame and the nth second package detection frame fail to match, it is also necessary to determine the target image of the missing package detection frame. In the embodiment of the present invention, when the second ratio is greater than the upper limit of the threshold range, the visible light image is determined to be the target image of the missing package detection frame; when the second ratio is less than the lower limit of the threshold range, the X-ray image is determined to be the target image of the missing package detection frame.

[0134] Specifically, Fig.10 A schematic diagram of a package detection frame that is missed in a visible light image provided by an embodiment of the present invention, such as Fig.10 As shown, when the second ratio is greater than the upper limit of the threshold range, for example, greater than 1.3, it means that the third distance between the nth first package detection frame and the n-1th first package detection frame in the visible light image is greater than the fourth distance between the nth second package detection frame and the n-1th second package detection frame in the X-ray image. Therefore, it can be determined that the package detection frame is missing in the visible light image, that is, the package detection frame corresponding to the nth second package detection frame in the X-ray image is missed in the visible light image.

[0135] Fig.11 A schematic diagram of a package detection frame missing from an X-ray image provided by an embodiment of the present invention, such as Fig.11 As shown, when the second ratio is less than the lower limit of the threshold range, for example, less than 1, it means that the third distance between the nth first package detection frame and the n-1th first package detection frame in the visible light image is less than the fourth distance between the nth second package detection frame and the n-1th second package detection frame in the X-ray image. Therefore, it can be determined that the package detection frame is missing in the X-ray image, that is, the package detection frame corresponding to the nth first package detection frame in the visible light image is missed in the X-ray image.

[0136] In this embodiment, the target image with the missing package detection frame can be determined based on the comparison result of the second ratio with the upper limit and the lower limit of the threshold range, so that the method of determining the target image is relatively simple and the efficiency of determining the target image is improved.

[0137] Further, when it is determined that the second ratio is greater than the upper limit of the threshold range, that is, when it is determined that the target image is a visible light image, it will be further determined whether the target package represented by the nth second package detection frame includes contraband. When it is determined that the target package represented by the nth second package detection frame includes contraband, a target first package detection frame will be added to the visible light image based on the second center point coordinates of the nth second package detection frame to select the target package. Exemplarily, the mapping coefficient of the first coordinate system where the visible light image is located to the second coordinate system where the X-ray image is located can be determined based on the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image, and the target center point coordinates of the target first package detection frame in the visible light image are determined based on the second center point coordinates and the mapping coefficients of the nth second package detection frame, thereby adding the target first package detection frame to the visible light image based on the size of the nth second package detection frame and the target center point coordinates.

[0138] Specifically, Fig.12 The second schematic diagram of adding a target first package detection frame in a visible light image provided by an embodiment of the present invention is as follows: Fig.12 As shown in the figure, when the package detection frame is missing in the visible light image and the target package corresponding to the second package detection frame corresponding to the nth package in the X-ray image contains contraband, the target center point coordinates of the supplementary target first package detection frame can be determined as (αx' n ,βy' n ), the size of the target first package detection frame is the same as the size of the nth second package detection frame in the X-ray image, that is, the width and height of the target first package detection frame are the same as the pixel width of the nth second package detection frame, such as Fig.11 Wherein, α and β are mapping coefficients of converting the second coordinate system where the X-ray image is located to the first coordinate system where the visible light image is located; optionally, the mapping coefficient can be determined according to the ratio of the coordinates of the lower right corner of the visible light image to the coordinates of the lower right corner of the X-ray image, for example, based on Fig.12 The coordinate value ratio of the boundary vertex A of the visible light image and the coordinate value ratio of the boundary vertex A' of the X-ray image shown in is determined.

[0139] It should be noted that after adding the target first package detection frame to the visible light image, the target first package detection frame and the nth second package detection frame can be used as a reference package detection frame pair as a basis for subsequent detection of other package detection frames.

[0140] It should also be added that, although a package is missed in the visible light image, a virtual target first package detection frame is added to the visible light package sequence, which can only guarantee the accuracy of subsequent package matching, and will not output the visible light image of the corresponding nth package in the current visible light image.

[0141] In this embodiment, the coordinates of the lower right corner of the visible light image and the coordinates of the lower right corner of the X-ray image can be selected as a reference to determine the mapping coefficient, and then based on the mapping coefficient, the target center point coordinates of the target first package detection frame in the visible light image are determined, and then the target first package detection frame is added to the visible light image according to the target center point coordinates and the size of the nth second package detection frame, so that the target package containing contraband can be quickly and accurately located in the visible light image.

[0142] Exemplarily, based on the above embodiment, when the second ratio is greater than the upper limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

[0143] Specifically, Fig.13 A schematic diagram of a case where a package detection frame is missed in a visible light image and a target package in an X-ray image does not include contraband, as provided in an embodiment of the present invention. Fig.13 As shown, the second package detection frame corresponding to the nth package in the X-ray image does not contain any contraband. Fig.14 A schematic diagram of deleting the nth second package detection frame from an X-ray image provided by an embodiment of the present invention, such as Fig.14 As shown, when the second ratio is greater than the upper limit of the threshold range, it can be determined that the nth first package detection frame is missing in the visible light image. At this time, after the X-ray image is detected for contraband, when it is determined that the target package represented by the nth second package detection frame does not include contraband, since there is no need to locate the target package in the visible light image, the nth second package detection frame in the X-ray image can be removed.

[0144] It should be noted that, at this time, the next left parcel detection frame adjacent to the nth second parcel detection frame in the X-ray image needs to be used as the new nth second parcel detection frame of the X-ray image, and the parcel matching coefficient between the new nth second parcel detection frame and the nth first parcel detection frame is determined according to the method in the above embodiment, that is, Δd is determined. n and Δd' n The ratio of is within the threshold range, thereby determining whether the new n-th second package detection frame matches the n-th first package detection frame, and repeating the above process until a reference package detection frame pair is detected.

[0145] In this embodiment, when the package detection frame is missing in the visible light image and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame can be deleted from the X-ray image, thereby saving system resources.

[0146] Exemplarily, based on the above embodiment, when the second ratio is less than the lower limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

[0147] Specifically, Fig.15 A schematic diagram of deleting the nth first package detection frame from a visible light image provided by an embodiment of the present invention, such as Fig.15 When the second ratio is less than the lower limit of the threshold range, it can be determined that the nth second package detection frame is missing in the X-ray image, that is, the nth package in the X-ray image is missed. Since it is impossible to detect whether the package contains contraband based on the visible light image, the nth first package detection frame in the visible light image can be removed.

[0148] In addition, at this time, it is necessary to use the next-to-left parcel detection frame adjacent to the nth first parcel detection frame in the visible light image as the new nth first parcel detection frame of the visible light image, and determine the reference parcel matching coefficient between the new nth first parcel detection frame and the nth second parcel detection frame according to the methods in the aforementioned embodiments, that is, determine Δd n and Δd' n The ratio of is within the threshold range, thereby determining whether the new n-th first package detection frame and the n-th second package detection frame match, and repeating the above process until a reference package detection frame pair is detected.

[0149] It should be noted that in this case, since the nth second package detection frame in the X-ray image is lost, it is impossible to determine whether the package corresponding to the nth second package detection frame contains prohibited items based on the X-ray image by using the contraband detection model. Therefore, in order to ensure the reliability of security inspection, it is necessary to manually open the package corresponding to the nth first package detection frame in the visible light image for inspection.

[0150] In this embodiment, when the package detection frame is missing in the X-ray image, since it is impossible to detect contraband based on the visible light image, the nth first package detection frame can be directly deleted from the visible light image, thereby saving system resources.

[0151] The package matching device based on visible light images and X-ray images provided by the present invention is described below. The package matching device based on visible light images and X-ray images described below and the package matching method based on visible light images and X-ray images described above can be referred to each other.

[0152] Fig.16 A schematic diagram of a package matching device based on visible light images and X-ray images provided in an embodiment of the present invention, referring to Fig.16As shown, the package matching device 1600 based on visible light image and X-ray image includes:

[0153] An acquisition module 1601 is used to acquire a visible light image and an X-ray image taken at the same time;

[0154] A determination module 1602, configured to respectively determine at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image;

[0155] The determination module 1602 is further used to determine whether the nth first package detection frame and the nth second package detection frame match based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, wherein the matching result is used to indicate whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer;

[0156] The determination module 1602 is further configured to determine a target image with a missing package detection frame when the nth first package detection frame and the nth second package detection frame fail to match each other;

[0157] An adding module 1603 is used to add a target first package detection frame in the visible light image based on the second center point coordinates of the nth second package detection frame when the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, and the target first package detection frame is used to select the target package.

[0158] In an exemplary embodiment, the adding module 1603 is specifically configured to:

[0159] Based on the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image, determine the mapping coefficient of converting the second coordinate system where the X-ray image is located to the first coordinate system where the visible light image is located; based on the second center point coordinates of the nth second package detection frame and the mapping coefficient, determine the target center point coordinates of the target first package detection frame in the visible light image; based on the size of the nth second package detection frame and the target center point coordinates, add the target first package detection frame in the visible light image.

[0160] In an exemplary embodiment, the determining module 1602 is specifically configured to:

[0161] A first coordinate system is constructed with the lower left corner of the visible light image as a first origin, and a second coordinate system is constructed with the lower left corner of the X-ray image as a second origin;

[0162] When n is 1, a first distance is determined based on the coordinates of the first center point of the nth first parcel detection frame and the coordinates of the first origin, and a second distance is determined based on the coordinates of the second center point of the nth second parcel detection frame and the coordinates of the second origin; the first first parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the visible light image, and the first second parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the X-ray image;

[0163] determining a first ratio of the first distance to the second distance;

[0164] When the first ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match;

[0165] When the first ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

[0166] In an exemplary embodiment, the determining module 1602 is specifically configured to:

[0167] When the first ratio is smaller than the lower limit of the threshold range, the visible light image is determined to be the target image of the missing package detection frame; when the first ratio is greater than the upper limit of the threshold range, the X-ray image is determined to be the target image of the missing package detection frame.

[0168] In an exemplary embodiment, the apparatus further comprises: a deletion module, wherein:

[0169] A deleting module is used to delete the nth second package detection frame from the X-ray image when the first ratio is less than a lower limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband.

[0170] In an exemplary embodiment, the deleting module is further configured to delete the nth first package detection frame from the visible light image when the first ratio is greater than an upper limit value of the threshold range.

[0171] In an exemplary embodiment, the determining module 1602 is specifically configured to:

[0172] When n is greater than 1 and the n-1th first package detection frame and the n-1th second package detection frame are matched successfully, a third distance is determined based on the first center point coordinates of the n-1th first package detection frame and the first center point coordinates of the n-1th first package detection frame, and a fourth distance is determined based on the second center point coordinates of the n-2nd package detection frame and the second center point coordinates of the n-1th second package detection frame; the n-1th first package detection frame is on the left side of the n-1th first package detection frame; the n-2nd package detection frame is on the left side of the n-1th second package detection frame;

[0173] determining a second ratio of the third distance to the fourth distance;

[0174] When the second ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match;

[0175] When the second ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

[0176] In an exemplary embodiment, the determining module 1602 is specifically configured to:

[0177] When the second ratio is greater than the upper limit value of the threshold range, the visible light image is determined to be the target image of the missing package detection frame; when the second ratio is less than the lower limit value of the threshold range, the X-ray image is determined to be the target image of the missing package detection frame.

[0178] In an exemplary embodiment, the deleting module is further configured to delete the nth first package detection frame from the visible light image when the second ratio is less than a lower limit of the threshold range.

[0179] In an exemplary embodiment, the deletion module is further used to delete the nth second package detection frame from the X-ray image when the second ratio is greater than an upper limit value of the threshold range and the target package represented by the nth second package detection frame does not include contraband.

[0180] The device of this embodiment can be used to execute the method of any one of the embodiments in the embodiment of the package matching method based on visible light images and X-ray images. Its specific implementation process and technical effects are similar to those in the embodiment of the package matching method based on visible light images and X-ray images. For details, please refer to the detailed description in the embodiment of the package matching method based on visible light images and X-ray images, which will not be repeated here.

[0181] Fig.17An example of a physical structure diagram of an electronic device is shown in FIG. Fig.17 As shown, the electronic device may include: a processor 1710, a communication interface 1720, a memory 1730 and a communication bus 1740, wherein the processor 1710, the communication interface 1720 and the memory 1730 communicate with each other through the communication bus 1740. The processor 1710 may call the logic instructions in the memory 1730 to execute a package matching method based on visible light images and X-ray images, the method comprising: acquiring a visible light image and an X-ray image taken at the same time; respectively determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image; based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, determining whether the nth first package detection frame and the nth second package detection frame match, wherein the matching The matching result is used to characterize whether the nth first package detection frame and the nth second package detection frame correspond to the same package, where n is a positive integer; in the case where the nth first package detection frame and the nth second package detection frame fail to match, the target image of the missing package detection frame is determined; in the case where the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, based on the second center point coordinates of the nth second package detection frame, a target first package detection frame is added to the visible light image, and the target first package detection frame is used to select the target package.

[0182] In addition, the logic instructions in the above-mentioned memory 1730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the package matching method based on visible light images and X-ray images provided by the above methods, and the method includes: acquiring a visible light image and an X-ray image taken at the same time; respectively determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image; based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, determining whether the nth first package detection frame matches the nth second package detection frame, wherein the matching result is used to characterize whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer; in the case where the nth first package detection frame and the nth second package detection frame fail to match, determining a target image where the package detection frame is missing; in the case where the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, based on the second center point coordinates of the nth second package detection frame, adding a target first package detection frame in the visible light image, the target first package detection frame being used to select the target package.

[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the package matching method based on visible light images and X-ray images provided by the above methods, the method comprising: acquiring a visible light image and an X-ray image taken at the same time; respectively determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image; and determining the nth first package detection frame and the nth second package detection frame based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image. whether the two package detection frames match, wherein the matching result is used to characterize whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer; in the case where the nth first package detection frame and the nth second package detection frame fail to match, determine the target image of the missing package detection frame; in the case where the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, based on the second center point coordinates of the nth second package detection frame, add a target first package detection frame in the visible light image, and the target first package detection frame is used to select the target package.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A package matching method based on visible light image and X-ray image, characterized in that: include: Obtain visible light images and X-ray images taken at the same time; respectively determining at least one first package detection frame in the visible light image and at least one second package detection frame in the X-ray image; Based on the first center point coordinates of the nth first package detection frame in the visible light image and the second center point coordinates of the nth second package detection frame in the X-ray image, determining whether the nth first package detection frame and the nth second package detection frame match, wherein the matching result is used to indicate whether the nth first package detection frame and the nth second package detection frame correspond to the same package, and n is a positive integer; In the case where the nth first package detection frame and the nth second package detection frame fail to match, determining a target image in which the package detection frame is missing; When the target image is the visible light image and the target package represented by the nth second package detection frame includes contraband, based on the second center point coordinates of the nth second package detection frame, a target first package detection frame is added to the visible light image, and the target first package detection frame is used to select the target package; The determining whether the nth first parcel detection frame matches the nth second parcel detection frame based on the first center point coordinate of the nth first parcel detection frame in the visible light image and the second center point coordinate of the nth second parcel detection frame in the X-ray image includes: A first coordinate system is constructed with the lower left corner of the visible light image as a first origin, and a second coordinate system is constructed with the lower left corner of the X-ray image as a second origin; When n is 1, a first distance is determined based on the coordinates of the first center point of the nth first parcel detection frame and the coordinates of the first origin, and a second distance is determined based on the coordinates of the second center point of the nth second parcel detection frame and the coordinates of the second origin; the first first parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the visible light image, and the first second parcel detection frame is a parcel detection frame with a center point with the largest horizontal coordinate in the X-ray image; determining a first ratio of the first distance to the second distance; When the first ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match; When the first ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

2. The package matching method based on visible light image and X-ray image according to claim 1, characterized in that: The adding a target first package detection frame in the visible light image based on the second center point coordinate of the nth second package detection frame includes: Determine, based on the coordinates of the upper right corner of the visible light image and the coordinates of the upper right corner of the X-ray image, a mapping coefficient for converting a second coordinate system where the X-ray image is located to a first coordinate system where the visible light image is located; Determining the target center point coordinates of the target first package detection frame in the visible light image based on the second center point coordinates of the nth second package detection frame and the mapping coefficient; Based on the size of the nth second package detection frame and the target center point coordinates, the target first package detection frame is added to the visible light image.

3. The package matching method based on visible light image and X-ray image according to claim 1, characterized in that: The step of determining a target image with a missing package detection frame includes: When the first ratio is less than a lower limit of the threshold range, determining that the visible light image is a target image with a missing package detection frame; When the first ratio is greater than the upper limit value of the threshold range, the X-ray image is determined to be a target image with a missing package detection frame.

4. The package matching method based on visible light image and X-ray image according to claim 1, characterized in that: The method further comprises: When the first ratio is less than the lower limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

5. The package matching method based on visible light image and X-ray image according to claim 1, characterized in that: The method further comprises: When the first ratio is greater than an upper limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

6. The package matching method based on visible light image and X-ray image according to claim 1, characterized in that: The determining whether the nth first parcel detection frame matches the nth second parcel detection frame based on the first center point coordinate of the nth first parcel detection frame in the visible light image and the second center point coordinate of the nth second parcel detection frame in the X-ray image includes: When n is greater than 1 and the n-1th first package detection frame and the n-1th second package detection frame are matched successfully, a third distance is determined based on the first center point coordinates of the n-1th first package detection frame and the first center point coordinates of the n-1th first package detection frame, and a fourth distance is determined based on the second center point coordinates of the n-2nd package detection frame and the second center point coordinates of the n-1th second package detection frame; the n-1th first package detection frame is on the left side of the n-1th first package detection frame; the n-2nd package detection frame is on the left side of the n-1th second package detection frame; determining a second ratio of the third distance to the fourth distance; When the second ratio is not within the threshold range, determining that the nth first package detection frame and the nth second package detection frame fail to match; When the second ratio is within a threshold range, it is determined that the nth first package detection frame and the nth second package detection frame are matched successfully.

7. The package matching method based on visible light image and X-ray image according to claim 6, characterized in that: The step of determining a target image with a missing package detection frame includes: When the second ratio is greater than an upper limit value of the threshold range, determining that the visible light image is a target image with a missing package detection frame; When the second ratio is less than the lower limit of the threshold range, the X-ray image is determined to be a target image with a missing package detection frame.

8. The package matching method based on visible light image and X-ray image according to claim 6, characterized in that: The method further comprises: When the second ratio is smaller than a lower limit of the threshold range, the nth first package detection frame is deleted from the visible light image.

9. The package matching method based on visible light image and X-ray image according to claim 6, characterized in that: The method further comprises: When the second ratio is greater than the upper limit of the threshold range and the target package represented by the nth second package detection frame does not include contraband, the nth second package detection frame is deleted from the X-ray image.

Images