Item association method, computer equipment and storage device

Abstract

The present application discloses an object association method, a computer device and a storage device. The method includes: obtaining a first video and a second video captured in the same time period using different imaging methods; determining first object contour features corresponding to each of a plurality of first object regions in the first video; and determining second object contour features corresponding to each of a plurality of second object regions in the second video; and associating a plurality of first object regions with a plurality of second object regions based on the determined contour features of each first object and the determined contour features of each second object. The above scheme can improve the accuracy of object association.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an object association method, computer equipment and storage device. Background Art

[0002] Security checks are currently widely used in various public places, such as subway stations, railway stations, airports, bus stations, etc. Security checks can be carried out on various places to avoid safety problems.

[0003] During the security inspection process, security inspection machines are usually used to conduct security inspections on people's packages. The packages can be displayed to security inspectors in the form of pseudo-color pictures. When security inspectors find dangerous items on the pseudo-color pictures, they cannot quickly determine the packages containing dangerous items. In some cases, they may even misidentify packages. The low accuracy of package association will also affect the normal security inspection work of security inspectors. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an item association method, a computer device and a storage device, which can improve the accuracy of item association.

[0005] In order to solve the above problems, the first aspect of the present application provides an object association method, which includes: acquiring a first video and a second video captured in the same time period using different imaging methods; determining a first object contour feature corresponding to each of a plurality of first object regions in the first video; and determining a second object contour feature corresponding to each of a plurality of second object regions in the second video; and associating the plurality of first object regions with the plurality of second object regions based on the determined first object contour features and the determined second object contour features.

[0006] In order to solve the above problem, the second aspect of the present application provides a computer device, which includes a memory and a processor coupled to each other, the memory stores program data, and the processor is used to execute the program data to implement any step in the above method.

[0007] In order to solve the above problem, the third aspect of the present application provides a storage device, which stores program data that can be executed by a processor, and the program data is used to implement any step in the above method.

[0008] The above scheme acquires the first video and the second video acquired by different imaging methods in the same time period; determines the first object contour features corresponding to each of the multiple first object regions in the first video; and determines the second object contour features corresponding to each of the multiple second object regions in the second video; and associates the multiple first object regions with the multiple second object regions based on the determined first object contour features and the determined second object contour features. Since the first video and the second video of the object are obtained by different imaging methods, the contour features of the object in the first video and the second video of the different imaging methods are basically the same. The first object regions and the second object regions are associated by the first object contour features corresponding to the multiple first object regions and the second object contour features corresponding to the multiple second object regions, thereby improving the accuracy of object association. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of the first embodiment of the method for associating items in the present application;

[0011] Figure 2 This application Figure 1 The schematic diagram of the process of the first embodiment of step S12;

[0012] Figure 3 It is a schematic diagram of an example of an embodiment of a video image frame of a radiographic imaging method of the present application;

[0013] Figure 4 This application Figure 1 The flowchart of the second embodiment of step S12 is shown in FIG.

[0014] Figure 5 This is a schematic diagram of an example of an embodiment of a video image frame of a natural imaging method of the present application;

[0015] Figure 6 This application Figure 1 A schematic diagram of a flow chart of an embodiment of the step before step S13;

[0016] Figure 7 This application Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0017] Figure 8 is a schematic diagram of an example of a first embodiment of the first object contour feature of the present application;

[0018] Fig. 9 is a schematic diagram of an example of a second embodiment of the first object contour feature of the present application;

[0019] Fig.10 This is a schematic diagram of an example of an embodiment of the second object contour feature of the present application;

[0020] Fig.11 It is a schematic diagram of an example of an embodiment of a feature set matched in the present application;

[0021] Fig.12 This application Figure 1 A schematic diagram of another embodiment of step S13;

[0022] Fig.13 is a schematic diagram of another embodiment of the feature group matched in the present application;

[0023] Fig.14 It is a structural schematic diagram of an embodiment of the article association device of the present application;

[0024] Fig.15 It is a structural schematic diagram of an embodiment of a computer device of the present application;

[0025] Fig.16 It is a structural diagram of an embodiment of a storage device of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0028] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The present application provides the following embodiments, and each embodiment is described in detail below.

[0030] See also Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for associating items in the present application. The method may include the following steps:

[0031] S11: Acquire a first video and a second video captured in the same time period using different imaging methods.

[0032] The first video and the second video of the object are obtained by using different imaging methods in the same time period. For example, during the security inspection process, the staff can use the security inspection machine to inspect the items of the personnel. During the time period of the security inspection of the items, the first video and the second video of the item can be obtained by using different imaging methods. The items can be parcels, luggage and other items carried by the personnel, and this application does not limit this.

[0033] In some implementations, one of the first video and the second video is shot using a natural imaging method.

[0034] In other embodiments, one of the first video and the second video is obtained by radiographic imaging.

[0035] This application is explained by taking the example that the first video is shot by natural imaging and the second video is shot by X-ray imaging, and this application does not impose any limitation on this.

[0036] In some embodiments, the first video is obtained by shooting the items placed on the conveying device using natural imaging. For example, during the security inspection process, staff or pedestrians can place items on the conveying device, where the items can be packages, luggage, etc., and the conveying device of the security inspection machine is used to place and transport the items. A camera device can be installed on or near the security inspection machine. During the process of the conveying device transporting the items, the camera device is used to shoot the first video of the items on the conveying device, and the image frame in the first video can be used as the first image frame of the item.

[0037] In some embodiments, a second video may be obtained by using a detection device to perform radiographic imaging on an item conveyed by a conveying device. When the conveying device conveys an item to the detection device, the detection device may be used to perform radiographic imaging on the item conveyed by the conveying device to obtain a second video. For example, an X-ray detection device is used to perform X-ray detection on an item on the conveying device, and a second video of the item is obtained by X-ray imaging, and an image frame of the second video may be used as a second image frame of the item.

[0038] S12: Determine first object contour features corresponding to each of the plurality of first object regions in the first video; and determine second object contour features corresponding to each of the plurality of second object regions in the second video.

[0039] Object detection is performed on the first video and the second video respectively to detect the areas where the objects are located in the first video and the second video, so as to determine the first object contour features corresponding to each of the multiple first object areas in the first video, and determine the second object contour features corresponding to each of the multiple second object areas in the second video.

[0040] Among them, when detecting multiple first object areas of the first video, object detection can be performed on each first image frame in the first video, each first image frame can include a first object area, and each first image frame can also include multiple first object areas. By detecting each first image frame in the first video, multiple first object areas in the first video can be obtained; similarly, by detecting each second image frame in the second video, multiple second object areas in the second video can be obtained.

[0041] After determining multiple first object areas of the first video and multiple second video areas of the second video, the object contours corresponding to the object areas can be detected to obtain multiple first object contour features corresponding to the multiple first object areas and multiple second object contour features corresponding to the multiple second object areas, wherein one object area can correspond to one object contour feature.

[0042] S13: Associating a plurality of first object regions with a plurality of second object regions based on the determined first object contour features and the determined second object contour features.

[0043] A first video and a second video of an object are obtained by adopting different imaging methods, and object detection is performed on the first video and the second video respectively. After obtaining first object contour features corresponding to multiple first object regions in the first video and second object contour features corresponding to multiple second object regions in the second video, since the contour features of the object are similar under different imaging methods, the first object contour features corresponding to each first object region and the second object contour features corresponding to each second object region are used to match the multiple first object regions in the first video and the multiple second object regions in the second video, that is, the multiple first object regions are associated with the multiple second object regions, so as to associate the first object regions in the first video with the second object regions in the second video.

[0044] In this embodiment, the first video and the second video collected by different imaging methods in the same time period are obtained; the first item contour features corresponding to each of the multiple first item areas in the first video are determined; and the second item contour features corresponding to each of the multiple second item areas in the second video are determined; based on the determined contour features of each first item and the determined contour features of each second item, the multiple first item areas and the multiple second item areas are associated. Since the contour features of the items in the first video and the second video of the different imaging methods are basically the same, the first item areas and the second item areas are associated by the first item contour features corresponding to the multiple first item areas and the second item contour features corresponding to the multiple second item areas, which can improve the accuracy of item association. Therefore, if dangerous items are found during the security inspection process, the items can be associated in time. In addition, since the above scheme does not rely on the hardware time synchronization of the first video and the second video of the item, the problem of item association is solved at the software level, making the scheme more universal.

[0045] In some embodiments, see Figure 2 In the above step S12, the imaging mode of one of the first video and the second video is a radiographic imaging mode; determining a plurality of first object regions in the first video; or determining a plurality of second object regions in the second video may include the following steps:

[0046] S1211: Perform grayscale value projection in a preset direction on each image frame of the video using the radiographic imaging method.

[0047] For the above step S12, the imaging method of one of the first video and the second video is a radiographic imaging method. If the first video adopts a radiographic imaging method, the specific implementation process of determining multiple first object areas in the first video in the above step S12 can refer to the specific implementation process of the steps of this embodiment. If the second video adopts a radiographic imaging method, the specific implementation process of determining multiple second object areas in the second video in the above step S12 can refer to the specific implementation process of the steps of this embodiment. This application takes the imaging method of the second video as a radiographic imaging method as an example for explanation, and this application does not limit this.

[0048] The grayscale value projection of the image frame of the video using the X-ray imaging method is performed in a preset direction, that is, the histogram statistics of the image frame of the video can be performed, for example, the grayscale value projection of the second image frame of the second video is performed in a preset direction, wherein the preset direction may include a horizontal direction and a vertical direction.

[0049] S1212: Determine multiple object regions in the video using radiographic imaging based on grayscale value projection.

[0050] After performing grayscale value projection on the image frame of the video in the horizontal and vertical directions, the peak value occupied by each object in each preset direction is calculated, and the peak value of the grayscale value projection in the preset direction is determined, wherein the preset direction includes the horizontal direction and the vertical direction, so that the maximum value and minimum value of the grayscale value projection in the horizontal direction, and the maximum value and minimum value of the grayscale value projection in the vertical direction can be determined.

[0051] The position between two adjacent peaks in the preset direction is used as the position of the object region in the preset direction. Specifically, based on the maximum and minimum values ​​of the grayscale value projection in the horizontal direction and the maximum and minimum values ​​of the grayscale value projection in the vertical direction, the coordinate position of the object region is determined, that is, the position of the object region is determined, and multiple second object regions corresponding to the image frame of the video are obtained.

[0052] See also Figure 3 , the imaging mode of the second video is a radiographic imaging mode as an example for explanation, and the present application does not impose any limitation on this. The second image frame 100 of the second video may include one or more objects 101, wherein the imaging mode of the second image frame 100 is a radiographic imaging mode, such as an X-ray imaging mode. The object 101 may be detected on the second image frame 100 to obtain multiple second object regions 102 in the second image frame 100. Specifically, the second image frame 100 may be projected with a grayscale value in a preset direction, and based on the grayscale value projection, the maximum value and the minimum value of the grayscale value projection in the horizontal direction, and the maximum value and the minimum value of the grayscale value projection in the vertical direction are determined, thereby determining the second object regions 102 of the multiple objects 101 in the second image frame 100.

[0053] In some embodiments, see Figure 4 In the above step S12, the imaging mode of one of the first video and the second video is a natural imaging mode; determining a plurality of first object regions in the first video; or determining a plurality of second object regions in the second video may include the following steps:

[0054] S1221: Correct each image frame of the video using natural imaging; wherein the correction includes direction correction and / or shape correction.

[0055] For the above step S12, the imaging mode of one of the first video and the second video is a natural imaging mode. If the first video adopts a natural imaging mode, the specific implementation process of determining the multiple first object areas in the first video in the above step S12 can refer to the specific implementation process of the steps of this embodiment. If the second video adopts a natural imaging mode, the specific implementation process of determining the multiple second object areas in the second video in the above step S12 can refer to the specific implementation process of the steps of this embodiment. This application is described by taking the imaging mode of the first video as a natural imaging mode as an example, and this application does not limit this.

[0056] In step S12, in order to align the position of the object area in the image frame using the radiographic imaging method and facilitate better matching of the object area later, the image frame of the video using the natural imaging method can be corrected before the object area detection is performed on the video; wherein the correction includes direction correction and / or shape correction, so that the area of ​​the object in the natural imaging video is more matched with the area of ​​the object in the radiographic imaging video. Wherein, the direction correction can be to rotate the image frame of the video at a preset angle or direction, and the shape correction can be to correct the image frame of the video to a preset shape, etc., and the present application is not limited thereto.

[0057] See also Figure 5, the imaging mode of the first video is taken as a natural imaging mode as an example, and the present application does not limit this. For example, the first image frame 200 of the first video is obtained by shooting the object 202 by the camera device (not shown) of the conveying device 201, wherein the object 202 is placed on the conveying device 201 for transmission, and the picture in the first image frame 200 is obtained from the shooting direction from near to far, and the picture of the conveying device 201 in the picture of the first image frame 200 is trapezoidal, and the picture of the conveying device 201 can be understood as the background image in the first image frame 200. In order to align with the position and direction of the object in the second image frame of the second video, the first image frame 200 of the first video can be corrected, that is, direction correction and / or shape correction is performed, for example, the direction of the first image frame 200 can be rotated 90°, and the picture of the conveying device 201 of the first image frame 200 is corrected, that is, the picture of the area where the conveying device 201 of the first image frame 200 is located is stretched into a rectangle, and the corrected first image frame 200 is used for object detection.

[0058] In some embodiments, the preset area 203 including the conveying device 201 in the first image frame 200 can be corrected. For example, if the preset area 203 including the object 202 in the conveying device 201 in the first image frame 200 is trapezoidal, the preset area 203 can be oriented and / or shaped. The direction of the preset area 203 can be rotated 90°, and the shape of the preset area 203 can be stretched and adjusted to a rectangle to obtain a corrected first image frame 200.

[0059] S1222: Using a preset target detection algorithm to perform object detection on each image frame of the video using the natural imaging method, and obtain multiple object regions in the video using the natural imaging method.

[0060] A preset target detection algorithm can be used to perform object detection on image frames of a video using a natural imaging method to obtain object regions in image frames of a video using a radiographic imaging method. For example, a preset target detection algorithm is used to perform object detection on a first video to obtain multiple first object regions in the first video. Among them, the preset target detection algorithm can be a detection algorithm using a deep neural network. For example, the preset target detection algorithm can include target detection algorithms such as Faster RCNN (Faster Region-based Convolutional Neural Networks), SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), and Cascade RCNN (Cascade Region-based Convolutional Neural Networks), and this application does not impose any restrictions on this.

[0061] In this embodiment, by performing direction correction and / or shape correction on the image frames of the video using the natural imaging method, the objects in the video using the natural imaging method and the video using the radiographic imaging method are placed in the same scene, which is more conducive to matching the object regions in the video using the natural imaging method and the video using the radiographic imaging method, thereby improving the feasibility of matching objects in videos using different imaging methods and improving the accuracy of matching multiple first object regions with multiple second object regions.

[0062] In some embodiments, see Figure 6 Before the above step S13 associates the plurality of first object regions with the plurality of second object regions based on the determined first object contour features and the determined second object contour features, the following steps are also included:

[0063] S21: performing foreground segmentation on the plurality of first object regions and the plurality of second object regions respectively to obtain foreground segmentation results of the plurality of first object regions and the plurality of second object regions.

[0064] Foreground segmentation can be performed on multiple first object areas of the first video and multiple second object areas of the second video respectively. For the first object area, since the first object area is obtained by the image frame of the first video using the natural imaging method, the image frame of the first video obtained under natural light can display the external information of the object, and the background image of the image frame is the information of the area where the conveyor belt is located, the foreground segmentation of the first object area can extract the object information in the foreground of the image, and obtain the foreground segmentation results of multiple first object areas. For the second object area, since the second object area is obtained by the second video using the ray imaging method, such as the image frame obtained by the X-ray imaging method, due to the penetrability of the X-ray, the information of the inside and edge of the object will be fully reflected in the image frame of the second video obtained by the X-ray imaging, and the background of the image frame of the X-ray imaging can be the information of the area where the conveyor belt is located. When the foreground segmentation of the first object area is performed, the foreground of the image frame can be extracted, that is, the background information of the area where the conveyor belt is located is removed, and the image frame including the information of the inside and edge of the object is obtained, thereby obtaining the foreground segmentation results of multiple first object areas.

[0065] In some embodiments, the OTSU algorithm may be used to perform foreground segmentation on the first article region and the second article region, respectively, to obtain foreground segmentation results for the first article region and the second article region. The OTSU algorithm divides the image into two parts, the background and the foreground, according to the grayscale characteristics of the image of the first article region or the second article region. Since variance is a measure of the uniformity of grayscale distribution, the greater the inter-class variance between the background and the foreground, the greater the difference between the two parts constituting the image. When part of the foreground is misclassified as the background or part of the background is misclassified as the foreground, the difference between the two parts will become smaller. Therefore, the segmentation that maximizes the inter-class variance between the background and the foreground means that the probability of misclassification of the background and the foreground is minimized.

[0066] S22: Based on the foreground segmentation results of the plurality of first object regions and the plurality of second object regions, correspondingly obtain first object contour features of the plurality of first object regions and second object contour features of the plurality of second object regions.

[0067] Based on the foreground segmentation results of the plurality of first article regions and the plurality of second article regions, internal region information of the first article region and the second article region is suppressed according to the positions of the first article region and the second article region to obtain external contour information of the first article region and the second article region, thereby obtaining the corresponding article contour features of the first article region and the second article region, that is, obtaining the first article contour feature corresponding to the first article region and the second article contour feature corresponding to the second article region.

[0068] The first video and the second video have a great difference in imaging methods. By acquiring the object contour features of the first object area and the second object area, the first object area and the second object area are converted to dimensions with the same features, that is, the dimensions of the object contour features, so that the first object area and the second object area can be matched through the contour features.

[0069] In some embodiments, see Figure 7 In the above step S13, based on the determined contour features of each first object and the determined contour features of each second object, associating the plurality of first object regions with the plurality of second object regions may include the following steps:

[0070] S1311: Based on the determined first object contour features, obtain at least one first feature group; wherein the first feature group includes N1 first object contour features that appear continuously in the first video, and N1 is an integer greater than 1.

[0071] Based on each first object contour feature determined in the first video, at least one first feature group is obtained, wherein N1 consecutive first object contour features in the first video can be used as the first feature group, that is, the first feature group contains N1 first object contour features that appear continuously in the first video, and N1 is an integer greater than 1. Continuous appearance means that the first object contour features appear continuously in sequence in the first video, that is, the order of appearance in the first video is continuous. For example, if N1 is 3, 3 consecutive first object contour features can be obtained as a first feature group, so that multiple first feature groups can be obtained in this way.

[0072] In some implementations, before step S1311, the first object contour features may be sorted. Specifically, as follows:

[0073] The first object contour features are sorted based on the time sequence in which the first object contour features appear in the first video. The first object contour features appearing in the first video may indicate that a first object region of the object is detected in the first video, and the first object contour features corresponding to the first object region are detected. The sorting of the first object contour features is determined based on the time sequence in which the objects appear in the first video.

[0074] In some embodiments, the time when the object or the first object outline feature appears completely in the first video can be used as reference information for sorting the first object outline features corresponding to the objects. Alternatively, the first object outline features corresponding to the objects can be sorted based on the time sequence in which the objects or the first object outline feature appear completely in the first video.

[0075] In some embodiments, if multiple objects or multiple first object contour features appear at the same time in the first video, the first object contour features at the same time may be sorted according to a preset arrangement direction of the first video or the image frames of the first video.

[0076] In some implementations, after sorting the plurality of first object contour features, the above step S1311 is performed to sequentially select N1 consecutive second object contour features in the first video from the sorted first object contour features as the first feature group.

[0077] See also Figure 8, among the multiple first item profile features, for example, among the first item profile features 1 to the first item profile features 6 arranged in chronological order, N1 is 3 for illustration, and three consecutive first item profile features can be selected in sequence from the first item profile features 1 to the first item profile features 6 as the first feature group 301. For example, three first item profile features can be selected in sequence from the first item profile features arranged in chronological order as the first feature group 301. The first item profile feature 1, the first item profile feature 2, and the first item profile feature 3 are selected as the first first feature group 301, and the first item profile feature 4, the first item profile feature 5, and the first item profile feature 6 are selected as the second first feature group 301.

[0078] In some embodiments, see Fig. 9 , among the first item profile features 1 to the first item profile features 6 arranged in chronological order, taking N1 as 3 as an example, three consecutive first item profile features can be selected in sequence according to the first preset step length as the first feature group according to the first item profile features arranged in chronological order, wherein the first preset step length is less than N1. For example, if the first preset step length is 1, the first item profile feature 1, the first item profile feature 2 and the first item profile feature 3 can be selected as the first first feature group 302, the first item profile feature 2, the first item profile feature 3 and the first item profile feature 4 can be selected as the second first feature group 302, and the first item profile feature 3, the first item profile feature 4 and the first item profile feature 5 can be selected as the third first feature group 302.

[0079] In the present application, at least one first feature group is obtained based on the determined contour features of each first object. The first feature group may also be obtained in other ways, which is not limited in the present application.

[0080] S1312: Based on the determined second object contour features, obtain at least one second feature group; wherein the second feature group includes N1 second object contour features that appear continuously in the second video.

[0081] In some implementations, before step S1312, the second object contour features may be sorted. Specifically, as follows:

[0082] Similarly, the process can refer to the sorting method of the first object contour features to sort the second object contour features in the second video.

[0083] The second object contour features are sorted based on the time sequence in which the second object contour features appear in the second video.

[0084] In some embodiments, the time when the object or the second object outline feature completely appears in the second video can be used as reference information for sorting the second object outline features corresponding to the objects. Alternatively, the second object outline features corresponding to the objects can be sorted based on the time sequence in which the object or the second object outline feature completely appears in the second video.

[0085] In some embodiments, if multiple objects or multiple second object contour features appear at the same time in the second video, the second object contour features at the same time may be sorted according to a preset arrangement direction of the second video or the image frame of the second video. This application does not limit this.

[0086] In step S1312, based on the determined second object contour features, N1 second object contour features may be selected in sequence to obtain at least one second feature group; wherein the second feature group includes N1 second object contour features that appear continuously in the second video.

[0087] In some implementations, N1 consecutive second object contour features in the second video may be selected in sequence as the second feature group from each second object contour feature with N2 second object contour features as a step length, where N2 is an integer greater than or equal to 1. N2 is less than N1.

[0088] In some embodiments, after sorting the plurality of second object contour features, N1 consecutive second object contour features in the second video can be selected in sequence as the second feature group from the sorted second object contour features with a step size of N2 second object contour features, where N2 is an integer greater than or equal to 1.

[0089] In some embodiments, see Fig.10 Among the sorted multiple second item profile features, for example, among the second item profile features 1 to the second item profile features 6 sorted in chronological order, taking N1 as 3 and N2 as 1 as an example, the second item profile features 1, 2 and 3 can be selected as the first second feature group 401, and with one second profile feature as the step length, the second item profile features 2, 3 and 4 can be selected as the second second feature group 401, and the second item profile features 3, 4 and 5 can be selected as the third second feature group 401. By analogy, multiple second feature groups 401 can be obtained.

[0090] In the present application, at least one second feature group is obtained based on the determined contour features of each second object. The second feature group may also be obtained in other ways, which is not limited in the present application.

[0091] In this embodiment, the above-mentioned step S1311 and step S1312 can be executed in parallel, or step S1311 can be executed before step S1312. This application does not limit the execution order of the above-mentioned step S1311 and step S1312.

[0092] S1313: Take each first feature group in at least one first feature group as a feature group to be processed, and perform the following operations: select a second feature group that matches the feature group to be processed from at least one second feature group; associate the first object area corresponding to the N1 first object contour features contained in the feature group to be processed with the second object area corresponding to the N1 second object contour features contained in the selected second feature group.

[0093] Each first feature group may be matched with each second feature group, a second feature group matching each first feature group may be selected, and the matched first feature groups and second feature groups may be associated.

[0094] Specifically, each first feature group in the at least one first feature group may be used as a feature group to be processed in sequence, and the following operations may be performed:

[0095] From at least one second feature group, a second feature group matching the feature group to be processed is selected, that is, a second feature group matching each first feature group is selected, and the first item area corresponding to the N1 first item profile features contained in the feature group to be processed (first feature group) is associated with the second item area corresponding to the N1 second item profile features contained in the selected second feature group. The N1 first item areas in the first feature group and the N2 second item areas in the matching second feature group are associated one by one according to the order of the item profile features.

[0096] In some embodiments, the second feature group that matches the feature group to be processed can be selected in the following manner: respectively obtain the similarity between the feature group to be processed and each second feature group. For example, to calculate the similarity between the first feature group and the second feature group, the cosine distance can be used to calculate the similarity between the first feature group and the second feature group, that is, to calculate the cosine similarity between the first feature group and the second feature group. The closer the cosine value between the first eigenvalue and the second eigenvalue is to 1, the higher the similarity between the first eigenvalue and the second eigenvalue. The present application can also use other similarity calculation methods to obtain the similarity between the first feature group and each second feature group, and the present application does not limit this.

[0097] A second feature group whose similarity meets the similarity requirement is selected as the second feature group that matches the feature group to be processed (the first feature group). The similarity requirement includes that the similarity between the feature group to be processed and each second feature group is the highest, and / or the similarity between the feature group to be processed and the second feature group exceeds a preset similarity threshold. For example, the second feature group with the highest similarity to the feature group to be processed can be used as the matching second feature group, so that the matching feature group to be processed (the first feature group) and the second feature group are associated.

[0098] In some embodiments, when the first feature group matches the second feature group, the N1 first item contour features included in the first feature group and the N1 second item contour features included in the second feature group can be associated respectively, so as to sequentially associate the first item regions corresponding to the N1 first item contour features and the second item regions corresponding to the N1 second item contour features, wherein the association can be performed sequentially according to the chronological order of the N1 first item contour features included in the first feature group and the N1 second item contour features included in the second feature group.

[0099] See also Fig.11 For example, the matched feature group includes a first feature group 500 matching a second feature group 600, and the first feature group 500 includes a first item outline feature 501, a first item outline feature 502, and a first item outline feature 503. The second feature group 600 includes a second item outline feature 601, a second item outline feature 602, and a second item outline feature 603. In the process of associating the first feature group 500 and the second feature group 600, the first item outline feature 501 may be associated with the second item outline feature 601, the first item outline feature 502 may be associated with the second item outline feature 602, and the first item outline feature 503 may be associated with the second item outline feature 603. In some embodiments, based on this matched feature group, other corresponding first item outline features may be associated with second item outline features in chronological order, and this application does not limit this.

[0100] In some embodiments, in the above step S13, the first item area and the second item area are associated using the first item contour features corresponding to the first item area within the preset time and the second item contour features corresponding to the second item area within the preset time. Specifically, at least one first feature group is obtained based on the multiple first item contour features corresponding to the multiple first item areas within the preset time, and at least one second feature group is obtained based on the multiple second item contour features corresponding to the multiple second item areas within the preset time. Thus, a matching second feature group is selected for each first feature group within the preset time to form a matching feature group. The corresponding first item area and the second item area in the matching feature group are associated.

[0101] In some embodiments, after finding the second feature group that matches each first feature group, each first feature group and the matching second feature group form a plurality of matching feature groups, and the first feature group and the second feature group with the highest matching degree can be selected from the plurality of matching feature groups, and the matching first feature group and the second feature group can be associated, and other first object contour features and second object contour features can be associated according to the matching first feature group and the second feature group, wherein the first feature group can be used as the starting point of the first contour feature, and the matching second feature group can be used as the starting point of the second contour feature, and the corresponding other first object contour features and second object contour features can be associated in chronological order.

[0102] In this embodiment, the first article area and the second article area are associated by utilizing the object contour features in each first article area and each second article area, and the object areas are associated by the object contour features, so that the objects imaged by radiography and natural light have similar features, thereby increasing the reliability of the object similarity.

[0103] In addition, at least one first feature group is obtained based on the determined first object contour features, at least one second feature group is obtained based on the determined second object contour features, a matching second feature group is selected for each first feature group, and each first feature group in at least one first feature group is used as a feature group to be processed, and the following operations are performed: from at least one second feature group, a second feature group matching the feature group to be processed is selected; and a first object area corresponding to the N1 first object contour features included in the feature group to be processed is associated with a second object area corresponding to the N1 second object contour features included in the selected second feature group. By jointly matching the N1 object contour features in the feature group, the computational complexity of the algorithm can be reduced, the time consumed by the algorithm can be reduced, and the credibility of the object association can be increased.

[0104] In some embodiments, see Fig.12In the above step S13, based on the determined contour features of each first object and the determined contour features of each second object, the plurality of first object regions and the plurality of second object regions are associated, and the following steps may also be included:

[0105] S1321: Based on the determined first object contour features, obtain at least one first feature group; wherein the first feature group includes N1 first object contour features that appear continuously in the first video, and N1 is an integer greater than 1.

[0106] S1322: Based on the determined second object contour features, obtain at least one second feature group; wherein the second feature group includes N1 second object contour features that appear continuously in the second video.

[0107] In this embodiment, the specific implementation process of step S1321 and step S1322 can refer to step S1311 and step S1312 in the above embodiment, and this application will not repeat them here.

[0108] S1323: Take each first feature group in at least one first feature group as a feature group to be processed, and perform the following operations: from at least one second feature group, select the second feature group and the feature group to be processed in sequence to obtain similarity; if the similarity is greater than a preset similarity threshold, associate the first item area corresponding to the N1 first item contour features included in the feature group to be processed with the second item area corresponding to the N1 second item contour features included in the selected second feature group; and associate the first item area corresponding to the N3 first item contour features included in the preset range of the feature group to be processed with the second item area corresponding to the N3 second item contour features included in the preset range of the selected second feature group.

[0109] Each first feature group may be matched with each second feature group, a second feature group matching each first feature group may be selected, and the matched first feature groups and second feature groups may be associated.

[0110] Specifically, each first feature group in the at least one first feature group may be used as a feature group to be processed in sequence, and the following operations may be performed:

[0111] From at least one second feature group, select the second feature group and the feature group to be processed (first feature group) in sequence to obtain similarity. If the similarity is greater than a preset similarity threshold, associate the first item area corresponding to the N1 first item profile features contained in the feature group to be processed (first feature group) with the second item area corresponding to the N1 second item profile features contained in the selected second feature group. And associate the first item area corresponding to the N3 first item profile features contained in the feature group to be processed (first feature group) within a preset range with the second item area corresponding to the N3 second item profile features contained in the selected second feature group within a preset range, wherein N3 is an integer greater than or equal to 1.

[0112] See also Fig.13 For example, the first feature group 700 and the second feature group 800 are matching feature groups, and the first feature group 700 includes the first item outline feature 701, the first item outline feature 702, and the first item outline feature 703. The second feature group 800 includes the second item outline feature 801, the second item outline feature 802, and the second item outline feature 803. In the process of associating the first feature group 700 and the second feature group 800, the first item outline feature 701 can be associated with the second item outline feature 801, the first item outline feature 702 can be associated with the second item outline feature 802, and the first item outline feature 703 can be associated with the second item outline feature 803. Furthermore, the first object area corresponding to the N3 first object outline features included in the preset range of the first feature group 700 and the second object area corresponding to the N3 second object outline features included in the preset range of the selected second feature group 800 can be associated in sequence, that is, the first object outline feature 704 can be associated with the second object outline feature 804, the first object outline feature 705 can be associated with the second object outline feature 805, and so on, and all the first object outlines and second object outlines within the preset time or within the preset range can be associated.

[0113] For the above implementation, the present application provides an item association device. Fig.14 , Fig.14 1 is a schematic diagram of the structure of an embodiment of the object association device of the present application. The object association device 60 includes an acquisition module 61, a detection module 62 and an association module 63, wherein the acquisition module 61, the detection module 62 and the association module 63 are connected.

[0114] The acquisition module 61 is used to acquire a first video and a second video captured in the same time period using different imaging methods.

[0115] The detection module 62 is used to determine first object contour features corresponding to each of the plurality of first object regions in the first video; and to determine second object contour features corresponding to each of the plurality of second object regions in the second video.

[0116] The association module 63 is used to associate a plurality of first object regions with a plurality of second object regions based on the determined first object contour features and the determined second object contour features.

[0117] In some embodiments, the imaging method of one of the first video and the second video is a radiographic imaging method; the imaging method of one of the first video and the second video is a natural imaging method. The acquisition module 61 can also be used to obtain a first video of an object placed on the conveying device using a natural imaging method; and obtain a second video obtained by the detection device using a radiographic imaging method for the object conveyed by the conveying device.

[0118] In some embodiments, the imaging method of one of the first video and the second video is a radiographic imaging method. The detection module 62 can also be used to perform a grayscale value projection in a preset direction on each image frame of the video using the radiographic imaging method; based on the grayscale value projection, determine multiple object areas in the video using the radiographic imaging method.

[0119] Wherein, based on the grayscale value projection, multiple object areas in the video using the radiographic imaging method are determined, including: determining the peak value of the grayscale value projection in a preset direction, wherein the preset direction includes a horizontal direction and a vertical direction; and taking the position between two adjacent peaks in the preset direction as the position of the object area in the preset direction.

[0120] In some embodiments, the imaging mode of one of the first video and the second video is a natural imaging mode. The detection module 62 can also be used to use a preset target detection algorithm to perform object detection on each image frame of the video using the natural imaging mode to obtain multiple object regions in the video using the radiographic imaging mode.

[0121] In some embodiments, the imaging mode of one of the first video and the second video is a natural imaging mode. The detection module 62 can be used to correct each image frame of the video using the natural imaging mode; wherein the correction includes direction correction and / or shape correction.

[0122] In some embodiments, the detection module 62 may also be used to perform foreground segmentation on the plurality of first object regions and the plurality of second object regions respectively to obtain foreground segmentation results for the plurality of first object regions and the plurality of second object regions. Based on the foreground segmentation results for the plurality of first object regions and the plurality of second object regions, first object contour features for the plurality of first object regions and second object contour features for the plurality of second object regions are correspondingly obtained.

[0123] In some embodiments, the association module 63 can also be used to obtain at least one first feature group based on the determined first object contour features; wherein the first feature group includes N1 first object contour features that appear continuously in the first video, and N1 is an integer greater than 1. Based on the determined second object contour features, at least one second feature group is obtained; wherein the second feature group includes N1 second object contour features that appear continuously in the second video. Each first feature group in the at least one first feature group is used as a feature group to be processed, and the following operations are performed: from the at least one second feature group, a second feature group that matches the feature group to be processed is selected; and the first object area corresponding to the N1 first object contour features included in the feature group to be processed is associated with the second object area corresponding to the N1 second object contour features included in the selected second feature group.

[0124] The step of selecting a matching second feature group for each first feature group includes: respectively obtaining the similarity between the first feature group and each second feature group; and selecting a second feature group whose similarity meets the similarity requirement as the second feature group matching the first feature group.

[0125] In some embodiments, the association module 63 may also be used to sort the first item outline features based on the time sequence of the appearance of the first item outline features in the first video. From the sorted first item outline features, N1 consecutive second item outline features in the first video are selected in sequence as the first feature group. And, based on the time sequence of the appearance of the second item outline features in the second video, sort the second item outline features. From the sorted second item outline features, N1 consecutive second item outline features in the second video are selected in sequence as the second feature group with N2 second item outline features as the step size, where N2 is an integer greater than or equal to 1.

[0126] In some embodiments, the association module 63 is used to select a second feature group that matches the feature group to be processed from at least one second feature group, including: respectively obtaining the similarity between the feature group to be processed and each second feature group; and selecting a second feature group whose similarity meets the similarity requirement as the second feature group that matches the feature group to be processed.

[0127] In some embodiments, the association module 63 is used to take each first feature group in at least one first feature group as a feature group to be processed, and perform the following operations: from at least one second feature group, sequentially select the second feature group and the feature group to be processed to obtain similarity; if the similarity is greater than a preset similarity threshold, associate the first item area corresponding to the N1 first item contour features included in the feature group to be processed with the second item area corresponding to the N1 second item contour features included in the selected second feature group; and associate the first item area corresponding to the N3 first item contour features included in the feature group to be processed within a preset range with the second item area corresponding to the N3 second item contour features included in the selected second feature group within a preset range.

[0128] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0129] For the above embodiment, the present application provides a computer device, see Fig.15 , Fig.15 1 is a schematic diagram of the structure of an embodiment of a computer device of the present application. The computer device 70 includes a memory 71 and a processor 72, wherein the memory 71 and the processor 72 are coupled to each other, the memory 71 stores program data, and the processor 72 is used to execute the program data to implement the steps in any embodiment of the above-mentioned method for associating items.

[0130] In this embodiment, the processor 72 may also be referred to as a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip having signal processing capabilities. The processor 72 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 72 may also be any conventional processor, etc.

[0131] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0132] The method of the above embodiment can be implemented in the form of a computer program, so the present application proposes a storage device, see Fig.16 , Fig.16 The storage device 80 stores program data 81 that can be run by a processor, and the program data can be executed by the processor to implement the steps of any embodiment of the above-mentioned method for associating items.

[0133] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0134] The storage device 80 of this embodiment can be a medium that can store program data, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program data. The server can send the stored program data to other devices for execution, or it can also execute the stored program data itself.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application.

[0139] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0140] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for associating items, It is characterized in that The method comprises: Acquire a first video and a second video captured in the same time period using different imaging methods; Determine a first object contour feature corresponding to each of a plurality of first object regions in the first video; and determine a second object contour feature corresponding to each of a plurality of second object regions in the second video; Associating the plurality of first object regions with the plurality of second object regions based on the determined first object contour features and the determined second object contour features, comprising: Based on the determined first object contour features, at least one first feature group is obtained; wherein the first feature group includes N1 first object contour features that appear continuously in the first video, and N1 is an integer greater than 1; Based on the determined second object contour features, at least one second feature group is obtained; wherein the second feature group includes the N1 second object contour features that appear continuously in the second video; Each of the at least one first feature group is used as a feature group to be processed, and the following operations are performed: Selecting a second feature group that matches the feature group to be processed from the at least one second feature group; associating a first object region corresponding to N1 first object contour features included in the feature group to be processed with a second object region corresponding to N1 second object contour features included in the selected second feature group; The selecting, from the at least one second feature group, a second feature group matching the feature group to be processed comprises: Respectively obtaining similarities between the feature group to be processed and each of the second feature groups; The second feature group whose similarity meets the similarity requirement is selected as the second feature group matching the feature group to be processed.

2. The method according to claim 1, It is characterized in that Before obtaining at least one first feature group based on the determined first object contour features, the method further includes: sorting the first object contour features based on the time sequence in which the first object contour features appear in the first video; and sorting the second object contour features based on the time sequence in which the second object contour features appear in the second video; The step of obtaining at least one first feature group based on the determined first object contour features comprises: From the sorted first object contour features, sequentially select the N1 consecutive first object contour features in the first video as the first feature group; The step of obtaining at least one second feature group based on the determined second object contour features comprises: From the sorted second object contour features, with N2 second object contour features as a step length, the N1 consecutive second object contour features in the second video are selected in sequence as the second feature group, wherein N2 is an integer greater than or equal to 1.

3. The method according to claim 1, It is characterized in that The method further comprises: The first feature groups in the at least one first feature group are respectively used as feature groups to be processed, and the following operations are performed: From the at least one second feature group, sequentially select the second feature group and the feature group to be processed to obtain similarity; if the similarity is greater than a preset similarity threshold, associate the first item area corresponding to the N1 first item contour features included in the feature group to be processed with the second item area corresponding to the N1 second item contour features included in the selected second feature group; and associate the first item area corresponding to the N3 first item contour features included in the feature group to be processed within a preset range with the second item area corresponding to the N3 second item contour features included in the selected second feature group within the preset range, wherein N3 is an integer greater than or equal to 1.

4. The method according to claim 1, It is characterized in that Before associating the plurality of first object regions with the plurality of second object regions based on the determined first object contour features and the determined second object contour features, the method further includes: Performing foreground segmentation on the plurality of first article regions and the plurality of second article regions respectively to obtain foreground segmentation results of the plurality of first article regions and the plurality of second article regions; Based on the foreground segmentation results of the plurality of first object regions and the plurality of second object regions, first object contour features of the plurality of first object regions and second object contour features of the plurality of second object regions are correspondingly obtained.

5. The method according to claim 1, It is characterized in that The imaging method of one of the first video and the second video is a radiographic imaging method; the determining of the plurality of first object regions in the first video; or the determining of the plurality of second object regions in the second video includes: Performing grayscale value projection in a preset direction on each image frame of the video, wherein the imaging method of the video is the ray imaging method; Based on the gray value projection, a plurality of object regions in the video using the radiographic imaging method are determined.

6. The method according to claim 5, It is characterized in that The determining, based on the gray value projection, a plurality of object regions in the video using the radiographic imaging method comprises: Determine a peak value of the grayscale value projection in the preset direction, wherein the preset direction includes a horizontal direction and a vertical direction; The position between two adjacent peaks in the preset direction is used as the position of the object area in the preset direction.

7. The method according to claim 1, It is characterized in that An imaging mode of one of the first video and the second video is a natural imaging mode; The determining of a plurality of first object regions in the first video; or the determining of a plurality of second object regions in the second video, comprises: A preset target detection algorithm is used to perform object detection on each image frame of the video using the natural imaging method to obtain multiple object regions in the video using the natural imaging method.

8. The method according to claim 1, It is characterized in that An imaging mode of one of the image frames of the first video and the second video is a natural imaging mode; Before determining a plurality of first object regions in the first video; or determining a plurality of second object regions in the second video, the method further includes: Each image frame of the video using the natural imaging method is corrected; wherein the correction includes direction correction and / or shape correction.

9. The method according to claim 1, It is characterized in that The obtaining of the first video and the second video captured in the same time period using different imaging methods comprises: The first video is obtained by shooting the object placed on the conveying device in a natural imaging manner; and The second video is obtained by acquiring the detection device using radiographic imaging of the article conveyed by the conveying device.

10. A computer device, It is characterized in that The invention comprises a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement the steps of the method according to any one of claims 1 to 9.

11. A storage device, It is characterized in that Program data that can be executed by a processor are stored, and the program data is used to implement the steps of the method according to any one of claims 1 to 9.

Images