Airport luggage detection method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides an airport luggage detection method and device, a storage medium and electronic equipment, and the method comprises the steps: extracting a target video frame and a reference video frame from target video data collected by connected target camera equipment; detecting whether luggage exists in the target video frame or not, and detecting whether luggage exists in the reference video frame or not; when it is detected that the target luggage exists in the target video frame, extracting a target attribute feature of the target luggage and a target environment feature of the target luggage from the target video frame; when it is detected that the reference luggage exists in the reference video frame, extracting reference attribute features of the reference luggage and reference environment features of the reference luggage from the reference video frame; detecting a target matching degree between the target luggage and the reference luggage; and generating target description information of the target luggage. According to the invention, the method solves a problem that the detection accuracy of the airport luggage is low, and achieves an effect of improving the detection accuracy of the airport luggage.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a method and device for detecting airport luggage, a storage medium, and an electronic device. Background Art

[0002] Baggage flow, passenger flow and information flow at civil airports are the three main processes to ensure the safe operation of civil aviation airports. Among them, whether the airport baggage flow is safe, smooth and efficient directly affects the normality of airport flights and passenger satisfaction.

[0003] In the related art, when passengers check in their luggage, a luggage tag will be attached to the luggage, which is a mark to identify the luggage and a voucher for the passenger to collect the checked luggage. At present, the barcode on the luggage tag is the only identification information of the luggage. When the luggage tag is contaminated or damaged, the luggage information cannot be effectively read.

[0004] In recent years, with the popularization of RFID (Radio Frequency Identification) technology, some airports have replaced traditional luggage tags with luggage tags with RFID chips, and use optical scanning, OCR (Optical Character Recognition), and RFID scanning equipment to scan luggage tag information to obtain tracking luggage information during the luggage handling process. Although this method has greatly improved the recognition rate of luggage, the cost of each luggage tag has also increased accordingly on the original basis. In addition, the solution of embedding luggage information with RFID chips and hanging them on the luggage still cannot solve the problem of luggage identification when the luggage tag falls off. It can only be found by offline registration of passenger flight information, luggage appearance information, etc. In addition, both processing methods will consume a large amount of paper luggage tags. Summary of the invention

[0005] The embodiments of the present application provide a method and device for detecting airport baggage, a storage medium, and an electronic device, so as to at least solve the problem of low accuracy of airport baggage detection in the related art.

[0006] According to an embodiment of the present application, a method for detecting luggage at an airport is provided, which is applied to a detection device, wherein the detection device is used to connect to a camera device at an airport, and the camera device is used to collect video data of luggage at the airport. The method comprises: extracting a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data before the target video frame; detecting whether there is luggage in the target video frame, and detecting whether there is luggage in the reference video frame; and extracting target attribute features of the target luggage and target environment features of the target luggage from the target video frame when detecting the presence of target luggage in the target video frame. The method comprises the steps of: detecting a target environment feature of the target baggage in the target video frame, wherein the target environment feature is used to indicate a target environment in which the target baggage is located in the target video frame; and extracting a reference attribute feature of the reference baggage and a reference environment feature of the reference baggage from the reference video frame when a reference baggage is detected in the reference video frame, wherein the reference environment feature is used to indicate a reference environment in which the reference baggage is located in the reference video frame; detecting a target matching degree between the target baggage and the reference baggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature; and generating target description information of the target baggage according to the target matching degree, wherein the target description information is used to describe a location of the target baggage in the airport.

[0007] In an exemplary embodiment, a target detection network is deployed in the detection device, and the extracting the target attribute features of the target luggage and the target environment features of the target luggage from the target video frame includes: inputting the target video frame into the target detection network to obtain the target attribute features and the target environment features output by the target detection network; extracting the reference attribute features of the reference luggage and the reference environment features of the reference luggage from the reference video frame includes: inputting the reference video frame into the target detection network to obtain the reference attribute features and the reference environment features output by the target detection network; wherein the target detection network is obtained by training an initial detection network using sample video data, wherein the sample video data includes sample video frames, sample attribute features, and sample environment features having corresponding relationships.

[0008] In an exemplary embodiment, detecting the target matching degree between the target luggage and the reference luggage based on the target attribute characteristics, the target environment characteristics, the reference attribute characteristics and the reference environment characteristics includes: detecting a first matching degree between the target attribute characteristics and the reference attribute characteristics, and detecting a second matching degree between the target environment characteristics and the reference environment characteristics; and detecting the target matching degree based on the first matching degree and the second matching degree.

[0009] In an exemplary embodiment, the detecting a first degree of matching between the target attribute feature and the reference attribute feature comprises: when the target attribute feature comprises a target angle corresponding to the target luggage, a target aspect ratio of the target luggage, and a target pixel value corresponding to the target luggage, and the reference attribute feature comprises a reference angle corresponding to the reference luggage, a reference aspect ratio of the reference luggage, and a reference pixel value corresponding to the reference luggage, detecting a first degree of matching between the target attribute feature and the reference attribute feature by performing the following steps: detecting a first angle difference between the target angle and the reference angle, detecting a first aspect ratio difference between the target aspect ratio and the reference aspect ratio, and detecting a pixel difference mean between the target pixel value and the reference pixel value; and determining the first degree of matching as being greater than or equal to a first degree of matching threshold when the first angle difference is less than a first angle difference threshold, the first aspect ratio difference is less than a first aspect ratio difference threshold, and the pixel difference mean is less than a pixel difference mean threshold.

[0010] In an exemplary embodiment, the detecting the mean pixel difference between the target pixel value and the reference pixel value comprises: adjusting the bounding box of the target luggage to a target size to obtain a target bounding box, and adjusting the bounding box of the reference luggage to the target size to obtain a reference bounding box, wherein the target bounding box and the reference bounding box each include m*n pixels; dividing the target bounding box into N pixels according to the diagonal of the target bounding box. 一 A bounding box is defined, and the reference bounding box is divided into N2 bounding boxes according to the diagonal of the reference bounding box, wherein N1 is equal to N2 and N1 is a positive integer; pixel differences between pixel points in the N1 bounding boxes and corresponding pixel points in the N2 bounding boxes are calculated to obtain N groups of pixel differences, wherein the i-th group of pixel differences in the N groups of pixel differences includes the N 一 The i-th bounding box 一a pixel difference between a pixel point in a bounding box and a corresponding pixel point in an i2-th bounding box of the N2 bounding boxes, N is equal to N1 and N2, i1 is equal to i2, i1 is a positive integer less than or equal to N1, and i2 is a positive integer less than or equal to N2; a value obtained by dividing a target product value by N is determined as the pixel difference mean, wherein the target product value is equal to a value obtained by summing the N product values, a j-th product value among the N product values ​​is equal to a value obtained by multiplying a j-th group of pixel differences among the N groups of pixel differences by a j-th weight among the N weights, wherein j is a positive integer less than or equal to N, a weight of a bounding box located in a target area among the N weights is greater than or equal to each weight among the N-1 weights, and the target area is an area in the target bounding box where a probability of including the target baggage is greater than or equal to a first probability threshold, or an area in the reference bounding box where a probability of including the reference baggage is greater than or equal to a second probability threshold.

[0011] In an exemplary embodiment, the detecting the second degree of matching between the target environment feature and the reference environment feature comprises: when the target environment feature comprises a first quantity of first candidate baggage around the target baggage in the target environment, a first aspect ratio of the first candidate baggage, a first angle corresponding to the first candidate baggage, and a first distance between the target baggage and the first candidate baggage, and the reference environment feature comprises a second quantity of second candidate baggage around the reference baggage in the reference environment, a second aspect ratio of the second candidate baggage, a second angle corresponding to the second candidate baggage, and a second distance between the reference baggage and the second candidate baggage, detecting the second degree of matching between the target environment feature and the reference environment feature by performing the following steps: detecting a quantity difference between the first quantity and the second quantity, detecting a second aspect ratio difference between the first aspect ratio and the second aspect ratio, detecting a second angle difference between the first angle and the second angle, and detecting a distance difference between the first distance and the second distance; and when the quantity difference is less than a quantity threshold, the second aspect ratio difference is less than a second aspect ratio difference threshold, the second angle difference is less than a second angle difference threshold, and the distance difference is less than a distance difference threshold, determining the second degree of matching as being greater than or equal to a second degree of matching threshold.

[0012] In an exemplary embodiment, detecting the target matching degree according to the first matching degree and the second matching degree includes: detecting whether the first matching degree is greater than or equal to a first matching degree threshold; if it is detected that the first matching degree is greater than or equal to the first matching degree threshold, detecting whether the second matching degree is greater than or equal to a second matching degree threshold; if it is detected that the second matching degree is greater than or equal to the second matching degree threshold, determining the target matching degree to be greater than or equal to the target matching degree threshold, wherein, if the target matching degree is greater than or equal to the target matching degree threshold, the target luggage is the same as the reference luggage.

[0013] In an exemplary embodiment, generating the target description information of the target baggage according to the target matching degree includes: when the target matching degree is greater than or equal to a target matching degree threshold, extracting a reference baggage identifier of the reference baggage, extracting a target position of the target baggage in the airport output by a target detection network, and generating the target description information carrying the reference baggage identifier and the target position; when the target matching degree is less than a target matching degree threshold, generating a target baggage identifier of the target baggage, extracting a target position of the target baggage in the airport output by the target detection network, and generating the target description information carrying the target baggage identifier and the target position, wherein the target detection network is further used to detect the position of the baggage in the airport in an input video frame; after generating the target description information of the target baggage according to the target matching degree, the method further includes: detecting whether a change in the target position of the target baggage in the airport meets a target change condition, obtaining a detection result, and when the detection result indicates that the change does not meet the target change condition, deleting the target baggage in the target video frame to obtain a candidate video frame.

[0014] According to another embodiment of the present application, there is provided an airport baggage detection device, which is applied to a detection device, wherein the detection device is used to connect to an airport camera device, and the camera device is used to collect video data of the baggage in the airport, and the device comprises: a first extraction module, which is used to extract a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data before the target video frame; a first detection module, which is used to detect whether there is baggage in the target video frame, and to detect whether there is baggage in the reference video frame; a second extraction module, which is used to extract the target attribute features of the target baggage and the reference video frame from the target video frame when the target baggage is detected in the target video frame. a target environment feature, wherein the target environment feature is used to indicate the target environment in which the target baggage is located in the target video frame; and when it is detected that there is a reference baggage in the reference video frame, extracting a reference attribute feature of the reference baggage and a reference environment feature of the reference baggage from the reference video frame, wherein the reference environment feature is used to indicate the reference environment in which the reference baggage is located in the reference video frame; a second detection module, configured to detect a target matching degree between the target baggage and the reference baggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature; and a generation module, configured to generate target description information of the target baggage according to the target matching degree, wherein the target description information is used to describe the location of the target baggage in the airport.

[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0018] Through the present application, the camera device can collect video data of the entire process of luggage in the airport. The luggage may be located in different environments in the airport. The detection device can detect the attribute characteristics and environmental characteristics of the luggage in the two video frames extracted from the video data collected by the camera device. It can be but not limited to detecting the matching degree between the attribute characteristics and environmental characteristics of the luggage in the two video frames to achieve the real-time position tracking of the luggage in the airport. Therefore, the problem of low accuracy of airport luggage detection can be solved, and the effect of improving the accuracy of airport luggage detection is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a hardware structure block diagram of a server device of an airport baggage detection method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for detecting airport baggage according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an architecture of an optional optimized SPPF network according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an optional target detection network according to an embodiment of the present application;

[0023] Figure 5 is an optional flow chart of detecting features of luggage through a target detection network according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of an optional airport baggage detection method according to an embodiment of the present application;

[0025] Figure 7 It is a structural block diagram of an airport baggage detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a server device of an airport baggage detection method according to an embodiment of the present application. Figure 1As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the airport baggage detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] First, the terms involved in the embodiments of the present application are explained as follows:

[0032] Input: Model input.

[0033] CNN: Convolutional Neural Network, convolutional network.

[0034] BN: Batch Normalization, data batch normalization.

[0035] RNN: Recurrent Neural Network.

[0036] Dropout: Randomly inactivate neurons to avoid overfitting.

[0037] In this embodiment, a method for detecting luggage at an airport is provided, which is applied to a detection device, wherein the detection device is used to connect to a camera device at an airport, and the camera device is used to collect video data of the luggage at the airport. Figure 2 is a flow chart of a method for detecting airport baggage according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0038] Step S202, extracting a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data and preceding the target video frame;

[0039] Step S204, detecting whether there is luggage in the target video frame, and detecting whether there is luggage in the reference video frame;

[0040] Step S206, when it is detected that there is a target baggage in the target video frame, extracting target attribute features of the target baggage and target environment features of the target baggage from the target video frame, wherein the target environment features are used to indicate the target environment in which the target baggage is located in the target video frame; and when it is detected that there is a reference baggage in the reference video frame, extracting reference attribute features of the reference baggage and reference environment features of the reference baggage from the reference video frame, wherein the reference environment features are used to indicate the reference environment in which the reference baggage is located in the reference video frame;

[0041] Step S208, detecting a target matching degree between the target luggage and the reference luggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature;

[0042] Step S210: generating target description information of the target luggage according to the target matching degree, wherein the target description information is used to describe the location of the target luggage in the airport.

[0043] Through the above steps, the camera device can collect video data of the entire process of the luggage in the airport. The luggage may be located in different environments in the airport. The detection device can detect the attribute characteristics and environmental characteristics of the luggage in the two video frames extracted from the video data collected by the camera device. It can be but not limited to detecting the matching degree between the attribute characteristics and environmental characteristics of the luggage in the two video frames to achieve the real-time position tracking of the luggage in the airport. Therefore, the problem of low accuracy of airport luggage detection can be solved, and the effect of improving the accuracy of airport luggage detection is achieved.

[0044] In the technical solution provided in the above step S202, the target video frame and the reference video frame may include but are not limited to two frames that are not continuous in time. For example, two frames may be randomly extracted from 20s of video data, or the target video frame and the reference video frame may include but are not limited to two frames that are continuous in time.

[0045] Optionally, in this embodiment, the camera equipment may include but is not limited to all cameras deployed in the airport, the video data may include but is not limited to video data collected by all cameras deployed in the airport, and the video data may be divided into multiple video frames according to unit time but is not limited to.

[0046] In the technical solution provided in the above step S204, it is possible but not limited to detect whether there is luggage in the video frame in the following manner: input the video frame into the object detection network, obtain a first confidence output by the object detection network, and determine that there is luggage in the video frame when the first confidence is greater than or equal to a first confidence threshold, and determine that there is no luggage in the video frame when the first confidence is less than the first confidence threshold.

[0047] Optionally, in this embodiment, there may be multiple pieces of luggage in the video frame. In this case, it is possible but not limited to determining all objects corresponding to multiple confidence levels detected that are greater than or equal to the first confidence threshold as luggage.

[0048] In the technical solution provided in the above step S206, the environment in which the luggage is located may change. For example, in the process from when the passenger completes the luggage check-in to when the luggage is loaded on the aircraft, the environment in which the luggage is located may change. It is understandable that the environment in which the luggage is located may change dynamically.

[0049] In an exemplary embodiment, a target detection network may be deployed in the detection device in the following manner, but is not limited to the following manner, wherein the target attribute features of the target luggage and the target environment features of the target luggage are extracted from the target video frame by: inputting the target video frame into the target detection network to obtain the target attribute features and the target environment features output by the target detection network; and the reference attribute features of the reference luggage and the reference environment features of the reference luggage are extracted from the reference video frame, including: inputting the reference video frame into the target detection network to obtain the reference attribute features and the reference environment features output by the target detection network; wherein the target detection network is obtained by training an initial detection network using sample video data, wherein the sample video data includes sample video frames, sample attribute features, and sample environment features having corresponding relationships.

[0050] Optionally, in this embodiment, the above method also includes: performing the rth round of training on the initial detection network by executing the following steps, where r is a positive integer: inputting the rth round of data used in the rth round of training into the detection network used in the rth round of training, and obtaining the rth group of attribute features and the rth group of environmental features output by the detection network corresponding to the rth round of training; detecting the first feature matching degree between the rth group of attribute features and the rth group of sample attribute features, and detecting the second feature matching degree between the rth group of environmental features and the rth group of sample environmental features; when the first feature matching degree is greater than or equal to the first feature matching degree threshold, and the second feature matching degree is greater than or equal to the second feature matching degree threshold, ending the training and determining the detection network corresponding to the rth round of training as the target detection network; when the first feature matching degree is less than the first feature matching degree threshold, and / or the second feature matching degree is less than the second feature matching degree threshold, adjusting the detection network corresponding to the rth round of training to obtain the detection network used for the r+1th round of training.

[0051] Through the embodiments of the present application, a target detection network is deployed on a detection device, and multiple video frames can be input into the target detection network, but it is not limited to the target detection network. The target detection network can detect the matching degree between the luggage in the multiple video frames in parallel, but it is not limited to the matching degree between the luggage in the multiple video frames, thereby improving the efficiency of video frame processing.

[0052] Optionally, in this embodiment, the target detection network may include, but is not limited to, an optimized SPPF (Spatial Pyramid Pooling-Fast) network. Figure 3 is a schematic diagram of an optional optimized SPPF network architecture according to an embodiment of the present application, such as Figure 3As shown, the optimized SPPF network can also include, but is not limited to, a Conv network and a Maxpool2d network. By changing all the C2f networks of the neck and the head to Conv networks, the parameters of the model are reduced and the training efficiency of the model is improved.

[0053] Optionally, in this embodiment, the angle, confidence, coordinates, etc. of the luggage in the video frame output by the target detection network can be obtained by, but not limited to, inputting the video frame into the target detection network, wherein the angle of the luggage can include, but is not limited to, the angle of the luggage relative to a rotating track or a conveyor belt, and the coordinates can be, but are not limited to, used to represent the position of the luggage in the airport.

[0054] Figure 4 is a schematic diagram of an optional target detection network according to an embodiment of the present application, such as Figure 4 As shown, the target detection network may include but is not limited to a Conv network and a Conv2d network. The video frame is input into the target detection network, and the angle, confidence, coordinates, etc. of the luggage in the video frame may be output through the Conv network and the Conv2d network but is not limited to.

[0055] In order to better understand the detection process of the target detection network in the embodiment of the present application, the process of detecting the attribute characteristics and environmental characteristics of luggage through the target detection network in the embodiment of the present application is explained and illustrated in combination with optional embodiments below, which can be applicable to but not limited to the embodiment of the present application.

[0056] Figure 5 is an optional flow chart of detecting features of luggage through a target detection network according to an embodiment of the present application, such as Figure 5 As shown, it may include but is not limited to the following:

[0057] STEP 1: Get the input image and obtain the feature value of [1, 3, 640, 640];

[0058] STEP2: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 64, 320, 320];

[0059] STEP3: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 128, 160, 160];

[0060] STEP4: Input the above eigenvalues ​​into the C2f network and obtain the eigenvalues ​​of [1, 128, 160, 160];

[0061] STEP5: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 256, 80, 80];

[0062] STEP6: Input the above eigenvalues ​​into the C2f network and obtain the eigenvalues ​​of [1, 256, 80, 80];

[0063] STEP7: Input the above eigenvalues ​​into the Conv network and obtain the eigenvalues ​​of [1, 512, 40, 40];

[0064] STEP8: Input the above eigenvalues ​​into the C2f network and obtain the eigenvalues ​​of [1, 512, 40, 40];

[0065] STEP9: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 512, 20, 20];

[0066] STEP10: Input the above eigenvalues ​​into the C2f network and obtain the eigenvalues ​​of [1, 512, 20, 20];

[0067] STEP11: Input the above eigenvalues ​​into the SPPF network for feature fusion to obtain the eigenvalues ​​of [1, 512, 20, 20]; STEP11.1: Process the input features through the Conv network, and copy the results into two parts, one of which is processed by Maxpool2d and merged with the other part;

[0068] STEP11.2: After the merged result, the output feature value of the SPPF network is obtained through the Conv network;

[0069] STEP12: Upsample the above eigenvalues ​​to obtain eigenvalues ​​of [1, 512, 40, 40];

[0070] STEP13: Combine the above eigenvalues ​​with the eigenvalues ​​obtained in STEP8 to obtain the eigenvalues ​​of [1, 1024, 40, 40];

[0071] STEP14: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 512, 40, 40];

[0072] STEP15: Upsample the above eigenvalues ​​to obtain eigenvalues ​​of [1, 512, 80, 80];

[0073] STEP16: Combine the above eigenvalues ​​with the eigenvalues ​​obtained in STEP6 to obtain the eigenvalues ​​of [1, 768, 80, 80];

[0074] STEP17: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 256, 80, 80];

[0075] STEP18: Input the SFP network feature value into the Conv network to obtain the feature value of [1, 256, 20, 20];

[0076] STEP 19: Perform interpolation and dimension matching on the above features to make the dimension become [1, 256, 80, 80];

[0077] STEP 19.1: Diffusion of the eigenvalues ​​of size [20,20] is performed 4 times based on the center point, so that the eigenvalues ​​of [20,20] become sparse eigenvalues ​​of [80,80];

[0078] STEP19.2: Perform linear interpolation on the blank features in the sparse eigenvalues ​​of [80,80] so that all eigenvalue positions have data;

[0079] STEP20: Add the above eigenvalues ​​and the eigenvalues ​​of STEP17 bit by bit to obtain a new eigenvalue with a dimension of [1, 256, 80, 80], and input its features into Detect1 to obtain detection head 1;

[0080] STEP21: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 256, 40, 40];

[0081] STEP22: Combine the above eigenvalues ​​with the eigenvalues ​​of STEP14 to obtain the eigenvalues ​​of [1, 768, 40, 40]. STEP23: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 512, 40, 40]. Input the features into Detect2 to obtain the detection head 2.

[0082] STEP24: Input the above eigenvalues ​​into the Conv network to obtain the eigenvalues ​​of [1, 512, 20, 20];

[0083] STEP25: Combine the above eigenvalues ​​with the eigenvalues ​​of SPPF to obtain the eigenvalues ​​of [1, 1024, 20, 20];

[0084] STEP26: Input the above feature values ​​into the Conv network to obtain the feature values ​​of [1, 512, 20, 20]; and input its features into Detect3 to obtain detection head 3.

[0085] It should be noted that, in the eigenvalues ​​in the above steps, the first digit indicates the number of images, the second digit indicates the number of channels of the image, and the third and fourth digits indicate the pixel size of the image. For example, taking the eigenvalue [1, 512, 20, 20] as an example, it means that the number of channels in the first image is 512 and the pixel size is 20*20.

[0086] Through the embodiments of the present application, the features of the SPPF are further fused with the features of the first detection head, thereby increasing the feature fusion capability of the FPN (Feature Pyramid Networks) structure. The fused features can better represent the features of the luggage itself and the surrounding environment, thereby improving the detection capability of the model and the accuracy of the model (or network) in detecting whether the luggage in different video frames is the same.

[0087] In the technical solution provided in the above step S208, when there are P pieces of luggage in the target video frame and Q pieces of luggage in the reference video frame, the matching degree between each piece of luggage in the P pieces of luggage and each piece of luggage in the Q pieces of luggage may be detected but is not limited to the detection.

[0088] In an exemplary embodiment, the target matching degree between the target luggage and the reference luggage may be detected according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature in the following manner, but is not limited to: detecting a first matching degree between the target attribute feature and the reference attribute feature, and detecting a second matching degree between the target environment feature and the reference environment feature; and detecting the target matching degree according to the first matching degree and the second matching degree.

[0089] Optionally, in this embodiment, it is also possible but not limited to first performing an operation of detecting a first degree of match between the target attribute feature and the reference attribute feature, and then performing an operation of detecting a second degree of match between the target environment feature and the reference environment feature, or first performing an operation of detecting a second degree of match between the target environment feature and the reference environment feature, and then performing an operation of detecting a first degree of match between the target attribute feature and the reference attribute feature.

[0090] In an exemplary embodiment, the first matching degree between the target attribute feature and the reference attribute feature may be detected in the following manner, but is not limited to: when the target attribute feature includes a target angle corresponding to the target luggage, a target aspect ratio of the target luggage, and a target pixel value corresponding to the target luggage, and the reference attribute feature includes a reference angle corresponding to the reference luggage, a reference aspect ratio of the reference luggage, and a reference pixel value corresponding to the reference luggage, the first matching degree between the target attribute feature and the reference attribute feature is detected by performing the following steps: detecting a first angle difference between the target angle and the reference angle, detecting a first aspect ratio difference between the target aspect ratio and the reference aspect ratio, and detecting a pixel difference average between the target pixel value and the reference pixel value; and when the first angle difference is less than a first angle difference threshold, the first aspect ratio difference is less than a first aspect ratio difference threshold, and the pixel difference average is less than a pixel difference average threshold, determining the first matching degree to be greater than or equal to a first matching degree threshold.

[0091] Optionally, in this embodiment, the angle corresponding to the luggage may include, but is not limited to, the angle of the luggage relative to a conveyor belt or a rotating track.

[0092] Optionally, in this embodiment, when the first angle difference is greater than or equal to a first angle difference threshold, and / or the first aspect ratio difference is greater than or equal to a first aspect ratio difference threshold, and / or the first pixel difference mean is greater than or equal to a first pixel difference mean threshold, the first matching degree is determined to be less than a first matching degree threshold.

[0093] In this way, as long as any one of the angle difference, aspect ratio difference and pixel difference mean does not meet the requirements, it can be detected that the luggage in the two video frames is different, thereby increasing the parameter range considered when detecting whether the two luggage are the same, and improving the accuracy of detecting whether the same luggage exists in the two video frames.

[0094] In an exemplary embodiment, the average pixel difference between the target pixel value and the reference pixel value may be detected in the following manner, but is not limited to: adjusting the bounding box of the target luggage to a target size to obtain a target bounding box, and adjusting the bounding box of the reference luggage to the target size to obtain a reference bounding box, wherein both the target bounding box and the reference bounding box include m*n pixel points; dividing the target bounding box into N pixels according to the diagonal of the target bounding box; 一A bounding box is defined, and the reference bounding box is divided into N2 bounding boxes according to the diagonal of the reference bounding box, wherein N1 is equal to N2 and N1 is a positive integer; pixel differences between pixel points in the N1 bounding boxes and corresponding pixel points in the N2 bounding boxes are calculated to obtain N groups of pixel differences, wherein the i-th group of pixel differences in the N groups of pixel differences includes the N 一 The i-th bounding box 一 a pixel difference between a pixel point in a bounding box and a corresponding pixel point in an i2-th bounding box of the N2 bounding boxes, N is equal to N1 and N2, i1 is equal to i2, i1 is a positive integer less than or equal to N1, and i2 is a positive integer less than or equal to N2; a value obtained by dividing a target product value by N is determined as the pixel difference mean, wherein the target product value is equal to a value obtained by summing the N product values, a j-th product value among the N product values ​​is equal to a value obtained by multiplying a j-th group of pixel differences among the N groups of pixel differences by a j-th weight among the N weights, wherein j is a positive integer less than or equal to N, a weight of a bounding box located in a target area among the N weights is greater than or equal to each weight among the N-1 weights, and the target area is an area in the target bounding box where a probability of including the target baggage is greater than or equal to a first probability threshold, or an area in the reference bounding box where a probability of including the reference baggage is greater than or equal to a second probability threshold.

[0095] Optionally, in this embodiment, the target bounding box and the reference bounding box may be, but are not limited to, both of target sizes. For example, the height of the target bounding box is the same as the height of the reference bounding box, and the width of the target bounding box is the same as the width of the reference bounding box.

[0096] Optionally, in this embodiment, the bounding box of the target luggage may be detected by a target detection network, but is not limited to the detection of the bounding box of the reference luggage by a target detection network.

[0097] Optionally, in this embodiment, the sum of the N weights may be, but is not limited to, 1. By setting the weight of a bounding box whose probability of including luggage is greater than or equal to a probability threshold among the multiple bounding boxes obtained by dividing the bounding box to be greater than the weights of the bounding boxes obtained by other divisions, the proportion of the bounding boxes that are relatively core and can reflect the luggage in the video frame when calculating the mean pixel difference value is increased. In this way, the accuracy of the calculated mean pixel difference value is improved.

[0098] In an exemplary embodiment, the second matching degree between the target environment feature and the reference environment feature may be detected in the following manner, but is not limited to: the target environment feature includes a first number of first candidate baggage around the target baggage in the target environment, a first aspect ratio of the first candidate baggage, a first angle corresponding to the first candidate baggage, and a first distance between the target baggage and the first candidate baggage; the reference environment feature includes a second number of second candidate baggage around the reference baggage in the reference environment, a second aspect ratio of the second candidate baggage, a second angle corresponding to the second candidate baggage, and a second distance between the reference baggage and the second candidate baggage. In this case, the second matching degree between the target environment feature and the reference environment feature is detected by performing the following steps: detecting the quantity difference between the first quantity and the second quantity, detecting the second aspect ratio difference between the first aspect ratio and the second aspect ratio, detecting the second angle difference between the first angle and the second angle, and detecting the distance difference between the first distance and the second distance; when the quantity difference is less than the quantity threshold, the second aspect ratio difference is less than the second aspect ratio difference threshold, the second angle difference is less than the second angle difference threshold, and the distance difference is less than the distance difference threshold, the second matching degree is determined to be greater than or equal to the second matching degree threshold.

[0099] Optionally, in this embodiment, when the quantity difference is greater than or equal to the quantity threshold, and / or the second aspect ratio difference is greater than or equal to the second aspect ratio difference threshold, and / or the second angle difference is greater than or equal to the second angle difference threshold, and / or the distance difference is greater than or equal to the distance difference threshold, the second matching degree is determined to be less than the second matching degree threshold.

[0100] Optionally, in this embodiment, the first distance and the second distance may include, but are not limited to, cosine distance, Euclidean distance, etc., and the present application does not impose any limitation on this.

[0101] In an exemplary embodiment, the target matching degree may be detected according to the first matching degree and the second matching degree in the following manner, but not limited to: detecting whether the first matching degree is greater than or equal to a first matching degree threshold; in the case of detecting that the first matching degree is greater than or equal to the first matching degree threshold, detecting whether the second matching degree is greater than or equal to a second matching degree threshold; in the case of detecting that the second matching degree is greater than or equal to the second matching degree threshold, determining the target matching degree to be greater than or equal to the target matching degree threshold, wherein, in the case that the target matching degree is greater than or equal to the target matching degree threshold, the target luggage is the same as the reference luggage.

[0102] Optionally, in this embodiment, the target matching degree can also be detected based on the first matching degree and the second matching degree in the following manner, but not limited to: first detect whether the second matching degree is greater than or equal to a second matching degree threshold; if the second matching degree is greater than or equal to the second matching degree threshold, continue to detect whether the first matching degree is greater than or equal to the first matching degree threshold; if the first matching degree is greater than or equal to the first matching degree threshold, determine that the target matching degree is greater than or equal to the target matching degree threshold; if the second matching degree is less than the second matching degree threshold, there is no need to detect the second matching degree between the target environment feature and the reference environment feature.

[0103] Optionally, in this embodiment, when the first matching degree is less than the first matching degree threshold, and / or the second matching degree is less than the second matching degree threshold, the target matching degree is determined to be less than the target matching degree threshold, wherein when the target matching degree is less than or equal to the target matching degree threshold, the target luggage is different from the reference luggage. For example, when the reference luggage is different from the target luggage, the target luggage may be newly added luggage.

[0104] Through the embodiments of the present application, it is possible to detect whether the luggage in two video frames is the same luggage based on the attribute features and environmental features of the luggage present in the two video frames extracted, thereby achieving accurate identification of luggage in different video frames and improving the accuracy of identifying luggage present in video frames.

[0105] In the technical solution provided in the above step S210, the target description information and the reference description information of the reference baggage may be the same or different but not limited to the same. The reference description information is used to describe the position of the reference baggage in the airport. For example, when the reference baggage is the same as the target baggage and the target baggage is not stranded or derailed, the target description information is different from the reference description information of the reference baggage. When the reference baggage is the same as the target baggage and the target baggage is stranded or derailed, the target description information is the same as the reference description information of the reference baggage.

[0106] In an exemplary embodiment, the target description information of the target baggage may be generated according to the target matching degree in the following manner, but not limited to: when the target matching degree is greater than or equal to a target matching degree threshold, extracting a reference baggage identifier of the reference baggage, extracting a target position of the target baggage in the airport output by a target detection network, and generating the target description information carrying the reference baggage identifier and the target position; when the target matching degree is less than a target matching degree threshold, generating a target baggage identifier of the target baggage, extracting a target position of the target baggage in the airport output by the target detection network, and generating the target description information carrying the target baggage identifier and the target position, wherein the target detection network is further used to detect the position of the baggage in the airport in an input video frame; after generating the target description information of the target baggage according to the target matching degree: detecting whether a change in the target position of the target baggage in the airport meets a target change condition, and obtaining a detection result; and when the detection result indicates that the change does not meet the target change condition, deleting the target baggage in the target video frame to obtain a candidate video frame.

[0107] Optionally, in this embodiment, the luggage identification of the luggage may include, but is not limited to, the luggage ID (Identity document, identity card identification number), or the luggage tag, etc.

[0108] Optionally, in this embodiment, the target detection network can also be obtained by, but is not limited to, training an initial detection network using sample video data, wherein the sample video data includes corresponding sample video frames, sample attribute features, sample environmental features, and sample positions.

[0109] Optionally, in this embodiment, the target location may include, but is not limited to, a rotating track for conveying luggage, a conveyor belt for conveying luggage, a luggage separation port or a confluence port, etc. in an airport.

[0110] Optionally, in this embodiment, the position can be, but is not limited to, used to indicate whether the position of the target luggage in the airport has changed and whether the position of the target luggage in the airport is within the allowed position range. When the detection result is used to indicate that the change does not meet the target change condition, it can indicate that the target position of the target luggage in the airport has not changed within the target time period, for example, the target luggage is stranded on the conveyor belt, or the target position of the target luggage in the airport has deviated, for example, the target luggage has fallen off the conveyor belt.

[0111] Optionally, in this embodiment, when the detection result is used to indicate that the change situation satisfies the target change condition, it can be indicated that the target position of the target luggage in the airport has changed within the target time length, for example, the target luggage is conveyed normally along the conveyor belt, or the target position of the target luggage in the airport has not deviated, for example, the target luggage has remained on the conveyor belt without derailment.

[0112] Through the embodiments of the present application, the stranded luggage or derailed luggage in the video frame is deleted, and the video frame from which the stranded luggage or derailed luggage is deleted is compared with other video frames, thereby avoiding the negative impact of these abnormal luggage on the luggage detection in other video frames, and improving the detection efficiency of the luggage in the video frame.

[0113] In order to better understand the detection process of the target detection network in the embodiment of the present application, the process of detecting the attribute characteristics and environmental characteristics of luggage through the target detection network in the embodiment of the present application is explained and illustrated in combination with optional embodiments below, which can be applicable to but not limited to the embodiment of the present application.

[0114] Figure 6 is a schematic diagram of an optional airport baggage detection method according to an embodiment of the present application, such as Figure 6 As shown, it may include but is not limited to the following:

[0115] STEP 1: Access the video stream and extract the image from the video stream.

[0116] STEP 2: Preprocess the image.

[0117] STEP 2.1: Preprocess the image and scale it to a specific size.

[0118] Optionally, the specific size is set to 640*640. During the image resizing process, the image is resized without changing its aspect ratio. The largest side is resized to 640, and the smaller side is filled with grayscale.

[0119] STEP 3: Input the processed image into the baggage object detection network (equivalent to the object detection network).

[0120] STEP4: Through the baggage target detection network, determine whether there is a baggage target (for example, luggage) in the camera area. If not, return to execute STEP1, otherwise continue to execute.

[0121] STEP 5: Get the coordinates of the luggage target frame, the angle of the luggage relative to the camera coordinate system, and the confidence level.

[0122] Optionally, the angle of the luggage relative to the camera coordinate system may include, but is not limited to, the angle of the luggage relative to a conveyor belt or a rotating track in the camera coordinate system.

[0123] STEP6: Determine whether the confidence of the target box is greater than the set threshold. If it is greater than the set threshold (equivalent to the confidence threshold), continue to execute, otherwise return to execute SPEP1.

[0124] STEP 7: Determine the angle direction of the detected luggage, convert the angle into the world coordinate system and remap it.

[0125] Optionally, to determine the angle direction of the suitcase relative to the camera coordinate system conveyor belt or conveyor turntable, the detected target frame may be rotated according to the detected angle, and two points on the upper left and lower right of the target detection frame are selected. Suppose the two points are KP_1 and KP_2, where the coordinates of KP_1 are (x kp_1 ,y kp_1 ), the coordinates of KP_2 are (x kp_2 ,y kp_2 ).

[0126] Assuming that the rotation matrix R represents the rotation from the camera coordinate system to the world coordinate system, and the rotation angle is α (the angle between the camera coordinate system and the world coordinate system), the rotation matrix can be expressed by, but is not limited to, formula (1):

[0127]

[0128] Wherein, R is the rotation matrix, α is the rotation angle, and α represents the angle between the camera coordinate system and the world coordinate system. It should be noted that the angles between different camera devices and the world coordinate system may be, but are not limited to, corresponding, for example, the same or different. The angle between the same camera device and the world coordinate system may be, but are not limited to, the same and fixed.

[0129] The point KP_1(x kp_1 ,y kp_1 ) and KP_2(x kp_2 ,y kp_2 ) is mapped to the world coordinate system:

[0130]

[0131] Among them, KP_1 w is the point KP_1 in the camera coordinate system corresponding to the point in the world coordinate system, KP_2 w is the point KP_2 in the camera coordinate system corresponding to the point in the world coordinate system, and α is the rotation angle.

[0132] The vectors of two points in the world coordinate system can be calculated by, but are not limited to, the following formula (4):

[0133]

[0134] Among them, is the point corresponding to the point KP1 in the camera coordinate system in the world coordinate system, is the point corresponding to the point KP2 in the camera coordinate system in the world coordinate system, and is the rotation angle.

[0135] make:

[0136] in,

[0137] Therefore, the vertical angle relative to the world coordinate system can be calculated by, but is not limited to, the following formula (5):

[0138]

[0139] Among them, θ w It is the vertical angle relative to the world coordinate system.

[0140] STEP8: Determine the luggage angle of the current target frame (equivalent to the target video frame) (equivalent to the angle corresponding to the luggage, i.e. θ w ) and the angle difference between the luggage angle of the previous target frame (equivalent to the reference video frame) is less than the set threshold (equivalent to the angle difference threshold), if not, return to STEP1, otherwise continue to the next step (it should be noted that if the deviation angles of the two luggage are similar, they may be the same luggage).

[0141] STEP9: Pick up the baggage boundary,

[0142] STEP 9.1: Eliminate the noise in the image. Use Gaussian filtering to eliminate the noise in the image. The noise in the image can be eliminated by, but is not limited to, formula (6):

[0143]

[0144] Where σ is the standard deviation of the Gaussian kernel, and x and y are the coordinates relative to the center of the kernel. By convolving this kernel with the image, a smoothed image can be obtained. G(x, y) is a two-dimensional Gaussian distribution.

[0145] STEP9.2: Calculate the gradient magnitude and gradient direction of the image.

[0146] The horizontal gradient can be calculated by, but is not limited to, formula (7):

[0147]

[0148] Among them, * represents the convolution operation, and I is the image processed by Gaussian. G x represents the horizontal gradient.

[0149] The vertical gradient can be calculated, but is not limited to, by formula (8):

[0150]

[0151] Among them, * represents the convolution operation, and I is the image processed by Gaussian, G y represents the vertical gradient.

[0152] The gradient magnitude G can be calculated, but is not limited to, by the following formula (9), and the gradient direction θ can be calculated by formula (10):

[0153]

[0154] Among them, G y represents the vertical gradient, and G x represents the horizontal gradient.

[0155] STEP9.3: Refine the edges, only retain the possible edge pixels, and suppress the non-maximum values; for each pixel point, check the gradient magnitudes of its neighboring pixels along the gradient direction. If the gradient magnitude of the current pixel is not the largest, set it to 0.

[0156] STEP9.4: Determine the strong edge and non-edge regions;

[0157] Among them, set the threshold T of the gradient magnitude G. When G >= T, mark it as the edge region, and when G < T, mark it as the non-edge region.

[0158] STEP9.5: Connect the edge regions to obtain the boundary of the suitcase;

[0159] STEP10: Determine whether the difference between the aspect ratio of the luggage in the current target frame and the aspect ratio of the luggage in the previous target frame is less than the set threshold. If not, return to STEP1; otherwise, continue to execute the next step (it should be noted that if the aspect ratios of two pieces of luggage are similar, they may be the same piece of luggage).

[0160] STEP11: Perform feature matching on the luggage to obtain two luggage matching values,

[0161] Among them, for feature matching, according to the boundaries of the two pieces of luggage, extract the pictures of the luggage for feature comparison and calculate the matching value. The specific calculation method is as follows,

[0162] STEP11.1: Scale the bounding boxes of the two pieces of luggage to the same size.

[0163] STEP11.2: Take the center point of the bounding box as the center and the line connecting the center point and any end point of the bounding box as the baseline, and divide it into three equal segments. In this way, three target box rectangles centered on the center point of the bounding box can be obtained.

[0164] STEP 11.3: Calculate the average value of the pixel difference in the three target box rectangles in the two images respectively. The calculation formula for the average value of the pixel difference of each target box is shown in (11):

[0165]

[0166] Wherein, Vx represents the mean difference between the pixels of each bounding box in the bounding box divided by the bounding box of the first image and the pixels of each bounding box in the bounding box divided by the bounding box of the second image, m and n represent the number of pixels in the width and height directions, Represents the pixel value of the first image at the pixel point (i, j), represents the pixel value of the second image at the pixel point (i, j). abs represents the absolute value. It should be noted that m and n can be but are not limited to being equal.

[0167] STEP 11.4: The average of the pixel differences corresponding to the three rectangular boxes may be calculated by, but is not limited to, the following formula (12), and then the average of the three is calculated to obtain the final matching value.

[0168] V mean =(0.3V1+0.4V2+0.3V3) / 3 (12)

[0169] Among them, V mean represents the matching value (equivalent to the mean pixel difference), V1 represents the mean difference between the pixels of the first bounding box in the bounding box divided by the bounding box of the first image and the pixels of the first bounding box in the bounding box divided by the bounding box of the second image; V2 represents the mean difference between the pixels of the second bounding box in the bounding box divided by the bounding box of the first image and the pixels of the second bounding box in the bounding box divided by the bounding box of the second image; V3 represents the mean difference between the pixels of the third bounding box in the bounding box divided by the bounding box of the first image and the pixels of the third bounding box in the bounding box divided by the bounding box of the second image.

[0170] STEP12: Determine the matching value V mean Is it greater than the set threshold? If so, return to STEP1. If not, proceed to the next step. It should be noted that the matching value V mean If the value (equivalent to the mean pixel difference) is greater than the set threshold, the luggage may be different.

[0171] STEP 13: Determine the distance and posture of other luggage around the luggage in the first picture and the distance and posture of other luggage around the luggage in the second picture.

[0172] STEP13.1: Calculate the number of other bags around the bag in the first image.

[0173] STEP 13.2: Calculate the spatial distance between each bag and the other bag to ensure that the relative positions remain unchanged.

[0174] STEP13.3: Calculate the posture of each surrounding baggage, such as aspect ratio and rotation angle.

[0175] STEP 13.4: Calculate the situation of other luggage around the luggage in the second figure one by one, and compare it with the situation in the first figure.

[0176] STEP 14: If the number of surrounding baggage is the same, and the distance and posture of the surrounding baggage meet the requirements, then the two baggage in the first and second images are determined to be the same baggage, otherwise proceed to the next step.

[0177] STEP 15: Calculate whether the distance from the baggage to the edge of the field of view is greater than the distance from other bags to prevent other bags from being out of the field of view due to different field of view directions.

[0178] STEP 16: Delete the comparison of other luggage that is out of view.

[0179] STEP 17: Determine whether there is a baggage separation port or a baggage collection port between two image frames.

[0180] Optionally, if there is a baggage separation or collection entrance, the video frames before and after the baggage separation or collection entrance are retained respectively for comparison with other video frames.

[0181] In addition, under the same camera, it is also necessary to monitor whether the luggage has fallen off to the outside of the conveyor belt or turntable. If the luggage is found to be stationary and the conveyor belt or turntable is moving again, it is determined that the luggage has fallen off or is attached to the current position, and the status of the luggage is updated, and the influence of the luggage on the matching of other luggage is eliminated.

[0182] STEP18: Delete the newly added or separated baggage.

[0183] STEP 19: Return to STEP 13 and recalculate the distance and posture of the surrounding luggage in the two images.

[0184] STEP 20: The procedure ends.

[0185] Through the embodiments of the present application, in the airport baggage tracking detection, the posture of the baggage and the direction relative to the world coordinate system are first determined. If the direction deviation relative to the world coordinate system exceeds the threshold, it can be determined as different baggage, thereby improving the detection efficiency of the target. The present application adopts methods such as boundary extraction and feature matching to gradually perform feature determination between two pieces of luggage to prevent errors in the determination of similar luggage. In addition, the present application further incorporates the surrounding features of the luggage, such as other surrounding luggage features, and integrates the surrounding environment into the feature comparison calculation without increasing the amount of calculation, thereby improving the tracking ability of luggage under different cameras. In addition, the present application fully considers the impact of baggage increase or decrease areas such as baggage separation ports and baggage confluence ports on feature matching under different cameras, and considers the processing method when the field of view is limited (when there is luggage not in the field of view), thereby improving the matching rate of luggage.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] In this embodiment, a device for detecting baggage at an airport is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are omitted. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0188] Figure 7 is a structural block diagram of an airport baggage detection device according to an embodiment of the present application, which is applied to a detection device, wherein the detection device is used to connect to an airport camera device, and the camera device is used to collect video data of the baggage in the airport, such as Figure 7 As shown, the device comprises:

[0189] A first extraction module 702 is used to extract a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data before the target video frame;

[0190] A first detection module 704, configured to detect whether there is luggage in the target video frame, and to detect whether there is luggage in the reference video frame;

[0191] The second extraction module 706 is configured to extract target attribute features and target environment features of the target baggage from the target video frame when the target baggage is detected to exist in the target video frame, wherein the target environment features are used to indicate the target environment in which the target baggage is located in the target video frame; and to extract reference attribute features and reference environment features of the reference baggage from the reference video frame when the reference baggage is detected to exist in the reference video frame, wherein the reference environment features are used to indicate the reference environment in which the reference baggage is located in the reference video frame;

[0192] A second detection module 708, configured to detect a target matching degree between the target baggage and the reference baggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature;

[0193] The generating module 710 is used to generate target description information of the target luggage according to the target matching degree, wherein the target description information is used to describe the location of the target luggage in the airport.

[0194] Through the above-mentioned device, the camera equipment can collect video data of the entire process of luggage in the airport. The luggage may be located in different environments in the airport. The detection equipment can detect the attribute characteristics and environmental characteristics of the luggage in the two video frames extracted from the video data collected by the camera equipment. It can be but not limited to detecting the matching degree between the attribute characteristics and environmental characteristics of the luggage in the two video frames to achieve the real-time position tracking of the luggage in the airport. Therefore, the problem of low accuracy of airport luggage detection can be solved, and the effect of improving the accuracy of airport luggage detection can be achieved.

[0195] In an exemplary embodiment, a target detection network is deployed in the detection device, and the first extraction module includes:

[0196] An input unit, used for inputting the target video frame into the target detection network to obtain the target attribute features and the target environment features output by the target detection network;

[0197] An extraction unit, used to extract reference attribute features of the reference baggage and reference environment features of the reference baggage from the reference video frame, comprising: inputting the reference video frame into the target detection network to obtain the reference attribute features and the reference environment features output by the target detection network;

[0198] The target detection network is obtained by training an initial detection network using sample video data, wherein the sample video data includes sample video frames, sample attribute features and sample environment features having corresponding relationships.

[0199] In an exemplary embodiment, the second detection module includes:

[0200] A first detection unit, configured to detect a first matching degree between the target attribute feature and the reference attribute feature, and to detect a second matching degree between the target environment feature and the reference environment feature;

[0201] The second detection unit is configured to detect the target matching degree according to the first matching degree and the second matching degree.

[0202] In an exemplary embodiment, the first detection unit is used to:

[0203] In the case where the target attribute feature includes a target angle corresponding to the target luggage, a target aspect ratio of the target luggage, and a target pixel value corresponding to the target luggage, and the reference attribute feature includes a reference angle corresponding to the reference luggage, a reference aspect ratio of the reference luggage, and a reference pixel value corresponding to the reference luggage, the first matching degree between the target attribute feature and the reference attribute feature is detected by performing the following steps:

[0204] Detecting a first angle difference between the target angle and the reference angle, detecting a first aspect ratio difference between the target aspect ratio and the reference aspect ratio, and detecting a mean pixel difference between the target pixel value and the reference pixel value;

[0205] When the first angle difference is less than a first angle difference threshold, the first aspect ratio difference is less than a first aspect ratio difference threshold, and the pixel difference mean is less than a pixel difference mean threshold, the first matching degree is determined to be greater than or equal to a first matching degree threshold.

[0206] In an exemplary embodiment, the first detection unit is further configured to:

[0207] The target baggage bounding box is adjusted to a target size to obtain a target bounding box, and the reference baggage bounding box is adjusted to the target size to obtain a reference bounding box, wherein the target bounding box and the reference bounding box each include m*n pixels;

[0208] According to the diagonal of the target bounding box, the target bounding box is divided into N 一bounding boxes, and divide the reference bounding box into N2 bounding boxes according to the diagonal of the reference bounding box, where N1 is equal to N2 and N1 is a positive integer;

[0209] Calculate the pixel differences between the pixel points in the N1 bounding boxes and the corresponding pixel points in the N2 bounding boxes to obtain N groups of pixel differences, wherein the i-th group of pixel differences in the N groups of pixel differences includes the N 一 The pixel difference between the pixel point in the i1th bounding box of the N2 bounding boxes and the corresponding pixel point in the i2th bounding box of the N2 bounding boxes, N equals N1 equals N2, i1 equals i2, i1 is a positive integer less than or equal to N1, and i2 is a positive integer less than or equal to N2;

[0210] A value obtained by dividing the target product value by N is determined as the pixel difference mean, wherein the target product value is equal to a value obtained by summing the N product values, the j-th product value among the N product values ​​is equal to a value obtained by multiplying the j-th group of pixel difference values ​​among the N groups of pixel difference values ​​by the j-th weight among the N weights, wherein j is a positive integer less than or equal to N, the weight of a bounding box located in a target area among the N weights is greater than or equal to each weight among the N-1 weights, and the target area is an area in the target bounding box where a probability of including the target baggage is greater than or equal to a first probability threshold, or an area in the reference bounding box where a probability of including the reference baggage is greater than or equal to a second probability threshold.

[0211] In an exemplary embodiment, the first detection unit is further configured to:

[0212] In a case where the target environment feature includes a first number of first candidate baggage around the target baggage in the target environment, a first aspect ratio of the first candidate baggage, a first angle corresponding to the first candidate baggage, and a first distance between the target baggage and the first candidate baggage, and the reference environment feature includes a second number of second candidate baggage around the reference baggage in the reference environment, a second aspect ratio of the second candidate baggage, a second angle corresponding to the second candidate baggage, and a second distance between the reference baggage and the second candidate baggage, a second matching degree between the target environment feature and the reference environment feature is detected by executing the following steps:

[0213] Detecting a quantity difference between the first quantity and the second quantity, detecting a second aspect ratio difference between the first aspect ratio and the second aspect ratio, detecting a second angle difference between the first angle and the second angle, and detecting a distance difference between the first distance and the second distance;

[0214] When the quantity difference is less than the quantity threshold, the second aspect ratio difference is less than the second aspect ratio difference threshold, the second angle difference is less than the second angle difference threshold, and the distance difference is less than the distance difference threshold, the second matching degree is determined to be greater than or equal to the second matching degree threshold.

[0215] In an exemplary embodiment, the second detection unit is used to:

[0216] Detecting whether the first matching degree is greater than or equal to a first matching degree threshold;

[0217] In the case of detecting that the first matching degree is greater than or equal to the first matching degree threshold, detecting whether the second matching degree is greater than or equal to a second matching degree threshold;

[0218] When it is detected that the second matching degree is greater than or equal to the second matching degree threshold, the target matching degree is determined to be greater than or equal to the target matching degree threshold, wherein when the target matching degree is greater than or equal to the target matching degree threshold, the target luggage is the same as the reference luggage.

[0219] In an exemplary embodiment, the generating module includes:

[0220] a processing unit, configured to extract, when the target matching degree is greater than or equal to a target matching degree threshold, a reference baggage identifier of the reference baggage, extract a target position of the target baggage in the airport output by a target detection network, and generate the target description information carrying the reference baggage identifier and the target position; and, when the target matching degree is less than a target matching degree threshold, generate a target baggage identifier of the target baggage, extract the target position of the target baggage in the airport output by the target detection network, and generate the target description information carrying the target baggage identifier and the target position, wherein the target detection network is further configured to detect the position of the baggage in the airport in an input video frame;

[0221] The device further includes: a processing module, which is used to detect whether a change in the target position of the target luggage in the airport satisfies a target change condition after generating the target description information of the target luggage according to the target matching degree, and obtain a detection result; if the detection result indicates that the change does not satisfy the target change condition, delete the target luggage in the target video frame to obtain a candidate video frame.

[0222] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0223] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0224] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0225] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0226] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0227] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0228] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0229] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0230] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0231] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0232] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting luggage at an airport, characterized in that: Applied to a detection device, the detection device is used to connect to a camera device at an airport, the camera device is used to collect video data of luggage in the airport, the method comprises: Extracting a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data before the target video frame; detecting whether there is luggage in the target video frame, and detecting whether there is luggage in the reference video frame; extracting target attribute features and target environment features of the target luggage from the target video frame when the target luggage is detected to be present in the target video frame, wherein the target environment features are used to indicate the target environment in which the target luggage is located in the target video frame; and extracting reference attribute features and reference environment features of the reference luggage from the reference video frame when the reference luggage is detected to be present in the reference video frame, wherein the reference environment features are used to indicate the reference environment in which the reference luggage is located in the reference video frame; Detecting a target matching degree between the target luggage and the reference luggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature; According to the target matching degree, target description information of the target luggage is generated, wherein the target description information is used to describe the location of the target luggage in the airport.

2. The method according to claim 1, characterized in that A target detection network is deployed in the detection device. The step of extracting the target attribute features of the target luggage and the target environment features of the target luggage from the target video frame comprises: inputting the target video frame into the target detection network to obtain the target attribute features and the target environment features output by the target detection network; The extracting the reference attribute features of the reference luggage and the reference environment features of the reference luggage from the reference video frame comprises: inputting the reference video frame into the target detection network to obtain the reference attribute features and the reference environment features output by the target detection network; The target detection network is obtained by training an initial detection network using sample video data, wherein the sample video data includes sample video frames, sample attribute features and sample environment features having corresponding relationships.

3. The method according to claim 1, characterized in that The detecting the target matching degree between the target luggage and the reference luggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature comprises: Detecting a first matching degree between the target attribute feature and the reference attribute feature, and detecting a second matching degree between the target environment feature and the reference environment feature; The target matching degree is detected according to the first matching degree and the second matching degree.

4. The method according to claim 3, characterized in that The detecting a first matching degree between the target attribute feature and the reference attribute feature includes: In the case where the target attribute feature includes a target angle corresponding to the target luggage, a target aspect ratio of the target luggage, and a target pixel value corresponding to the target luggage, and the reference attribute feature includes a reference angle corresponding to the reference luggage, a reference aspect ratio of the reference luggage, and a reference pixel value corresponding to the reference luggage, the first matching degree between the target attribute feature and the reference attribute feature is detected by performing the following steps: Detecting a first angle difference between the target angle and the reference angle, detecting a first aspect ratio difference between the target aspect ratio and the reference aspect ratio, and detecting a mean pixel difference between the target pixel value and the reference pixel value; When the first angle difference is less than a first angle difference threshold, the first aspect ratio difference is less than a first aspect ratio difference threshold, and the pixel difference mean is less than a pixel difference mean threshold, the first matching degree is determined to be greater than or equal to a first matching degree threshold.

5. The method according to claim 4, characterized in that The detecting the mean pixel difference between the target pixel value and the reference pixel value comprises: adjusting the bounding box of the target luggage to a target size to obtain a target bounding box, and adjusting the bounding box of the reference luggage to the target size to obtain a reference bounding box, wherein the target bounding box and the reference bounding box each include m*n pixel points; Divide the target bounding box into N1 bounding boxes according to the diagonal of the target bounding box, and divide the reference bounding box into N2 bounding boxes according to the diagonal of the reference bounding box, where N1 is equal to N2 and N1 is a positive integer; Calculate pixel differences between pixel points in the N1 bounding boxes and corresponding pixel points in the N2 bounding boxes to obtain N groups of pixel differences, wherein the i-th group of pixel differences in the N groups of pixel differences includes the pixel differences between the pixel points in the i1-th bounding box of the N1 bounding boxes and the corresponding pixel points in the i2-th bounding box of the N2 bounding boxes, N equals N1 equals N2, i1 equals i2, i1 is a positive integer less than or equal to N1, and i2 is a positive integer less than or equal to N2; A value obtained by dividing the target product value by N is determined as the pixel difference mean, wherein the target product value is equal to a value obtained by summing the N product values, the j-th product value among the N product values ​​is equal to a value obtained by multiplying the j-th group of pixel difference values ​​among the N groups of pixel difference values ​​by the j-th weight among the N weights, wherein j is a positive integer less than or equal to N, the weight of a bounding box located in a target area among the N weights is greater than or equal to each weight among the N-1 weights, and the target area is an area in the target bounding box where a probability of including the target baggage is greater than or equal to a first probability threshold, or an area in the reference bounding box where a probability of including the reference baggage is greater than or equal to a second probability threshold.

6. The method according to claim 3, characterized in that The detecting a second matching degree between the target environment feature and the reference environment feature includes: In a case where the target environment feature includes a first number of first candidate baggage around the target baggage in the target environment, a first aspect ratio of the first candidate baggage, a first angle corresponding to the first candidate baggage, and a first distance between the target baggage and the first candidate baggage, and the reference environment feature includes a second number of second candidate baggage around the reference baggage in the reference environment, a second aspect ratio of the second candidate baggage, a second angle corresponding to the second candidate baggage, and a second distance between the reference baggage and the second candidate baggage, a second matching degree between the target environment feature and the reference environment feature is detected by executing the following steps: Detecting a quantity difference between the first quantity and the second quantity, detecting a second aspect ratio difference between the first aspect ratio and the second aspect ratio, detecting a second angle difference between the first angle and the second angle, and detecting a distance difference between the first distance and the second distance; When the quantity difference is less than the quantity threshold, the second aspect ratio difference is less than the second aspect ratio difference threshold, the second angle difference is less than the second angle difference threshold, and the distance difference is less than the distance difference threshold, the second matching degree is determined to be greater than or equal to the second matching degree threshold.

7. The method according to claim 3, characterized in that The detecting the target matching degree according to the first matching degree and the second matching degree includes: Detecting whether the first matching degree is greater than or equal to a first matching degree threshold; In the case of detecting that the first matching degree is greater than or equal to the first matching degree threshold, detecting whether the second matching degree is greater than or equal to a second matching degree threshold; When it is detected that the second matching degree is greater than or equal to the second matching degree threshold, the target matching degree is determined to be greater than or equal to the target matching degree threshold, wherein when the target matching degree is greater than or equal to the target matching degree threshold, the target luggage is the same as the reference luggage.

8. The method according to claim 1, characterized in that The step of generating target description information of the target luggage according to the target matching degree includes: When the target matching degree is greater than or equal to the target matching degree threshold, the reference baggage identifier of the reference baggage is extracted, and the target position of the target baggage in the airport output by the target detection network is extracted, and the target description information carrying the reference baggage identifier and the target position is generated; when the target matching degree is less than the target matching degree threshold, the target baggage identifier of the target baggage is generated, and the target position of the target baggage in the airport output by the target detection network is extracted, and the target description information carrying the target baggage identifier and the target position is generated, wherein the target detection network is further used to detect the position of the baggage in the airport in the input video frame; After generating the target description information of the target luggage according to the target matching degree, the method further includes: detecting whether a change in the target position of the target luggage in the airport satisfies a target change condition, obtaining a detection result, and if the detection result indicates that the change does not satisfy the target change condition, deleting the target luggage in the target video frame to obtain a candidate video frame.

9. An airport luggage detection device, characterized in that: Applied to a detection device, the detection device is used to connect to a camera device at an airport, the camera device is used to collect video data of luggage in the airport, and the device includes: A first extraction module is used to extract a target video frame and a reference video frame from the target video data collected by the connected target camera device, wherein the reference video frame is a previous video frame extracted from the target video data before the target video frame; A first detection module, configured to detect whether there is luggage in the target video frame, and to detect whether there is luggage in the reference video frame; a second extraction module, configured to extract target attribute features and target environment features of the target baggage from the target video frame when the target baggage is detected to exist in the target video frame, wherein the target environment features are used to indicate the target environment in which the target baggage is located in the target video frame; and to extract reference attribute features and reference environment features of the reference baggage from the reference video frame when the reference baggage is detected to exist in the reference video frame, wherein the reference environment features are used to indicate the reference environment in which the reference baggage is located in the reference video frame; A second detection module, configured to detect a target matching degree between the target luggage and the reference luggage according to the target attribute feature, the target environment feature, the reference attribute feature and the reference environment feature; A generating module is used to generate target description information of the target luggage according to the target matching degree, wherein the target description information is used to describe the location of the target luggage in the airport.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 8 are implemented.

Images