Auxiliary luggage identification method

Abstract

The invention relates to the technical field of logistics, in particular to a luggage auxiliary identification method. According to the method, an enhanced semantic segmentation model is used for carrying out image recognition on all luggage with potential recognition requirements, image features of the luggage are extracted, mapping between the image features of each luggage picture and corresponding luggage information is established, and the image features of all the luggage are classified and stored as a feature library. When the luggage to be identified is needed, acquiring an image of the luggage to be identified, extracting image features of the luggage to be identified by using an enhanced semantic segmentation model, comparing and searching image features which are the same as or similar to the image features of the luggage to be identified in a feature library by using a clustering mode identification model, and finally, identifying the luggage to be identified on the basis of the searched image features. And luggage information of the to-be-identified luggage image is obtained according to the corresponding mapping relation, so that a luggage tracking and identifying effect is achieved.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a luggage auxiliary identification method. Background Art

[0002] At airports, stations and logistics transfer centers, there are a large number of luggage that need to be sorted and identified. How to ensure the safe, accurate and efficient sorting of luggage is a very important issue. Taking civil aviation airports as an example, there are a large number of flights and passengers at busy domestic airports every day. The safe, smooth and efficient operation of the passenger checked luggage flow is a key factor in ensuring the safety of civil aviation airport operations, which directly affects the normality of flights and the overall satisfaction of passengers. Looking around the world, luggage mishandling and leakage have always been the main source of complaints from civil aviation passengers, and the cost paid by the civil aviation industry is also very high. Even the implicit and geometrically multiplied safety risks are difficult to estimate. RFID technology is generally regarded as an important solution for the construction of baggage handling systems in the civil aviation field for the needs of passengers to check in luggage due to its high recognition rate. When the luggage passes through the important nodes of automatic diversion or sorting, the RFID equipment will uniformly organize the sorting and identification of the luggage. After successfully identifying and binding the luggage serial number, the processing results will be fed back to the PLC and elevation control system of the baggage handling system as an important basis for luggage diversion or sorting. In addition, the widespread application of RFID technology has effectively improved the rate of baggage transportation information collection and the accuracy of baggage tracking by airports and airlines, and realized the real-time sharing and precise positioning of baggage information. Thanks to the technical characteristics of RFID technology that is not affected by conditions such as barcode position, bending, background light, print quality, and debris covering the barcode surface, it has a more generalized advantage than laser recognition, camera recognition and other technologies.

[0003] However, due to the antenna design and metal interference of RFID technology, it is difficult to constrain the RF range to a specific spatial range. The application of existing RFID technology requires the baggage handling system to increase the distance between the luggage to more than 350mm, and cannot well handle the problems of disordered sorting and identification and inaccurate tracking caused by the luggage being connected or stacked when the luggage distance is less than 350mm. The problems caused by the luggage reading, wrong binding, and missed binding also further limit the success rate of luggage sorting, identification, and binding from 99% to a higher level. Taking the security inspection link as an example, if the luggage sorting, identification, and binding are abnormal, the luggage will not be able to be diverted normally. A large number of luggage blocked in the bag opening room will not only increase the workload of security inspectors, but also disrupt normal airport operations; similarly, in the automatic sorting link, if the luggage sorting, identification, and binding are abnormal, the luggage with unclear purpose information will not be able to be sorted normally. A large number of luggage discarded at the bag abandonment port will not only increase the workload of sorters, but also greatly increase the risk of luggage sorting errors. In the context of informatization construction, artificial intelligence technology is becoming more and more mature, and the widespread application of machine learning has also provided solutions to complex problems and large amounts of data processing. The use of image recognition technology based on machine learning is a proven and effective technology that can also be used for baggage identification. However, if the technical means of scanning the entire image feature database and then locating specific luggage is adopted, it has the disadvantages of weak luggage identification ability and insufficient real-time processing capabilities. This cannot adapt to the reality of the surge in civil aviation passenger traffic and the convergence of luggage styles. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a baggage auxiliary identification method, which is used to cooperate with RFID identification technology to assist in identifying baggage, and solve the problem that the existing RFID technology cannot accurately handle the disordered sorting and identification caused by connected or stacked baggage when the luggage spacing is less than 350mm, and cannot be accurately tracked.

[0005] The present invention discloses a luggage auxiliary identification method through an embodiment, comprising the following steps:

[0006] Establishing a training image set, constructing a first enhanced semantic segmentation model, constructing a first clustering pattern recognition model, training the first enhanced semantic segmentation model based on the training image set, and obtaining a second enhanced semantic segmentation model and a training feature set; training the first clustering pattern recognition model based on the training feature set, and obtaining a second clustering pattern recognition model;

[0007] Collecting luggage information with potential recognition requirements, using the second enhanced semantic segmentation model to recognize images of luggage with potential recognition requirements, extracting first image features, establishing a mapping between the first image features and the corresponding luggage information, and storing the first image features of all luggage into a feature library by category;

[0008] During the baggage identification process, if the RFID device cannot effectively read the baggage information with potential identification requirements, an image of the baggage that cannot be effectively read is collected and recorded as the baggage image to be identified. The second enhanced semantic segmentation model is used to identify the baggage image to be identified, and the second image features are extracted. The second clustering pattern recognition model is used to compare and search for the first image features that are the same or similar to the second image features in the feature library, and the baggage information of the baggage image to be identified is obtained based on the mapping relationship of the found first image features.

[0009] It is worth noting that, as described in the background technology, for baggage information with potential identification needs, when the distance between baggage is less than 350 mm or there is overlap, the RFID device cannot effectively identify the baggage. Therefore, all baggage may be at a distance of 350 mm or in an overlapped state. Therefore, all baggage is baggage with potential identification needs, but not all baggage requires auxiliary identification using this method.

[0010] Further, a mapping between the first image feature and the corresponding luggage information is established, and the first image features of all luggage are classified and stored as a feature library, including: storing all the first image features extracted by the second enhanced semantic segmentation model in the form of a B+ tree based on the neo4j graph database; constructing a B+ tree data structure by classifying the luggage by color, shape and size in the B+ tree, and storing the features other than the color, shape and size of the luggage in the leaf nodes of the B+ tree, and storing them in a chain with a timestamp. Generally, the image features of the luggage extracted according to the enhanced semantic segmentation model have various dimensions and different expressions. Therefore, the data structure of the B+ tree is constructed with the three most basic features of color, shape and size of the luggage, and the leaf nodes of the B+ tree are used to store other features and store them in a chain with a timestamp. Excluding appearance descriptions such as color, shape, and size, other features are less interpretable and therefore are recorded uniformly, and the index can be increased or decreased later according to the average retrieval time and hit rate.

[0011] Furthermore, the second enhanced semantic segmentation model is deployed on the terminal, and the second clustering pattern recognition model is deployed on the cloud. As described in the background technology, when the RFID recognition technology cannot well recognize the luggage with a distance of less than 350 mm between connected and stacked luggage, the second semantic segmentation model deployed on the terminal is used to extract the image features of the luggage that cannot be recognized by RFID, and then the second clustering pattern recognition model deployed on the cloud is used to assist in identifying the luggage through image recognition.

[0012] Furthermore, the first enhanced semantic segmentation model includes an enhanced semantic segmentation model based on DeepLabV3+ and Vision Transformer; the first clustering pattern recognition model includes a clustering pattern recognition model based on GMMs-MoCoV3.

[0013] Furthermore, the enhanced semantic segmentation model based on DeepLabV3+ and Vision Transformer includes a feature extraction module, a global feature learning module based on VisionTransformer, and a fully connected layer module; wherein the feature extraction module serves as an input module of the first enhanced semantic segmentation model; the feature extraction module is connected to the global feature learning module, and the fully connected layer module is used to integrate the outputs of the connected feature extraction module and the global feature learning module.

[0014] Furthermore, the backbone network of DeepLabV3+ in the feature extraction module includes the Xception network; the HWC format images in the training image set are converted into BCHW format input tensors and sent to the backbone network of DeepLabV3+ for multiple scale feature extraction, and the contextual information of different scales in the image is enhanced by using the dilated convolution and spatial pyramid pooling techniques; the extracted features are saved in the form of feature maps. Among them, the Xception network as the backbone network of DeepLabV3+ can reduce the training time and improve the model accuracy. The feature map records the category information of each pixel in the image, covering the regional features of the luggage in each image.

[0015] Furthermore, the global feature learning module based on VisionTransformer includes: an embedding layer, which is used to divide the feature map extracted by the backbone network of DeepLabV3+ into image blocks of fixed size, and linearly embed each image block and add position encoding, and convert the training image data to be input into a serialized representation that can be input into the Transformer encoder; a Transformer encoding layer, which is composed of multiple Transformer encoders connected in series, and is used to input the serialized represented data into the encoder, each Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network for capturing global information and learning feature representation; a fully connected layer for clustering, which is used to cluster the luggage features after passing through the Transformer encoding layer.

[0016] Further, training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training a GMMs classification module:

[0017] According to the total number N of all luggage with potential recognition needs, the luggage is classified and stored as the number of categories K in the feature library, and the luggage feature matrix X is defined as:

[0018]

[0019] Among them, x iis the feature vector corresponding to the image feature of the i-th piece of luggage;

[0020] Define the baggage category matrix Y:

[0021] Y=[y1,y2,...,y N ]

[0022] Among them, y i is the category of the i-th piece of luggage;

[0023] Use K-means clustering to initialize the mean, covariance matrix and weights of the GMMs classification module;

[0024] The GMMs classification module is trained using the maximum expectation algorithm, and the following steps are iteratively performed until the GMMs classification module converges:

[0025] Calculate the responsibility value of each data point belonging to each Gaussian component:

[0026]

[0027] Among them, γ ik is the responsibility p(x i |k) is the probability density of the i-th sample under the k-th Gaussian component, π k is the weight of the kth component;

[0028] Update the mean, covariance matrix, and weights of the Gaussian components:

[0029]

[0030]

[0031]

[0032] Further, training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training a similar baggage matching module:

[0033] Perform different data augmentation operations on a specific category of luggage one by one to obtain two sets of image views with different perspectives from the original luggage image, which are defined as query view and key view respectively;

[0034] The query view and key view from the same baggage are respectively used as positive samples, so that the similarity between the two can be maximized; the query view and key views from other different baggage are used as negative samples, so that the similarity between the query view and the negative samples can be minimized;

[0035] Use ViT as the backbone network to encode the query view, positive samples and negative samples respectively, and obtain the query feature vector q, positive sample vector k+ and negative sample vector k-;

[0036] Use the InfoNCE loss function to measure the similarity between the query feature vector q and the positive sample vector k+, and keep as large a gap as possible with the negative sample vector k-;

[0037] The loss is calculated using back-propagation and the parameters of the similar baggage matching module are updated through the optimizer.

[0038] Further, using the second enhanced semantic segmentation model to identify the luggage image to be identified, extracting second image features, using the second clustering pattern recognition model to compare and search for first image features that are identical or similar to the second image features in the feature library, and obtaining luggage information of the luggage image to be identified based on the mapping relationship of the found first image features includes:

[0039] Using the second enhanced semantic segmentation model to identify the luggage image to be identified, and extracting second image features;

[0040] The probability of the second image feature of the luggage image to be identified in each category is calculated through the GMMs classification module:

[0041]

[0042] Use the MoCoV3 detection module to match and search for the most similar luggage to be identified in the sub-library under the category with the highest probability:

[0043]

[0044] Among them, x b is the second image feature of the luggage image to be identified.

[0045] Further, obtaining a luggage image to be identified, using a second enhanced semantic segmentation model to identify the luggage image to be identified, extracting second image features, using a second clustering pattern recognition model to compare and search for first image features that are identical or similar to the second image features in a feature library, and obtaining luggage information of the luggage image to be identified based on a mapping relationship of the found first image features also includes:

[0046] If the probability of the second image feature of the luggage image to be identified in each category If the confidence of the most similar luggage to be identified found by the MoCoV3 detection module is lower than the preset value or the confidence of the most similar luggage to be identified found by the MoCoV3 detection module is lower than the preset value, the sub-library under the category corresponding to the current maximum probability is excluded, the probability of the second image feature of the luggage to be identified image in each category is recalculated, and then the most similar luggage to be identified is found according to the sub-library matching under the category corresponding to the recalculated maximum probability; if the ownership of the luggage to be identified is still not determined after the recalculation times reach the preset times, the luggage to be identified is specially marked.

[0047] Furthermore, the specially marked baggage is transferred to the manual code supplement process.

[0048] The technical principle of the present invention is: use the enhanced semantic segmentation model to perform image recognition on all luggage with potential recognition needs, extract its image features, establish a mapping between the image features of each luggage picture and its corresponding luggage information, and store the image features of all luggage in a feature library by category. When encountering luggage to be recognized, obtain the image of the luggage to be recognized, use the enhanced semantic segmentation model to extract the image features of the luggage to be recognized, and then use the clustering pattern recognition model to compare and search for image features that are the same or similar to the image features of the luggage to be recognized in the feature library. Finally, based on the found image features, the luggage information of the luggage image to be recognized is obtained according to its corresponding mapping relationship, which plays a role in tracking and identifying the luggage.

[0049] Compared with the prior art, the present invention has the following beneficial effects: compared with the technical means of locating specific luggage by scanning the entire image feature database, the present invention has the characteristic of small computational complexity; combined with RFID identification technology, when the RFID identification method cannot identify luggage in a connected or stacked state with a spacing of less than 350 mm, the present auxiliary identification method is used to identify the luggage, thereby achieving a more efficient and accurate whole-process luggage tracking, identification and sorting effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present invention and facilitate further understanding of the technical effects, technical features and purposes of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings. The accompanying drawings constitute an essential part of the specification and are used together with the embodiments of the present invention to illustrate the technical solution of the present invention, but do not constitute a limitation to the present invention.

[0051] Figure 1 A flow chart of the method described in the present invention;

[0052] Figure 2 A flow chart of the method described in the present invention in combination with RFID identification technology;

[0053] Figure 3 A schematic diagram of a B+ tree according to the present invention;

[0054] Figure 4 A schematic diagram of model interaction of each module in the method described in the present invention;

[0055] Figure 5 Schematic diagram of the device structure used in the embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described below are only intended to explain the technical solutions of the present invention, rather than to limit the present invention. In addition, the parts described in the embodiments or the accompanying drawings are only examples of the relevant parts of the present invention, rather than the entirety of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art should naturally belong to the protection scope of the present invention.

[0057] like Figure 1 As shown, the present invention proposes a luggage auxiliary identification method through the following embodiments, comprising the following steps:

[0058] Establish a training image set, construct a first enhanced semantic segmentation model, construct a first clustering pattern recognition model, train the first enhanced semantic segmentation model based on the training image set, obtain a second enhanced semantic segmentation model and a training feature set; train the first clustering pattern recognition model based on the training feature set to obtain a second clustering pattern recognition model. Collect luggage information with potential recognition requirements, use the second enhanced semantic segmentation model to recognize images of luggage with potential recognition requirements, extract first image features, establish a mapping between the first image features and the corresponding luggage information, and store the first image features of all luggage in a feature library by category. If the RFID device cannot effectively read the luggage information with potential recognition requirements during the luggage recognition process, collect images of luggage that cannot be effectively read, record them as luggage images to be recognized, use the second enhanced semantic segmentation model to recognize the luggage images to be recognized, extract second image features, use the second clustering pattern recognition model to compare and search for first image features that are the same or similar to the second image features in the feature library, and obtain luggage information of the luggage images to be recognized based on the mapping relationship of the first image features found.

[0059] It is worth noting that the so-called luggage with potential identification needs refers to, as described in the background technology, when the distance between luggage is less than 350mm or there is a bag overlap, the RFID device cannot effectively identify the luggage. Therefore, all luggage may be within a distance of 350mm or in a bag overlap state. Therefore, all luggage is luggage with potential identification needs, but not all luggage may need this method for auxiliary identification. Therefore, when this technology is applied, Figure 5As shown in the figure, RFID recognition technology is combined with this image-assisted recognition technology, with RFID recognition technology as the main technology and image recognition technology as the auxiliary technology to assist in baggage recognition. Semantic segmentation is used to assign a label to each pixel in the image, so as to achieve more fine-grained extraction, and then the pattern recognition of the Gaussian mixture model is superimposed to recognize the baggage. The solution combining semantic segmentation and classification pattern recognition can better handle baggage recognition and tracking tasks under extreme conditions, so as to achieve more efficient and accurate baggage tracking, identification and sorting effects throughout the entire process.

[0060] It is worth noting that the accuracy of the feature extraction of the enhanced semantic segmentation model for luggage is also probabilistic. Performing feature extraction on luggage only once does not guarantee that the luggage can be fully identified in the future. Therefore, it is necessary to perform multiple iterative extractions on the feature map of the luggage. During multiple extractions, the score of the existing feature map output by the enhanced semantic segmentation model is compared with the score of the currently extracted feature map, and the one with the higher score is selected as the feature map of the luggage and stored in the feature library. Specifically, Figure 2 As shown in the figure, during the process of baggage check-in, security check, transportation, sorting and loading, as long as the baggage passes through a node with collection function, such as check-in, security check, sorting, etc., or is captured by a camera on the sorting conveyor belt, its image information is completely collected, and then the edge device schedules the enhanced semantic segmentation model to extract features from the baggage. When the enhanced semantic segmentation model extracts baggage features, it outputs a score for the baggage feature map: the higher the score, the higher the accuracy of the feature map extracted in this round. Therefore, for a specific baggage, each time a feature map is extracted, the score output when the feature map is currently extracted is compared with the score of the current feature map, and a feature map corresponding to a higher score is used as the image feature of the current baggage and stored in the feature library. After multiple rounds of algorithm iterative processing, the baggage feature map has the most complete and accurate features, and accordingly, the subsequent feature matching tasks it supports are more reliable.

[0061] like Figure 3As shown, in this embodiment, a mapping between the first image feature and the corresponding luggage information is established, and the first image features of all luggage are classified and stored as a feature library, including: storing all the first image features extracted by the second enhanced semantic segmentation model in the form of a B+ tree based on the neo4j graph database; constructing a data structure of the B+ tree by classifying the luggage by color, shape and size in the B+ tree, and storing the features other than the color, shape and size of the luggage in the leaf nodes of the B+ tree, and storing them in a chain with a timestamp. Generally, the image features of the luggage extracted by the enhanced semantic segmentation model have various dimensions and different expressions, so the data structure of the B+ tree is constructed with the three most basic features of color, shape and size of the luggage, and the leaf nodes of the B+ tree are used to store other features and store them in a chain with a timestamp. Excluding appearance descriptions such as color, shape, and size, other features are less interpretable, so they are recorded uniformly, and the index can be increased or decreased later according to the average retrieval time and hit rate.

[0062] It is worth noting that, taking the application of the auxiliary identification method to the identification of checked baggage at the airport as an example, the baggage information should at least include: the owner of the baggage, the owner's flight number, and the flight take-off and landing times.

[0063] like Figure 2 , 4 As shown, in this embodiment, the auxiliary identification method is still used in the identification of checked baggage at the airport as an example, and the second enhanced semantic segmentation model is arranged at the terminal, such as the camera or video camera at the check-in counter, the camera or video camera on the conveyor belt at the sorting counter, etc.; the second clustering pattern recognition model is arranged in the cloud, that is, the control center for sorting checked baggage at the airport, or the sorting center. As described in the background technology, in the airport baggage sorting process, the RFID identification technology at the sorting outlet cannot well identify the baggage in a connected or stacked state when the distance between the baggage is less than 350mm. The second semantic segmentation model arranged at the terminal (that is, the camera or video camera at the sorting counter) is used to extract the image features of the baggage that the RFID cannot identify, and then the second clustering pattern recognition model arranged in the cloud is used to assist in the identification of the baggage through image recognition, and the baggage information is identified before sorting. After the passenger checks in, the baggage is affixed with an RFID tag, such as Figure 5 The camera at the check-in station as shown in the figure collects the image information of the baggage, uses the enhanced semantic segmentation model to extract the image features, then establishes the relevant mapping, and then establishes the feature library. The checked baggage enters the sorting and loading process, and the baggage is transported on the conveyor belt and goes through the following process: Figure 5 When the RFID identification device encounters the above-mentioned baggage that cannot be identified, the RFID identification technology can be used to identify the baggage for sorting. Figure 5The camera at the device shown captures the image of the current luggage, uses the enhanced semantic segmentation model to extract image features, and then uses the clustering pattern recognition model to compare and search for image features that are the same or similar to the image features of the luggage that cannot be recognized at present in the feature library, and obtains the luggage information of the luggage to be recognized at present based on the mapping relationship of the found image features.

[0064] In this embodiment, establishing a training image set includes: collecting and labeling luggage image data taken by a terminal device, and performing denoising and correcting image labels on the image data; expanding the luggage image data by rotating, scaling, cropping, flipping or adjusting brightness; and normalizing the luggage image data into HWC format images to obtain a training image set. These operations can increase the robustness and generalization ability of the model.

[0065] In this embodiment, the first enhanced semantic segmentation model includes a feature extraction module, a global feature learning module based on VisionTransformer, and a fully connected layer module; wherein the feature extraction module serves as the input module of the first enhanced semantic segmentation model; the feature extraction module is connected to the global feature learning module, and the fully connected layer module is used to integrate the outputs of the connected feature extraction module and the global feature learning module.

[0066] In this embodiment, the backbone network of DeepLabV3+ of the feature extraction module includes the Xception network; the HWC format image in the training image set is converted into the input tensor in the BCHW format and sent to the backbone network of DeepLabV3+ for multiple scale feature extraction, and the context information of different scales in the image is enhanced by using the hole convolution and spatial pyramid pooling technology; the extracted features are saved in the form of feature maps. Among them, the Xception network as the backbone network of DeepLabV3+ can reduce the training time and improve the model accuracy. The feature map records the category information of each pixel in the image, covering the regional features of the luggage in each image.

[0067] In this embodiment, the global feature learning module based on VisionTransformer includes: an embedding layer, which is used to divide the feature map extracted by the backbone network of DeepLabV3+ into image blocks of fixed size, and linearly embed each image block and add position encoding, and convert the training image data to be input into a serialized representation that can be input into the Transformer encoder; a Transformer encoding layer, which is composed of multiple Transformer encoders connected in series, and is used to input the serialized represented data into the encoder, each Transformer encoder includes a multi-head self-attention mechanism and a feedforward neural network for capturing global information and learning feature representation; a fully connected layer for clustering, which is used to cluster the baggage features after passing through the Transformer encoding layer.

[0068] In this embodiment, the first enhanced semantic segmentation model is trained based on the training image set to obtain the second enhanced semantic segmentation model and the training feature set, including: dividing the training image set and the verification image set during the training process, and determining the initial optimizer, learning efficiency, attenuation strategy and number of iterations, evaluating the performance of the first enhanced semantic segmentation model based on the verification image set, and continuously iteratively adjusting the parameters of the first enhanced semantic segmentation model until a final converged model is obtained, and the final converged model is the second enhanced semantic segmentation model.

[0069] In this embodiment, the first clustering pattern recognition model based on GMMs-MoCoV3 includes a GMMs classification module and a similar baggage matching module, wherein the similar baggage matching module is a MoCoV3 detection module.

[0070] In this embodiment, training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training a GMMs classification module:

[0071] According to the total number N of all luggage with potential recognition needs, the luggage is classified and stored as the number of categories K in the feature library, and the luggage feature matrix X is defined as:

[0072]

[0073] Among them, x i is the feature vector corresponding to the image feature of the i-th piece of luggage;

[0074] Define the baggage category matrix Y:

[0075] Y=[y1,y2,...,y N ]

[0076] Among them, y iis the category of the i-th piece of luggage;

[0077] Use K-means clustering to initialize the mean, covariance matrix and weights of the GMMs classification module;

[0078] The GMMs classification module is trained using the maximum expectation algorithm, and the following steps are iteratively performed until the GMMs classification module converges:

[0079] Calculate the responsibility value of each data point belonging to each Gaussian component:

[0080]

[0081] Among them, γ ik is the responsibility p(x i |k) is the probability density of the i-th sample under the k-th Gaussian component, π k is the weight of the kth component;

[0082] Update the mean, covariance matrix, and weights of the Gaussian components:

[0083]

[0084]

[0085]

[0086] In this embodiment, training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training the similar baggage matching module:

[0087] Different data augmentation operations are performed on a specific category of luggage one by one to obtain two sets of image views with different perspectives from the original luggage image, which are defined as query views and key views respectively; the query view and key view from the same luggage are respectively used as positive samples to maximize the similarity between the two, and the query view and key views from other different luggage are used as negative samples to minimize the similarity between the query view and the negative sample; ViT is used as the backbone network to encode the query view, positive sample and negative sample respectively to obtain the query feature vector q, positive sample vector k+ and negative sample vector k-; the InfoNCE loss function is used to measure the similarity between the query feature vector q and the positive sample vector k+, and to keep the gap with the negative sample vector k- as large as possible; the loss is calculated using backpropagation, and the parameters of the MoCoV3 detection module are updated through the optimizer.

[0088] It is worth noting here that the query view and key view from the same luggage constitute positive samples, and the query view and key views from other different luggage constitute negative samples. This is achieved through image enhancement, such as image cropping, flipping and other operations.

[0089] In this embodiment, the first clustering pattern recognition model is trained based on the training feature set to obtain the second clustering pattern recognition model, including: dividing the training image set and the verification image set during the training process, and determining the initial optimizer, learning efficiency, attenuation strategy and number of iterations, evaluating the performance of the first clustering pattern recognition model according to the verification image set, and continuously iteratively adjusting the parameters of the first clustering pattern recognition model until a final converged model is obtained, and the final converged model is the second clustering pattern recognition model.

[0090] In this embodiment, obtaining a luggage image to be identified, using a second enhanced semantic segmentation model to identify the luggage image to be identified, extracting second image features, using a second clustering pattern recognition model to compare and search for first image features that are identical or similar to the second image features in a feature library, and obtaining luggage information of the luggage image to be identified based on a mapping relationship of the found first image features includes:

[0091] First, the luggage image to be identified is identified using a second enhanced semantic segmentation model to extract second image features;

[0092] The probability of the second image feature of the luggage image to be identified in each category is calculated through the GMMs classification module:

[0093]

[0094] Finally, the MoCoV3 detection module is used to match and search for the most similar luggage to be identified in the sub-library under the category with the highest probability:

[0095]

[0096] Among them, x b is the second image feature of the luggage image to be identified.

[0097] In this embodiment, if Figure 2 As shown, obtaining a luggage image to be identified, using a second enhanced semantic segmentation model to identify the luggage image to be identified, extracting second image features, using a second clustering pattern recognition model to compare and search for first image features that are identical or similar to the second image features in a feature library, and obtaining luggage information of the luggage image to be identified based on a mapping relationship of the found first image features also includes:

[0098] If the probability of the second image feature of the luggage image to be identified in each category If the confidence of the most similar luggage to be identified found by the MoCoV3 detection module is lower than the preset value or the confidence of the most similar luggage to be identified found by the MoCoV3 detection module is lower than the preset value, the sub-library under the category corresponding to the current maximum probability is excluded, and the probability of the second image feature of the luggage to be identified in each category is recalculated; then the most similar luggage to be identified is searched according to the sub-library under the category corresponding to the recalculated maximum probability; if the ownership of the luggage to be identified is still not determined after the recalculation times reach the preset times, the luggage to be identified is specially marked.

[0099] In this embodiment, if Figure 2 As shown, the specially marked baggage is transferred to the manual coding process. During the manual coding, the worker matches and identifies the information of the baggage. If the baggage is identified, the baggage information is input into the baggage sorting system. At the same time, the system updates the characteristics of the current baggage so that the characteristic diagram can be found in the characteristic library when there is a need for baggage identification in the future.

[0100] It should be noted that the above embodiments are only for more clearly illustrating the technical solutions of the present invention. Those skilled in the art will understand that the implementation methods of the present invention are not limited to the above contents, and obvious changes, replacements or substitutions based on the above contents do not exceed the scope of the technical solutions of the present invention; other implementation methods will naturally fall within the scope of the present invention without departing from the concept of the present invention.

Claims

1. A baggage auxiliary identification method, characterized in that: include Establishing a training image set, constructing a first enhanced semantic segmentation model, constructing a first clustering pattern recognition model, training the first enhanced semantic segmentation model based on the training image set, and obtaining a second enhanced semantic segmentation model and a training feature set; Training the first clustering pattern recognition model based on the training feature set to obtain a second clustering pattern recognition model; Collecting luggage information with potential recognition requirements, using a second enhanced semantic segmentation model to recognize the image of the luggage with potential recognition requirements, extracting first image features, establishing a mapping between the first image features and the corresponding luggage information, and storing the first image features of all luggage into a feature library by category; During the baggage identification process, if the RFID device cannot effectively read the baggage information with potential identification requirements, an image of the baggage that cannot be effectively read is collected and recorded as a baggage image to be identified. The baggage image to be identified is identified by using a second enhanced semantic segmentation model, and second image features are extracted. A second clustering pattern recognition model is used to compare and search for a first image feature that is identical or similar to the second image feature in the feature library, and baggage information of the baggage image to be identified is obtained based on a mapping relationship of the found first image features.

2. The luggage auxiliary identification method according to claim 1, characterized in that: The step of establishing a mapping between the first image feature and the corresponding luggage information, and storing the first image features of all luggage into a feature library by category comprises: Storing all first image features extracted by the second enhanced semantic segmentation model in the form of a B+ tree based on a neo4j graph database; The data structure of the B+ tree is constructed by classifying the luggage by color, shape and size. The leaf nodes in the B+ tree store the features other than the color, shape and size of the luggage and store them in a chained manner using timestamps.

3. The luggage auxiliary identification method according to claim 1, characterized in that: The first enhanced semantic segmentation model includes an enhanced semantic segmentation model based on DeepLabV3+ and Vision Transformer; The first clustering pattern recognition model includes a clustering pattern recognition model based on GMMs-MoCoV3.

4. The luggage auxiliary identification method according to claim 3, characterized in that: Also includes: The enhanced semantic segmentation model based on DeepLabV3+ and Vision Transformer includes a feature extraction module, a global feature learning module based on Vision Transformer, and a fully connected layer module; The feature extraction module serves as an input module of the first enhanced semantic segmentation model; The feature extraction module is connected to the global feature learning module, and the fully connected layer module is used to integrate and connect the outputs of the feature extraction module and the global feature learning module.

5. The luggage auxiliary identification method according to claim 4, characterized in that: The backbone network of the feature extraction module includes an Xception network; Convert the HWC format images in the training image set into BCHW format input tensors and feed them into the backbone network of DeepLabV3+ for multiple scale feature extraction, and use dilated convolution and spatial pyramid pooling techniques to enhance context information of different scales in the image; The extracted features are saved in the form of feature maps.

6. The luggage auxiliary identification method according to claim 5, characterized in that: The global feature learning module based on VisionTransformer includes: An embedding layer, which is used to divide the feature map extracted by DeepLabV3+ into image blocks of fixed size, linearly embed each image block and add position encoding, and convert the training image data to be input into a serialized representation that can be input into the Transformer encoder; A Transformer encoding layer, which is composed of multiple Transformer encoders connected in series and is used to input the serialized data into the encoder. Each Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network to capture global information and learn feature representation; The fully connected layer for clustering is used to cluster the baggage features after passing through the Transformer encoding layer.

7. The luggage auxiliary identification method according to claim 6, characterized in that: Training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training a GMMs classification module: According to the total number N of all luggage with potential recognition needs, the luggage is classified and stored as the number of categories K in the feature library, and the luggage feature matrix X is defined as: Among them, x i is the feature vector corresponding to the image feature of the i-th piece of luggage; Define the baggage category matrix Y: Y=[y1,y2,...,y N ] Among them, y i is the category of the i-th piece of luggage; Use K-means clustering to initialize the mean, covariance matrix and weights of the GMMs classification module; The GMMs classification module is trained using the maximum expectation algorithm, and the following steps are iteratively performed until the GMMs classification module converges: Calculate the responsibility value of each data point belonging to each Gaussian component: Among them, γ ik is the responsibility p(x i |k) is the probability density of the i-th sample under the k-th Gaussian component, π k is the weight of the kth component; Update the mean, covariance matrix, and weights of the Gaussian components:

8. The luggage auxiliary identification method according to claim 7, characterized in that: Training the first clustering pattern recognition model based on the training feature set to obtain the second clustering pattern recognition model includes training a similar baggage matching module: Perform different data augmentation operations on a specific category of luggage one by one to obtain two sets of image views with different perspectives from the original luggage image, which are defined as query view and key view respectively; The query view and key view from the same baggage are respectively used as positive samples, and the query view and key views from other different bags are used as negative samples. Use ViT as the backbone network to perform feature encoding on the query view, positive samples and negative samples respectively, to obtain a query feature vector q, a positive sample vector k+ and a negative sample vector k-; Use the InfoNCE loss function to measure the similarity between the query feature vector q and the positive sample vector k+, and keep as large a gap as possible with the negative sample vector k-; The loss is calculated using back propagation and the parameters of the similar baggage matching module are updated through an optimizer.

9. The luggage auxiliary identification method according to claim 8, characterized in that: The step of using the second enhanced semantic segmentation model to identify the luggage image to be identified, extracting second image features, using the second clustering pattern recognition model to compare and search for first image features that are identical or similar to the second image features in the feature library, and obtaining luggage information of the luggage image to be identified based on a mapping relationship of the found first image features includes: Using a second enhanced semantic segmentation model to identify the luggage image to be identified, and extracting second image features; The probability of the second image feature of the luggage image to be identified in each category is calculated by the GMMs classification module: Use the similar baggage matching module to match and search for the most similar baggage to be identified in the sub-library under the category with the highest probability: Among them, x b is the second image feature of the luggage image to be identified.

10. The luggage auxiliary identification method according to claim 9, characterized in that: The step of acquiring a luggage image to be identified, identifying the luggage image to be identified using a second enhanced semantic segmentation model, extracting a second image feature, and using a second clustering pattern recognition model to compare and search in the feature library for a first image feature that is identical or similar to the second image feature, and acquiring luggage information of the luggage image to be identified based on a mapping relationship of the found first image feature also includes: If the probability of the second image feature of the luggage image to be identified in each category If the confidence level of the second image feature of the luggage to be identified is lower than a preset value or the confidence level of the most similar luggage to be identified found by the similar luggage matching module is lower than a preset value, the sub-library under the category corresponding to the current maximum probability is excluded, and the probability of the second image feature of the luggage to be identified in each category is recalculated; Then, the most similar luggage to be identified is found by matching the sub-libraries under the category corresponding to the recalculated maximum probability; When the ownership of the luggage to be identified is still not determined after the recalculation times reach the preset times, the luggage to be identified is specially marked.

Images