Vision-based luggage attribute analysis method, analysis device and storage medium

Abstract

The application discloses a visual-based luggage attribute analysis method, which comprises the following steps: preparing a training data set for luggage attribute analysis; collecting a first image set containing luggage with multiple attributes; detecting the luggage type and a luggage target area of each image in the first image set by using a luggage type detection model, and taking the result of the luggage target area detection as a training data set; preprocessing the training data set; and training a luggage attribute recognition model for recognizing the function, color and size attributes of luggage by using the preprocessed training data set and a deep learning network. The application also provides a luggage attribute analysis device and a computer readable storage medium.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a vision-based method, analysis device, and storage medium for baggage attribute analysis. Background Technology

[0002] Staff typically use traditional manual registration methods to collect baggage information, including type, function, and color. This traditional method is costly and lacks automation, failing to meet the needs of smart city development. With rapid economic development and rising living standards, people travel with large amounts of luggage. Therefore, the role of non-intrusive baggage attribute analysis technology is becoming increasingly important for achieving information-based baggage management. Summary of the Invention

[0003] In view of the prior art, the technical problem solved by this application is to provide a vision-based, non-intrusive baggage attribute analysis method, analysis device, and storage medium that can realize automated baggage attribute analysis and help realize intelligent management of passenger baggage.

[0004] To address the aforementioned technical problems, this application provides a vision-based baggage attribute analysis method, which includes:

[0005] Prepare the training dataset for baggage attribute analysis: Collect a first set of images containing baggage with multiple attributes, and use a baggage type detection model to detect baggage type and baggage target region for each image in the first set of images. Use the results of baggage target region detection as the training dataset.

[0006] Preprocessing of the training dataset: preprocessing the baggage target region to remove interference and perform data augmentation; and

[0007] We used the preprocessed training dataset to train a deep learning-based network to train a luggage attribute recognition model that identifies luggage functions, colors, and size attributes.

[0008] In one possible implementation, during the preparation of the training dataset for baggage attribute analysis,

[0009] Prepare training data for the baggage type detection model: Collect a second image set containing baggage with multiple attributes. Divide the second image set into training data, validation data, and test data in an 8:1:1 ratio. Label the baggage type and baggage area on the training data in the second image set.

[0010] Training the baggage type detection model: The attention mechanism module is added to the starting position of each unit module of the backbone network of the Yolox model to construct the baggage type detection model; and the baggage type detection model is trained by combining the Focal Loss loss function.

[0011] Baggage type and target detection: Using a trained baggage type detection model, baggage type and target detection are performed on each image in the training dataset for baggage attribute analysis. The center point of the baggage target and the detection box of the region related to the baggage target are obtained in each image in the training dataset for baggage attribute analysis.

[0012] The first image set is cropped based on the center point of the baggage target and the detection box to obtain the baggage target area.

[0013] In one possible implementation, the center point coordinates of any baggage target are denoted as (x, y), and the height and width of the detection box of the baggage target are H and W, respectively. The step of cropping the first image set based on the center point and detection box of the baggage target to obtain the baggage target region is as follows:

[0014] Calculate the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2) of the coordinate frame;

[0015] Use the top left and bottom right corners as two diagonal points to crop and obtain a rectangular luggage target area;

[0016] in

[0017] In one possible implementation, the steps of training a deep learning-based network using the preprocessed training dataset include:

[0018] Each luggage target region in the training dataset is preprocessed sequentially with denoising and data augmentation to obtain the third image set;

[0019] Data augmentation preprocessing is performed on each luggage target area in the dataset to obtain the fourth image set;

[0020] Training the luggage color attribute recognition model: Label the third image set with color attributes, and input the labeled third image set into the first deep learning network for training;

[0021] Training the luggage size attribute recognition model: Label the size attributes for the fourth image set, and input the labeled third image set into the second deep learning network for training;

[0022] Training the luggage functional attribute recognition model: Label the functional color attributes for the fourth image set, and input the labeled third image set into the third deep learning network for training;

[0023] Training the baggage attribute recognition model: Calculate the accuracy of the three trained baggage color attribute recognition models, baggage size attribute recognition models, and baggage function attribute recognition models. Select the main branch of the attribute recognition model with the highest accuracy as the main branch of the fourth deep learning network. Freeze the first n units of the main branch and use the remaining units of the main branch to train the other two attribute recognition models again. Output a single-input three-output baggage attribute recognition model.

[0024] In one possible implementation, n takes the value 10.

[0025] In one possible implementation, the baggage type detection model detects baggage types including shoulder bags, backpacks, handbags, waist bags, suitcases, and tote bags. Specifically, based on the baggage type detection model's results, baggage targets belonging to the same type undergo batch preprocessing to remove interference.

[0026] For the target area of ​​shoulder bag type luggage, the upper two-thirds of the luggage target area is blacked out in the X-axis direction;

[0027] For backpack-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction;

[0028] For handbag-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction;

[0029] For the luggage target area of ​​the handbag, the upper third of the luggage target area is blacked out in the X-axis direction;

[0030] For suitcase-type luggage target areas, the upper half of the luggage target area is colored black along the X-axis; and

[0031] For waist pack type luggage target areas, divide the luggage target area into four equal areas along the Y-axis, and color the first and fourth areas on the Y-axis black.

[0032] In one possible implementation, the preprocessing steps for data augmentation of the baggage target area include:

[0033] The size of each luggage target region in the unified training dataset;

[0034] Perform an affine transformation on each luggage target region in the training dataset;

[0035] Flip each luggage target region in the training dataset horizontally;

[0036] Each luggage target region in the training dataset is vectorized.

[0037] Normalize each channel of each baggage target area in the training dataset.

[0038] This application also provides a baggage attribute detection device, which includes:

[0039] The preparation unit for the training dataset of baggage attribute analysis: It is used to collect a first set of images containing baggage with multiple attributes, and to use a baggage type detection model to detect baggage type and baggage target region for each image in the first set of images, and use the results of baggage target region detection as the training dataset.

[0040] The preprocessing unit for the training dataset: it is used for preprocessing the baggage target region to remove distractions and perform data augmentation; and

[0041] Training unit of baggage attribute recognition model: It is used to train a deep learning network using a preprocessed training dataset to train a baggage attribute recognition model that recognizes baggage function, color and size attributes.

[0042] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the vision-based baggage attribute analysis method according to any one of claims 1-7.

[0043] This application provides a vision-based baggage attribute analysis method, analysis device, and readable storage medium. It utilizes a baggage detection model to detect the baggage type and target region in an image, and then uses these target regions to form a training dataset to train a baggage attribute recognition model that identifies baggage functions, colors, and sizes. Therefore, this vision-based, non-intrusive baggage attribute analysis method, compared to manual methods, can automatically analyze baggage attributes based on vision, which is beneficial for realizing intelligent baggage management. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This application presents a vision-based baggage attribute analysis method as an embodiment.

[0046] Figure 2(a) is an image containing a baggage target according to an embodiment of this application;

[0047] Figure 2(b) is a schematic diagram of the result after detecting and cropping 2(a) using the baggage type detection model.

[0048] Figure 3 This is a flowchart illustrating the steps of training a deep learning network using a preprocessed training dataset, as described in an embodiment of this application.

[0049] Figure 4(a) shows the luggage target area of ​​a shoulder bag type according to an embodiment of this application;

[0050] Figure 4(b) is a schematic diagram of the result of blackening two-thirds of the upper part of 4(a).

[0051] Figure 5(a) shows the target area of ​​handbag-type luggage in an embodiment of this application;

[0052] Figure 5(b) is a schematic diagram of the result of blackening one-third of the upper part of 5(a);

[0053] Figure 6(a) shows the baggage target area of ​​an embodiment of this application;

[0054] Figure 6(b) is a schematic diagram of the result of blackening half of the upper part of Figure 6(a). Detailed Implementation

[0055] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0056] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0057] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0059] The visual-based baggage attribute analysis method, analysis device, and computer-readable storage medium of this application will now be described in detail with reference to the accompanying drawings.

[0060] Referring to the figures, the steps of the vision-based baggage attribute analysis method provided in this application embodiment include:

[0061] Step S100: Prepare the training dataset for baggage attribute analysis: Collect a first image set containing baggage with multiple attributes, and use the baggage type detection model to detect baggage type and baggage target region for each image in the first image set. Use the results of baggage target region detection as the training dataset.

[0062] Step S200: Preprocessing of the training dataset: preprocessing the baggage target region for denoising and data augmentation; and

[0063] Step S300: Use the preprocessed training dataset to train a deep learning network to train a luggage attribute recognition model that identifies luggage function, color and size attributes.

[0064] Understandably, baggage attributes include baggage function, baggage color, baggage size, and baggage type. In the steps described above, a baggage detection model can be used to detect the baggage type and target region in the image. Then, the target regions are used to form a training dataset to train a baggage attribute recognition model that identifies baggage function, color, and size. Therefore, this vision-based, non-intrusive baggage attribute analysis method, compared to manual methods, can automatically analyze baggage attributes based on vision, which is beneficial for the realization of intelligent baggage management.

[0065] In one embodiment, reference is made to Figure 1 In step S100, during the preparation of the training dataset for baggage attribute analysis,

[0066] Step S110: Prepare training data for the baggage type detection model: Collect a second image set containing baggage with multiple attributes. Divide the second image set into training data, validation data, and test data in an 8:1:1 ratio. Label the baggage type and baggage area on the training data in the second image set.

[0067] Step S120: Train the baggage type detection model: Add the attention mechanism module to the starting position of each unit module of the backbone network of the Yolox model to construct the baggage type detection model; and train the baggage type detection model by combining the Focal Loss loss function.

[0068] Step S130: Baggage type and target detection: Using the trained baggage type detection model, baggage type and target detection are performed on each image in the training dataset for baggage attribute analysis, and the center point of the baggage target and the detection box of the region related to the baggage target are obtained in each image in the training dataset for baggage attribute analysis.

[0069] Step S140: Crop the first image set according to the center point of the baggage target and the detection box to obtain the baggage target area.

[0070] In one embodiment, for step S140, the center point coordinates of any baggage target are denoted as (x, y), and the height and width of the detection box of the baggage target are H and W, respectively. The step of cropping the first image set based on the center point and detection box of the baggage target to obtain the baggage target region is as follows:

[0071] Calculate the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2) of the coordinate frame;

[0072] Use the top left and bottom right corners as two diagonal points to crop and obtain a rectangular luggage target area;

[0073] in

[0074] As shown in Figures 2(a) and 2(b), Figure 2(a) is an image containing baggage targets, and Figure 2(b) is the result of detecting and cropping 2(a) using a baggage type detection model.

[0075] It is worth noting that in step S120, when training the baggage type detection model, the number of training loops is set to 500, and the batch-size for each training iteration is set to 64.

[0076] In one embodiment, reference is made to Figure 3 In step S300, the step of training a deep learning network using the preprocessed training dataset includes:

[0077] Step S310: Perform preprocessing of noise reduction and data augmentation on each luggage target region in the training dataset in sequence to obtain the third image set;

[0078] Step S320: Perform data augmentation preprocessing on each luggage target area in the dataset to obtain the fourth image set;

[0079] Step S330: Train the luggage color attribute recognition model: Label the third image set with color attributes, and input the labeled third image set into the first deep learning network for training;

[0080] Step S340: Train the luggage size attribute recognition model: Label the size attributes for the fourth image set, and input the labeled third image set into the second deep learning network for training;

[0081] Step S350: Train the luggage functional attribute recognition model: Label the functional color attributes for the fourth image set, and input the labeled third image set into the third deep learning network for training;

[0082] Step S360: Train the baggage attribute recognition model: Calculate the accuracy of the three trained baggage color attribute recognition models, baggage size attribute recognition models, and baggage function attribute recognition models. Select the main branch of the attribute recognition model with the highest accuracy as the main branch of the fourth deep learning network. Freeze the first n units (blocks) of the main branch and use the remaining units of the main branch to train the other two attribute recognition models again. Output a single-input three-output baggage attribute recognition model.

[0083] In one specific embodiment, the first deep learning network, the second deep learning network, the third deep learning network, and the fourth deep learning network can be, but are not limited to, the EfficientNet learning network. The color attributes include ten categories: red, black, speckled, yellow, gray, blue, green, purple, and brown; the functional attributes include carrying, daily use, and sports; and the size attributes include three categories: large, medium, and small.

[0084] In one embodiment, when the first deep learning network uses the EfficientNet learning network, and the main branch of the learning network has 16 blocks, the value of n is set to 10. For example, when baggage color attribute recognition model, baggage size attribute recognition model, and baggage function attribute recognition model are obtained by training the first, second, and third deep learning networks respectively, if the baggage color attribute recognition model has the highest accuracy among the three recognition models, then the main branch of the baggage color attribute recognition model is used as the main branch of the fourth deep learning network, and the first 10 blocks are frozen. Two of the remaining 6 blocks are used to retrain the baggage size attribute, and the other two of the remaining 6 blocks are used to retrain the baggage function attribute. It can be understood that when training the three models of size, function, and color respectively, the branch with the highest accuracy output of the evaluation metric is used as the main branch. The main purpose of the main branch is to ensure the accuracy of the overall merged network. If a branch with low accuracy is selected as the main branch and then merged with the other three branches, the accuracy of the merged network will decrease.

[0085] It is worth noting that the baggage type detection model detects baggage types including shoulder bags, backpacks, handbags, waist bags, suitcases, and tote bags. In one embodiment, in steps S200 and S310, based on the baggage type results obtained from the baggage type detection model, baggage targets belonging to the same baggage type undergo batch preprocessing to remove interference, specifically:

[0086] For the target area of ​​shoulder bag type luggage, the upper two-thirds of the luggage target area is blacked out in the X-axis direction;

[0087] For backpack-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction;

[0088] For handbag-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction;

[0089] For the luggage target area of ​​the handbag, the upper third of the luggage target area is blacked out in the X-axis direction;

[0090] For suitcase-type luggage target areas, the upper half of the luggage target area is colored black along the X-axis; and

[0091] For waist pack type luggage target areas, divide the luggage target area into four equal areas along the Y-axis, and color the first and fourth areas on the Y-axis black.

[0092] It's worth noting that the X and Y axes refer to the X and Y axes of the image coordinate system, respectively, with the origin of the image coordinate system at the top left corner. In the image processing, luggage targets are placed according to human visual perception. Thus, it's understandable that the main body of a detected shoulder bag is typically located in the lower third of the image, the main body of a detected backpack in the lower two-thirds, the main body of a detected handbag in the lower two-thirds, the main body of a detected suitcase in the lower half, and the main body of a detected waist bag in the center of the image, potentially occupying three-quarters of the image. Since color attribute recognition training doesn't require the entire luggage target, only the majority of the main body, the aforementioned interference removal process eliminates most background interference, further facilitating the training of a more accurate recognition model.

[0093] Figure 4(a) shows the target area for a shoulder bag, and Figure 4(b) shows the result of blackening two-thirds of the upper part of Figure 4(a); Figure 5(a) shows the target area for a handbag, and Figure 5(b) shows the result of blackening one-third of the upper part of Figure 5(a).

[0094] Figure 6(a) shows the target area for suitcases, and Figure 6(b) shows... Figure 6a This is a schematic diagram showing the result of blackening half of the upper region.

[0095] In one embodiment, the preprocessing step of data augmentation for the baggage target area in steps S200, S310, and S320 includes:

[0096] The size of each luggage target region in the unified training dataset;

[0097] Perform an affine transformation on each luggage target region in the training dataset;

[0098] Flip each luggage target region in the training dataset horizontally;

[0099] Each luggage target region in the training dataset is vectorized.

[0100] Normalize each channel of each baggage target area in the training dataset.

[0101] It's worth noting that the size of each baggage target area is uniformly 240*240; the `transforms.RandomAffine` function is used to perform an affine transformation on each baggage target area; the `transforms.RandomHorizontalFlip` function is used to horizontally flip each baggage target area; the `transforms.ToTensor` function is used for vector transformation; and the `transforms.Normalize` function is used for normalization. To ensure the model has sufficiently strong generalization ability, a series of data augmentation techniques are used to meet the data requirements of complex scenarios.

[0102] This application also provides a baggage attribute detection device, which includes a preparation unit for a training dataset for baggage attribute analysis, a preprocessing unit for the training dataset, and a training unit for a baggage attribute recognition model.

[0103] Specifically, the preparation unit for the training dataset of baggage attribute analysis is used to: collect a first image set containing baggage with multiple attributes, use a baggage type detection model to detect baggage type and baggage target region for each image in the first image set, and use the results of baggage target region detection as the training dataset.

[0104] Specifically, the preprocessing unit of the training dataset is used for: preprocessing the baggage target region for denoising and data augmentation;

[0105] Specifically, the training unit of the baggage attribute recognition model is used to train a deep learning network using a preprocessed training dataset to train a baggage attribute recognition model that identifies baggage function, color, and size attributes.

[0106] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the vision-based baggage attribute analysis method.

[0107] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A vision-based method for analyzing baggage attributes, characterized in that, include: Prepare the training dataset for baggage attribute analysis: Collect a first set of images containing baggage with multiple attributes, and use a baggage type detection model to detect baggage type and baggage target region for each image in the first set of images. Use the results of baggage target region detection as the training dataset. Preprocessing of the training dataset: De-interference and data augmentation preprocessing of the baggage target area; as well as Using the preprocessed training dataset, a deep learning-based network is trained to develop a luggage attribute recognition model that identifies luggage functions, colors, and sizes. This includes: Each luggage target region in the training dataset is preprocessed sequentially with denoising and data augmentation to obtain the third image set; Data augmentation preprocessing is performed on each luggage target area in the dataset to obtain the fourth image set; Training the luggage color attribute recognition model: Label the third image set with color attributes, and input the labeled third image set into the first deep learning network for training; Training the luggage size attribute recognition model: Label the size attributes for the fourth image set, and input the labeled third image set into the second deep learning network for training; Training the luggage functional attribute recognition model: Label the functional color attributes for the fourth image set, and input the labeled third image set into the third deep learning network for training; Training the baggage attribute recognition model: Calculate the accuracy of the three trained baggage color attribute recognition models, baggage size attribute recognition models, and baggage function attribute recognition models. Select the main branch of the attribute recognition model with the highest accuracy as the main branch of the fourth deep learning network. Freeze the first n units of the main branch and use the remaining units of the main branch to train the other two attribute recognition models again. Output a single-input three-output baggage attribute recognition model.

2. The vision-based baggage attribute analysis method as described in claim 1, characterized in that, In preparing the training dataset for baggage attribute analysis, Prepare training data for the baggage type detection model: Collect a second image set containing baggage with multiple attributes. Divide the second image set into training data, validation data, and test data in an 8:1:1 ratio. Label the baggage type and baggage area on the training data in the second image set. Training the baggage type detection model: The attention mechanism module is added to the starting position of each unit module of the backbone network of the Yolox model to construct the baggage type detection model; and the baggage type detection model is trained by combining the Focal Loss loss function. Baggage type and target detection: Using a trained baggage type detection model, baggage type and target detection are performed on each image in the training dataset for baggage attribute analysis. The center point of the baggage target and the detection box of the region related to the baggage target are obtained in each image in the training dataset for baggage attribute analysis. The first image set is cropped based on the center point of the baggage target and the detection box to obtain the baggage target area.

3. The vision-based baggage attribute analysis method as described in claim 2, characterized in that, Let the center point coordinates of any baggage target be (x, y), and the height and width of the detection box of the baggage target be H and W, respectively. The step of cropping the first image set based on the center point and detection box of the baggage target to obtain the baggage target region is as follows: Calculate the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2) of the coordinate frame. Use the top left and bottom right corners as two diagonal points to crop and obtain a rectangular luggage target area; in .

4. The vision-based baggage attribute analysis method as described in claim 1, characterized in that, n takes the value 10.

5. The vision-based baggage attribute analysis method as described in claim 1, characterized in that, The baggage type detection model identifies baggage types including shoulder bags, backpacks, handbags, waist bags, suitcases, and tote bags. Based on the baggage type detection model's results, baggage targets belonging to the same type undergo batch preprocessing to remove interference. Specifically: For the target area of ​​shoulder bag type luggage, the upper two-thirds of the luggage target area is blacked out in the X-axis direction; For backpack-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction; For handbag-type luggage target areas, the upper third of the luggage target area is blacked out in the X-axis direction; For the luggage target area of ​​the handbag, the upper third of the luggage target area is blacked out in the X-axis direction; For suitcase-type luggage target areas, the upper half of the luggage target area is colored black along the X-axis; and For waist pack type luggage target areas, divide the luggage target area into four equal areas along the Y-axis, and color the first and fourth areas on the Y-axis black.

6. The vision-based baggage attribute analysis method as described in claim 1, characterized in that... The preprocessing steps for data augmentation of the baggage target area include: The size of each luggage target region in the unified training dataset; Perform an affine transformation on each luggage target region in the training dataset; Flip each luggage target region in the training dataset horizontally; Each luggage target region in the training dataset is vectorized. Normalize each channel of each baggage target area in the training dataset.

7. A baggage attribute detection device, based on the vision-based baggage attribute analysis method of claim 1, characterized in that, include: The preparation unit for the training dataset of baggage attribute analysis: It is used to collect a first set of images containing baggage with multiple attributes, and to use a baggage type detection model to detect baggage type and baggage target region for each image in the first set of images, and use the results of baggage target region detection as the training dataset. The preprocessing unit for the training dataset is used for preprocessing the baggage target area to remove interference and augment data. as well as Training unit of baggage attribute recognition model: It is used to train a deep learning network using a preprocessed training dataset to train a baggage attribute recognition model that recognizes baggage function, color and size attributes.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the vision-based baggage attribute analysis method according to any one of claims 1-6.

Images