A luggage position detection method, device and equipment
By acquiring baggage image data and calculating the baggage's three-dimensional spatial position and size data, the problem of inaccurate baggage location detection in the aircraft's belly compartment was solved, enabling fast and accurate baggage retrieval and efficient operation of the robotic arm.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-03
- Publication Date
- 2026-03-27
AI Technical Summary
Densely packed and overlapping baggage in the aircraft's belly compartment can lead to inaccurate baggage location detection, affecting flight efficiency and safety.
By acquiring baggage image data, determining the keyframe image when the baggage trajectory stops, calculating baggage bounding box data and depth image, obtaining baggage depth value and size data, and finally determining the three-dimensional spatial position of the baggage.
It enables quick and accurate location determination of luggage, improves luggage retrieval efficiency, ensures that the robotic arm can accurately cover the luggage area, avoids collisions, and improves efficiency.
Smart Images
Figure CN120599039B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of luggage position detection, and further relates to a luggage position detection method, device and equipment. BACKGROUND
[0002] Before an airplane takes off, a passenger may not be able to continue boarding due to personal reasons, and needs to take away the luggage that has been placed in the belly compartment of the airplane. Quickly and accurately determining the position of the luggage is crucial to improving the efficiency and safety of flight operations. However, due to the fact that the luggage in the belly compartment of the airplane is often dense and may be blocked or overlapped with each other, the blocking between the luggage may cause some luggage to be unable to be directly observed, thereby affecting the accuracy of the luggage position detection. SUMMARY
[0003] The technical problem to be solved by the present application is to provide a luggage position detection method, device and equipment to quickly and accurately determine the position of the luggage.
[0004] To solve the above technical problem, the technical solution of the present application is as follows:
[0005] In a first aspect, the present application provides a luggage position detection method, comprising:
[0006] obtaining luggage image data;
[0007] determining a key frame image at a time when a luggage trajectory stops according to the luggage image data; the key frame image contains a time stamp;
[0008] determining luggage bounding box data and a depth image according to the key frame image;
[0009] obtaining a luggage depth value according to the luggage bounding box data and the depth image;
[0010] determining a three-dimensional spatial position of the luggage according to the luggage depth value and a preset parameter;
[0011] determining luggage size data according to the luggage bounding box data, the luggage depth value and the preset parameter;
[0012] determining luggage position data according to the three-dimensional spatial position of the luggage and the luggage size data.
[0013] Optionally, determining a key frame image at a time when a luggage trajectory stops according to the luggage image data comprises:
[0014] determining a luggage trajectory stop time according to the luggage image data and a preset speed threshold;
[0015] determining a key frame image according to the luggage image data and the luggage trajectory stop time.
[0016] Optionally, based on the keyframe image, the luggage bounding box data is determined, including:
[0017] The keyframe image is preprocessed to obtain a preprocessed image;
[0018] Feature extraction is performed on the preprocessed image to obtain an initial feature map;
[0019] Feature fusion is performed based on the initial feature map to obtain a fused feature map;
[0020] Based on the fused feature map, the baggage bounding box data is determined.
[0021] Optionally, determining a depth image based on the keyframe image includes:
[0022] Based on the keyframe image and To obtain feature data;
[0023] Based on the feature data and This yields fused feature data;
[0024] Based on the fusion feature data and Determine the depth image;
[0025] in, For position Feature data at the location, Location in keyframe image pixel values, Position in the convolution kernel The weight values, where b is the bias term. For position The fusion feature data of channel c For position Low-level feature data of channel c For position Low-level feature data of channel c Location in depth image The depth value, f is the regression function.
[0026] Optionally, the luggage depth value is obtained based on the luggage bounding box data and the depth image, including:
[0027] The depth image is cropped based on the luggage bounding box data to obtain the cropped area;
[0028] The luggage depth value is obtained based on the average depth of the cut area.
[0029] Optionally, the three-dimensional spatial position of the luggage is determined based on the luggage depth value and preset parameters, including:
[0030] acquire preset parameters; the preset parameters include a camera focal length and a principal point coordinate;
[0031] determine a three-dimensional space position of the luggage according to the luggage depth value, the preset parameters and 、 、
[0032] wherein, , ;
[0033] wherein, x is a three-dimensional space horizontal coordinate of the luggage, u is a horizontal coordinate in an image coordinate system, z is a luggage depth value, i.e., a three-dimensional space depth coordinate of the luggage, f is a camera focal length, y is a three-dimensional space vertical coordinate of the luggage, v is a vertical coordinate in the image coordinate system, is a horizontal coordinate of a pixel of a center point of a bounding box, is a vertical coordinate of a pixel of the center point of the bounding box.
[0034] Optionally, according to the luggage bounding box data, the luggage depth value and the preset parameters, luggage size data is determined, including:
[0035] determine an actual width, an actual length and an actual height of the luggage according to the luggage bounding box data, the luggage depth value, the preset parameters and 、 、
[0036] determine luggage size data according to the actual width, the actual length and the actual height of the luggage;
[0037] wherein, k is the actual width of the luggage, w is a pixel width of the bounding box, z is the luggage depth value, f is the camera focal length, g is the actual height of the luggage, and h is a pixel height of the bounding box.
[0038] A second aspect of the present application provides a luggage position detection device, including:
[0039] an acquisition module, configured to acquire luggage image data;
[0040] The processing module is configured to determine a key frame image when the luggage trajectory stops according to the luggage image data; the key frame image comprises a time stamp; determine luggage bounding box data and a depth image according to the key frame image; obtain a luggage depth value according to the luggage bounding box data and the depth image; determine a three-dimensional spatial position of the luggage according to the luggage depth value and a preset parameter; determine luggage size data according to the luggage bounding box data, the luggage depth value and the preset parameter; and determine luggage position data according to the three-dimensional spatial position of the luggage and the luggage size data.
[0041] In a third aspect, the present application provides a computing device, comprising: a processor, and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method according to the first aspect.
[0042] In a fourth aspect, the present application provides a computer readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the method according to the first aspect.
[0043] The above scheme of the present application has at least the following advantages:
[0044] The above scheme of the present application has at least the following advantages: BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 is a flowchart of the luggage position detection method in the embodiment of the present application;
[0046] Figure 2 is a structural schematic diagram of the luggage position detection device in the embodiment of the present application. DETAILED DESCRIPTION
[0047] Exemplary embodiments of the present application will be described in detail with reference to the drawings, wherein the same or similar components are designated by the same reference numerals, and the repetitive description will be omitted. In the drawings:
[0048] AsFigure 1 As shown, the embodiment of the present application proposes a luggage position detection method, comprising the following steps:
[0049] Step 101, acquiring luggage image data;
[0050] Step 102, determining a key frame image when a luggage trajectory stops according to the luggage image data; the key frame image contains a time stamp;
[0051] Step 103, determining luggage bounding box data and a depth image according to the key frame image;
[0052] Step 104, obtaining a luggage depth value according to the luggage bounding box data and the depth image;
[0053] Step 105, determining a three-dimensional spatial position of the luggage according to the luggage depth value and a preset parameter;
[0054] Step 106, determining luggage size data according to the luggage bounding box data, the luggage depth value and the preset parameter;
[0055] Step 107, determining luggage position data according to the three-dimensional spatial position of the luggage and the luggage size data.
[0056] The luggage position detection method proposed by the embodiment of the present application has the advantages that the luggage image data is acquired, the key frame image when the luggage trajectory stops is determined according to the luggage image data, the luggage bounding box data and the depth image are determined according to the key frame image, the luggage depth value is obtained according to the luggage bounding box data and the depth image, the three-dimensional spatial position of the luggage is determined according to the luggage depth value and the preset parameter, the luggage size data is determined according to the luggage bounding box data, the luggage depth value and the preset parameter, and finally the luggage position data is determined according to the three-dimensional spatial position of the luggage and the luggage size data, which is beneficial to quickly and accurately determining the luggage position and improving the luggage searching efficiency.
[0057] In an optional embodiment of the present application, step 101 comprises:
[0058] acquiring the luggage image data using a monocular camera; the luggage image data comprises a plurality of luggage images arranged in time sequence.
[0059] Specifically, the cost of acquiring the luggage image data using the monocular camera is low, the flexibility is good, and the quality of the luggage image is high; by acquiring a plurality of luggage images arranged in time sequence, the speed and trajectory of the luggage can be conveniently calculated, and the key frame and time when the luggage stops moving can be determined.
[0060] In an optional embodiment of the present application, the step of determining the key frame image at the stopping time of the luggage trajectory according to the luggage image data in step 102 comprises:
[0061] Step 1021, determining the stopping time of the luggage trajectory according to the luggage image data and a preset speed threshold value;
[0062] Specifically, the instantaneous speed of the luggage is determined according to the luggage image data and , wherein v t is the instantaneous speed of the luggage at the t th frame, is the position coordinate of the luggage at the t th frame, is the position coordinate of the luggage at the t-1 th frame, is the time interval between the two frames.
[0063] When the instantaneous speed of the luggage is less than the preset speed threshold value (such as 0.1 m / s), the stopping time of the luggage trajectory is determined. Here, when the instantaneous speed of the luggage in a certain frame image is less than the preset speed threshold value, the stopping time of the luggage trajectory can be determined according to the time stamp on the image.
[0064] Step 1022, determining the key frame image according to the luggage image data and the stopping time of the luggage trajectory.
[0065] Specifically, the image in the luggage image data with the same time stamp as the stopping time of the luggage trajectory is determined as the key frame image, which is used for subsequent determination of the luggage position.
[0066] In an optional embodiment of the present application, the step of determining the luggage bounding box data according to the key frame image in step 103 comprises:
[0067] Step 10311, pre-processing the key frame image to obtain a pre-processed image;
[0068] Specifically, the pixel values of the key frame image can be normalized according to the preset size information to improve the subsequent processing efficiency.
[0069] Step 10312, performing feature extraction on the pre-processed image to obtain an initial feature map;
[0070] Specifically, the pre-processed image can be feature-extracted using a backbone network, i.e., the pre-processed image is feature-extracted according to , wherein, is the initial feature map of the i th scale extracted by the backbone network, is the mapping function of the backbone network, is the parameter of the backbone network, and I is the pre-processed image. The initial feature map includes feature maps of different scales .
[0071] Step 10313, performing feature fusion according to the initial feature maps to obtain a fused feature map;
[0072] Specifically, according to performing feature fusion on the initial feature maps to obtain a fused feature map, wherein P is the fused feature map, is a 1x1 convolution, used to adjust the number of channels of the low-level feature maps, is an up-sampling operation, used to adjust the resolution of the high-level feature maps, is a low-level feature map in the initial feature maps, is a high-level feature map in the initial feature maps.
[0073] Step 10314, determining the luggage bounding box data according to the fused feature map.
[0074] Specifically, first, according to the fused feature map and 、 、 、 performing prediction on the bounding box parameters to obtain predicted bounding box parameters; then performing transformation on the predicted bounding box data and 、 、 、 to obtain the luggage bounding box coordinates, i.e., the luggage bounding box data.
[0075] wherein, 、 are offset amounts of the center points of the bounding boxes, is a Sigmoid function, used to limit the offset amount in the range of [0, 1], is a scaling factor of the width of the bounding box, is a scaling factor of the height of the bounding box, 、 、 、 are convolution operations, used to predict the bounding box parameters from the fused feature map, is an exponential function, used to ensure that the scaling factor is positive, and P is the fused feature map, is a horizontal coordinate of the center point of the luggage bounding box, is a vertical coordinate of the center point of the luggage bounding box, is a width of the luggage bounding box, is a height of the luggage bounding box, is a horizontal coordinate of the center point of the anchor box, is a vertical coordinate of the center point of the anchor box, is a width of the anchor box, and is a height of the anchor box.
[0076] Here, the predicted bounding box parameters include the offset of the bounding box center point, and the scaling factors for the bounding box width and height.
[0077] In an optional embodiment of the present invention, step 103, determining the depth image based on the keyframe image, includes:
[0078] Step 10321, based on the keyframe image and To obtain feature data;
[0079] Specifically, the feature data includes low-level feature maps and high-level feature maps, and the spatial dimensions (i.e., width and height) of the low-level and high-level feature maps are the same. It should be noted that i and j are the current position indices of the feature maps in the feature data, and m and n are the local indices of the convolution kernel, ranging from [...]. [k / 2,k / 2] (assuming the kernel size is odd, such as 3×3, then the kernel size is k=3), i+m and j+n are the indices in the keyframe image corresponding to the current position of the kernel.
[0080] Step 10322, based on the feature data and This yields fused feature data;
[0081] Step 10323, based on the fused feature data and Determine the depth image;
[0082] in, For position Feature data at the location, Location in keyframe image pixel values, Position in the convolution kernel The weight values, where b is the bias term. For position The fusion feature data of channel c For position Low-level feature data of channel c For position Low-level feature data of channel c Location in depth image The depth value is f, where f is the regression function, usually a linear layer or a convolutional layer, followed by an activation function (such as ReLU).
[0083] In an optional embodiment of the present invention, step 104 includes:
[0084] Step 1041: Crop the depth image based on the luggage bounding box data to obtain the cropped area;
[0085] Specifically, a corresponding region in the luggage bounding box data is cropped according to the bounding box coordinates of the luggage bounding box data, as a cropped region.
[0086] In step 1042, a luggage depth value is obtained according to the average depth of the cropped region.
[0087] Specifically, the average depth of the cropped region is obtained according to the cropped region and:
[0088] , that is, the luggage depth value. Wherein, z is the luggage depth value, is the minimum value of the top-left corner vertex of the bounding box in the horizontal direction (x-axis) of the image, is the maximum value of the bottom-right corner vertex of the bounding box in the horizontal direction (x-axis), is the minimum value of the top-left corner vertex of the bounding box in the vertical direction (y-axis) of the image, is the maximum value of the bottom-right corner vertex of the bounding box in the vertical direction (y-axis).
[0089] In an optional embodiment of the present application, step 105 includes:
[0090] In step 1051, a preset parameter is obtained; the preset parameter includes a camera focal length and a principal point coordinate;
[0091] Specifically, the theoretical focal length can be obtained through the technical specifications of the camera or the lens specification book, and the camera focal length can be calculated by shooting a known size calibration object (such as a checkerboard) and extracting the position information of the calibration object in the image, combined with geometric transformation and optimization algorithm; the principal point coordinate of the camera can be calculated by shooting the calibration object and extracting the position information of the calibration object in the image, combined with the internal parameters (such as focal length, distortion coefficient, etc.) and external parameters (such as the pose and position of the camera) of the camera.
[0092] In step 1052, a three-dimensional space position of the luggage is determined according to the luggage depth value, the preset parameter, and , ,
[0093] Wherein, ,
[0094] Wherein, x is the three-dimensional space horizontal coordinate of the luggage, u is the horizontal coordinate in the image coordinate system, z is the luggage depth value, that is, the three-dimensional space depth coordinate of the luggage, f is the camera focal length, y is the three-dimensional space vertical coordinate of the luggage, v is the vertical coordinate in the image coordinate system, is the horizontal coordinate of the center point of the bounding box, is the vertical coordinate of the center point of the bounding box.
[0095] Here, the three-dimensional spatial position of the luggage mainly includes a three-dimensional spatial horizontal coordinate of the luggage, a three-dimensional spatial vertical coordinate of the luggage, and a three-dimensional spatial depth coordinate of the luggage.
[0096] In an optional embodiment of the present application, step 106 comprises:
[0097] Step 1061, determining actual width, actual length, and actual height of the luggage according to the luggage bounding box data, the luggage depth value, the preset parameters, and 、 、 .
[0098] Step 1062, determining luggage size data according to the actual width, the actual length, and the actual height of the luggage.
[0099] wherein k is the actual width of the luggage, w is the pixel width of the bounding box, z is the luggage depth value, f is the camera focal length, g is the actual height of the luggage, and h is the pixel height of the bounding box.
[0100] Here, the luggage size data includes the actual width, the actual length, and the actual height of the luggage. Calculating the luggage size data can improve the accuracy when using a mechanical arm to grab the luggage later.
[0101] In an optional embodiment of the present application, the luggage position data in step 107 includes the three-dimensional spatial position of the luggage and the luggage size data.
[0102] Specifically, according to the determined luggage position data, it is not only convenient to search for the luggage in the aircraft belly compartment, but also the motion path of the mechanical arm can be planned according to the luggage size data in the luggage position data, so as to ensure that the mechanical arm can cover the entire range of the luggage, avoid collision, and improve efficiency.
[0103] One specific embodiment of the luggage position detection method proposed in the embodiments of the present application comprises:
[0104] Step 111, acquiring luggage image data;
[0105] A monocular camera is used to collect multiple pieces of luggage image data arranged in time sequence.
[0106] Step 112, determining a key frame image;
[0107] When the instantaneous speed of the luggage in a certain frame image is less than a preset speed threshold, the stop time of the luggage trajectory can be determined according to the timestamp on the image, and the corresponding luggage image is found according to the timestamp to determine the key frame image.
[0108] Step 113, determining luggage bounding box data and a depth image;
[0109] After preprocessing the keyframe images, feature extraction and feature fusion are used to determine the baggage bounding box data and the depth image.
[0110] Step 114: Determine the baggage depth value;
[0111] The corresponding region in the depth image is cropped out based on the baggage bounding box data, and then the average depth of the cropped region is used as the baggage depth value.
[0112] Step 115: Determine the three-dimensional spatial location of the luggage;
[0113] By calibrating the camera, the camera focal length and principal point coordinates are obtained. Then, using relevant formulas, the three-dimensional spatial coordinates of the luggage are calculated as the three-dimensional spatial position of the luggage.
[0114] Step 116: Determine the luggage dimensions;
[0115] Based on the baggage bounding box data, baggage depth value, camera focal length, and principal point coordinates, the actual width, actual length, and actual height of the baggage are calculated using relevant formulas, which serve as the baggage size data.
[0116] Step 117: Determine the luggage location data.
[0117] Baggage location data includes the baggage's three-dimensional spatial location and baggage size data.
[0118] The luggage location detection method proposed in this invention calculates the three-dimensional spatial location and size data of the luggage based on the acquired luggage image, which facilitates subsequent luggage location based on the three-dimensional spatial location and improves the search efficiency. In addition, the method can also calculate and plan the motion path of the robotic arm based on the luggage size data to ensure that the robotic arm can cover the entire range of the luggage, avoid collisions and improve efficiency.
[0119] like Figure 2 As shown, an embodiment of the present invention provides a luggage location detection device 200, comprising:
[0120] Module 201 is used to acquire baggage image data;
[0121] The processing module 202 is configured to: determine a keyframe image of the time when the luggage trajectory stops based on the luggage image data; the keyframe image includes a timestamp; determine luggage bounding box data and a depth image based on the keyframe image; obtain a luggage depth value based on the luggage bounding box data and the depth image; determine the three-dimensional spatial position of the luggage based on the luggage depth value and preset parameters; determine luggage size data based on the luggage bounding box data, the luggage depth value, and the preset parameters; and determine luggage location data based on the luggage three-dimensional spatial position and the luggage size data.
[0122] Optionally, based on the baggage image data, determine the keyframe image when the baggage trajectory stops, including:
[0123] The time when the luggage trajectory stops is determined based on the luggage image data and the preset speed threshold.
[0124] Based on the baggage image data and the baggage trajectory stopping time, keyframe images are determined.
[0125] Optionally, based on the keyframe image, the luggage bounding box data is determined, including:
[0126] The keyframe image is preprocessed to obtain a preprocessed image;
[0127] Feature extraction is performed on the preprocessed image to obtain an initial feature map;
[0128] Feature fusion is performed based on the initial feature map to obtain a fused feature map;
[0129] Based on the fused feature map, the baggage bounding box data is determined.
[0130] Optionally, determining a depth image based on the keyframe image includes:
[0131] Based on the keyframe image and To obtain feature data;
[0132] Based on the feature data and This yields fused feature data;
[0133] Based on the fusion feature data and Determine the depth image;
[0134] in, For position Feature data at the location, Location in keyframe image pixel values, Position in the convolution kernel The weight values, where b is the bias term. for a position , fusion feature data of channel c, for a position , low-level feature data of channel c, for a position , low-level feature data of channel c, for a position in a depth image, f is a regression function.
[0135] Optionally, according to the luggage bounding box data and the depth image, a luggage depth value is obtained, including:
[0136] According to the luggage bounding box data, the depth image is cropped to obtain a cropped region;
[0137] According to the average depth of the cropped region, a luggage depth value is obtained.
[0138] Optionally, according to the luggage depth value and a preset parameter, a three-dimensional spatial position of the luggage is determined, including:
[0139] A preset parameter is obtained; the preset parameter includes a camera focal length and a principal point coordinate;
[0140] According to the luggage depth value, the preset parameter, and , , , a three-dimensional spatial position of the luggage is determined;
[0141] wherein, , ;
[0142] wherein, x is a three-dimensional spatial horizontal coordinate of the luggage, u is a horizontal coordinate in an image coordinate system, z is a luggage depth value, i.e., a three-dimensional spatial depth coordinate of the luggage, f is a camera focal length, y is a three-dimensional spatial vertical coordinate of the luggage, v is a vertical coordinate in an image coordinate system, is a horizontal coordinate of a bounding box center point pixel, is a vertical coordinate of a bounding box center point pixel.
[0143] Optionally, according to the luggage bounding box data, the luggage depth value, and the preset parameter, luggage size data is determined, including:
[0144] According to the luggage bounding box data, the luggage depth value, the preset parameter, and , , , an actual width, an actual length, and an actual height of the luggage are determined;
[0145] According to the actual width, the actual length, and the actual height of the luggage, luggage size data is determined;
[0146] wherein k is the actual width of the luggage, w is the pixel width of the bounding box, z is the luggage depth value, f is the camera focal length, g is the actual height of the luggage, and h is the pixel height of the bounding box.
[0147] The luggage position detection device provided in the embodiment of the present application obtains luggage image data, determines a key frame image when a luggage track stops according to the luggage image data, then determines luggage bounding box data and a depth image according to the key frame image, obtains a luggage depth value according to the luggage bounding box data and the depth image, determines a three-dimensional space position of the luggage according to the luggage depth value and preset parameters, determines luggage size data according to the luggage bounding box data, the luggage depth value and the preset parameters, and finally determines luggage position data according to the three-dimensional space position of the luggage and the luggage size data, which is conducive to quickly and accurately determining the luggage position and improving luggage searching efficiency.
[0148] It should be noted that the device corresponds to the above method, and all implementation manners in the above method embodiments are applicable to the device embodiments and can achieve the same technical effects. The embodiments will not be described again.
[0149] The embodiment of the present application also provides a computing device, comprising a processor and a memory storing a computer program, wherein the computer program is executed by the processor to perform the method in any one of the above embodiments. All implementation manners in the above method embodiments are applicable to the device embodiments and can achieve the same technical effects. The embodiments will not be described again.
[0150] The embodiment of the present application also provides a computer readable storage medium storing instructions, wherein the instructions are executed on a computer to make the computer perform the method in any one of the above embodiments. All implementation manners in the above method embodiments are applicable to the device embodiments and can achieve the same technical effects. The embodiments will not be described again.
[0151] It should be noted that in the device and method of the present application, each component or each step can be obviously decomposed and / or recombined. These decompositions and / or recombination should be regarded as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can be naturally executed in time sequence according to the order of description, but do not need to be necessarily executed in time sequence. Some steps can be executed in parallel, cross or independently of each other.
[0152] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Exclusion of such elements is only present if it is expressly stated that these elements are excluded. Further, it is to be understood that the description of the embodiments of the present application is not limited to the order of the steps of the methods described herein and that unless otherwise specified, the steps of the methods described herein can be performed in any order. Additionally, features described with respect to certain examples can be combined in other examples.
[0153] The above description is considered that the preferred embodiments of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application described, can also make a number of improvements and refinements, these improvements and refinements should also be considered the scope of protection of the present application.
Claims
1. A method for detecting the location of luggage, characterized in that, include: Acquire baggage image data; Based on the baggage image data, determine the keyframe image when the baggage trajectory stops; The keyframe image contains a timestamp; Based on the keyframe images, determine the baggage bounding box data and depth image; The luggage depth value is obtained based on the luggage bounding box data and the depth image; The three-dimensional spatial position of the luggage is determined based on the luggage depth value and preset parameters; The luggage size data is determined based on the luggage boundary frame data, the luggage depth value, and the preset parameters; Based on the three-dimensional spatial location of the luggage and the luggage size data, determine the luggage location data; The determination of keyframe images at the point where the luggage trajectory stops, based on the luggage image data, includes: Based on luggage image data and Determine the instantaneous speed of the luggage; among which, Let be the instantaneous velocity of the luggage in frame t. Let be the coordinates of the luggage's position in frame t. Let the coordinates of the luggage be the position coordinates in frame t-1. The time interval between two frames; when the instantaneous speed of the luggage is less than a preset speed threshold, the luggage trajectory stops. Based on the luggage image data and the luggage trajectory stopping time, keyframe images are determined; The determination of luggage bounding box data based on the keyframe images includes: The keyframe image is preprocessed to obtain a preprocessed image; according to Feature extraction is performed on the preprocessed image to obtain an initial feature map; where, This is the initial feature map of the i-th scale extracted from the backbone network. The mapping function for the backbone network, is the parameter of the backbone network, and I is the preprocessed image; according to The initial feature map is fused to obtain a fused feature map, where P is the fused feature map. For 1×1 convolution, For upsampling operation, This refers to the low-level feature map in the initial feature map. This refers to the high-level feature map within the initial feature map; Based on the fusion feature map and , , , Predict the bounding box parameters to obtain the predicted bounding box parameters; based on the predicted bounding box parameters and , , , The transformation is performed to obtain the baggage bounding box coordinates, i.e., the baggage bounding box data; in, , These are all offsets of the center point of the bounding box. For the Sigmoid function, It is the bounding box width scaling factor. It is a scaling factor for the bounding box height. , , , All are convolution operations. Let P be an exponential function, and P be the fused feature map. The x-coordinate of the center point of the baggage bounding box. The ordinate of the center point of the baggage bounding box. This is the width of the luggage bounding box. The height of the luggage boundary frame. The x-coordinate of the center point of the anchor frame. The ordinate of the center point of the anchor frame. The width of the anchor frame. The height of the anchor frame; Determining the depth image based on the keyframe image includes: Based on the keyframe image and To obtain feature data; Based on the feature data and This yields fused feature data; Based on the fusion feature data and Determine the depth image; in, For position Feature data at the location, Location in keyframe image pixel values, Position in the convolution kernel The weight values, where b is the bias term. For position The fusion feature data of channel c For position Low-level feature data of channel c For position High-level feature data of channel c Location in depth image The depth value, where f is the regression function; The baggage depth value is obtained based on the baggage bounding box data and the depth image, including: The depth image is cropped based on the luggage bounding box data to obtain the cropped area; Based on the cutting area and This yields the average depth of the cut-out area, i.e., the luggage depth value; where z is the luggage depth value. This is the minimum value in the horizontal direction of the top-left vertex of the bounding box. This represents the maximum value of the bottom right vertex of the bounding box in the horizontal direction. This is the minimum value in the vertical direction of the top-left vertex of the bounding box. This represents the maximum value in the vertical direction of the bottom right vertex of the bounding box. This is the numerical matrix of the depth map corresponding to the clipped region.
2. The baggage location detection method according to claim 1, characterized in that, Based on the luggage depth value and preset parameters, the three-dimensional spatial position of the luggage is determined, including: Obtain preset parameters; the preset parameters include camera focal length and principal point coordinates; Based on the luggage depth value, the preset parameters, and , , To determine the three-dimensional spatial location of the luggage; in, , ; Where x is the horizontal coordinate of the luggage in three-dimensional space, u is the horizontal coordinate in the image coordinate system, z is the luggage depth value, i.e., the luggage's three-dimensional depth coordinate, f is the camera focal length, y is the luggage's three-dimensional vertical coordinate, and v is the vertical coordinate in the image coordinate system. The x-coordinate of the center point of the bounding box is 1 pixel. This represents the y-coordinate of the center point of the bounding box in pixels.
3. The baggage location detection method according to claim 2, characterized in that, Based on the luggage boundary frame data, the luggage depth value, and the preset parameters, the luggage size data is determined, including: Based on the luggage boundary data, the luggage depth value, and the preset parameters, , , Determine the actual width, length, and height of the luggage; Determine the luggage dimensions based on the luggage's actual width, length, and height. Where k is the actual width of the luggage, w is the pixel width of the bounding box, z is the luggage depth value, f is the camera focal length, g is the actual height of the luggage, and h is the pixel height of the bounding box.
4. A baggage location detection device, characterized in that, include: The acquisition module is used to acquire baggage image data; The processing module is used to determine the keyframe image when the luggage trajectory stops based on the luggage image data; The keyframe image contains a timestamp; based on the keyframe image, baggage bounding box data and depth image are determined; based on the baggage bounding box data and depth image, baggage depth value is obtained; based on the baggage depth value and preset parameters, the three-dimensional spatial position of the baggage is determined; based on the baggage bounding box data, baggage depth value and preset parameters, baggage size data is determined; based on the baggage three-dimensional spatial position and baggage size data, baggage location data is determined. The determination of keyframe images at the point where the luggage trajectory stops, based on the luggage image data, includes: Based on luggage image data and Determine the instantaneous speed of the luggage; among which, Let be the instantaneous velocity of the luggage in frame t. Let be the coordinates of the luggage's position in frame t. Let the coordinates of the luggage be the position coordinates in frame t-1. The time interval between two frames; when the instantaneous speed of the luggage is less than a preset speed threshold, the luggage trajectory stops. Based on the luggage image data and the luggage trajectory stopping time, keyframe images are determined; The determination of luggage bounding box data based on the keyframe images includes: The keyframe image is preprocessed to obtain a preprocessed image; according to Feature extraction is performed on the preprocessed image to obtain an initial feature map; where, This is the initial feature map of the i-th scale extracted from the backbone network. The mapping function for the backbone network, is the parameter of the backbone network, and I is the preprocessed image; according to The initial feature map is fused to obtain a fused feature map, where P is the fused feature map. For 1×1 convolution, For upsampling operation, This refers to the low-level feature map in the initial feature map. This refers to the high-level feature map within the initial feature map; Based on the fusion feature map and , , , Predict the bounding box parameters to obtain the predicted bounding box parameters; based on the predicted bounding box parameters and , , , The transformation is performed to obtain the baggage bounding box coordinates, i.e., the baggage bounding box data; in, , These are all offsets of the center point of the bounding box. For the Sigmoid function, It is a scaling factor for the bounding box width. It is a scaling factor for the bounding box height. , , , All are convolution operations. Let P be an exponential function, and P be the fused feature map. The x-coordinate of the center point of the baggage bounding box. The ordinate of the center point of the baggage bounding box. This is the width of the luggage boundary frame. The height of the luggage boundary frame. The x-coordinate of the center point of the anchor frame. The ordinate of the center point of the anchor frame. The width of the anchor frame. The height of the anchor frame; wherein, determining the depth image based on the keyframe image includes: Based on the keyframe image and To obtain feature data; Based on the feature data and This yields fused feature data; Based on the fusion feature data and Determine the depth image; in, For position Feature data at the location, Location in keyframe image pixel values, Position in the convolution kernel The weight values, where b is the bias term. For position The fusion feature data of channel c For position Low-level feature data of channel c For position High-level feature data of channel c Location in depth image The depth value, where f is the regression function; The baggage depth value is obtained based on the baggage bounding box data and the depth image, including: The depth image is cropped based on the luggage bounding box data to obtain the cropped area; Based on the cutting area and This yields the average depth of the cut-out area, i.e., the luggage depth value; where z is the luggage depth value. This is the minimum value in the horizontal direction of the top-left vertex of the bounding box. This represents the maximum value of the bottom right vertex of the bounding box in the horizontal direction. This is the minimum value in the vertical direction of the top-left vertex of the bounding box. This represents the maximum value in the vertical direction of the bottom right vertex of the bounding box. This is the numerical matrix of the depth map corresponding to the clipped region.
5. A computing device, characterized in that, include: A processor, a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method as described in any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 3.
Citation Information
Patent Citations
Luggage size identification method and device based on depth image
CN110838142A
Airport luggage management system
CN118504951A