Lockhole detection method and device based on boundary perception feature fusion and electronic equipment

Abstract

The application relates to the technical field of machine vision and intelligent port machine perception, and provides a lock hole detection method and device based on boundary perception feature fusion and electronic equipment. In view of the characteristics that the size of a container lock hole is small, the boundary is obvious, but the lock hole is easily disturbed by a complex background and similar textures, the method performs multi-scale feature extraction on a work scene graph to obtain shallow features rich in edge responses and deep features rich in semantics; the shallow features are used as boundary priors to construct a spatial gating weight graph, the deep features are modulated by space positions, the response of the deep features is enhanced at the lock hole boundary, and the response of the deep features is suppressed in the background area; the modulated deep features and the shallow features are multi-scale fused, are sent into a rotary detection head, and output rotary frame parameter information and category information matched with the direction of the lock hole. The application realizes selective enhancement of low-level details on high-level semantics through boundary perception gating, effectively suppresses interference, and significantly improves the positioning accuracy of the lock hole and the stability of direction estimation in a complex work scene.

Description

Technical Field

[0001] This application relates to the fields of machine vision and intelligent port machinery perception technology, and in particular to a keyhole detection method, device and electronic device based on boundary perception feature fusion. Background Technology

[0002] In the wave of port automation and intelligence, container handling equipment such as quay cranes and yard cranes are increasingly demanding higher precision and real-time performance in their operational sensing. During container grabbing and stacking, spreaders need to accurately identify and locate the locking holes in the corner fittings of containers to provide crucial visual input for subsequent alignment and locking. Therefore, locking hole detection is a key step in the automated loading and unloading process.

[0003] In existing technologies, there are several keyhole detection schemes based on traditional machine vision or deep learning. Traditional methods typically use manually designed features (such as gradient histograms) combined with classifiers for recognition. However, these methods have limited adaptability and poor robustness when faced with complex backgrounds, lighting variations, and diverse target poses. While some existing deep learning methods have improved recognition capabilities, their general object detection frameworks and feature fusion strategies have not been optimized for the specific challenges of keyhole detection.

[0004] Specifically, in the context of spreader camera operation, keyhole detection still faces core technical challenges: First, keyhole targets are typically very small, occupying only a few pixels in the image, resulting in sparse effective features and high detection difficulty. Second, keyholes exhibit significant directionality depending on the container's posture and imaging perspective, making it difficult for traditional horizontal rectangular bounding boxes to tightly enclose the target, easily introducing irrelevant background information and affecting positioning accuracy. Third, keyhole areas are often accompanied by strong interference such as container surface textures, rust, shadows, reflections, and stains, leading to blurred target boundary details and a high likelihood of false positives and false negatives. Finally, existing multi-scale feature fusion methods commonly used in detection models typically focus more on transmitting high-level semantic information, neglecting the response and utilization of local structural details such as boundaries and contours contained in low-level features. Therefore, current technology still lacks a detection method that can both perform direction-sensitive modeling of keyhole targets and enhance the ability to express boundary details in complex backgrounds. Summary of the Invention

[0005] In view of this, this application provides a keyhole detection method, device and electronic device based on boundary-aware feature fusion, which can solve the problem of insufficient detection accuracy and robustness in container keyhole detection due to small target size, variable orientation, complex background and blurred boundary.

[0006] This application provides a keyhole detection method, apparatus, and electronic device based on boundary-aware feature fusion. The following describes this application from multiple aspects, and the embodiments and beneficial effects described below can be referenced interchangeably.

[0007] Firstly, this application provides a keyhole detection method based on boundary-aware feature fusion, comprising:

[0008] Obtain operational scene diagrams containing lock holes of containers in container grabbing or stacking scenarios;

[0009] Feature extraction is performed on the work scene image to obtain shallow and deep features corresponding to the keyhole. The shallow features include at least one of edge, texture and local detail information, while the deep features have stronger semantic expressive power than the shallow features.

[0010] Based on shallow features, a gating weight map is obtained. The numerical distribution of the gating weight map reflects the degree of boundary salience at different spatial locations in the work scenario map.

[0011] Modulated deep features are obtained by modulating deep features based on gating weight graphs;

[0012] Shallow features and modulated deep features are fused to obtain multi-scale fused features;

[0013] By using multi-scale fusion features as input data for the rotating detection head, the category information and rotation box parameter information corresponding to the keyhole are obtained, thereby determining the position of the keyhole.

[0014] According to the embodiments of this application, the above-described technical solution of this application has at least the following beneficial effects:

[0015] By rotating the detection head to output rotating frame parameter information that matches the actual direction of the keyhole and the corresponding category information of the keyhole, the defects of traditional horizontal rectangular frames that introduce a lot of irrelevant background information are completely avoided, and the spatial positioning accuracy of the keyhole is significantly improved.

[0016] Meanwhile, by adaptively generating a gated weight map from shallow features rich in local structural information such as boundary contours, the deep semantic features are precisely modulated, which effectively enhances the model's ability to respond to target boundary details and suppresses strong background interference such as box surface texture, rust, shadow, reflection and stains.

[0017] Then, the shallow high-resolution detail features are fused with the deep strong semantic features modulated by boundary awareness at multiple scales. This multi-scale fused features have both high-level semantic discrimination ability and low-level fine spatial expression ability, effectively overcoming the detection problem of sparse effective features of small-scale keyholes and reducing false detection and false negative rates.

[0018] In one possible implementation of the first aspect above, a gated weight map is obtained based on shallow features, including:

[0019] Local smoothing operations are performed on shallow features to obtain smoothed features;

[0020] Boundary response features are obtained based on smooth features and shallow features;

[0021] The shallow features and boundary response features are concatenated to obtain the concatenated features;

[0022] The concatenated features are convolved and mapped to obtain the mapped features.

[0023] By using the mapped features as input data for a preset activation function, a gating weight map is obtained.

[0024] According to the implementation of this application, noise interference in shallow features is removed by local smoothing operation, and the boundary response features of the keyhole are accurately extracted based on the difference between the smoothed features and the original shallow features. After being concatenated with the original shallow features, the gated weight map is generated by convolution mapping and activation function processing. This not only significantly enhances the perception ability of the gated weight map of keyhole boundary contours and other key local structures, but also adaptively highlights the target area and suppresses background interference areas such as box surface texture and rust.

[0025] In one possible implementation of the first aspect above, boundary response features are obtained based on smooth features and shallow features, including:

[0026] The boundary response feature is obtained by taking the absolute value of the difference between the smooth feature and the shallow feature.

[0027] According to the implementation method of this application, the boundary response features are obtained by taking the absolute value of the difference between the smooth features and the shallow features. The operation logic is simple and intuitive, and the amount of computation is extremely low, which can fully meet the real-time requirements of keyhole detection in port automated loading and unloading scenarios.

[0028] In one possible implementation of the first aspect above, the deep features are modulated based on the gated weight graph to obtain modulated deep features, including:

[0029] The modulated deep features are obtained by performing element-wise multiplication between the gated weight map and the deep features.

[0030] According to the implementation method of this application, feature modulation is achieved by performing element-wise multiplication between the gated weight map and the deep features. The operation method is simple and efficient, does not significantly increase the computational load of the model, and can ensure the real-time requirements of the keyhole detection system. At the same time, this method can use the gated weights generated based on boundary awareness to adaptively spatially weight the deep semantic features, accurately enhance the feature response of the keyhole target area, especially the boundary contour area, and effectively suppress the feature expression of background interference areas such as box surface texture, rust, and shadows, so that the modulated deep features are more focused on the key information of the keyhole.

[0031] In one possible implementation of the first aspect above, shallow features and deep modulation features are fused to obtain multi-scale fused features, including:

[0032] Projecting and mapping shallow features yields shallow mapped features;

[0033] Projection mapping is performed on the modulation deep features to obtain deep mapping features;

[0034] Shallow and deep mapping features are concatenated along the channel dimension to obtain multi-scale fused features.

[0035] According to the embodiments of this application, by projecting and mapping shallow features and modulation deep features respectively, the feature dimensions and number of channels of the two can be unified, eliminating the dimension mismatch problem during feature fusion, while effectively compressing redundant information and reducing subsequent computational overhead.

[0036] In one possible implementation of the first aspect above, feature extraction is performed on the work scene map to obtain shallow and deep features corresponding to the keyhole, including:

[0037] By using a pre-defined neural network model, features are extracted from the work scene image to obtain shallow and deep features.

[0038] According to the embodiments of this application, by using a preset neural network model for feature extraction, shallow features with high spatial resolution and rich local details can be automatically and efficiently learned and separated from the work scene map, thus improving the efficiency and accuracy of feature extraction.

[0039] Secondly, this application provides a keyhole detection device based on boundary-aware feature fusion, comprising:

[0040] The data acquisition module is used to acquire operational scene images containing lock holes of containers in container grabbing or stacking scenarios;

[0041] The feature extraction module is used to extract features from the work scene image to obtain shallow features and deep features corresponding to the keyhole. The shallow features include at least one of edge, texture and local detail information, while the deep features have stronger semantic expressive power than the shallow features.

[0042] The weight calculation module is used to obtain a gated weight map based on shallow features. The numerical distribution of the gated weight map reflects the degree of boundary salience at different spatial locations in the task scenario map.

[0043] The modulation module is used to modulate deep features based on the gated weight map to obtain modulated deep features;

[0044] The feature fusion module is used to fuse shallow features and deep modulation features to obtain multi-scale fused features;

[0045] The detection module is used to take multi-scale fused features as input data for the rotating detection head to obtain category information and rotation box parameter information corresponding to the keyhole, and then determine the position of the keyhole.

[0046] Thirdly, this application provides an electronic device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the keyhole detection method based on boundary-aware feature fusion as disclosed in the first aspect and any possible implementation thereof.

[0047] Fourthly, this application provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the keyhole detection method based on boundary-aware feature fusion as disclosed in the first aspect and any possible implementation thereof.

[0048] Fifthly, this application provides a computer program product comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the keyhole detection method based on boundary-aware feature fusion as disclosed in the first aspect and any possible implementation thereof.

[0049] The beneficial effects of the second to fifth aspects can be found in the first aspect and the beneficial effects of any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0050] Figure 1 This is a flowchart of the keyhole detection process in an embodiment of this application;

[0051] Figure 2This is a comparison diagram of the detection boxes in the embodiments of this application;

[0052] Figure 3 This is a flowchart of the gating weight graph calculation in the embodiments of this application;

[0053] Figure 4 This is a flowchart illustrating the feature fusion process in an embodiment of this application.

[0054] Figure 5 This is a schematic diagram of the overall framework of the keyhole detection model in the embodiments of this application;

[0055] Figure 6 This is a flowchart illustrating the processing flow of the boundary-aware semantic modulation module in this application embodiment.

[0056] Figure 7 This is a frame diagram of the keyhole detection device in the embodiments of this application;

[0057] Figure 8 This is a block diagram of the electronic device in the embodiments of this application;

[0058] Figure 9 This is a block diagram of a system-on-chip (SoC) in the embodiments of this application.

[0059] Figure label:

[0060] 1. Horizontal detection box; 2. Rotated detection box; 3. Keyhole; 4. Vertex; P2. Shallow feature; P3. First deep feature; P4. Second deep feature; P5. Third deep feature; 1000. Keyhole detection device; 1010. Data acquisition module; 1020. Feature extraction module; 1030. Weight calculation module; 1040. Modulation module; 1050. Feature fusion module; 1060. Detection module. Detailed Implementation

[0061] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0062] The technical problems to be solved by the embodiments of this application will be described below.

[0063] As described in the background section, traditional machine vision methods rely on manually designed gradient histograms and other features combined with classifiers for recognition. This approach has limited adaptability and poor robustness in scenarios with complex backgrounds, varying lighting, and diverse target poses. While some implementations utilize deep learning methods to improve recognition capabilities, they employ general object detection frameworks and feature fusion strategies without addressing the specific challenges of keyhole detection. Specifically, this manifests as difficulty in effectively extracting sparse, effective features from small-scale keyholes. For instance, traditional horizontal bounding boxes cannot tightly enclose directional keyhole targets, easily introducing irrelevant backgrounds. Furthermore, they struggle to resist strong interference from surface textures, rust, shadows, reflections, and stains, leading to false positives and false negatives.

[0064] Furthermore, existing multi-scale feature fusion methods focus more on the transmission of high-level semantic information, and are insufficient in responding to and utilizing the details of local structures such as boundaries and contours contained in low-level features. Ultimately, this results in low positioning accuracy and weak anti-interference ability of keyhole detection, which cannot meet the high requirements of port automated loading and unloading equipment for operational perception accuracy and real-time performance.

[0065] Therefore, to address the aforementioned issues, this application provides a keyhole detection method based on boundary-aware feature fusion. This method involves acquiring an operational scene image containing container keyholes in a port container grabbing or stacking scenario, and extracting features from it to obtain shallow and deep features. A gating weight map is then generated based on the shallow features, and the deep features are modulated. Finally, the shallow features and the modulated deep features are fused to obtain a multi-scale fused feature. This fused feature is then input into a rotating detection head to output the keyhole category information and rotation box parameter information.

[0066] This method combines a boundary-aware feature fusion mechanism with rotation detection technology. It addresses the problem of insufficient localization accuracy caused by the inability of traditional horizontal rectangular bounding boxes to tightly enclose directional keyhole targets, while also enhancing the model's responsiveness to keyhole boundary details through gating weight graph modulation. This method not only suppresses interference from complex backgrounds such as box surface textures, rust, and shadows, but also combines high-level semantic discrimination capabilities with low-level fine-grained spatial representation capabilities through multi-scale fusion features. This effectively overcomes the challenge of detecting sparse effective features for small-scale keyholes, reducing false positives and false negatives.

[0067] To better understand the keyhole detection method based on boundary-aware feature fusion according to the embodiments of this application, the following description is provided in conjunction with the appendix to the specification. Figure 1 —6. The solutions corresponding to the methods in the embodiments of this application will be described in detail.

[0068] Reference Figure 1 , Figure 1 This is a flowchart of the keyhole detection process in an embodiment of this application.

[0069] like Figure 1 As shown, a keyhole detection method based on boundary-aware feature fusion in this application embodiment includes steps S10-S60.

[0070] S10 involves acquiring a scene diagram of the operation containing lock holes of containers in a container grabbing or stacking scenario (corresponding to...). Figure 1 S1 in the middle.

[0071] One way to obtain images of the work scene is by capturing images from on-site cameras.

[0072] S20, perform feature extraction on the work scene map to obtain shallow and deep features corresponding to the keyhole (corresponding to...). Figure 1 (S2 in the middle).

[0073] In some embodiments, a preset neural network model is used to extract features from the work scene map to obtain shallow features and deep features.

[0074] For example, the neural network models mentioned above can be the Residual Network (ResNet) series, VGGNet, or more modern Transformer architectures. These models, pre-trained on large image datasets (such as ImageNet), already possess powerful general feature extraction capabilities.

[0075] It should be noted that shallow features are those with high spatial resolution and rich local details but low semantic abstraction, such as edges, textures, and local details. Deep features, on the other hand, are those with strong semantic expressive power but low spatial resolution. They have a larger receptive field, enabling them to capture more global and abstract semantic information. Their strong expressive power is beneficial for target classification.

[0076] The structure and training methods of the aforementioned neural network model can be referenced from existing technologies, and will not be elaborated upon in detail here.

[0077] S30, based on shallow features, obtains the gated weight map (corresponding to...). Figure 1 (S3 in the middle).

[0078] Understandably, since a keyhole is a structured target, its boundary is one of its most prominent features, and shallow features precisely preserve this high-frequency information. Therefore, by using shallow features, it is possible to effectively identify which regions may contain the target's boundary, thereby generating a gated weight map with high response values ​​in these regions. This solves the problem in existing methods where deep semantic information and shallow detail information are simply added or concatenated, leading to the dilution or neglect of boundary information.

[0079] S40, based on the gated weight map, the deep features are modulated to obtain the modulated deep features (corresponding to...). Figure 1(S4 in the middle).

[0080] In some embodiments, the gated weight map is multiplied element-wise with the deep features to obtain the modulated deep features.

[0081] Understandably, while deep features are rich in semantic information, they may also introduce significant background noise due to their large receptive field, such as the texture of the box around the keyhole or rust. Each value in the gating weight map (ranging from 0 to 1) acts as a regulator, directly affecting the feature vector at the corresponding spatial location in the deep feature map. If the gating weight value is close to 1, it means that the location is likely a target boundary, and the deep semantic features at that location will be fully preserved or enhanced. Conversely, if the weight value is close to 0, it means that the location is a flat background area, and the deep semantic features at that location will be effectively suppressed.

[0082] Through this element-wise multiplication modulation, the embodiments of this application can modulate high-level semantics with pixel-level or feature unit-level precision by utilizing low-level details. This results in the final modulated deep features retaining strong semantic discriminative power while greatly reducing background noise interference, making feature representation purer and more efficient.

[0083] S50 fuses shallow features and modulated deep features to obtain multi-scale fused features (corresponding to...). Figure 1 (S5 in the middle).

[0084] Understandably, after the previous modulation step, the deep features have focused on the target itself. At this point, fusing them with the shallow features that retain the most original and accurate spatial location information yields a powerful feature representation that combines high-level semantic discrimination capabilities with low-level precise localization capabilities. This fusion method is not a simple feature superposition, but rather a carefully considered guided and modulated process, enabling the generation of fused features particularly effective for small targets like keyholes. This solves the problem of insufficient fusion of semantic and spatial information in traditional feature pyramid networks.

[0085] S60 uses the multi-scale fused features as input data for the rotating detection head to obtain category information and rotation box parameter information corresponding to the keyhole, thereby realizing the detection of the keyhole (corresponding to...). Figure 1 (S6 in the middle).

[0086] It should be noted that the aforementioned rotating frame parameters may include: keyhole center position parameters, dimensional parameters, and orientation parameters. The process for calculating the category information corresponding to the keyhole and the rotating frame parameters using the rotating detection head can be found in existing technologies and will not be elaborated upon here.

[0087] refer to Figure 2 , Figure 2A comparison diagram of the detection boxes in the embodiments of this application is shown.

[0088] like Figure 2 As shown, both the horizontal detection box 1 and the rotating detection box 2 have four vertices 4. When the keyhole 3 is tilted, the horizontal detection box 1 needs to be expanded to the rectangular area of ​​the tilted contour of the keyhole 3 to completely enclose it, resulting in a large amount of irrelevant background area within the box. In contrast, the rotating detection box 2 can adaptively adjust the boundary angle according to the direction of the keyhole 3, so that its two sets of opposite sides are aligned with the major and minor axes of the keyhole 3, respectively, closely fitting the contour of the keyhole 3, and significantly reducing the irrelevant background area within the box.

[0089] Understandably, due to the potential tilting of the container during lifting, the keyhole may appear at an arbitrary angle in the image. The horizontal detection frame 1 does not carry directional information about the keyhole 3; its width and height are merely the projected lengths of the keyhole 3 on the image coordinate axes, not its own major and minor axis dimensions, making it difficult to provide a complete pose reference for downstream alignment operations. However, by using a rotating detection head, a rotating detection frame 2 that matches the actual orientation of the keyhole 3 can be directly predicted. Its center, long side, short side, and rotation angle correspond to the actual center, major and minor axis dimensions, and orientation of the keyhole 3, respectively. This not only tightly fits the contour of the keyhole 3 to suppress background interference but also provides a directly usable pose reference for downstream alignment operations. By inputting the high-quality fusion features generated in the preceding steps into the rotating detection head, high-precision and robust detection of the keyhole 3 in complex scenes is finally achieved.

[0090] To facilitate understanding of the embodiments of this application, steps S30 and S50 will be described in detail below with reference to the accompanying drawings.

[0091] refer to Figure 3 , Figure 3 A flowchart illustrating the gating weight graph calculation process in an embodiment of this application is shown.

[0092] like Figure 3 As shown, in some embodiments, a gating weight map is obtained based on shallow features, including steps S31-S35.

[0093] S31 performs local smoothing operations on shallow features to obtain smoothed features.

[0094] It should be noted that the above local smoothing operation can be average pooling or Gaussian blur, the purpose of which is to blur some subtle textures while preserving the overall features of the keyhole area.

[0095] S32, based on smooth features and shallow features, yields boundary response features.

[0096] In some embodiments, the boundary response features are obtained by taking the absolute value of the difference between the smooth features and the shallow features.

[0097] It should be noted that the above operation is similar to edge detection, which can effectively highlight areas where the intensity changes drastically in the feature map, i.e., potential object boundaries.

[0098] S33, the shallow features and boundary response features are concatenated to obtain the concatenated features.

[0099] In some embodiments, shallow features and boundary response features are concatenated along the channel dimension to obtain the concatenated features.

[0100] It should be noted that combining the original detailed information with the enhanced boundary information provides richer input for subsequent weight generation. The splicing process can be referenced from existing techniques, and will not be elaborated upon in this article.

[0101] S34, perform convolution mapping on the spliced ​​features to obtain the mapped features.

[0102] It should be noted that one or more 1x1 or 3x3 convolutional layers are typically used to learn a non-linear mapping from concatenated features to gated weights, and the number of output channels can be adjusted to obtain the mapped features.

[0103] S35, using the mapped features as input data for a preset activation function, obtains a gated weight map.

[0104] For example, the activation function mentioned above can be the sigmoid function or other activation functions; this article does not limit the activation function.

[0105] Understandably, by extracting boundary responses and then learning weights through concatenation and convolution mapping, the resulting gated weight map can accurately reflect the contour information of targets such as keyholes in the image. Compared to implicitly learning attention, this explicit boundary-oriented approach allows the model to focus more on the structure of the keyhole itself, rather than similar but unrelated textures around it (such as text or stains on a box). This provides high-quality, physically meaningful guiding signals for subsequent deep feature modulation, enhancing the method's robustness against interference in complex backgrounds.

[0106] refer to Figure 4 , Figure 4 A flowchart illustrating the feature fusion process in an embodiment of this application is shown.

[0107] In some embodiments, shallow features and modulated deep features are fused to obtain multi-scale fused features, including steps S51-S53.

[0108] S51 involves projecting and mapping the shallow features to obtain the shallow mapped features.

[0109] like Figure 4 As shown, step S51 above corresponds to step S51 in the figure, which performs projection mapping on shallow features.

[0110] For example, projection mapping is implemented through a 1x1 convolutional layer. Its main function is to adjust the number of channels in the feature map to unify them to a preset dimension, so as to facilitate subsequent stitching operations. At the same time, it can also perform linear combination between channels, which plays the role of feature integration.

[0111] S52, project and map the modulation deep features to obtain deep mapping features.

[0112] like Figure 4 As shown, step S52 above corresponds to step S52 in the figure, which performs projection mapping on the deep modulation features.

[0113] It is understandable that this step can refer to step S51 above, and will not be repeated here.

[0114] S53 concatenates shallow and deep mapping features along the channel dimension to obtain multi-scale fusion features.

[0115] like Figure 4 As shown, step S53 above corresponds to step S53 in the figure, which concatenates the shallow mapping features and the deep mapping features.

[0116] After obtaining shallow and deep mapping features with consistent channel dimensions, they are concatenated along the channel dimension. Unlike element-wise addition, this concatenation operation stacks the channels of the two feature maps together, forming a new feature map with a deeper number of channels. For example, concatenating a 64-channel shallow mapping feature and a 64-channel deep mapping feature results in a 128-channel multi-scale fused feature.

[0117] The overall process of this application will be described in detail below with reference to the accompanying drawings.

[0118] refer to Figure 5 , Figure 5 A schematic diagram of the overall framework of the keyhole detection model in an embodiment of this application is shown.

[0119] like Figure 5As shown, the keyhole detection model mainly includes a backbone network, a boundary-aware feature pyramid fusion (BAPF) module, and an oriented boundingbox (OBB). The backbone network is the aforementioned neural network model, and the boundary-aware feature pyramid fusion module includes multiple boundary-aware semantic modulation (BASM) modules.

[0120] The operational scene image containing the lock holes of the container is used as the input image of the backbone network. After feature extraction by the backbone network, features of different resolutions are obtained, including shallow feature P2, first deep feature P3, second deep feature P4, and third deep feature P5. The shallow feature P2, first deep feature P3, second deep feature P4, and third deep feature P5 are then fused from top to bottom or bottom to top.

[0121] Taking top-down fusion as an example, the third deep feature P5 and the second deep feature P4 are used as input data for a BASM. After fusion, the first fused feature is obtained. Then, the first fused feature is fused with the first deep feature P3 into another BASM to obtain the second fused feature. Subsequently, the second fused feature is fused with the shallow feature P2 into another BASM to obtain the third fused feature.

[0122] The third and second fusion features are input into a BASM (Balanced Scale Analyzer) for fusion to obtain the fourth fusion feature. This fourth fusion feature is then fused with the first fusion feature into another BASM for fusion to obtain the fifth fusion feature. Finally, the third, fourth, and fifth fusion features are used as multi-scale fusion features and input into the OBB (On-Board Detection) head to obtain the aforementioned category information and bounding box parameter information.

[0123] The following section provides a detailed explanation of the specific processing steps of the boundary-aware semantic modulation module.

[0124] refer to Figure 6 , Figure 6 A flowchart illustrating the processing flow of the boundary-aware semantic modulation module in an embodiment of this application is shown.

[0125] like Figure 6As shown, taking the fusion process of the aforementioned shallow feature P2 and the second fusion feature as an example, the boundary response (i.e., the aforementioned smooth feature) is first obtained based on the low-resolution feature (i.e., the aforementioned shallow feature P2). After concatenating the low-resolution feature and the boundary response, the input is fed into a gating network, which calculates the gating weight map. The gating network includes two 1×1 convolutional kernels (i.e., the 1×1 conv in the figure), a SiLU activation function, and a sigmoid activation function.

[0126] Subsequently, based on the gated weight map, the low-resolution features and high-resolution features (i.e., the second fusion features mentioned above) are semantically modulated to obtain the modulated deep features. Then, the low-resolution features and the modulated deep features are fused to obtain the fused features (i.e., the third fusion features mentioned above).

[0127] It should be noted that the process of fusing other features using BASM can be found above, and will not be elaborated upon here.

[0128] refer to Figure 7 , Figure 7 A frame diagram of the keyhole detection device in an embodiment of this application is shown.

[0129] like Figure 7 As shown, a keyhole detection device 1000 based on boundary-aware feature fusion in this embodiment of the application includes a data acquisition module 1010, a feature extraction module 1020, a weight calculation module 1030, a modulation module 1040, a feature fusion module 1050, and a detection module 1060.

[0130] Among them, the data acquisition module 1010 is used to acquire the operation scene diagram containing the lock holes of the container in the container grabbing or stacking scenario;

[0131] The feature extraction module 1020 is used to extract features from the work scene image to obtain shallow features and deep features corresponding to the keyhole. The shallow features include at least one of edge, texture and local detail information, and the semantic expression ability of the deep features is stronger than that of the shallow features.

[0132] The weight calculation module 1030 is used to obtain a gated weight map based on shallow features. The numerical distribution of the gated weight map reflects the degree of boundary salience at different spatial locations in the work scene map.

[0133] The modulation module 1040 is used to modulate deep features based on the gated weight map to obtain modulated deep features;

[0134] The feature fusion module 1050 is used to fuse shallow features and modulated deep features to obtain multi-scale fused features;

[0135] The detection module 1060 is used to take the multi-scale fused features as input data for the rotating detection head to obtain the category information and rotation box parameter information corresponding to the keyhole, and then determine the position of the keyhole.

[0136] In the apparatus of this application embodiment, each module executes the method of the above embodiment, and its specific functions and corresponding technical effects can be referred to the above embodiment. Figures 1-6 The methods explained will not be repeated here.

[0137] This application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. When the processor loads and executes the instruction or program, the electronic device performs a keyhole detection method based on boundary-aware feature fusion as described in the above embodiments. The specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-6 The keyhole detection method based on boundary-aware feature fusion explained earlier will not be elaborated here.

[0138] The following is combined Figure 8 The electronic devices described in this application will be described in detail.

[0139] refer to Figure 8 The diagram shows a block diagram of an electronic device 1200 according to one embodiment of this application. The electronic device 1200 may include one or more processors 1201 coupled to a controller hub 1203. In at least one embodiment, the controller hub 1203 communicates with the processor 1201 via a multi-branch bus such as a front side bus (FSB) 1210, a point-to-point interface such as a quick path interconnect (QPI), or a similar connection. The processor 1201 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 1203 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0140] Electronic device 1200 may also include a coprocessor 1202 and a memory 1204 coupled to a controller hub 1203. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in this application), with memory 1204 and coprocessor 1202 directly coupled to processor 1201 and controller hub 1203, which resides on a single chip with the IOH. Memory 1204 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, coprocessor 1202 is a dedicated processor, such as, for example, a high-throughput MIC (many integerized core) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of coprocessor 1202 are indicated by dashed lines. Figure 8 middle.

[0141] As a computer-readable storage medium, memory 1204 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDDs), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0142] In one embodiment, electronic device 1200 may further include a network interface controller (NIC) 1206. Network interface 1206 may include a transceiver for providing a radio interface for electronic device 1200 to communicate with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, network interface 1206 may be integrated with other components of electronic device 1200. Network interface 1206 can implement the functions of the communication unit in the above embodiments.

[0143] Electronic device 1200 may further include input / output (I / O) device 1205. I / O device 1205 may include: a user interface designed to enable a user to interact with electronic device 1200; a peripheral component interface designed to enable peripheral components to also interact with electronic device 1200; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 1200.

[0144] It is worth noting that, Figure 8 This is merely an example. That is, although... Figure 8 The electronic device 1200 shown includes multiple devices such as a processor 1201, a coprocessor 1202, a controller hub 1203, and a memory 1204. However, in practical applications, devices using the methods of this application may include only a portion of the devices in the electronic device 1200. For example, it may include only the processor 1201 and the network interface 1206. Figure 8 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 1204, which is a computer-readable storage medium, stores instructions or programs that, when executed on a computer, perform a keyhole detection method based on boundary-aware feature fusion as described in the above embodiments. Specific details can be found in the methods described in the above embodiments, and will not be repeated here.

[0145] Now for reference Figure 9 The diagram shown is a block diagram of a system-on-chip (SoC) 1300 according to an embodiment of this application. Figure 9 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 9 In this SoC 1300, the following are included: an interconnect unit 1350 coupled to an application processor 1310; a system proxy unit 1380; a bus controller unit 1390; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0146] The static random access memory (SRAM) cell 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1300 to perform a keyhole detection method based on boundary-aware feature fusion according to the above embodiments, the specific method of which can be referred to in the above embodiments and will not be repeated here.

[0147] This application provides a computer-readable storage medium storing at least one instruction or at least one program. The instruction or program is loaded and executed by a processor to implement the keyhole detection method based on boundary-aware feature fusion described in the above embodiments. Its specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-6 The keyhole detection method based on boundary-aware feature fusion explained earlier will not be elaborated here.

[0148] This application provides a computer program product, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device implements a keyhole detection method based on boundary-aware feature fusion as described in the above embodiments. Its specific functions and corresponding technical effects can be found in the above embodiments. Figures 1-6 The keyhole detection method based on boundary-aware feature fusion explained earlier will not be elaborated here.

[0149] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0150] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0151] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0152] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, and digital signals) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0153] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0154] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0156] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A keyhole detection method based on boundary-aware feature fusion, characterized in that, include: Obtain operational scene diagrams containing lock holes of containers in container grabbing or stacking scenarios; Feature extraction is performed on the work scene image to obtain shallow features and deep features corresponding to the keyhole. The shallow features include at least one of edge, texture and local detail information, and the deep features have stronger semantic expressive power than the shallow features. Based on the shallow features, a gating weight map is obtained, and the numerical distribution of the gating weight map reflects the degree of boundary salience at different spatial locations in the work scenario map; The deep features are modulated based on the gated weight map to obtain modulated deep features; The shallow features and the modulated deep features are fused to obtain multi-scale fused features; The multi-scale fusion features are used as input data for the rotating detection head to obtain category information and rotation frame parameter information corresponding to the keyhole, thereby determining the position of the keyhole.

2. The method according to claim 1, characterized in that, The process of obtaining the gated weight map based on the shallow features includes: Perform local smoothing operations on the shallow features to obtain smoothed features; Based on the smoothing features and the shallow features, the boundary response features are obtained; The shallow features and the boundary response features are concatenated to obtain the concatenated features; Perform convolution mapping on the spliced ​​features to obtain the mapped features; The mapping features are used as input data for a preset activation function to obtain the gated weight map.

3. The method according to claim 2, characterized in that, The boundary response features obtained based on the smooth features and the shallow features include: The boundary response feature is obtained by taking the absolute value of the difference between the smooth feature and the shallow feature.

4. The method according to claim 1, characterized in that, The process of modulating the deep features based on the gated weight map to obtain modulated deep features includes: The modulation deep features are obtained by performing element-wise multiplication between the gated weight map and the deep features.

5. The method according to claim 1, characterized in that, The process of fusing the shallow features and the modulated deep features to obtain multi-scale fused features includes: The shallow features are projected and mapped to obtain shallow mapped features; The deep modulation features are projected and mapped to obtain deep mapping features; The shallow mapping features and the deep mapping features are concatenated along the channel dimension to obtain the multi-scale fusion features.

6. The method according to claim 1, characterized in that, The step of extracting features from the work scene image to obtain shallow and deep features corresponding to the keyhole includes: By using a preset neural network model, features are extracted from the work scene image to obtain the shallow features and the deep features.

7. A keyhole detection device based on boundary-aware feature fusion, characterized in that, include: The data acquisition module is used to acquire operational scene images containing lock holes of containers in container grabbing or stacking scenarios; The feature extraction module is used to extract features from the work scene image to obtain shallow features and deep features corresponding to the keyhole. The shallow features include at least one of edge, texture and local detail information, and the deep features have stronger semantic expressive power than the shallow features. The weight calculation module is used to obtain a gated weight map based on the shallow features. The numerical distribution of the gated weight map reflects the degree of boundary salience at different spatial locations in the work scenario map. A modulation module is used to modulate the deep features based on the gated weight map to obtain modulated deep features; The feature fusion module is used to fuse the shallow features and the modulation deep features to obtain multi-scale fused features; The detection module is used to take the multi-scale fused features as input data for the rotating detection head to obtain category information and rotation frame parameter information corresponding to the keyhole, and then determine the position of the keyhole.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the keyhole detection method based on boundary-aware feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the keyhole detection method based on boundary-aware feature fusion as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the keyhole detection method based on boundary-aware feature fusion as described in any one of claims 1 to 6.

Images