Object recognition method, apparatus, electronic device, storage medium, and program product

Abstract

The application provides an object recognition method and device, electronic equipment, storage medium and program product, which can be applied to the technical field of computer vision and object detection. The method comprises the following steps: extracting feature information and local position information of a plurality of object parts from an input image; enhancing the plurality of feature information respectively according to the correlation between the feature information of the plurality of object parts, to obtain enhanced feature information; combining the plurality of enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object comprises object parts corresponding to the plurality of enhanced feature information in the feature set; and determining the object type and overall position information of the complete object corresponding to each of the at least one feature set according to the at least one feature set and the local position information of the plurality of object parts.

Description

Technical Field

[0001] This application relates to the field of computer vision and object detection technology, and more specifically, to an object recognition method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Object detection is a crucial support for ensuring the safety and normal operation of specific scenarios (such as security checks, industrial production, and medical settings). Accurate identification in object detection can avoid security risks, production errors, or diagnostic biases caused by missed detections. For object detection, related technologies mainly determine the presence of a target by detecting the overall appearance features of the object (such as shape and texture).

[0003] However, relying on the overall appearance features of an object is difficult to deal with scenarios where the overall appearance of the target has no uniform pattern (such as object camouflage, diverse packaging forms, and the emergence of new forms); when the overall appearance of the target is irregular, it is difficult to effectively identify it by relying solely on overall features, which can easily lead to missed detections and fail to meet actual detection needs. Summary of the Invention

[0004] In view of this, this application provides an object identification method, apparatus, electronic device, storage medium, and program product.

[0005] One aspect of this application provides an object recognition method, comprising: extracting feature information and local location information of multiple object components from an input image; enhancing the multiple feature information according to the correlation between the feature information of the multiple object components to obtain enhanced feature information; combining the multiple enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, the complete object including the object components corresponding to the multiple enhanced feature information in the feature set; and determining the object type and overall location information of the complete object corresponding to each of the at least one feature set based on the at least one feature set and the local location information of the multiple object components.

[0006] Another aspect of this application provides an object recognition device, comprising: an extraction module for extracting feature information and local location information of multiple object components from an input image; an enhancement module for enhancing the multiple feature information according to the correlation between the feature information of the multiple object components to obtain enhanced feature information; a combination module for combining the multiple enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object includes the object components corresponding to the multiple enhanced feature information in the feature set; and a determination module for determining the object type and overall location information of the complete object corresponding to each of the at least one feature set, based on the at least one feature set and the local location information of the multiple object components.

[0007] Another aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0008] Another aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0009] Another aspect of this application provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0010] According to embodiments of this application, by extracting feature information and local location information of multiple object components and performing enhancement processing based on the correlation between different feature information, a feature set corresponding to the complete object can be dynamically combined, thereby effectively overcoming the limitations of traditional target detection methods that rely on overall appearance features. Through component-level feature fusion and combination reasoning, the robustness of detection of disguised targets, diverse encapsulated objects and new forms of objects is significantly improved, the risk of missed detection caused by changes in overall appearance is reduced, and the universality is enhanced in scenarios with flexible changes in component positions and new combination modes. Attached Figure Description

[0011] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0012] Figure 1 An exemplary system architecture for which object recognition methods can be applied according to embodiments of this application is illustrated.

[0013] Figure 2 A flowchart illustrating an object identification method according to an embodiment of this application is shown.

[0014] Figure 3 The illustration shows an example diagram of a process for determining the overall position information of an object component according to an embodiment of this application.

[0015] Figure 4 A block diagram of an object recognition device according to an embodiment of this application is shown schematically.

[0016] Figure 5 A block diagram of an electronic device suitable for implementing an object recognition method according to an embodiment of this application is shown schematically. Detailed Implementation

[0017] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0019] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0020] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0021] In related technologies, object detection mainly relies on detecting the overall appearance features of an object (such as shape and texture) to determine the presence of a target. However, depending on the overall appearance features of an object, it is difficult to deal with scenarios where the overall appearance of the target does not follow a uniform pattern (such as object camouflage, diverse packaging forms, or the emergence of new forms). When the overall appearance of the target is unpredictable, relying solely on overall features is insufficient for effective identification, easily leading to missed detections and failing to meet practical detection needs.

[0022] In some examples, convolutional neural networks are used to extract the overall appearance features of an image (such as shape, texture, color distribution, etc.), then candidate regions are generated using region candidate box networks or anchor point mechanisms, and finally, the bounding boxes of the target and the class probabilities are output through classification and regression networks.

[0023] However, traditional methods rely on the statistical regularities of the overall appearance of the target in the training data. When the appearance features of the target are inconsistent with the distribution of the training set due to camouflage, diverse packaging, or new forms (e.g., explosives are obscured, industrial components are restructured), the model has difficulty extracting effective features, resulting in missed detections. When the target is composed of multiple key components (such as detonators and batteries in explosives), traditional detectors can only process each local area independently. It is difficult to infer the potential overall target through the correlation between components (e.g., batteries are often spatially adjacent to wires), leading to detection failures when components are scattered or partially obscured.

[0024] In view of this, embodiments of this application provide an object recognition method, apparatus, electronic device, storage medium, and program product. The method includes: extracting feature information and local location information of multiple object components from an input image; enhancing the multiple feature information according to the correlation between the feature information of the multiple object components to obtain enhanced feature information; combining the multiple enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object includes the object components corresponding to the multiple enhanced feature information in the feature set; and determining the object type and overall location information of the complete object corresponding to each of the at least one feature set according to the at least one feature set and the local location information of the multiple object components.

[0025] According to embodiments of this application, by extracting feature information and local location information of multiple object components and performing enhancement processing based on the correlation between different feature information, a feature set corresponding to the complete object can be dynamically combined, thereby effectively overcoming the limitations of traditional target detection methods that rely on overall appearance features. Through component-level feature fusion and combination reasoning, the robustness of detection of disguised targets, diverse encapsulated objects and new forms of objects is significantly improved, the risk of missed detection caused by changes in overall appearance is reduced, and the universality is enhanced in scenarios with flexible changes in component positions and new combination modes.

[0026] Figure 1 An exemplary system architecture for which object recognition methods can be applied according to embodiments of this application is illustrated.

[0027] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1As shown, the system architecture 100 according to this embodiment may include a terminal device 101, an image acquisition unit 102, a server 103, and a network 104. The network 104 serves as a medium for providing a communication link between the terminal device 101, the image acquisition unit 102, and the server 103. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal device 101 to interact with server 103 via network 104 to receive or send messages, etc. For example, an operator can initiate an identification task through the human-machine interface of terminal device 101 and generate instructions containing task identifiers and collection parameters. Terminal device 101 can be various electronic devices with a display screen and web browsing support, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] The image acquisition unit 102 can be an image acquisition device. For example, according to instructions, the image acquisition unit 102 can capture an image of a target area (such as a workstation on a conveyor belt) in the field of view to obtain an input image. The image acquisition unit 102 can include visible light imaging devices, X-ray imaging devices, infrared thermal imaging devices, millimeter wave and terahertz devices, lidar and 3D imaging devices, etc., depending on actual needs. The specific type is not limited here.

[0031] Server 103 can be a processor that provides various services. For example, after receiving an image sent by image acquisition unit 102, it can extract component features and locations, and output the feature information and local location information of each of the multiple object components. Alternatively, the computing module within the server can calculate the correlation (such as cosine similarity) between all component feature vectors obtained in the previous step, and use the correlation values ​​as weights to aggregate and weight the feature vectors of each component.

[0032] It should be noted that the object recognition method provided in this application embodiment can generally be executed by server 103. Correspondingly, the object recognition device provided in this application embodiment can generally be located in server 103. The object recognition method provided in this application embodiment can also be executed by a server or server cluster that is different from server 103 and capable of communicating with terminal device 101 and / or server 103. Correspondingly, the object recognition device provided in this application embodiment can also be located in a server or server cluster that is different from server 103 and capable of communicating with terminal device 101 and / or server 103. Alternatively, the object recognition method provided in this application embodiment can also be executed by terminal device 101, or by other terminal devices different from terminal device 101. Correspondingly, the object recognition device provided in this application embodiment can also be located in terminal device 101, or in other terminal devices different from terminal device 101.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, image acquisition units, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, image acquisition units, networks, and servers can be included.

[0034] Figure 2 A flowchart illustrating an object recognition method according to an embodiment of this application is shown. Figure 2 As shown, the method includes operations S210~S240.

[0035] In operation S210, feature information and local position information of multiple object parts are extracted from the input image.

[0036] In operation S220, based on the correlation between the feature information of multiple object components, the feature information of multiple components is enhanced to obtain enhanced feature information.

[0037] In operation S230, multiple enhanced feature information are combined into at least one feature set. Each feature set corresponds to a complete object, and the complete object includes object components corresponding to the multiple enhanced feature information in the feature set.

[0038] In operation S240, based on at least one feature set and the local position information of each of the multiple object components, the object type and overall position information of the complete object corresponding to each of the at least one feature set are determined.

[0039] In embodiments of this application, the input image can be raw image data to be processed, which can be acquired by an image acquisition device, and includes the target object and its components to be identified. The object components can be independently identifiable key parts of the image, which are basic elements constituting the complete target, such as detonators or batteries in explosives, or screws or sensors in industrial components. Feature information can be a numerical representation extracted from each object component through feature extraction, and can be a feature vector encoding the visual attributes (e.g., texture, shape) of the component.

[0040] Local location information can be a description of the spatial location of each object component in the input image, represented by bounding box coordinates or center point. Correlation can be the degree of association between the feature information of multiple object components, quantified by calculating feature similarity (such as cosine similarity) or attention weights, reflecting the semantic or spatial dependencies between components. Enhanced feature information can be an optimized feature representation obtained by weighted fusion or transformation of the original feature information based on correlation, incorporating contextual information from other related components.

[0041] A feature set can be a grouping of multiple enhanced feature information pieces formed by specific combination rules. The enhanced feature information within each feature set is highly correlated semantically or spatially, collectively describing a complete entity target. A complete object can be a whole target entity composed of multiple object components combined according to specific structural relationships, such as an explosive device or mechanical assembly, which acts as an independent identifiable unit in the image. The object type can be the category label or classification identifier to which the complete object belongs. The overall location information can be the overall spatial location and extent of the complete object in the image, represented by bounding boxes that cover all component parts.

[0042] For example, a region candidate box network is used to generate candidate component regions. Then, a region interest alignment operation is performed on each region to extract a fixed-dimensional feature vector. At the same time, the bounding box coordinates of each region are recorded as local location information. The similarity between the features of all components is calculated as the correlation through a self-attention mechanism. Then, the feature information of each component is weighted and fused with the correlation as the weight to obtain enhanced feature information.

[0043] For example, each object component is treated as a graph node, augmented feature information is used as node features, local location information is used to construct spatial edge relationships, node representations are learned through graph convolutional networks, and then graph segmentation algorithms are used to treat tightly connected node subgraphs as a feature set; for each feature set, graph pooling is used to obtain a graph-level representation, which is then input into a classification network to predict the object type; at the same time, the overall location information is regressed based on the location information of all nodes in the feature set.

[0044] According to embodiments of this application, by extracting feature information and local location information of multiple object components and performing enhancement processing based on the correlation between different feature information, a feature set corresponding to the complete object can be dynamically combined, thereby effectively overcoming the limitations of traditional target detection methods that rely on overall appearance features. Through component-level feature fusion and combination reasoning, the robustness of detection of disguised targets, diverse encapsulated objects and new forms of objects is significantly improved, the risk of missed detection caused by changes in overall appearance is reduced, and the universality is enhanced in scenarios with flexible changes in component positions and new combination modes.

[0045] As you can understand, the above text has already provided an overall explanation of how to perform object recognition. The following text will provide a detailed explanation of how to obtain the feature information and local location information of the object components.

[0046] According to an embodiment of this application, for operation S210, extracting feature information and local position information of multiple object components from the input image includes: performing multi-scale feature extraction on the input image to obtain a multi-scale feature map; performing local target localization on the scale feature map to obtain local position information of multiple object components; extracting initial feature information corresponding to the multiple local position information from the multi-scale feature map of the input image; and preprocessing the multiple initial feature information to obtain multiple feature information with the same dimension.

[0047] In the embodiments of this application, multi-scale feature extraction refers to extracting feature representations of an input image under different receptive fields through different layers of a convolutional neural network. Multi-scale features include features at various scales that contain detailed information and semantic information. A multi-scale feature map can be multiple feature maps of different resolutions obtained after multi-scale feature extraction, with each scale corresponding to a different level of abstraction and detail.

[0048] Local target localization refers to detecting and locating the specific positions of each object component on a multi-scale feature map, generating the bounding box coordinates or center point position of each component. Initial feature information can be the original feature vectors corresponding to each object component extracted from the multi-scale feature map based on local location information. Preprocessing can involve standardizing the initial feature information, such as dimensionality adjustment, normalization, and dimensionality reduction, to address feature dimension standardization and spatial alignment issues.

[0049] In one feasible embodiment, image features can be obtained using an image encoder, which can be a backbone network such as a residual network (ResNet), a shifted window transformer, or an internal image representation network.

[0050] For example, for multi-scale feature extraction, a residual network can be used as the backbone of feature extraction. By progressively downsampling through convolutional and pooling layers, feature maps at different levels can be extracted. A feature pyramid network structure can be constructed to fuse deep semantic features with shallow detail features from top to bottom, generating a multi-scale feature map pyramid with strong semantic information, thereby obtaining multi-scale feature maps, with each scale corresponding to a different receptive field and level of detail.

[0051] For example, local object localization on scale feature maps can include: using a region candidate box network on each scale feature map, setting multiple anchor boxes on the feature map using a sliding window approach, predicting the probability that each anchor box contains an object, and adjusting the bounding box parameters; generating candidate boxes from feature maps of different scales, detecting large targets on small-scale feature maps and small targets on large-scale feature maps, ensuring that objects of various sizes can be effectively detected; filtering candidate boxes with high overlap, retaining the most representative candidate boxes, and obtaining accurate local location information.

[0052] For example, the region interest alignment (roialign) method is used to set a uniform output size to determine the final spatial resolution of all features. Each initial feature information is regarded as a complete region, and bilinear interpolation is used for sampling. For each pixel on the output feature map, it is mapped back to the original feature map coordinates corresponding to the initial feature information. The four nearest pixels are found and bilinear interpolation is performed. The weights are calculated and interpolation is performed to obtain multiple feature information of the same dimension.

[0053] Traditional techniques using single-scale feature extraction struggle to simultaneously adapt to components with significant size differences; small components are prone to being missed, while large components suffer from incomplete features. Furthermore, traditional sliding window or fixed anchor point methods exhibit coarse localization and limited bounding box regression accuracy. To address these issues, embodiments of this application utilize multi-scale features to ensure appropriate feature representations for components ranging from small to large. Multi-scale regions are recommended to be combined with precise feature alignment (multiple features of the same dimension) to reduce localization errors, making it particularly suitable for irregularly shaped components.

[0054] According to an embodiment of this application, for operation S220, based on the correlation between the feature information of each of the multiple object components, the multiple feature information is enhanced to obtain enhanced feature information, including: determining attention weights that characterize the correlation between two different feature information among the multiple feature information based on the feature information of each of the multiple object components; and using the attention weights to enhance the multiple feature information to obtain enhanced feature information corresponding to each of the multiple feature information.

[0055] According to a specific embodiment of this application, for the feature information of any one of the multiple object components, the similarity between the feature information of the object component and the feature information of the other object components can be calculated, and the obtained similarity can be used as the attention weight between the corresponding feature information.

[0056] Then, the attention weights between the feature information of the object component and other feature information are weighted with the corresponding feature information; and the weighted feature information is fused with the feature information of the object component to obtain the enhanced feature information corresponding to the object component.

[0057] In this embodiment, by using attention weights among various feature information to weight each feature information, information complementary to the current feature information can be extracted from other feature information by utilizing the correlation between the current feature information and other feature information. Then, the weighted feature information is fused with the current feature information to enhance the current feature information.

[0058] According to another specific embodiment of this application, determining the attention weights that characterize the correlation between two different feature information among multiple object components based on their respective feature information includes: taking the feature information of each of the multiple object components as a channel feature of the input, concatenating the multiple feature information into an input feature map; and using an attention mechanism to determine the attention weight matrix of the input feature map, wherein the attention weight in the i-th row and j-th column of the attention weight matrix is ​​used to characterize the correlation between the i-th feature information and the j-th feature information in the input feature map, where i and j are both positive integers.

[0059] In the embodiments of this application, channel features can be used where the feature information of each object component is treated as an independent data channel during the feature concatenation process. The input feature map can be a high-dimensional data structure formed by concatenating multiple channel features along a specific dimension, with each channel corresponding to the feature information of an object component. The attention weight matrix can be calculated through an attention mechanism, where the attention weight in the i-th row and j-th column is used to characterize the correlation between the i-th and j-th feature information in the input feature map, where i and j are both positive integers.

[0060] For example, multiple parallel attention heads can be used, each calculating attention weights from a different feature subspace. The outputs of the multiple heads are then concatenated and linearly transformed to obtain the final enhanced feature information.

[0061] For example, regarding feature concatenation, suppose there are N object parts, each corresponding to a D-dimensional feature vector. These feature vectors are concatenated along the channel dimension to form an input feature map of dimension N×D, where N represents the number of features (number of channels) and D represents the dimension of each feature. The concatenated input feature map is then subjected to layer normalization to eliminate the dimensional differences between features.

[0062] During the attention weight calculation process, the input feature map can be mapped to a query matrix Q, a key matrix K, and a value matrix V through three independent linear transformation layers. Specifically, a pre-trained parameter matrix W can be used. Q W K W V Perform matrix multiplication with the input feature map to obtain the corresponding query matrix Q, key matrix K, and value matrix V, W. QW K W V The shape of each attention head is D×Dk, where Dk represents the number of attention heads. Next, the dot product between the query matrix and the transposed key matrix is ​​calculated, and then divided by a scaling factor (which can be the square root of the feature dimension) to prevent gradient vanishing. The attention score can be converted into an attention weight matrix using the softmax function. The attention weight matrix has an N×N shape, representing the correlation between N object parts (feature information). In this attention weight matrix, the sum of all weights is ensured to be 1. The correlation between the N object parts (feature information) is essentially the importance / influence of each feature information on another feature information; the attention weight matrix can also represent the importance distribution of each feature.

[0063] For example, for an N×N attention weight matrix A, the attention weights A ij The correlation between the feature information in the i-th row and the feature information in the j-th column is represented. The concatenation order of the N feature information in the row and column is fixed. Therefore, the attention weight A ij This also represents the correlation between the i-th feature and the j-th feature in the input feature map. Here, i and j are both positive integers less than or equal to the number of feature information N.

[0064] In the embodiments of this application, by concatenating the feature information of each object component as channels to obtain an input feature map, and processing the input feature map based on an attention mechanism, it is possible to construct the spatial topological relationship of multiple object components, and use the attention mechanism to dynamically focus on the spatial correlation between object components according to the spatial context information expressed by the input feature map, thereby better extracting the correlation between feature information.

[0065] According to another specific embodiment of this application, an attention weight matrix is ​​applied to the input feature map to obtain an enhanced feature map; and the enhanced feature map is split into enhanced feature information corresponding to multiple feature information according to the splicing order of multiple feature information when splicing the input feature map.

[0066] For the attention weight matrix constructed using the input feature map, during the feature enhancement stage, the attention weight matrix can be multiplied by the value matrix V to obtain a weighted feature representation. The enhanced features are then residually connected to the original input feature map, preserving the original information while incorporating the attention enhancement effect. A two-layer feedforward neural network (containing fully connected layers and activation functions) is then used for further nonlinear transformation to enhance the expressive power of the features, resulting in the corresponding enhanced feature map. Subsequently, the N features in the enhanced feature map can be separated into N D-dimensional enhanced features according to the concatenation order of the N features.

[0067] Taking the detection of explosives (complete components) as an example, the feature information of three object components—detonator, battery, and wire—can first be extracted from the image (e.g., each a 512-dimensional vector). These vectors are then concatenated into a 3×512 input feature map. Next, an attention weight matrix among the three components is calculated using a self-attention mechanism. Finally, the attention weight matrix is ​​applied to obtain enhanced feature information, where the detonator feature may receive a higher weight due to its strong correlation with the explosive, thus playing a greater role in subsequent identification.

[0068] Traditional techniques employ fixed strategies such as average pooling or max pooling to fuse features, which struggle to adaptively emphasize important features, resulting in insufficient feature discrimination. For example, in explosives detection, detonator and battery features are treated equally, but in practice, detonator features are more decisive for identification. The embodiments of this application, through the fusion of an attention weight matrix and the input feature map, can dynamically adjust the feature importance in the input feature map based on the attention weight matrix incorporating spatial context information. This allows key component features to receive higher weights, significantly improving the clarity of the classification boundaries of object components.

[0069] According to an embodiment of this application, for operation S230, combining multiple enhanced feature information into at least one feature set includes: identifying the object component combination relationship of the multiple enhanced feature information to obtain an identification result, wherein the identification result is used to indicate whether the object components corresponding to the multiple enhanced feature information are used to form the same complete object; and combining the multiple enhanced feature information into at least one feature set according to the identification result.

[0070] In the embodiments of this application, combination relationship recognition can refer to the process of calculating and analyzing whether object components corresponding to multiple enhanced feature information belong to the same complete object semantically and spatially. This is used to establish the attribution relationship between object components and the complete object, providing a logical basis for the combination of feature sets. The recognition result can be the output of combination relationship recognition, represented in the form of probability values ​​or binary labels, used to indicate which object components should be combined into the same feature set.

[0071] In one feasible embodiment, a Graph Neural Network (GNN) is used to implement the complete combination process from enhanced feature information to a feature set, including: defining each enhanced feature information as a node in the graph, with the node feature being the corresponding enhanced feature vector; and constructing edge connections between nodes based on spatial distance and feature similarity. For spatially adjacent edges, if the distance between the bounding boxes of two parts in the image is less than a threshold (e.g., 50 pixels), a connection is established; for feature-similar edges, the cosine similarity between feature vectors is calculated, and if it is greater than a threshold (e.g., 0.7), a connection is established, thereby constructing an undirected graph G=(V, E), where V is the set of nodes and E is the set of edges.

[0072] Message propagation and feature aggregation are performed on the graph using a multi-layer graph convolutional network. An edge classifier calculates the probability of a combination relationship between each pair of nodes. For example, the input is the concatenation of features of two nodes, and the output is the probability of the relationship between the two nodes. This probability is then compared with a set threshold (e.g., 0.5), and edges with a probability greater than the threshold are identified as valid combination relationships. Based on the identification results, connected subgraphs are extracted from the graph. Each connected component corresponds to a potential feature set, and the node features within each connected component are weighted and fused. Each feature set is scored, and invalid sets with too few nodes (e.g., less than 2) or low feature consistency are filtered out to obtain the final feature set.

[0073] Traditional combination methods based on fixed rules or simple distance thresholds are difficult to understand the semantic relationships between object components. For example, combining based solely on spatial distance may lead to the incorrect combination of unrelated neighboring components. Combination methods based on template matching or fixed spatial relationships are sensitive to changes in the position of object components and will fail to recognize components when their relative positions change.

[0074] To address this technical problem, the embodiments of this application improve the accuracy of combination by learning complex association patterns between object components through graph neural networks. The improvement is particularly significant in scenarios where the components are sparsely distributed or have an irregular layout. Furthermore, the combination relationship recognition based on feature semantic similarity is insensitive to changes in spatial layout, and improves the stability of combination recognition in test scenarios where the positions of components change randomly.

[0075] According to an embodiment of this application, object component combination relationship identification is performed on multiple enhanced feature information to obtain identification results, including: obtaining pre-set text prompt words, wherein the text prompt words include a first prompt word for identifying the object component corresponding to each enhanced feature information and a second prompt word for identifying the object component combination relationship; using a multimodal large model, identifying the object component corresponding to each of the multiple enhanced feature information according to the first prompt word; and generating identification results based on the multiple object components according to the second prompt word, wherein the identification results include multiple enhanced feature information that have object component combination relationships, and the enhanced feature information corresponding to multiple object components that make up the same complete object has the same object component combination relationship.

[0076] In embodiments of this application, the text prompts can be predefined or automatically generated natural language descriptive text, used to guide and constrain the understanding and reasoning process of the large model. A multimodal large model can refer to a large pre-trained artificial intelligence model capable of simultaneously processing and fusing multiple modal information (e.g., images, text), which can be used to receive image features (enhanced feature information) and text instructions (text prompts), utilizing its vast prior knowledge to perform cross-modal alignment, understanding, and reasoning, and output structured recognition results.

[0077] The object component combination relationship refers to the association between multiple object components that belong to the same higher-level complete object in terms of structure, function, or semantics. The recognition result can be the output of a multimodal large model after processing the input based on prompt words, indicating which enhanced feature information (corresponding components) have combination relationships.

[0078] For example, in the identification of hazardous devices, the first prompt could be: identifying the type of object component corresponding to each feature, and the second prompt could be: identifying whether multiple object components can be combined into a complete object. Guided by the first prompt, the multimodal large model can identify the corresponding object component type based on the semantics of multiple enhanced feature information. Guided by the second prompt, the multimodal large model can perform relational reasoning based on pre-trained knowledge in the expert database and / or reference examples given in the second prompt (if present), through understanding historical experience and / or reference examples. For example, based on pre-trained knowledge (containing hazardous device components and their connection relationships), relational reasoning can be performed on the identified object components. According to pre-trained knowledge, "detonator," "explosive," and "gasket" usually appear together, forming a "core part of a hazardous device." Guided by the second prompt, the identification result of whether multiple object components are used to form a complete object is obtained based on the result of relational reasoning.

[0079] According to embodiments of this application, traditional supervised learning-based methods heavily rely on large amounts of labeled component and combination relationship data. When faced with new products and new component combinations (especially the identification of objects with frequently changing shapes, such as hazardous devices), it is necessary to re-collect data and retrain the model, resulting in high costs and long cycles. Expert systems based on fixed rules have complex rule designs and struggle to handle unforeseen situations. Embodiments of this application utilize the common sense internalized in large models and zero-shot learning capabilities. By modifying text prompts (such as adding new component names or describing new combinations), the system can identify entirely new combination relationships without retraining the visual model. This achieves a paradigm shift from data-driven to knowledge-guided approaches. Combination recognition no longer relies on surface associations but is based on functional semantics, resulting in more reasonable recognition results and greater reliability for complex and non-standard objects.

[0080] According to embodiments of this application, object component combination relationship identification is performed on multiple enhanced feature information to obtain identification results, including: converting multiple enhanced feature information into a graph structure, wherein the graph structure includes multiple nodes and multiple edges, nodes represent enhanced feature information, and the edge between two nodes indicates that the similarity between the enhanced feature information represented by the two nodes satisfies the similarity condition; using a graph neural network, the graph structure is divided into multiple subgraphs according to the similarity represented by the edges, wherein the number of nodes and the number of edges in the subgraphs satisfy the corresponding constraint conditions, and the enhanced feature information represented by multiple nodes in the subgraphs has the same object component combination relationship; using the multiple subgraphs as identification results, so as to combine multiple enhanced feature information with the same object component combination relationship into a feature set according to the identification results.

[0081] In the embodiments of this application, the graph structure can be a mathematical structure consisting of a set of nodes and a set of edges connecting the nodes, used to intuitively represent entities (nodes) and the relationships (edges) between entities. Nodes are the carriers of features. Edges can indicate that the similarity between the enhanced feature information represented by two nodes has reached a preset threshold (similarity condition).

[0082] Similarity criteria can be used to determine whether two enhanced feature information are sufficiently similar, thus serving as a quantitative or qualitative standard for establishing connecting edges in a graph structure (e.g., feature vector cosine similarity greater than 0.7). Graph neural networks are deep learning models used to process graph-structured data. They can update node representations by aggregating neighbor node information and learn the overall properties of the graph or subgraph.

[0083] Subgraph partitioning can be the process of dividing the original complete graph structure into several smaller subgraphs that are internally tightly connected but sparsely connected to each other. Constraints can be rules used in the subgraph partitioning process to determine whether a subgraph is legal and valid (e.g., a subgraph must contain at least 2 nodes, and the number of edges must satisfy a certain ratio to the number of nodes).

[0084] Taking the automatic detection grouping of industrial parts based on graph convolutional networks as an example, after obtaining the enhanced feature information of N object part regions (each object part region corresponds to an object part), N nodes can be created using the enhanced feature information. According to the pre-set similarity conditions, the cosine similarity between each pair of node feature vectors is calculated to obtain an undirected weighted graph G=(V,E), where V is the node set and E is the edge set. This graph structure encapsulates the enhanced feature information of all object parts and the (feature and space) similarity between different enhanced feature information.

[0085] Graph convolutional networks can be composed of L stacked graph convolutional layers. Nodes can update their own features by aggregating the feature information from the previous layer of all their neighboring nodes and combining it with their own feature information. Through multi-layer propagation, nodes that were originally based on simple similarity connections gradually incorporate information from multi-hop neighbors into their features. Nodes belonging to the same potential component (such as all piston-related object parts) will have increasingly similar features in high-dimensional space.

[0086] Using the final node features and original edge information output by the trained graph convolutional network, community detection (subgraph partitioning) can be performed, employing node representation-based clustering algorithms (such as spectral clustering). The similarity matrix of node features is calculated, and then spectral clustering is used to divide the nodes into K clusters (K can be predicted by the model or selected based on metrics such as modularity). This results in M ​​subgraphs {G1, G2, ..., GM} that satisfy the constraints. Each subgraph Gi contains a group of nodes. Each subgraph Gi corresponds to a recognition result, indicating that the nodes (object component features) it contains belong to the same object component combination (the same engine module). For example, the recognition result could be: {Group 1: [Component ID_Detonator 1, Component ID_Explosive 1, Component ID_Gasket 1], Group 2: [Component ID_Explosive 2, Component ID_Switch 1], ...}.

[0087] It is understandable that graph neural networks, through message passing mechanisms, can model complex and indirect dependencies between components and utilize the information of the graph structure itself to enhance node representations, thereby achieving improvements in accuracy (especially in complex relationship scenarios), robustness (anti-noise and occlusion), and adaptability (learnability) compared to traditional methods.

[0088] The embodiments of this application utilize graph neural networks to allow node features to aggregate information from multi-hop neighbors through multi-layer message passing. Even if the components are not directly similar, the graph neural network can learn the potential associations between multiple object components through the intermediate nodes in the graph structure, thereby more accurately classifying them into the same subgraph (combination) and improving the recognition accuracy when the component relationships are indirect and implicit.

[0089] According to an embodiment of this application, for operation S240, determining the object type and overall position information of the complete object corresponding to each of the at least one feature set based on the local position information of each of the multiple object components includes: determining the object type of the complete object corresponding to each feature set based on multiple enhanced feature information in the at least one feature set; and for a single feature set, integrating the local position information corresponding to the multiple enhanced feature information to obtain the overall position information covering the object components corresponding to the multiple enhanced feature information.

[0090] In the embodiments of this application, integration can refer to the process of performing geometric calculations or fusion processing on multiple local location information to generate overall location information.

[0091] According to embodiments of this application, an overall bounding box and a predicted category score can be generated for automatically discovered candidate box combinations.

[0092] For example, in security X-ray image detection, it is necessary to identify luggage on the conveyor belt of the security equipment. Multiple suspicious object components are detected by scanning the X-ray image, such as a blocky high-density object A, a coiled wire object B, a small electronic component C, and a blocky low-density object D. For each object component, based on its enhanced feature information, such as shape, density, material properties, spatial context information, and local location information, analysis reveals that the enhanced feature information of object components A, B, C, and D are spatially adjacent and semantically highly correlated. For example, explosives, wires, detonators, and gunpowder often appear in combination. Therefore, these suspicious objects are grouped into the same feature set S1.

[0093] Based on the knowledge base, this combination pattern of "high-energy explosive block + wire + detonator + gunpowder powder" strongly points to a complete object: an improvised explosive device (IED). Therefore, the object type of the complete object corresponding to feature set S1 can be determined to be an explosive. Furthermore, based on the local position information (four small boxes) of object components A, B, C, and D in S1, this position information is integrated and calculated (e.g., calculating the smallest bounding rectangle that completely contains these four small boxes). The result is the overall position information of the complete object (explosive) (a complete large box containing the explosive, wire, detonator, and gunpowder powder).

[0094] Traditional methods typically perform global classification of images or identify isolated objects individually, such as identifying "a battery" and "a roll of tape," but they cannot determine whether these constitute a hazardous device. Furthermore, they are prone to missing detections due to object obstruction, deformation, or component separation, and they are also prone to misclassifying unrelated but nearby ordinary items as hazardous materials.

[0095] The embodiments of this application associate multiple related object components through feature sets, and infer the complete object from the combination patterns of the object components. Even if the detonator of an explosive is obscured, as long as the explosive block and the wire are identified, it can still be judged as an explosive with a high probability, significantly reducing the false alarm rate. At the same time, since multiple features need to match a specific pattern simultaneously, the false alarm rate of misidentifying a single ordinary item as a dangerous item is also greatly reduced.

[0096] According to embodiments of this application, determining the object type of a complete object corresponding to each feature set based on multiple enhanced feature information in at least one feature set includes: fusing multiple enhanced feature information to obtain fused features; wherein the fused features are obtained through at least one of the following methods: average pooling of multiple enhanced feature information; weighted summation of multiple enhanced feature information using their respective attention weights; sequential linear transformation and nonlinear activation of the features concatenated by a multilayer perceptron; multiple convolution operations on the fused features to strengthen the local correlations within the fused features to obtain enhanced fused features; and target recognition of the enhanced fused features to obtain the object type.

[0097] In the embodiments of this application, feature fusion can be a process of integrating multiple enhanced feature information into a single feature vector, aiming to extract and retain key information from multiple component features to form a comprehensive representation of the overall object. Average pooling can be used to calculate the average value of multiple enhanced feature information along the feature dimension to generate a fused feature. Attention weight weighted summation can be based on calculated attention weights to perform a weighted summation of multiple enhanced feature information, where the weights represent the importance of each component feature. The multilayer perceptron is a feedforward artificial neural network containing an input layer, hidden layers, and an output layer, capable of performing nonlinear transformations on the concatenated features.

[0098] Linear transformation refers to affine transformation of input features using weight matrices and bias vectors. Nonlinear activation involves applying nonlinear functions (such as ReLU and Sigmoid) after a linear transformation, introducing nonlinear expressive power. Fusion features can be a single feature vector obtained through fusion, integrating information from multiple component features. Convolution operations involve performing convolution operations on fusion features, extracting local feature patterns through sliding convolution kernels, and strengthening local correlations. Enhanced fusion features refer to features processed through multiple convolution operations, possessing stronger local correlations and higher-level semantic information. Object recognition can be a task that classifies objects based on enhanced fusion features to determine their types.

[0099] Taking the scenario of explosive type identification based on multi-component feature fusion and convolutional enhancement as an example, in the analysis of security inspection X-ray images, it is necessary to identify different types of explosive devices. Each explosive device consists of multiple components (such as detonators, batteries, wires, explosives, etc.). The system first detects each component and extracts its features, and then identifies the complete explosive device type (such as "improvised explosive device", "plastic explosives", etc.) through the embodiments of this application.

[0100] Suppose four components are detected: detonator, battery, wire, and suspicious powder, corresponding to four enhanced feature information (F1, F2, F3, F4). A weighted summation method based on attention weights is used for fusion. First, the attention weight of each enhanced feature information can be calculated. For example, a multilayer perceptron can be used to calculate the weighted summation to obtain the fused feature.

[0101] Local correlations can be strengthened through convolution operations. For example, the fused features can be reshaped into a two-dimensional feature map. Two convolution operations are applied to output the convolutional feature map, followed by a ReLU activation function and batch normalization. Global average pooling is then performed to obtain a multi-dimensional feature vector from the convolutional feature map, which is the enhanced fused feature.

[0102] In the target recognition stage, multidimensional feature vectors can be input into a classifier. The classifier's first fully connected layer performs dimensionality reduction by connecting the ReLU activation function and the Dropout function to obtain dimensionality-reduced features. Then, a second fully connected layer transforms these dimensionality-reduced features into a number of categories, for example, 5 categories (improvised explosive devices, plastic explosives, grenades, fireworks, and others). Finally, a Softmax layer outputs the probability of each category, and the category with the highest probability is taken as the object type, such as "improvised explosive devices".

[0103] According to embodiments of this application, traditional methods typically classify components by simply concatenating their features, without fully exploring the internal structure of the features, resulting in limited discriminative power. Furthermore, they neglect local correlation patterns within the fused features, making it difficult to capture subtle but important feature combination patterns. Embodiments of this application highlight key component features and suppress secondary features through attention weighting, making the fused features more representative. Convolutional operations extract local feature patterns, learning key patterns such as "a specific distance between detonators and batteries." In the identification of complex explosive devices, convolutional layers can learn spatial arrangement patterns between components. Even if the appearance of components changes, as long as the arrangement pattern is similar, correct identification can be achieved, improving recognition accuracy.

[0104] According to an embodiment of this application, for a single feature set, the local location information corresponding to multiple enhanced feature information is integrated to obtain the overall location information of the object component covering the multiple enhanced feature information, including at least one of the following: using a multilayer perceptron to perform regression prediction on the local location information corresponding to the multiple enhanced feature information to obtain the overall location information; determining the minimum bounding rectangle corresponding to the multiple local location information, and determining the location information of the minimum bounding rectangle as the overall location information.

[0105] In the embodiments of this application, regression prediction can refer to using a multilayer perceptron to predict the overall location information based on multiple local location information. The minimum bounding rectangle can be the rectangle with the smallest area that can completely enclose the given multiple local location information, which can be obtained by calculating the extreme values ​​of the coordinates of the midpoints of all the multiple local location information.

[0106] For example, if it is confirmed that four object parts belong to the same complete object, the integration method can be automatically selected based on the number and distribution characteristics of the object parts (which can be determined by the variance of the distance between object parts). For instance, when the number of object parts is greater than three and their distribution is scattered, multilayer perceptron regression is used; otherwise, minimum bounding rectangle calculation is used. Minimum bounding rectangle calculation obtains the overall position information of the complete object by calculating the extreme values ​​of all bounding box vertices. The multilayer perceptron can normalize the coordinates of the four local bounding boxes through a multilayer fully connected network and then concatenate them into a 16-dimensional vector to output the predicted overall bounding box coordinates.

[0107] Figure 3 The illustration shows an example diagram of a process for determining the overall position information of an object component according to an embodiment of this application.

[0108] like Figure 3 As shown, taking the identification of explosives in security X-ray images as an example, the object component 31 of the acquired suspicious object (complete object) is used. The object component 31 includes a block-shaped high-density object A, a coiled wire object B, and a small electronic component C. Multiple object components have their own local position information 32.

[0109] Local location information 32 may include the local bounding box of a blocky high-density object A, the local bounding box of a linear object B, and the local bounding box of a small electronic component C. Each local bounding box coordinate corresponds to four coordinate values. Multiple nodes (4 local bounding boxes × 4 coordinate values) corresponding to the local location information 32 are used as input information and input to the input layer 331 of the multilayer perceptron 33. After passing through multiple processing units in the hidden layer 332, different combination patterns of the input information are learned to obtain a multidimensional preliminary linear combination result 333. The linear combination result 333 is then subjected to nonlinear activation processing to obtain an activated multidimensional feature vector 334. The activated multidimensional feature vector 334 is then mapped to the target space through the output layer 335 to output a linear result as the minimum bounding rectangle 34. The four coordinate points of the minimum bounding rectangle 34 are used as the complete location information 35 of the object component 31.

[0110] Traditional technologies typically employ a "detect the whole first, then classify" paradigm, or detect each component independently. For a complete object composed of multiple separate components (such as an explosive device), the former may miss detection due to the target's lack of compactness, while the latter will output multiple scattered bounding boxes (such as separately boxing out a linear object B), failing to intuitively indicate the complete range of the entire threat and increasing the risk of misjudgment. The embodiments of this application, through integrated operations, can output a unified bounding box covering all key components, whether using multilayer perceptron regression or geometric minimum bounding rectangle. This allows a single large red box on the security inspection screen to clearly mark the entire range of the explosive device, greatly improving the intuitiveness of human-computer interaction and the completeness of positioning.

[0111] In one feasible embodiment, the object recognition method can be implemented based on an end-to-end object recognition system. The object recognition system can include candidate box detection, candidate box feature extraction, detection candidate box combination, overall target position regression and score prediction, integrated into a complete system to achieve end-to-end output from image input to overall object recognition result, which is suitable for application scenarios such as security inspection and object detection.

[0112] Based on the above object recognition method, the present invention also provides an object recognition device. The following will be combined with... Figure 4 The device is described in detail.

[0113] Figure 4 A block diagram of an object recognition device according to an embodiment of this application is shown schematically.

[0114] like Figure 4 As shown, the object recognition device 400 includes an extraction module 410, an enhancement module 420, a combination module 430, and a determination module 440.

[0115] Extraction module 410 is used to extract feature information and local location information of multiple object parts from the input image. In one embodiment, extraction module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0116] The enhancement module 420 is used to enhance multiple feature information based on the correlation between the feature information of each of the multiple object components, thereby obtaining enhanced feature information. In one embodiment, the enhancement module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0117] The combination module 430 is used to combine multiple enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object includes object components corresponding to the multiple enhanced feature information in the feature set. In one embodiment, the combination module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0118] The determining module 440 is configured to determine the object type and overall position information of the complete object corresponding to each of the at least one feature set, based on at least one feature set and the local position information of each of the multiple object components. In one embodiment, the determining module 440 may be used to perform the operation S240 described above, which will not be repeated here.

[0119] According to the embodiments of this application, based on the extraction module 410, enhancement module 420, combination module 430 and determination module 440 in the object recognition device 400, by extracting feature information and local position information of multiple object parts, and performing enhancement processing based on the correlation between different feature information, a feature set corresponding to the complete object can be dynamically combined, thereby effectively overcoming the limitation of traditional target detection methods that rely on overall appearance features. Through part-level feature fusion and combination reasoning, the robustness of detection of disguised targets, diverse encapsulated objects and new forms of objects is significantly improved, the risk of missed detection caused by changes in overall appearance is reduced, and the universality is enhanced in scenarios with flexible changes in part position and new combination modes.

[0120] According to an embodiment of this application, the enhancement module 420 includes a first enhancement submodule, configured to determine an attention weight representing the correlation between two different feature information among the multiple object components based on their respective feature information; and a second enhancement submodule, configured to enhance the multiple feature information using the attention weight to obtain enhanced feature information corresponding to the multiple feature information respectively.

[0121] According to an embodiment of this application, the first enhancement submodule includes: a first enhancement unit, used to take the feature information of each of multiple object components as a channel feature of the input, and to concatenate the multiple feature information into an input feature map; and a second enhancement unit, used to use an attention mechanism to determine the attention weight matrix of the input feature map, wherein the attention weight in the i-th row and j-th column of the attention weight matrix is ​​used to characterize the correlation between the i-th feature information and the j-th feature information in the input feature map, and i and j are both positive integers.

[0122] According to an embodiment of this application, the second enhancement submodule includes: a third enhancement unit, used to apply the attention weight matrix to the input feature map to obtain an enhanced feature map; and a fourth enhancement unit, used to split the enhanced feature map into enhanced feature information corresponding to multiple feature information according to the splicing order of multiple feature information when splicing the input feature map.

[0123] According to an embodiment of this application, the combination module 430 includes an identification submodule and a combination submodule. The identification submodule is used to identify the object component combination relationship of multiple enhanced feature information to obtain an identification result, wherein the identification result is used to indicate whether the object components corresponding to the multiple enhanced feature information are used to form the same complete object; and the combination submodule is used to combine the multiple enhanced feature information into at least one feature set according to the identification result.

[0124] According to an embodiment of this application, the identification submodule includes an acquisition unit and a component identification unit. The acquisition unit is used to acquire pre-set text prompts, wherein the text prompts include a first prompt for identifying the object components corresponding to each enhanced feature information and a second prompt for identifying the combination relationship of object components. The component identification unit is used to utilize a multimodal large model to identify the object components corresponding to each of the multiple enhanced feature information according to the first prompt; and to generate an identification result based on the multiple object components according to the second prompt, wherein the identification result includes multiple enhanced feature information that have a combination relationship of object components, and the enhanced feature information corresponding to multiple object components that make up the same complete object has the same object component combination relationship.

[0125] According to embodiments of this application, the recognition submodule includes: a conversion unit, a partitioning unit, and an information combination unit. The conversion unit converts multiple enhanced feature information into a graph structure, wherein the graph structure includes multiple nodes and multiple edges, nodes represent enhanced feature information, and an edge between two nodes indicates that the similarity between the enhanced feature information represented by the two nodes satisfies a similarity condition. The partitioning unit uses a graph neural network to divide the graph structure into multiple subgraphs based on the similarity represented by the edges, wherein the number of nodes and edges in each subgraph satisfies corresponding constraints, and the enhanced feature information represented by each node in the subgraph has a combination relationship with the same object component. The information combination unit uses the multiple subgraphs as recognition results to combine multiple enhanced feature information with the same object component combination relationship into a feature set based on the recognition results.

[0126] According to an embodiment of this application, the determining module 440 includes a type determining submodule and an integration submodule. The type determining submodule is used to determine the object type of the complete object corresponding to each feature set based on multiple enhanced feature information in at least one feature set; the integration submodule is used to integrate the local position information corresponding to the multiple enhanced feature information for a single feature set to obtain the overall position information covering the object components corresponding to the multiple enhanced feature information.

[0127] According to embodiments of this application, the type determination submodule includes: a feature fusion unit, a convolution unit, and a feature recognition unit. The feature fusion unit is used to fuse multiple enhanced feature information to obtain fused features; wherein the fused features are obtained through at least one of the following methods: average pooling of multiple enhanced feature information; weighted summation of multiple enhanced feature information using their respective attention weights; or sequential linear transformation and nonlinear activation of the concatenated features by a multilayer perceptron. The convolution unit is used to perform multiple convolution operations on the fused features to strengthen the local correlations within the fused features, obtaining enhanced fused features. The feature recognition unit is used to perform target recognition on the enhanced fused features to obtain the object type.

[0128] According to embodiments of this application, the integration submodule includes at least one of the following: a prediction unit and a rectangle determination unit. The prediction unit is used to perform regression prediction on the local position information corresponding to multiple enhanced feature information using a multilayer perceptron to obtain overall position information; the rectangle determination unit is used to determine the minimum bounding rectangle corresponding to the multiple local position information, and to determine the position information of the minimum bounding rectangle as the overall position information.

[0129] According to an embodiment of this application, the extraction module 410 includes: a scale feature extraction submodule, a localization submodule, a feature information extraction submodule, and a preprocessing submodule. The scale feature extraction submodule is used to extract multi-scale features from the input image to obtain a multi-scale feature map; the localization submodule is used to perform local target localization on the scale feature map to obtain local position information for each of the multiple object components; the feature information extraction submodule is used to extract initial feature information corresponding to the multiple local position information from the multi-scale feature map of the input image; and the preprocessing submodule is used to preprocess the multiple initial feature information to obtain multiple feature information with the same dimension.

[0130] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0131] For example, any plurality of the extraction module 410, enhancement module 420, combination module 430, and determination module 440 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the extraction module 410, enhancement module 420, combination module 430, and determination module 440 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the extraction module 410, enhancement module 420, combination module 430, and determination module 440 may be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0132] It should be noted that the apparatus portion in the embodiments of this application corresponds to the method portion in the embodiments of this application. The description of the apparatus portion is specifically referred to in the method portion, and will not be repeated here.

[0133] Figure 5 A block diagram of an electronic device suitable for implementing an object recognition method according to an embodiment of this application is shown schematically.

[0134] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0135] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0136] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0137] According to embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0138] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0139] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0140] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0142] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this application.

[0143] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0144] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0145] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0147] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An object recognition method, comprising: Extract the feature information and local location information of multiple object components from the input image; Based on the correlation between the feature information of multiple object components, the feature information of each of the multiple feature information is enhanced to obtain enhanced feature information; The multiple enhanced feature information pieces are combined into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object includes object components corresponding to the multiple enhanced feature information pieces in the feature set; and Based on at least one set of features and the local location information of each of the multiple object components, determine the object type and overall location information of the complete object corresponding to each of the at least one set of features.

2. The method according to claim 1, wherein, The step of combining multiple enhanced feature information into at least one feature set includes: The object component combination relationship is identified by performing object component combination relationship identification on multiple enhanced feature information, and an identification result is obtained, wherein the identification result is used to indicate whether the object components corresponding to multiple enhanced feature information are used to form the same complete object; and Based on the recognition results, the multiple enhanced feature information is combined into at least one feature set.

3. The method according to claim 2, wherein, The process of identifying the object component combination relationship from multiple enhanced feature information to obtain the identification result includes: Obtain pre-set text prompts, wherein the text prompts include a first prompt for identifying the object component corresponding to each enhanced feature information and a second prompt for identifying the combination relationship of the object components; Using a multimodal large model, the object components corresponding to each of the multiple enhanced feature information are identified according to the first prompt word; and the recognition result is generated based on the multiple object components according to the second prompt word, wherein the recognition result includes multiple enhanced feature information that have object component combination relationships, and the enhanced feature information corresponding to the multiple object components that make up the same complete object has the same object component combination relationship.

4. The method according to claim 2, wherein, The process of identifying the object component combination relationship from multiple enhanced feature information to obtain the identification result includes: The multiple enhanced feature information is converted into a graph structure, wherein the graph structure includes multiple nodes and multiple edges, the nodes represent the enhanced feature information, and the edge between two nodes indicates that the similarity between the enhanced feature information represented by the two nodes satisfies the similarity condition; The graph structure is divided into multiple subgraphs based on the similarity of the edge representations using a graph neural network. The number of nodes and edges in each subgraph satisfies the corresponding constraints, and the enhanced feature information represented by each node in the subgraph has the same object component combination relationship. Multiple subgraphs are used as the recognition results, and multiple enhanced feature information that have the same object component combination relationship are combined into a feature set according to the recognition results.

5. The method according to claim 1, wherein, The step of determining the object type and overall location information of the complete object corresponding to each of the at least one feature set based on the local location information of each of the multiple object components includes: Based on multiple enhanced feature information from at least one feature set, determine the object type of the complete object corresponding to each feature set; For a single feature set, the local location information corresponding to the multiple enhanced feature information is integrated to obtain the overall location information covering the object components corresponding to the multiple enhanced feature information.

6. The method according to claim 5, wherein, The step of determining the object type of the complete object corresponding to each feature set based on multiple enhanced feature information from at least one feature set includes: Multiple enhanced feature informations are fused to obtain fused features; wherein, the fused features are obtained through at least one of the following methods: average pooling of multiple enhanced feature informations; weighted summation of multiple enhanced feature informations using their respective attention weights; or a multilayer perceptron sequentially performing linear transformation and nonlinear activation on the features concatenated from multiple enhanced feature informations. Multiple convolution operations are performed on the fused features to strengthen the local correlations within the fused features, resulting in enhanced fused features; The enhanced fusion features are used for target identification to obtain the object type.

7. The method according to claim 5, wherein, For a single feature set, the local location information corresponding to the multiple enhanced feature information is integrated to obtain the overall location information covering the object components corresponding to the multiple enhanced feature information, including at least one of the following: The overall location information is obtained by using a multilayer perceptron to perform regression prediction on the local location information corresponding to the multiple enhanced feature information. Determine the minimum bounding rectangle corresponding to the multiple local position information, and determine the position information of the minimum bounding rectangle as the overall position information.

8. The method according to claim 1, wherein, The step of enhancing multiple feature information based on the correlation between the feature information of each of the multiple object components to obtain enhanced feature information includes: Based on the feature information of each of the multiple object components, determine the attention weight that characterizes the correlation between two different feature information among the multiple feature information; The attention weights are used to enhance multiple feature information to obtain enhanced feature information corresponding to each of the multiple feature information.

9. The method according to claim 8, wherein, The step of determining attention weights that characterize the correlation between two different pieces of feature information among the multiple object components based on their respective feature information includes: The feature information of each of the multiple object components is used as a channel feature of the input, and the multiple feature information is concatenated into an input feature map; Using an attention mechanism, an attention weight matrix for the input feature map is determined, wherein the attention weight in the i-th row and j-th column of the attention weight matrix is ​​used to characterize the correlation between the i-th feature information and the j-th feature information in the input feature map, where i and j are both positive integers.

10. The method according to claim 8 or 9, wherein, The step of enhancing multiple feature information using the attention weights to obtain enhanced feature information corresponding to each of the multiple feature information includes: The attention weight matrix is ​​applied to the input feature map to obtain the enhanced feature map; and According to the stitching order of the multiple feature information when stitching the input feature map, the enhanced feature map is split into multiple enhanced feature information corresponding to the feature information respectively.

11. The method according to any one of claims 1 to 10, wherein, The extraction of feature information and local location information of multiple object components from the input image includes: Multi-scale feature extraction is performed on the input image to obtain a multi-scale feature map; Local target localization is performed on the scale feature map to obtain the local position information of each of the multiple object components; Extract initial feature information corresponding to the multiple local location information from the multi-scale feature map of the input image; The initial feature information is preprocessed to obtain multiple feature information with the same dimension.

12. An object recognition device, characterized in that, The identification device includes: The extraction module is used to extract the feature information and local location information of multiple object parts from the input image; The enhancement module is used to enhance the multiple feature information respectively based on the correlation between the feature information of multiple object components to obtain enhanced feature information; A combination module is used to combine multiple enhanced feature information into at least one feature set, wherein the feature set corresponds to a complete object, and the complete object includes object components corresponding to the multiple enhanced feature information in the feature set; The determining module is used to determine the object type and overall position information of the complete object corresponding to each of the at least one feature set based on the local position information of each of the multiple object components.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program or instructions stored thereon, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program or instructions, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

Images