Key point recognition method and related device

Abstract

The application discloses a key point identification method and related equipment, and relates to the technical field of data processing, and comprises the following steps: in response to a key point identification instruction, obtaining a to-be-detected image, processing the to-be-detected image by using a preset deep learning model to obtain an initial key point identification result, and correcting the initial key point identification result based on prior knowledge and a key part topology fusion strategy to obtain a target key point identification result. The application constructs a deep learning model based on an OBB rotating bounding box regression function. The deep learning model can obtain more accurate initial key point identification results based on multi-scale fusion features and multi-semantics hierarchical fusion features. On this basis, the initial key point identification results are corrected based on prior knowledge and a key part topology fusion strategy, and more accurate target key point identification results are obtained.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to key point recognition methods and related equipment. Background Technology

[0002] Traditional keypoint detection relies on pictorial structure template matching, constraining the relative positions of components through a spring deformation relationship model. While this can describe geometric priors, its robustness drops sharply under special scenarios such as low signal-to-noise ratio, drastic lighting changes, occlusion, or sudden changes in target pose. Deep learning methods are divided into top-down and bottom-up approaches: the former detects keypoints first and then regresses them target-by-target, represented by networks such as HRNet and Graph-PCNN, but is limited by detection box errors and computational cost; the latter performs global detection first and then clustering, represented by networks such as OpenPose and PifPaf, relying on high-resolution heatmaps and affinity fields, but the association error rate increases when the image is blurred or the target is truncated, and integer linear programming is time-consuming. The above models generally have a large number of parameters and high memory consumption, making it difficult to run in real time on the device, and they are not specifically designed for extreme conditions such as low light, strong noise, and small targets, resulting in poor keypoint detection performance. Summary of the Invention

[0003] The main objective of this application is to provide a key point recognition method and related equipment, aiming to solve the technical problem of how to improve the accuracy of key point recognition.

[0004] To achieve the above objectives, this application proposes a key point identification method, which includes:

[0005] In response to key point recognition instructions, acquire the image to be detected;

[0006] The image to be detected is processed using a preset deep learning model to obtain initial keypoint recognition results. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF module and a PAN structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct the initial detection box.

[0007] Based on prior knowledge and a topology fusion strategy for key parts, the initial key point identification result is inferred and corrected to obtain the target key point identification result.

[0008] In one embodiment, the step of processing the image to be detected using a preset deep learning model to obtain initial key point recognition results further includes:

[0009] The image to be detected is processed using a rotating bounding box object detection and key point recognition network in a preset deep learning model to obtain multi-scale fusion features and multi-semantic level fusion features corresponding to the image to be detected.

[0010] The multi-scale fusion features and the multi-semantic level fusion features are processed using the detection head module in the preset deep learning model to determine the initial detection box corresponding to the image to be detected, the regression parameters corresponding to the initial detection box, and the regression parameters corresponding to the key point coordinates.

[0011] Based on the regression parameters corresponding to the initial detection box, the initial detection box is adjusted to obtain the target detection box;

[0012] Based on the target detection box, the regression parameters corresponding to the key point coordinates, and the Gaussian heatmap, the key point regression module in the preset deep learning model is used to perform key point regression operation to obtain the initial key point recognition result.

[0013] In one embodiment, the step of reasoning and correcting the initial key point identification result based on prior knowledge and a key part topology fusion strategy to obtain the target key point identification result further includes:

[0014] Based on prior knowledge of the coordinates of key parts of the aircraft in the physical coordinate system, a physical space topological relationship is constructed, and based on the initial key point recognition results, a topological relationship of pixel coordinates on the image is constructed.

[0015] Based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of pixel coordinates on the image, the initial key point recognition result is inferred and corrected to obtain the target key point recognition result.

[0016] In one embodiment, the step of reasoning and correcting the initial key point recognition result based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of the pixel coordinates on the image to obtain the target key point recognition result further includes:

[0017] A matching analysis is performed on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result;

[0018] Based on the matching analysis results, reliable key point identification results are determined from the initial key point identification results;

[0019] Based on the key component topology fusion strategy, the reliable key point identification results, and the known relative relationships between different key components, the initial key point identification results are inferred and corrected to obtain the target key point identification results.

[0020] In one embodiment, the step of performing matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result further includes:

[0021] A matching analysis is performed on the first vector in the physical space topology and the second vector in the pixel coordinate topology of the image to calculate the magnitude difference and the included angle between the first vector and the second vector.

[0022] Determine whether the difference in modulus length is less than or equal to a preset threshold for the difference in modulus length to obtain a first determination result, and determine whether the included angle value is less than or equal to a preset threshold for the included angle to obtain a second determination result;

[0023] Based on the first judgment result and the second judgment result, the matching analysis result is obtained.

[0024] In one embodiment, before the step of processing the image to be detected using a preset deep learning model to obtain the initial key point recognition result, the method further includes:

[0025] Acquire sample data and the current deep learning model, wherein the data processing result corresponding to the sample data is the first recognition result;

[0026] The deep learning model is processed using the current deep learning model to obtain a second recognition result;

[0027] Based on a preset loss function, the loss between the first recognition result and the second recognition result is calculated to obtain a ranking loss value, which includes the target loss Lobj and the predicted box loss Lbox.

[0028] Determine whether the sorting loss value is less than or equal to a preset sorting loss threshold;

[0029] If the value is greater than the target value, adjust the parameters of the current deep learning model. Based on the adjusted parameters, return to the step of using the current deep learning model to process the deep learning model and obtain the second recognition result. Continue until the ranking loss value is less than or equal to the preset ranking loss threshold to obtain the preset deep learning model.

[0030] Furthermore, to achieve the above objectives, this application also proposes a key point recognition device, which includes:

[0031] The acquisition module is used to acquire the image to be detected in response to a key point recognition command;

[0032] An image processing module is used to process the image to be detected using a preset deep learning model to obtain initial keypoint recognition results. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF (Special Power Filtering) module and a PAN (Positioning Panel) structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct initial detection boxes.

[0033] The reasoning correction module is used to reason and correct the initial key point identification result based on prior knowledge and the key part topology fusion strategy to obtain the target key point identification result.

[0034] In one embodiment, the image processing module further includes:

[0035] The first data processing unit is used to process the image to be detected using a rotating bounding box target detection and key point recognition network in a preset deep learning model, so as to obtain the multi-scale fusion features and multi-semantic level fusion features corresponding to the image to be detected.

[0036] The second data processing unit is used to process the multi-scale fusion features and the multi-semantic level fusion features using the detection head module in the preset deep learning model, and to determine the initial detection box corresponding to the image to be detected, the regression parameters corresponding to the initial detection box, and the regression parameters corresponding to the key point coordinates.

[0037] An adjustment unit is used to adjust the initial detection box based on the regression parameters corresponding to the initial detection box to obtain the target detection box;

[0038] The third data processing unit is used to perform key point regression operation using the key point regression module in the preset deep learning model based on the target detection box, the regression parameters corresponding to the key point coordinates, and the Gaussian heatmap, to obtain the initial key point recognition result.

[0039] In one embodiment, the inference correction module further includes:

[0040] The construction unit is used to construct the physical space topology based on the coordinates of key parts of the aircraft in the physical coordinate system of the aircraft body, and to construct the topology of pixel coordinates on the image based on the initial key point recognition results.

[0041] The first inference correction unit is used to infer and correct the initial key point recognition result based on the key part topology fusion strategy, the physical space topology relationship and the topology relationship of the pixel coordinates on the image, so as to obtain the target key point recognition result.

[0042] In one embodiment, the inference correction module further includes:

[0043] The matching analysis unit is used to perform matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image, and obtain the matching analysis result.

[0044] The determining unit is used to determine reliable key point recognition results in the initial key point recognition results based on the matching analysis results;

[0045] The second reasoning correction unit is used to reason and correct the initial key point identification result based on the key part topology fusion strategy, the reliable key point identification result, and the known relative relationship between different key parts, so as to obtain the target key point identification result.

[0046] In one embodiment, the inference correction module further includes:

[0047] The first calculation unit is used to perform matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image, so as to calculate the magnitude difference and the included angle value corresponding to the first vector and the second vector.

[0048] The first judgment unit is used to determine whether the modulus difference is less than or equal to a preset modulus difference threshold, to obtain a first judgment result, and to determine whether the included angle is less than or equal to a preset included angle threshold, to obtain a second judgment result.

[0049] The obtaining unit is used to obtain a matching analysis result based on the first judgment result and the second judgment result.

[0050] In one embodiment, the key point recognition device further includes a model training module, which further includes:

[0051] An acquisition unit is used to acquire sample data and the current deep learning model, wherein the data processing result corresponding to the sample data is the first recognition result;

[0052] The fourth data processing unit is used to process the deep learning model using the current deep learning model to obtain the second recognition result;

[0053] The second calculation unit is used to calculate the loss between the first recognition result and the second recognition result based on a preset loss function to obtain a ranking loss value, wherein the ranking loss value includes the target loss Lobj and the predicted box loss Lbox;

[0054] The second judgment unit is used to determine whether the sorting loss value is less than or equal to a preset sorting loss threshold.

[0055] The iterative training unit is used to adjust the parameters of the current deep learning model if the ranking loss value is greater than the preset ranking loss threshold, and return to the step of using the current deep learning model to process the deep learning model and obtain the second recognition result based on the current deep learning model with the adjusted parameters, until the ranking loss value is less than or equal to the preset ranking loss threshold, and obtain the preset deep learning model.

[0056] In addition, to achieve the above objectives, this application also proposes a key point recognition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the key point recognition method as described above.

[0057] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the key point recognition method described above.

[0058] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the key point recognition method described above.

[0059] One or more technical solutions proposed in this application have at least the following technical effects:

[0060] This application proposes a keypoint recognition method and related equipment, relating to the field of data processing technology. In related technologies, traditional keypoint detection relies on pictorial structure template matching, constraining the relative positions of components through a spring deformation relationship model. While this can describe geometric priors, the priors fail and robustness drops sharply in special scenarios such as low signal-to-noise ratio, drastic lighting changes, occlusion, or sudden changes in target pose. Deep learning methods are divided into top-down and bottom-up approaches: the former detects keypoints first and then regresses them target-by-target, represented by networks such as HRNet and Graph-PCNN, but is limited by detection box errors and computational cost; the latter performs global detection first and then clustering, represented by OpenPose and PifPaf, relying on high-resolution heatmaps and affinity fields. The association error rate increases when images are blurred or targets are truncated, and integer linear programming is time-consuming. These models generally have large parameter counts and high memory consumption, making them difficult to run in real-time on the device, and they are not designed for extreme scenarios such as low light, strong noise, and small targets. Compared to the previous method of specially designed end conditions that resulted in poor keypoint detection performance, this application first acquires the image to be detected in response to a keypoint recognition instruction. Then, it processes the image using a preset deep learning model to obtain an initial keypoint recognition result. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function and includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF (Special Perception Policy) module and a PAN (Positioning Area) structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct an initial detection box. Finally, based on prior knowledge and a key part topology fusion strategy, the initial keypoint recognition result is inferred and corrected to obtain the target keypoint recognition result.

[0061] It is understood that this application constructs a deep learning model based on the OBB rotated bounding box regression function. This deep learning model can obtain more accurate initial key point recognition results based on multi-scale fusion features and multi-semantic level fusion features. On this basis, combined with prior knowledge and key part topology fusion strategy, the initial key point recognition results are inferred and corrected to obtain more accurate target key point recognition results. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating an embodiment of the key point identification method of this application.

[0065] Figure 2 This is a flowchart illustrating Embodiment 2 of the key point identification method of this application;

[0066] Figure 3 This is a flowchart illustrating Embodiment 3 of the key point identification method of this application;

[0067] Figure 4 This is a schematic diagram of the module structure of the key point recognition device in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the key point recognition method in this application embodiment.

[0069] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0070] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0071] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0072] The main solution in this application's embodiments is:

[0073] In this embodiment, for ease of description, the key point recognition device will be used as the execution subject in the following description.

[0074] Due to the limitations of related technologies, traditional keypoint detection relies on pictorial structure template matching, which constrains the relative positions of components through a spring deformation relationship model. While this can describe geometric priors, the priors become invalid and robustness drops sharply in special scenarios such as low signal-to-noise ratio, drastic changes in lighting, occlusion, or sudden changes in target pose. Deep learning methods are divided into two categories: top-down and bottom-up. The former detects keypoints first and then regresses them one by one, represented by networks such as HRNet and Graph-PCNN, but is limited by detection box errors and computational cost. The latter detects globally first and then clusters, represented by networks such as OpenPose and PifPaf, which rely on high-resolution heatmaps and affinity fields. The association error rate increases when the image is blurred or the target is truncated, and integer linear programming is time-consuming. The above models generally have a large number of parameters and high memory consumption, making it difficult to run in real time on the edge. They are also not specifically designed for extreme conditions such as low light, strong noise, and small targets, resulting in poor keypoint detection performance.

[0075] This application provides a solution in which: first, in response to a keypoint recognition instruction, an image to be detected is acquired; then, the image to be detected is processed using a preset deep learning model to obtain an initial keypoint recognition result. The preset deep learning model is constructed based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF (Special Programming Factorization) module and a PAN (Positioning Area) structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is constructed based on an anchor-based method, which is used to pre-construct an initial detection box. Finally, based on prior knowledge and a key part topology fusion strategy, the initial keypoint recognition result is inferred and corrected to obtain the target keypoint recognition result.

[0076] It is understood that this application constructs a deep learning model based on the OBB rotated bounding box regression function. This deep learning model can obtain more accurate initial key point recognition results based on multi-scale fusion features and multi-semantic level fusion features. On this basis, combined with prior knowledge and key part topology fusion strategy, the initial key point recognition results are inferred and corrected to obtain more accurate target key point recognition results.

[0077] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or key point recognition device capable of performing the above functions. The following description uses a key point recognition device as an example to illustrate this embodiment and the subsequent embodiments.

[0078] Based on this, embodiments of this application provide a key point recognition method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the key point identification method of this application.

[0079] In this embodiment, the key point recognition method includes steps S100~S300:

[0080] Step S100: In response to the key point recognition instruction, acquire the image to be detected;

[0081] It should be noted that a keypoint recognition command is a signal or command that triggers the system to execute a keypoint recognition task. It can originate from the user interface (such as clicking a button or entering a command) or from other automated systems (such as monitoring systems or data stream processing systems). Receiving the command signifies that the system has begun executing the keypoint recognition task, initiating subsequent image processing and analysis procedures.

[0082] Upon receiving the instruction, the system needs to acquire one or more images for keypoint recognition. These images can come from various sources, such as cameras, image databases, and real-time video streams.

[0083] In this application, specific application scenarios may include:

[0084] In an intelligent surveillance system, when abnormal activity is detected in a certain area, a keypoint recognition command is automatically triggered. The system then acquires the current image frame from the surveillance camera and inputs it into the keypoint recognition model to determine the target's pose and the location of key parts. The starting point of this process is "acquiring the image to be detected in response to the keypoint recognition command."

[0085] Step S200: The image to be detected is processed using a preset deep learning model to obtain initial key point recognition results. The preset deep learning model is built based on the OBB rotated bounding box regression function. The preset deep learning model includes a rotated bounding box object detection and key point recognition network structure and an output layer structure. The rotated bounding box object detection and key point recognition network also includes an SPPF module and a PAN structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a key point regression module. The detection head module is built based on an anchor-based method, which is used to pre-build initial detection boxes.

[0086] It should be noted that the preset deep learning model is a deep learning model specifically designed for object detection and keypoint recognition. It incorporates Oriented Bounding Box (OBB) regression, enabling it to handle rotated targets and simultaneously output object detection boxes and keypoint coordinates.

[0087] It should be noted that the model consists of the following components:

[0088] This network architecture for rotating bounding box object detection and keypoint recognition can simultaneously perform object detection (outputting rotated bounding boxes) and keypoint recognition (outputting keypoint coordinates). It supports rotated bounding box (OBB) regression, meaning the model can detect the rotation angle of the object, not just the traditional axis-aligned bounding box (AABB).

[0089] The SPPF (Spatial Pyramid Pooling-Fast) module is used to generate and fuse features at different scales. This module replaces the traditional single large pooling kernel (such as 9×9 or 13×13) with multiple cascaded small pooling kernels (such as 5×5), thus improving running speed while retaining original functionality. It can fuse feature maps from different receptive fields, enhancing the model's ability to detect targets at different scales.

[0090] The PAN (Path Aggregation Network) structure is used to fuse features from different semantic levels. Building upon the traditional Feature Pyramid Network (FPN), the PAN structure introduces bottom-up path augmentation. It not only utilizes high-level features for object detection but also fully leverages shallow features (such as edges and shapes) to improve detection accuracy, performing particularly well in small object detection. Furthermore, the PAN structure is simple and computationally inexpensive, adding only a small number of convolutional operations.

[0091] The output layer structure comprises a detection head module and a keypoint regression module. The detection head module is built using an anchor-based approach. It pre-constructs initial detection boxes (anchor boxes), and then the network adjusts the position and size of these boxes to match the actual location of the target, enabling rapid generation of candidate boxes and subsequent optimization. The keypoint regression module is used to regress and locate keypoints based on the detected target bounding boxes. It accurately predicts the location of keypoints by learning the offset between the keypoints and the center of the bounding boxes.

[0092] In the Top-Down keypoint detection branch, the keypoints required for the target are mainly regressed based on the target bounding box. The corresponding offset to be regressed is:

[0093]

[0094]

[0095] in, and This represents the normalized offset of keypoints in the horizontal and vertical directions. and The values ​​represent the horizontal and vertical coordinates of the ground truth keypoints in the image pixel coordinate system. truth.x and truth.y represent the x and y coordinates of the center of the ground truth bounding box, and truth.w and truth.h represent the width and height of the ground truth bounding box.

[0096] In the Bottom-Up keypoint heatmap prediction branch, the following Gaussian kernel is mainly used to generate the heatmap to be predicted (generally 4 times the size of the image downsampled):

[0097]

[0098] in, The key points are mapped to their coordinates on the heatmap. That is, the coordinates of each point on the heat map. It is related to the target size ( and The relevant standard deviation. If two Gaussian distributions of a certain class overlap, the one with the largest inter-element difference is used directly. Furthermore, since the predicted heatmap scale is generally smaller than the original image, to more accurately predict the location of key points, an additional offset due to the downsampling coordinate bias can be added. The predictions show that the scale of the bias plot is consistent with that of the heatmap.

[0099] In this embodiment, the specific workflow is as follows:

[0100] First, the image to be detected is input. Then, features are extracted from the image through a backbone network (e.g., a C3 module combined with Darknet53). These features are then fused at multiple scales using the SPPF module to enhance the detection capability for targets of different sizes. The features are further fused semantically using a PAN structure to improve detection accuracy. In the output layer, the detection head module generates initial bounding boxes using an anchor-based method and performs accurate regression on the keypoints within each bounding box using a keypoint regression module. Finally, the initial keypoint recognition results are output, including the rotated bounding box of the target and the keypoint coordinates.

[0101] Understandably, in this application, the OBB regression function allows the model to detect rotating targets, improving its adaptability to complex pose and orientation changes. The SPPF module enhances the model's ability to detect targets at different scales through multi-scale feature fusion, performing particularly well in detecting small and large targets. The PAN structure fuses features from different semantic levels, fully utilizing the advantages of shallow and deep features to improve detection accuracy and robustness. The anchor-based method pre-constructs initial detection boxes, and the network adjusts these boxes to match the target, improving detection speed and efficiency.

[0102] It is understood that, in this embodiment, by combining OBB regression, multi-scale feature fusion (SPPF), semantic hierarchical fusion (PAN), and anchor-based detection head, the model can achieve high-precision key point recognition in complex scenes.

[0103] Specifically, the step of processing the image to be detected using a preset deep learning model to obtain the initial key point recognition result further includes steps S210~S240:

[0104] Step S210: The image to be detected is processed using the rotating bounding box target detection and key point recognition network in the preset deep learning model to obtain the multi-scale fusion features and multi-semantic level fusion features corresponding to the image to be detected.

[0105] It should be noted that multi-scale fusion features refer to features generated through specific modules of the model (such as the SPPF module), which can capture target information at different scales (sizes) in the image. Specifically:

[0106] It's worth noting that the SPPF module, by cascading multiple small pooling kernels (e.g., 5×5) instead of the traditional single large pooling kernel (e.g., 9×9 or 13×13), improves running speed while retaining original functionality. It can fuse feature maps from different receptive fields, thereby enhancing the model's ability to detect targets at different scales. For example, small targets may require smaller receptive fields to capture details, while large targets require larger receptive fields to understand the overall structure. After processing by the SPPF module, the model can generate multi-scale fused features containing rich information from local details to global structure, which helps improve the accuracy of target detection and keypoint recognition.

[0107] It should be noted that multi-semantic level fusion features refer to features generated through a specific structure of the model (such as the PAN structure), which can fuse information from different semantic levels.

[0108] It's important to note that the PAN structure introduces bottom-up path augmentation on top of the traditional Feature Pyramid Network (FPN). This not only utilizes high-level features for object detection but also fully leverages shallow features (such as edges and shapes). Shallow features typically contain more detailed information (e.g., edges, textures), while high-level features contain more abstract semantic information (e.g., the overall shape and category of the target). By fusing these two types of features, the PAN structure can better understand the details and semantics of the target. After processing with the PAN structure, the model can generate multi-semantic level fusion features. These features combine shallow and high-level semantic information, helping to improve the model's adaptability to complex scenes and its detection accuracy.

[0109] It is understood that in this embodiment, by processing the image to be detected through the rotating bounding box target detection and key point recognition network, the model can generate two important features. These features provide a rich information foundation for subsequent target detection and key point recognition, which helps to improve the performance and robustness of the entire system.

[0110] Step S220: Use the detection head module in the preset deep learning model to process the multi-scale fusion features and the multi-semantic level fusion features to determine the initial detection box corresponding to the image to be detected, the regression parameters corresponding to the initial detection box, and the regression parameters corresponding to the key point coordinates.

[0111] It should be noted that the detection head module is a crucial part of the deep learning model used for object detection and keypoint recognition. Its main function is to generate specific detection results based on extracted features (such as multi-scale fusion features and multi-semantic level fusion features). It typically includes the following core tasks:

[0112] Object detection: Determining the location of an object in an image, usually achieved by generating a bounding box.

[0113] Key point identification: Determining the location of key points of a target, usually achieved by regressing the coordinates of key points.

[0114] It should be noted that the initial detection box refers to the target location initially predicted by the model, usually represented as a bounding box. In this embodiment, the detection head module uses an anchor-based method to generate the initial detection box. Specifically:

[0115] It's important to note that Anchor Boxes refer to a predefined series of candidate boxes (anchor points) of fixed size and proportion in an image, covering the entire image. Furthermore, the detection head module learns the offset between the target and the anchor points, adjusting the position and size of the anchor points to more closely approximate the actual target location. The initial detection boxes are the adjusted anchor points, representing the model's preliminary prediction of the target location.

[0116] It should be noted that the regression parameters of the initial detection box are used to adjust the position and size of the initial detection box to make it closer to the actual target. The regression parameters of the initial detection box include:

[0117] Center point offset: This represents the offset between the center of the detection box and the center of the anchor point box.

[0118] Width and height adjustment: This indicates the adjustment ratio of the width and height of the detection box relative to the anchor point box.

[0119] Rotation angle: For rotated bounding boxes (OBB), the rotation angle of the detection box also needs to be adjusted.

[0120] Understandably, these regression parameters enable the model to more accurately pinpoint the location and shape of the target.

[0121] It should be noted that the regression parameters corresponding to the keypoint coordinates are used to adjust the position of the keypoints to make them closer to the actual keypoint positions. The regression parameters of the initial detection box include: keypoint offset (representing the offset of the keypoint relative to the center of the detection box or a certain reference point).

[0122] Understandably, these regression parameters enable the model to more accurately pinpoint the location of key points.

[0123] Understandably, by adjusting the regression parameters, the model can more accurately locate targets and key points, reducing false positives and false negatives. Combining multi-scale and multi-semantic level features, the detection head module can better handle target detection and key point recognition tasks in complex scenes, especially when the target scale varies greatly or there is severe occlusion. The initial detection boxes and key point coordinates form the basis for further optimization and correction, such as through reasoning and correction based on prior knowledge and topological relationships.

[0124] In this embodiment, by adjusting the positions of the anchor point frame and key points, a basis is provided for subsequent optimization and correction, thereby improving the detection accuracy and robustness of the entire system.

[0125] Step S230: Based on the regression parameters corresponding to the initial detection box, adjust the initial detection box to obtain the target detection box;

[0126] It should be noted that the adjustment process optimizes the initial detection box based on regression parameters to make it more accurately match the actual target. The specific steps are as follows:

[0127] First, adjust the center position of the initial detection box based on the center point offset in the regression parameters, thereby ensuring that the center of the detection box is closer to the actual center of the target.

[0128] Furthermore, the width and height of the initial detection box are scaled according to the width and height adjustment ratio in the regression parameters, thereby making the size of the detection box closer to the actual size of the target.

[0129] Furthermore, based on the rotation angle in the regression parameters, the rotation angle of the initial detection box is adjusted, thereby making the direction of the detection box closer to the actual direction of the target, which is especially suitable for detecting rotating targets.

[0130] Finally, after the above adjustments, the initial detection box was optimized into an object detection box. The object detection box is the final detection result, which more accurately represents the position, size, and orientation of the object.

[0131] Step S240: Based on the target detection box, the regression parameters corresponding to the key point coordinates, and the Gaussian heatmap, perform key point regression operation using the key point regression module in the preset deep learning model to obtain the initial key point recognition result.

[0132] It's important to note that a Gaussian heatmap is an image representing the probability distribution of keypoint locations. Keypoints have higher pixel values, with surrounding pixels showing gradually decreasing values. The Gaussian heatmap provides thermal distribution information about keypoint locations, helping the module determine their precise location. Peak positions in the heatmap typically correspond to the actual locations of the keypoints.

[0133] In this embodiment, the module first determines the approximate search range of the key points based on the position and size of the target detection box.

[0134] Furthermore, by using the regression parameters corresponding to the key point coordinates, the positions of the key points are initially adjusted to make them closer to the actual key point positions within the target detection box.

[0135] Furthermore, the module analyzes the thermal distribution in the Gaussian heatmap to identify peak locations. These peak locations are then used as the precise locations of key points, allowing for further adjustment of their coordinates.

[0136] Understandably, Gaussian heatmaps provide probability distribution information of key point locations, which can help modules more accurately locate key points in complex scenes (such as occlusion or blur).

[0137] Understandably, this process significantly improves the accuracy and robustness of keypoint recognition, providing a more reliable input for subsequent optimization and correction.

[0138] Step S300: Based on prior knowledge and the key part topology fusion strategy, the initial key point identification result is inferred and corrected to obtain the target key point identification result.

[0139] It should be noted that after a deep learning model (such as a rotated bounding box object detection and keypoint recognition network) processes the image to be detected, it outputs initial keypoint recognition results. These results include the rotated bounding box of the object and the coordinates of the keypoints. However, due to the influence of complex scenes (such as occlusion, low signal-to-noise ratio, illumination changes, etc.), the initial recognition results may contain errors or be inaccurate.

[0140] It should be noted that prior knowledge refers to structured information known in a specific domain and relevant to the target. In keypoint recognition tasks, prior knowledge can include:

[0141] The geometry of the target: for example, the relative positions and proportions between key parts of an aircraft, such as its wings, fuselage, and tail.

[0142] Topological relationships: the connections and spatial layout between key components, for example, wings are always located on both sides of the fuselage, and tail fins are located at the rear of the fuselage.

[0143] Physical constraints: The physical characteristics of the target, such as the distance range between key points and the angular relationship.

[0144] This prior knowledge can help the model better understand the structure of the target, thereby correcting the initial recognition results.

[0145] It should be noted that the topology fusion strategy utilizes prior knowledge of topological relationships to reason and correct the initial keypoint identification results. The specific steps are as follows:

[0146] First, based on the initial keypoint recognition results, the topological relationships between keypoints are constructed in the image coordinate system. For example, the distances and angles between keypoints are calculated. Then, based on prior knowledge, a topological relationship model of the target in physical space is constructed. For example, the relative positions and proportions of key aircraft parts in the aircraft's coordinate system are known.

[0147] Furthermore, for each keypoint, its vector relationship (such as direction and distance) in image space is calculated. For example, the vector in image space is matched with the prior vector in physical space. If the magnitude and angle of the vectors meet a preset threshold, the vector is considered a match, confirming the reliability of the recognition result.

[0148] Furthermore, vector matching analysis is used to determine the reliability of the initial keypoint identification results. If the matching results for some keypoints are unreliable, corrections are necessary. Using known reliable keypoints and prior topological relationships, the accurate locations of other keypoints are inferred through geometric transformations and spatial mapping. For example, if the keypoints on the wing are accurately identified, the locations of the fuselage keypoints can be calculated using their known proportional relationship with the fuselage.

[0149] Ultimately, after the aforementioned reasoning and correction process, the target key point recognition results are more accurate and robust. This result not only considers the output of the deep learning model but also incorporates prior knowledge and topological constraints, effectively reducing false positives and false negatives, and performing exceptionally well in complex scenarios.

[0150] In this application, the practical application scenario can be:

[0151] In an aircraft keypoint recognition task, a deep learning model initially identifies keypoints on the wings, fuselage, and tail. However, due to occlusion, the keypoint recognition result for the tail may be inaccurate. In this case, by using prior knowledge (such as the known proportional relationship between the wings and tail) and topology fusion strategies (such as vector matching analysis), the accurate location of the tail keypoints can be inferred, thereby correcting the initial recognition result.

[0152] It is understood that, in this embodiment, through vector matching and inference computation, this strategy can significantly improve the accuracy and robustness of key point recognition, especially in complex scenes and under low signal-to-noise ratio conditions.

[0153] This application proposes a keypoint recognition method and related equipment, relating to the field of data processing technology. In related technologies, traditional keypoint detection relies on pictorial structure template matching, constraining the relative positions of components through a spring deformation relationship model. While this can describe geometric priors, the priors fail and robustness drops sharply in special scenarios such as low signal-to-noise ratio, drastic lighting changes, occlusion, or sudden changes in target pose. Deep learning methods are divided into top-down and bottom-up approaches: the former detects keypoints first and then regresses them target-by-target, represented by networks such as HRNet and Graph-PCNN, but is limited by detection box errors and computational cost; the latter performs global detection first and then clustering, represented by OpenPose and PifPaf, relying on high-resolution heatmaps and affinity fields. The association error rate increases when images are blurred or targets are truncated, and integer linear programming is time-consuming. These models generally have large parameter counts and high memory consumption, making them difficult to run in real-time on the device, and they are not designed for extreme scenarios such as low light, strong noise, and small targets. Compared to the previous method of specially designed end conditions that resulted in poor keypoint detection performance, this application first acquires the image to be detected in response to a keypoint recognition instruction. Then, it processes the image using a preset deep learning model to obtain an initial keypoint recognition result. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function and includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF (Special Perception Policy) module and a PAN (Positioning Area) structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct an initial detection box. Finally, based on prior knowledge and a key part topology fusion strategy, the initial keypoint recognition result is inferred and corrected to obtain the target keypoint recognition result.

[0154] It is understood that this application constructs a deep learning model based on the OBB rotated bounding box regression function. This deep learning model can obtain more accurate initial key point recognition results based on multi-scale fusion features and multi-semantic level fusion features. On this basis, combined with prior knowledge and key part topology fusion strategy, the initial key point recognition results are inferred and corrected to obtain more accurate target key point recognition results.

[0155] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2The step of reasoning and correcting the initial key point identification result based on prior knowledge and key part topology fusion strategy to obtain the target key point identification result further includes steps A100~A200:

[0156] Step A100: Based on the prior knowledge of the coordinates of key parts of the aircraft in the physical coordinate system, construct the physical space topology, and based on the initial key point recognition results, construct the topology of pixel coordinates on the image.

[0157] It should be noted that prior knowledge refers to the known relative positions and proportions of key aircraft components (such as wings, fuselage, and tail) in the aircraft's physical coordinate system. For example, the position of the wing relative to the fuselage, and the relative angle between the tail and the fuselage.

[0158] In this embodiment, a topological model is constructed based on these known physical coordinates and relationships to describe the geometric and spatial relationships between key components. These relationships include distance, angle, and relative position. This provides a reference standard for subsequent key point correction, ensuring that the key point identification results conform to the rationality of the physical structure.

[0159] Furthermore, based on the initial recognition results, the topological relationships between key points, such as distances and angles, are calculated. For example, the distances and angles between key points on the wing and key points on the fuselage are calculated, and an image-based topological relationship is provided for comparison and correction with the physical space topological relationship.

[0160] Step A200: Based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of pixel coordinates on the image, the initial key point recognition result is inferred and corrected to obtain the target key point recognition result.

[0161] The key component topology fusion strategy refers to a reasoning strategy based on prior knowledge and topological relationships, used to correct the initial key point recognition results. Specifically, the key component topology fusion strategy includes fusing physical spatial topological relationships and image topological relationships, as well as reasoning corrections.

[0162] Integrate physical space topology and image topology: Compare the physical space topology constructed in step A100 with the topology of pixel coordinates on the image.

[0163] Inference and Correction: First, the vectors (such as direction and distance) in the image's topological relationships are compared with those in the physical topological relationships. If they match, the initial recognition result is considered reliable; otherwise, correction is needed. Further, known proportions and angular relationships in the physical topological relationships are used to adjust mismatched keypoints. For example, if the position of a keypoint deviates significantly from its expected position in the physical topological relationships, its position is recalculated based on known relative relationships. For missing or inaccurate keypoints, inference calculations are performed using known reliable keypoints and topological relationships to determine their accurate positions.

[0164] It should be noted that the target key point identification result refers to the corrected key point coordinates, which are more consistent with the physical structure and topological relationships.

[0165] Understandably, by combining prior knowledge and topological relationships, the corrected results have higher accuracy and robustness in complex scenarios (such as occlusion and low signal-to-noise ratio).

[0166] Specifically, the step of reasoning and correcting the initial key point recognition result based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of the pixel coordinates on the image to obtain the target key point recognition result further includes steps A210~A230:

[0167] Step A210: Perform a matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result;

[0168] In physical space topology, the first vector refers to a vector constructed based on prior knowledge, describing the relative position and orientation of critical components of an aircraft in the physical coordinate system. For example, the physical distance and orientation between critical points on the wing and critical points on the fuselage.

[0169] The second vector in the topological relationship of pixel coordinates on an image refers to a vector constructed based on the initial keypoint recognition results, describing the relative position and orientation between keypoints in the image. For example, the pixel distance and orientation between detected wing keypoints and fuselage keypoints in an image.

[0170] It should be noted that matching analysis refers to comparing the first vector and the second vector to determine whether they match. A matching vector indicates that the initial keypoint recognition result is consistent with prior physical knowledge, while a non-matching vector indicates that there may be errors.

[0171] It should be noted that the matching analysis method can calculate the difference between two vectors, including the magnitude (distance) and the included angle.

[0172] Specifically, the step of performing a matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result further includes steps A211~A213:

[0173] Step A211: Perform a matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to calculate the difference in magnitude and the included angle between the first vector and the second vector.

[0174] It should be noted that the modulus difference refers to the difference between the modulus (length) of the first vector and the second vector.

[0175] It should be noted that the included angle value refers to the angle between the first vector and the second vector.

[0176] Step A212: Determine whether the modulus difference is less than or equal to a preset modulus difference threshold to obtain a first determination result, and determine whether the included angle value is less than or equal to a preset included angle threshold to obtain a second determination result;

[0177] It should be noted that the preset modulus difference threshold refers to the maximum allowable modulus difference.

[0178] It should be noted that the preset included angle threshold refers to the maximum allowed included angle value.

[0179] In this embodiment, it is determined whether the difference in modulus length is less than or equal to a preset modulus length difference threshold, and whether the included angle value is less than or equal to a preset included angle threshold.

[0180] Step A213: Based on the first judgment result and the second judgment result, obtain the matching analysis result.

[0181] In this embodiment, the final matching analysis result is obtained by combining the first judgment result (magnitude matching) and the second judgment result (direction matching). If both the magnitude and direction match (i.e., both judgment results are "matched"), the first vector and the second vector are considered to match. If either the magnitude or direction does not match (i.e., either judgment result is "not matched"), the vectors are considered not to match. Identifying whether each vector pair matches provides a basis for subsequent key point reliability judgments.

[0182] Understandably, this process ensures the accuracy and reliability of vector matching analysis by quantifying the differences between vectors and setting reasonable thresholds, providing an important basis for the subsequent screening and correction of key point identification results.

[0183] Step A220: Based on the matching analysis results, determine the reliable key point identification results in the initial key point identification results;

[0184] A reliable keypoint recognition result means that the initial keypoint recognition result corresponding to the matched vector is considered reliable.

[0185] In this embodiment, for matching vectors, the keypoint recognition results are confirmed to be reliable. For non-matching vectors, the keypoint recognition results are marked as unreliable.

[0186] Step A230: Based on the key part topology fusion strategy, the reliable key point identification results, and the known relative relationships between different key parts, the initial key point identification results are inferred and corrected to obtain the target key point identification results.

[0187] It should be noted that reliable keypoint identification results can be used as a reference for correction, ensuring that the corrected keypoints maintain a consistent topological relationship with known reliable points.

[0188] The known relative relationships between different key parts refer to using the relative positions, distances, and angles in prior knowledge to deduce and correct unreliable key points.

[0189] In this embodiment, for unreliable key points, reasoning and correction are performed using known topological relationships and physical structures. For example, if the key point identification results for the wing are reliable, but the key points for the tail are unreliable, the position of the tail key point is recalculated using the known relative relationship between the wing and the tail.

[0190] Furthermore, based on reliable keypoints and known topological relationships, the corrected positions of unreliable keypoints are calculated. Geometric transformations (such as translation and rotation) and scaling adjustments are used to ensure that the corrected keypoints are consistent with the physical structure. Finally, the target keypoint identification results are output, i.e., the final corrected keypoint positions.

[0191] Understandably, this process, through gradual screening and correction, significantly improves the accuracy and robustness of key point identification, especially performing exceptionally well in complex scenarios.

[0192] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before the step of processing the image to be detected using a preset deep learning model to obtain the initial key point recognition result, the method further includes steps B100 to B500:

[0193] Step B100: Obtain sample data and the current deep learning model, wherein the data processing result corresponding to the sample data is the first recognition result;

[0194] Sample data refers to the dataset used to train and validate deep learning models. The processing results of this data (i.e., the true labels) are known and are called the first recognition results.

[0195] The current deep learning model refers to the model that is being trained or optimized, and its initial state may be a pre-trained model or a randomly initialized model.

[0196] The first identification result refers to the true labeling result of the sample data, which is used to evaluate the performance of the model.

[0197] Step B200: Process the deep learning model using the current deep learning model to obtain a second recognition result;

[0198] In this embodiment, sample data is input into the current deep learning model. The model performs forward propagation based on the current parameters and outputs prediction results (such as detection boxes and keypoint locations).

[0199] The second recognition result refers to the model's prediction results, including the object detection box (position, size, angle) and keypoint coordinates. These results will be used for subsequent loss calculations to evaluate model performance.

[0200] Step B300: Based on a preset loss function, calculate the loss between the first recognition result and the second recognition result to obtain a ranking loss value, which includes the target loss Lobj and the predicted box loss Lbox;

[0201] Target loss (Lobj): measures the accuracy of the model in predicting whether a target exists or not, and is usually expressed as binary cross-entropy (BCE).

[0202] Lbox loss (predicted bounding box): measures the difference between the predicted bounding box and the ground truth bounding box. Commonly used methods include CIoU Loss, which takes into account the center distance, overlap, and scale differences of the bounding boxes.

[0203] The loss function for detection includes target loss (Lobj) and predicted bounding box loss (Lbox). The target loss uses BCE (Binary CrossEntropy) Loss, and the Lbox loss uses CIoU Loss. Taking into account the distance between the predicted bounding box and the ground truth center, the degree of overlap, and scale differences, the target bounding box prediction is trained. The specific formula for CIoU Loss is as follows:

[0204]

[0205]

[0206]

[0207] In addition, to achieve key point recognition, Wing Loss, commonly used in face recognition, was added as the key point regression loss to the original loss function. The calculation formula is as follows:

[0208]

[0209] in,

[0210] It should be noted that in the CIoU formula, IoU represents the intersection-over-union ratio (IoU) between the predicted and ground truth boxes, D represents the Euclidean distance between the center points of the predicted and ground truth boxes, Dc represents the diagonal length of the smallest bounding box containing both the predicted and ground truth boxes, α represents a coefficient used to adjust the weight of the aspect ratio loss, and θ represents the aspect ratio consistency loss (measuring the difference in aspect ratio between the predicted and ground truth boxes). In the θ expression, w and h are the width and height of the predicted box. The actual frame width. This represents the actual bounding box height. In Wing expressions, x represents the error. This represents the boundary value and controls the range of influence of the Log part (amplifying small errors). The smoothness of the log curve is controlled by C, which is a constant term.

[0211] Compared to common L1 or L2 losses, Wing Loss is more sensitive to small and moderate errors, which helps in more accurate localization and improves robustness to outliers. One of the design goals of Wing Loss is to reduce sensitivity to outliers. In practical applications, datasets may contain label noise or inaccurate annotations. Wing Loss mitigates the impact of outliers on the loss by introducing a non-linear curve, thereby improving the model's robustness. The loss function curve of Wing Loss exhibits a linear trend when the prediction is close to the true value, which helps the model learn more stably when it is close to the correct prediction, improving the convergence speed. Wing Loss is not limited to facial landmark detection; its design philosophy and advantages have certain universality in other regression problems, thus enabling its application in multiple fields.

[0212] In this embodiment, for each sample, the target loss and the predicted bounding box loss are calculated. Then, these loss values ​​are summed to obtain the ranking loss value, which is used to evaluate the overall performance of the model.

[0213] Step B400: Determine whether the sorting loss value is less than or equal to a preset sorting loss threshold;

[0214] The preset ranking loss threshold is a predefined threshold used to determine whether the model's performance meets the requirements. If the loss value is lower than or equal to this threshold, the model performance is considered good enough, and training can be stopped; otherwise, return to the step of optimizing model parameters.

[0215] Step B500: If the value is greater than the target value, adjust the parameters of the current deep learning model. Based on the adjusted parameters, return to the step of using the current deep learning model to process the deep learning model and obtain the second recognition result, until the ranking loss value is less than or equal to the preset ranking loss threshold, and obtain the preset deep learning model.

[0216] In this embodiment, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. The model parameters are then updated based on the gradient, typically using optimization algorithms such as SGD or Adam.

[0217] Furthermore, the updated model is used to reprocess the sample data to obtain a new second recognition result. The loss value is repeatedly calculated to determine whether the threshold condition is met. This process continues until the ranking loss value is less than or equal to the preset threshold. When the ranking loss value is less than or equal to the preset threshold, the model is considered to have been optimized to a satisfactory level. At this point, the current deep learning model becomes the preset deep learning model and can be used for subsequent tasks.

[0218] Understandably, this process is a typical deep learning model training and optimization process. By iteratively adjusting the model parameters, the model's output results are made closer to the real annotations, thereby obtaining a pre-set deep learning model with optimized performance.

[0219] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the key point identification method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0220] It should be noted that all user-related data involved in this application (such as user attribute data, user behavior data, and user geographical location, etc., the data types here should be modified according to the adaptability of the solution content) were obtained with the user's permission or consent; that is to say, when this application is applied to specific products or technologies, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0221] For example, when it is necessary to obtain a user's current geographical location, a location acquisition prompt can be displayed on the user's terminal. After receiving confirmation from the user regarding the location acquisition prompt, the terminal can obtain the user's current geographical location.

[0222] This application also provides a key point recognition device, please refer to... Figure 4 The key point recognition device includes:

[0223] Acquisition module 10, the acquisition module is used to acquire the image to be detected in response to the key point recognition command;

[0224] Image processing module 20 is used to process the image to be detected using a preset deep learning model to obtain initial key point recognition results. The preset deep learning model is built based on the OBB rotated bounding box regression function. The preset deep learning model includes a rotated bounding box object detection and key point recognition network structure and an output layer structure. The rotated bounding box object detection and key point recognition network also includes an SPPF module and a PAN structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a key point regression module. The detection head module is built based on the anchor-based method, which is used to pre-build the initial detection box.

[0225] The reasoning correction module 30 is used to reason and correct the initial key point identification result based on prior knowledge and key part topology fusion strategy to obtain the target key point identification result.

[0226] In one embodiment, the image processing module further includes:

[0227] The first data processing unit is used to process the image to be detected using a rotating bounding box target detection and key point recognition network in a preset deep learning model, so as to obtain the multi-scale fusion features and multi-semantic level fusion features corresponding to the image to be detected.

[0228] The second data processing unit is used to process the multi-scale fusion features and the multi-semantic level fusion features using the detection head module in the preset deep learning model, and to determine the initial detection box corresponding to the image to be detected, the regression parameters corresponding to the initial detection box, and the regression parameters corresponding to the key point coordinates.

[0229] An adjustment unit is used to adjust the initial detection box based on the regression parameters corresponding to the initial detection box to obtain the target detection box;

[0230] The third data processing unit is used to perform key point regression operation using the key point regression module in the preset deep learning model based on the target detection box, the regression parameters corresponding to the key point coordinates, and the Gaussian heatmap, to obtain the initial key point recognition result.

[0231] In one embodiment, the inference correction module further includes:

[0232] The construction unit is used to construct the physical space topology based on the coordinates of key parts of the aircraft in the physical coordinate system of the aircraft body, and to construct the topology of pixel coordinates on the image based on the initial key point recognition results.

[0233] The first inference correction unit is used to infer and correct the initial key point recognition result based on the key part topology fusion strategy, the physical space topology relationship and the topology relationship of the pixel coordinates on the image, so as to obtain the target key point recognition result.

[0234] In one embodiment, the inference correction module further includes:

[0235] The matching analysis unit is used to perform matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image, and obtain the matching analysis result.

[0236] The determining unit is used to determine reliable key point recognition results in the initial key point recognition results based on the matching analysis results;

[0237] The second reasoning correction unit is used to reason and correct the initial key point identification result based on the key part topology fusion strategy, the reliable key point identification result, and the known relative relationship between different key parts, so as to obtain the target key point identification result.

[0238] In one embodiment, the inference correction module further includes:

[0239] The first calculation unit is used to perform matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image, so as to calculate the magnitude difference and the included angle value corresponding to the first vector and the second vector.

[0240] The first judgment unit is used to determine whether the modulus difference is less than or equal to a preset modulus difference threshold, to obtain a first judgment result, and to determine whether the included angle is less than or equal to a preset included angle threshold, to obtain a second judgment result.

[0241] The obtaining unit is used to obtain a matching analysis result based on the first judgment result and the second judgment result.

[0242] In one embodiment, the key point recognition device further includes a model training module, which further includes:

[0243] An acquisition unit is used to acquire sample data and the current deep learning model, wherein the data processing result corresponding to the sample data is the first recognition result;

[0244] The fourth data processing unit is used to process the deep learning model using the current deep learning model to obtain the second recognition result;

[0245] The second calculation unit is used to calculate the loss between the first recognition result and the second recognition result based on a preset loss function to obtain a ranking loss value, wherein the ranking loss value includes the target loss Lobj and the predicted box loss Lbox;

[0246] The second judgment unit is used to determine whether the sorting loss value is less than or equal to a preset sorting loss threshold.

[0247] The iterative training unit is used to adjust the parameters of the current deep learning model if the ranking loss value is greater than the preset ranking loss threshold, and return to the step of using the current deep learning model to process the deep learning model and obtain the second recognition result based on the current deep learning model with the adjusted parameters, until the ranking loss value is less than or equal to the preset ranking loss threshold, and obtain the preset deep learning model.

[0248] The key point recognition device provided in this application, employing the key point recognition method in the above embodiments, can solve the technical problem of key point recognition. Compared with related technologies, the beneficial effects of the key point recognition device provided in this application are the same as those of the key point recognition method provided in the above embodiments, and other technical features in the key point recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0249] This application provides a key point recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the key point recognition method in Embodiment 1 above.

[0250] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the key point recognition device in the embodiments of this application. The key point recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5The key point recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0251] like Figure 5 As shown, the key point recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the key point recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the key point identification device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show key point identification devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0252] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0253] The key point recognition device provided in this application, employing the key point recognition method described in the above embodiments, can solve the technical problem of key point recognition. Compared with related technologies, the beneficial effects of the key point recognition device provided in this application are the same as those of the key point recognition method described in the above embodiments, and other technical features of this key point recognition device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0254] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0255] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0256] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the key point recognition method in the above embodiments.

[0257] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0258] The aforementioned computer-readable storage medium may be included in the key point recognition device; or it may exist independently and not be assembled into the key point recognition device.

[0259] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the keypoint recognition device, cause the keypoint recognition device to:

[0260] In response to key point recognition instructions, acquire the image to be detected;

[0261] The image to be detected is processed using a preset deep learning model to obtain initial keypoint recognition results. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF module and a PAN structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct the initial detection box.

[0262] Based on prior knowledge and a topology fusion strategy for key parts, the initial key point identification result is inferred and corrected to obtain the target key point identification result.

[0263] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0264] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0265] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0266] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described key point recognition method, thereby solving the technical problem of key point recognition. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the key point recognition method provided in the above embodiments, and will not be repeated here.

[0267] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the key point recognition method described above.

[0268] The computer program product provided in this application can solve the technical problem of key point recognition. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the key point recognition method provided in the above embodiments, and will not be repeated here.

[0269] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A key point recognition method, characterized by, The key point identification method includes: In response to key point recognition instructions, acquire the image to be detected; The image to be detected is processed using a preset deep learning model to obtain initial keypoint recognition results. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF module and a PAN structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct the initial detection box. Based on prior knowledge and a key component topology fusion strategy, the initial key point identification result is inferred and corrected to obtain the target key point identification result. The step of reasoning and correcting the initial key point identification result based on prior knowledge and key part topology fusion strategy to obtain the target key point identification result further includes: Based on prior knowledge of the coordinates of key parts of the aircraft in the physical coordinate system, a physical space topological relationship is constructed, and based on the initial key point recognition results, a topological relationship of pixel coordinates on the image is constructed. Based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of pixel coordinates on the image, the initial key point recognition result is inferred and corrected to obtain the target key point recognition result.

2. The key point recognition method as described in claim 1, characterized in that, The step of processing the image to be detected using a preset deep learning model to obtain the initial key point recognition result further includes: The image to be detected is processed using a rotating bounding box object detection and key point recognition network in a preset deep learning model to obtain multi-scale fusion features and multi-semantic level fusion features corresponding to the image to be detected. The multi-scale fusion features and the multi-semantic level fusion features are processed using the detection head module in the preset deep learning model to determine the initial detection box corresponding to the image to be detected, the regression parameters corresponding to the initial detection box, and the regression parameters corresponding to the key point coordinates. Based on the regression parameters corresponding to the initial detection box, the initial detection box is adjusted to obtain the target detection box; Based on the target detection box, the regression parameters corresponding to the key point coordinates, and the Gaussian heatmap, the key point regression module in the preset deep learning model is used to perform key point regression operation to obtain the initial key point recognition result.

3. The key point recognition method as described in claim 1, characterized in that, The step of reasoning and correcting the initial key point recognition result based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of pixel coordinates on the image to obtain the target key point recognition result further includes: A matching analysis is performed on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result; Based on the matching analysis results, reliable key point identification results are determined from the initial key point identification results; Based on the key component topology fusion strategy, the reliable key point identification results, and the known relative relationships between different key components, the initial key point identification results are inferred and corrected to obtain the target key point identification results.

4. The key point recognition method as described in claim 3, characterized in that, The step of performing a matching analysis on the first vector in the physical space topology and the second vector in the topology of the pixel coordinates on the image to obtain the matching analysis result further includes: A matching analysis is performed on the first vector in the physical space topology and the second vector in the pixel coordinate topology of the image to calculate the magnitude difference and the included angle between the first vector and the second vector. Determine whether the difference in modulus length is less than or equal to a preset threshold for the difference in modulus length to obtain a first determination result, and determine whether the included angle value is less than or equal to a preset threshold for the included angle to obtain a second determination result; Based on the first judgment result and the second judgment result, the matching analysis result is obtained.

5. The key point recognition method as described in claim 1, characterized in that, Before the step of processing the image to be detected using a preset deep learning model to obtain the initial key point recognition result, the method further includes: Acquire sample data and the current deep learning model, wherein the data processing result corresponding to the sample data is the first recognition result; The deep learning model is processed using the current deep learning model to obtain a second recognition result; Based on a preset loss function, the loss between the first recognition result and the second recognition result is calculated to obtain a ranking loss value, which includes the target loss Lobj and the predicted box loss Lbox. Determine whether the sorting loss value is less than or equal to a preset sorting loss threshold; If the value is greater than the target value, adjust the parameters of the current deep learning model. Based on the adjusted parameters, return to the step of using the current deep learning model to process the deep learning model and obtain the second recognition result. Continue until the ranking loss value is less than or equal to the preset ranking loss threshold to obtain the preset deep learning model.

6. A key point recognition device, characterized in that, The key point recognition device includes: The acquisition module is used to acquire the image to be detected in response to a key point recognition command; An image processing module is used to process the image to be detected using a preset deep learning model to obtain initial keypoint recognition results. The preset deep learning model is built based on the OBB (Rotated Bounding Box) regression function. The preset deep learning model includes a rotated bounding box object detection and keypoint recognition network structure and an output layer structure. The rotated bounding box object detection and keypoint recognition network also includes an SPPF (Special Power Filtering) module and a PAN (Positioning Panel) structure. The SPPF is used to generate and fuse features at different scales, and the PAN structure is used to fuse features at different semantic levels. The output layer structure includes a detection head module and a keypoint regression module. The detection head module is built based on an anchor-based method, which is used to pre-construct initial detection boxes. The reasoning correction module is used to reason and correct the initial key point identification result based on prior knowledge and key part topology fusion strategy to obtain the target key point identification result. The key point recognition device is also used to achieve: Based on prior knowledge of the coordinates of key parts of the aircraft in the physical coordinate system, a physical space topological relationship is constructed, and based on the initial key point recognition results, a topological relationship of pixel coordinates on the image is constructed. Based on the key part topology fusion strategy, the physical space topology relationship, and the topology relationship of pixel coordinates on the image, the initial key point recognition result is inferred and corrected to obtain the target key point recognition result.

7. A key point recognition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the key point recognition method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the key point recognition method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the key point recognition method as described in any one of claims 1 to 5.

Images