Object positioning and posture recognition method based on 3D vision

Abstract

The invention provides an object positioning and posture recognition method based on 3D vision. The method comprises the steps that 3D point cloud data and image data corresponding to the 3D point cloud data are collected; preprocessing the image data to obtain a quality evaluation result, and executing image restoration processing based on the quality evaluation result to generate optimized image data; inputting the optimized image data into a pre-constructed conditional diffusion model based on hierarchical feature guidance, and generating feature representation of the optimized image data; and inputting the feature representation of the optimized image data into a differential transformer network to obtain a position and attitude prediction result of the object. According to the invention, through innovative image quality evaluation based on dual representation interaction driving, a conditional diffusion model based on hierarchical feature guidance and the design of a differential transformer network, the precision and robustness of object positioning and posture recognition in a complex environment are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and computer vision technology, and in particular to an object positioning and posture recognition method based on 3D vision technology. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, the precise positioning and gesture recognition of objects are playing an increasingly important role in industrial production. At present, the positioning and gesture recognition of objects in industrial environments mainly use traditional computer vision methods based on 2D images and point cloud processing methods based on a single 3D sensor.

[0003] Traditional 2D image vision methods achieve object recognition and positioning through image feature matching, but their performance degrades significantly under complex lighting and occlusion conditions. Although the point cloud processing method of a single 3D sensor can obtain the depth information of an object, it is difficult to ensure positioning accuracy due to data noise and occlusion problems.

[0004] In recent years, although some researchers have tried to introduce deep learning technology to solve these problems, the existing simple deep learning models fail to fully utilize the geometric features of 3D data, resulting in unstable recognition results. In addition, complex lighting and occlusion in industrial environments seriously affect image quality, and there is a lack of effective processing and feature extraction methods for 3D point cloud data. These problems all affect the accuracy of positioning and recognition. Summary of the invention

[0005] In view of this, the present application provides a 3D vision-based object positioning and posture recognition method, which improves the accuracy of object positioning and posture recognition.

[0006] The present application provides a method for object positioning and posture recognition based on 3D vision, including: Collect 3D point cloud data and image data corresponding to the 3D point cloud data; Preprocessing the image data to obtain a quality assessment result, and performing image restoration processing based on the quality assessment result to generate optimized image data; Inputting the optimized image data into a pre-built conditional diffusion model guided by hierarchical features to generate a feature representation of the optimized image data; Inputting the feature representation of the optimized image data into a differential transformer network to obtain the position and posture prediction results of the object; According to the position and posture prediction results of the object, the precise spatial position and posture information of the object is output.

[0007] The embodiment of the present application also provides a 3D vision-based object positioning and posture recognition device, the device comprising: processor; and, A memory communicatively connected to the processor; wherein, The memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the above-mentioned object positioning and gesture recognition method based on 3D vision.

[0008] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned 3D vision-based object positioning and posture recognition method.

[0009] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned 3D vision-based object positioning and posture recognition method.

[0010] This application has the following technical effects: Through the innovative dual-representation interactively driven image quality assessment method, the image processing capability in complex environments is improved, providing high-quality input data for subsequent feature extraction.

[0011] The conditional diffusion model design guided by hierarchical features realizes the generation and optimization of high-quality features, effectively integrating the geometric features of 3D point cloud data and the visual features of image data.

[0012] The differential transformer network is used for object positioning and posture recognition. The overall performance and robustness of the system are significantly improved through multi-scale feature fusion and adaptive weight mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the disclosed embodiments in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 It is a flow chart of a method for object positioning and posture recognition based on 3D vision provided in an embodiment of the present application; Figure 2 It is a flowchart of the image data preprocessing method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a two-stream neural network provided in an embodiment of the present application; Figure 4 It is a structural diagram of the conditional diffusion model provided in the embodiment of the present application; Figure 5is a schematic diagram of the structure of a differential transformer network provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of a 3D vision-based object positioning and posture recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0016] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0017] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0018] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0019] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0020] The object positioning and posture recognition method based on 3D vision provided in the embodiment of the present application has the following overall process: Figure 1 As shown, the following steps are included: Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

[0021] S1: Collect 3D point cloud data and image data corresponding to the 3D point cloud data.

[0022] In this solution, laser scanners (such as LiDAR) can be used to efficiently obtain high-precision 3D point cloud data. These devices generate 3D point clouds by emitting laser beams and measuring the return time to calculate the distance of objects.

[0023] S2: Preprocess the image data to obtain a quality assessment result, and based on the quality assessment result, perform image restoration processing to generate optimized image data.

[0024] The image preprocessing process in step S2 is as follows: Figure 2 As shown, it specifically includes two stages: quality assessment and image restoration.

[0025] In the quality assessment stage, we first build a two-stream neural network structure, such as Figure 3 As shown in Figure 1, the network extracts low-level features and high-level features of image data respectively. For low-level features, a multi-scale convolution layer is used to extract basic features including edges and textures, where the multi-scale convolution layer includes 3×3, 5×5 and 7×7 convolution kernels. For example, for a car body image, low-level features include basic features such as the texture and edge contour of the car body surface. For high-level features, a deep convolution network with an attention mechanism is used to extract semantic level features, such as the semantic information of car doors, windows and other components.

[0026] In order to improve the effect of feature extraction, this application adopts a contrastive learning strategy to optimize the feature extraction parameters of the two-stream neural network. Specifically, images of the same area from different perspectives are constructed as positive sample pairs, and images of different areas are constructed as negative sample pairs. The network parameters are optimized by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. For example, for a local area of ​​the vehicle body, images collected from different angles should have similar feature representations.

[0027] Then, based on the pre-designed interaction module, the dual representation features are fused. The cross-attention mechanism is used to calculate the correlation matrix between the low-level feature map and the high-level feature map, and the weights of different features are dynamically adjusted. For example, when scratches are detected on the surface of the car body, the low-level features can capture the texture features of the scratches, while the high-level features can identify that this is a defect in the door area. The interaction of the two features can more accurately evaluate the image quality.

[0028] In the image restoration stage, adaptive denoising is first performed on image areas with quality scores below a preset threshold (such as 0.5). The algorithm dynamically adjusts denoising parameters based on the type and degree of noise, such as using non-local mean denoising for noise caused by uneven illumination and wavelet threshold denoising for sensor noise.

[0029] Next, the local adaptive histogram equalization technique is used to improve the image details. The specific method is to divide the image into multiple small blocks (such as 16×16 pixels), perform histogram equalization on each block, and use bilinear interpolation to ensure smooth transitions between blocks. For example, for a local area of ​​the car body with insufficient light, the details are made clearer by increasing the contrast of the area.

[0030] Finally, the overall image quality is optimized through a multi-scale fusion strategy. The image is decomposed into pyramid structures of different scales, and enhancement processing is performed on each scale. Then, the final optimized image data is obtained through weighted fusion. The weight coefficient is adaptively determined by the quality score, and the area with a lower quality score is given a higher enhancement weight.

[0031] In S2, the image data is preprocessed to obtain a quality assessment result, including: S2.1: Based on the constructed two-stream neural network, low-level features and high-level features of the image data are extracted respectively to obtain dual representation features including the low-level features and the high-level features, wherein: the low-level features include edge and texture information of the image data, and the high-level features include semantic information of the image data.

[0032] More specifically, S2.1.1: for the low-level features, a multi-scale convolution layer is used to extract basic features including edges and textures, wherein the multi-scale convolution layer includes at least one of 3×3, 5×5 and 7×7 convolution kernels.

[0033] For the extraction of low-level features in the two-stream neural network, a multi-scale convolution layer is first used for processing. Specifically, a multi-scale convolution module containing three sizes of convolution kernels, 3×3, 5×5, and 7×7, is designed. Among them, the 3×3 convolution kernel is mainly used to capture the fine texture information in the image, such as subtle scratches on the surface of the car body, subtle changes in the paint surface, etc.; the 5×5 convolution kernel focuses on extracting medium-scale edge features, such as the contours of car body parts and window frames; the 7×7 convolution kernel focuses on larger-scale regional features, such as the boundary information of large parts such as doors and hoods. Through this multi-scale convolution design, the basic features of different scales in the image can be fully captured. In addition, a batch normalization layer and a ReLU activation function are introduced after each convolution layer to enhance the nonlinear expression ability of feature extraction.

[0034] Therefore, using at least one of the convolution kernels of 3×3, 5×5 and 7×7 can be applied to a variety of different scenarios and improve the prediction accuracy for different regional features.

[0035] S2.1.2: For the high-level features, a deep convolutional network with an attention mechanism is used to extract the semantic level features; For the extraction of high-level features, a deep convolutional network with an attention mechanism is used. The deep convolutional network backbone adopts the ResNet structure, which effectively alleviates the gradient vanishing problem of the deep network through residual connections.

[0036] In addition, a self-attention module is added between convolutional layers to enable the network to adaptively focus on key areas in the image.

[0037] For example, when processing car body images, the attention mechanism will give priority to areas with significant semantic information, such as door handles, car logos, and characteristic parts.

[0038] In other words, this high-level feature not only contains the semantic information of the object, but also retains the spatial relationship between the components, providing an important basis for subsequent positioning and posture recognition.

[0039] S2.1.3: Construct images of the same area from different perspectives as positive sample pairs, and images of different areas as negative sample pairs. Optimize the feature extraction parameters of the two-stream neural network by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs.

[0040] In order to further improve the effect of feature extraction, this application innovatively introduces a contrastive learning strategy to optimize the feature extraction parameters of the two-stream neural network. First, images of the same vehicle body area are collected from different perspectives to construct positive sample pairs.

[0041] For example, two images of the front door area taken from angles of 45 degrees and 60 degrees respectively constitute a pair of positive samples.

[0042] At the same time, images of different areas are selected as negative samples, such as images of the front door and trunk.

[0043] During the training process, the contrast loss function is used to guide the network to learn feature representation. By maximizing the feature similarity between positive sample pairs and minimizing the feature similarity between negative sample pairs, the network can learn more discriminative feature representations. In addition, in order to enhance the robustness of the model, different lighting conditions, different shooting distances and other variables are also considered when constructing sample pairs. Through this strategy, the network can extract more robust and discriminative features, effectively improving the accuracy of subsequent processing.

[0044] It should be noted that in order to adapt to the needs of different scenarios, this application provides flexible adjustment space for the specific parameter settings of the multi-scale convolution layer. For example, when processing high-resolution images, a larger convolution kernel (such as 9×9) can be added; in scenarios where computing resources are limited, only convolution kernels of 3×3 and 5×5 can be used. Similarly, for the specific implementation of the attention mechanism, different schemes can be selected according to actual needs, such as channel attention, spatial attention, or a combination of the two. The sample construction strategy in contrastive learning can also be adjusted according to the specific application scenario, such as increasing or decreasing the range of perspective differences, adjusting the ratio of positive and negative samples, etc. This flexible design makes this solution adaptable and scalable.

[0045] S2.2: Based on a pre-designed interaction module, the dual representation features are fused, and the relative weights of the low-level features and the high-level features are dynamically adjusted through an attention mechanism to obtain fused features.

[0046] After obtaining the dual representation features, these features need to be effectively integrated through the feature fusion module.

[0047] First, an interactive module is designed to achieve the fusion of dual representations, and a cross-attention mechanism is used to calculate the correlation matrix between low-level feature maps and high-level feature maps. For example, when scratches are detected on the surface of the car body, the low-level features can capture the texture features of the scratches, while the high-level features can identify that this is a defect located in the door area. Through the dynamic weight adjustment of the attention mechanism, the system can adaptively balance the importance of low-level features and high-level features according to the characteristics of different regions, thereby obtaining a more comprehensive and accurate feature representation.

[0048] S2.3: Based on the fused features, determine an image quality score as the quality assessment result.

[0049] Specifically, the interactive module is implemented using a multi-layer perceptron structure. First, the low-level features and high-level features are mapped to the same feature space, and then the attention weights between them are calculated. This process can be expressed as: first calculate the query matrix Q, key matrix K and value matrix V, then use the softmax function to get the attention weights, and finally weighted fusion to get the fused features. This mechanism ensures that the advantages of features at different levels can be fully utilized during the feature fusion process, improving the quality evaluation results of feature representation.

[0050] And based on the quality assessment result, performing image restoration processing to generate optimized image data, including: S2.4: For image regions whose quality scores are lower than a preset threshold, the denoising parameters are adaptively adjusted according to the type and degree of noise to obtain a denoised image.

[0051] Based on the fused features, a fully connected layer network is used to calculate the image quality score. The fully connected layer network maps the fused features to a normalized quality score between 0 and 1, where 1 represents the highest quality and 0 represents the lowest quality. For example, a quality score of 0.9 indicates good image quality, while 0.3 indicates poor image quality that needs to be restored. This scoring process not only takes into account the basic quality attributes of the image, but also includes the quality requirements related to specific tasks, providing an important basis for subsequent image restoration processing.

[0052] For image areas with quality scores below a preset threshold (such as 0.5), the system starts an adaptive denoising process. The denoising algorithm dynamically adjusts parameters based on the type and degree of noise. For example, for noise caused by uneven illumination, a non-local mean denoising algorithm is used; for sensor noise, a wavelet threshold denoising method is used. The algorithm first estimates the statistical characteristics of the noise, and then adaptively sets the denoising parameters based on these characteristics. This adaptive mechanism ensures that the denoising process can effectively remove noise interference while retaining useful information.

[0053] S2.5: Divide the denoised image into blocks, and use a local adaptive histogram equalization technique to process the image of each block to obtain an enhanced image.

[0054] To further improve image quality, the system uses local adaptive histogram equalization technology to enhance the image. The specific method is to divide the image into multiple small blocks, with a typical block size of 16×16 pixels. Each block is processed with histogram equalization, and bilinear interpolation is used to ensure a smooth transition between blocks to avoid block artifacts. This local processing method can better adapt to the lighting conditions of different areas of the image and improve the visibility of details. For example, for a local area of ​​the car body with insufficient light, the contrast of the area is increased to make the details more clearly visible.

[0055] S2.6: Decompose the enhanced image into pyramid structures of different scales through a multi-scale fusion strategy, enhance the images at each scale separately, and fuse the images at each scale in a weighted manner to obtain the optimized image data, wherein the weight value of the image at each scale is determined based on the quality score.

[0056] Finally, the image is optimized as a whole through a multi-scale fusion strategy. This process first decomposes the image into a pyramid structure of different scales, and performs enhancement processing on each scale. In the processing of different scales, the system will select appropriate enhancement parameters according to the characteristics of the scale. For example, at higher scales, the main focus is on adjusting the overall brightness and contrast, while at lower scales, more attention is paid to enhancing details. After the processing is completed, the results of each scale are synthesized into the final optimized image through weighted fusion. The weight coefficient is adaptively determined based on the previous quality score. The area with a lower quality score will be given a higher enhancement weight, which ensures that more processing resources can be invested in areas that need improvement.

[0057] It should be noted that during the entire image preprocessing process, the system will continuously monitor the processing effect. After each processing step is completed, the intermediate results will be evaluated for quality. If the processing effect of a certain step is found to be unsatisfactory, the system will automatically adjust the parameters for optimization. In addition, considering the differences in requirements for different application scenarios, each processing module in this solution has reserved a parameter adjustment interface. Operators can flexibly adjust processing parameters according to specific needs, such as denoising intensity, contrast enhancement degree, etc. This design greatly improves the practicality and adaptability of the solution.

[0058] Step S3: Inputting the optimized image data into a pre-built conditional diffusion model guided by hierarchical features to generate a feature representation of the optimized image data.

[0059] In step S3, if Figure 4 As shown, a feature extraction network based on the PointNet++ architecture is first used to extract the geometric features of 3D point cloud data.

[0060] For the car body point cloud, the network captures local and global structural information through layered sampling and grouping operations. For example, the first layer extracts local surface features (such as flatness and curvature), the second layer extracts component-level features (such as the geometry of doors and hoods), and the third layer extracts the contour features of the entire vehicle.

[0061] At the same time, ResNet50 is used as the backbone network to extract the visual features of image data, and the transmission of detail information is maintained through residual connections. In the conditional diffusion process, there are two stages: forward diffusion and reverse denoising.

[0062] In the forward diffusion stage, Gaussian noise is gradually injected into the features to generate a noise feature sequence of T time steps. In the reverse denoising stage, a denoising network with a U-Net structure is designed to gradually restore the contaminated features.

[0063] In order to ensure that the generated features are consistent with the physical properties of the input data, this application designs a conditional embedding network to encode the physical constraints of the object (such as body size, symmetry) as a conditional vector. During the diffusion process, the conditional vector is injected into the denoising network at each time step to guide the feature generation process to meet the physical constraints.

[0064] In addition, this application also constructs a feature pyramid structure to organize and guide the feature generation process. Figure 4 As shown in the figure, the pyramid contains three levels: the bottom layer keeps the original resolution to preserve basic features; the middle layer uses 2x downsampling to preserve component-level features; the top layer uses 4x downsampling to preserve overall-level features. At different stages of the diffusion process, feature guidance information of the corresponding levels is introduced respectively: the top-level features are introduced in the early stage of diffusion to provide global guidance, the middle-level and top-level features are introduced in the middle stage of diffusion to provide detail constraints, and the bottom-level features are introduced in the late stage of diffusion to restore fine structures.

[0065] S3 specifically includes: S3.1: Use a feature extraction network based on the PointNet++ architecture to extract geometric features of the 3D point cloud data.

[0066] When processing 3D point cloud data, this application uses a feature extraction network based on the PointNet++ architecture. The feature extraction network captures local and global structural information through layered sampling and grouping operations.

[0067] First, local surface features are extracted in the first layer, including geometric properties such as surface flatness and curvature. For example, for the point cloud data of the car body, features such as surface flatness and curvature can be identified.

[0068] Then, component-level features are extracted in the second layer, such as the geometric shapes of doors, hoods, and other components. Finally, overall contour features are extracted in the third layer to obtain the overall shape information of the vehicle body. Through this hierarchical feature extraction method, the system can fully obtain the geometric feature information of the object.

[0069] In order to improve the effect of feature extraction, the network adopts a hierarchical sampling and grouping strategy. In the sampling stage, the farthest point sampling algorithm is used to select key points to ensure that the sampling points can be evenly distributed on the surface of the object. In the grouping stage, the neighborhood of each key point is grouped by ball query to construct local feature areas. By setting different ball radii, local features of different scales can be obtained. In addition, the network also introduces a dense connection mechanism to fuse features at different levels and enhance the expressiveness of features.

[0070] S3.2: A backbone network extraction network based on ResNet50 is used to extract visual features of the image data, and multi-level visual features are obtained through residual connections.

[0071] For image data processing, the system uses ResNet50 as the backbone network for visual feature extraction.

[0072] The deep structure based on ResNet50 can effectively learn the hierarchical representation of images, and its unique residual connection mechanism can effectively alleviate the gradient vanishing problem in deep network training.

[0073] Specifically, the deep structure network based on ResNet50 contains multiple residual blocks, each of which consists of multiple convolutional layers, and implements residual learning through shortcut connections. This design not only improves the training efficiency of the network, but also maintains the effective transmission of detailed information. For example, when processing car body images, low-level features retain detailed texture information, while high-level features contain semantic-level understanding.

[0074] S3.3: Based on a conditional diffusion process involving T time steps, the noise component in the feature is gradually removed at each time step through a denoising network with a U-Net structure to obtain a preliminary feature representation; In the process of conditional diffusion, the system designs a processing flow including forward diffusion and reverse denoising. In the forward diffusion stage, a noise feature sequence of T time steps is generated by gradually injecting Gaussian noise.

[0075] For example, from t=1 to t=T, the geometric and visual information in the features are gradually contaminated by noise. The noise intensity at each time step is predefined and usually follows a rule of changing from small to large. This gradual noise injection process helps the model learn multi-scale representations of features.

[0076] In the reverse denoising stage, the system uses a denoising network with a U-Net structure to gradually restore the contaminated features.

[0077] The U-Net network consists of two parts: an encoder and a decoder, which use jump connections to achieve the fusion of features at different levels. At each time step, the denoising network receives the current noise features and time encoding as input, predicts and removes the noise component. This step-by-step denoising method can better preserve the structural information of the features. In order to improve the denoising effect, the network also introduces a self-attention mechanism, which enables it to focus on the key areas in the features.

[0078] S3.4: Encode the physical constraints of the object into a conditional vector, and inject the conditional vector into the denoising network at each time step to ensure that the generated features are consistent with the physical properties of the input data, so as to obtain the feature representation of the optimized image data.

[0079] In order to ensure that the generated features are consistent with the physical properties of the input data, this application innovatively introduces a conditional control mechanism.

[0080] First, the physical constraints of the object are encoded into conditional vectors, which include physical properties such as body size and symmetry. The generation of the conditional vector adopts an encoder-decoder structure to compress the high-dimensional physical constraint information into a low-dimensional conditional space. Then, during the denoising process at each time step, the conditional vector is injected into the denoising network to guide the feature generation process to meet the physical constraints. For example, when generating body features, the conditional vector ensures that the generated results maintain bilateral symmetry and that the relative position relationship between components meets the actual assembly requirements.

[0081] It should be noted that in order to adapt to different application scenarios, each network module in this solution provides flexible configuration options.

[0082] For example, the number of sampling points and ball query radius of the PointNet++ network can be adjusted according to the point cloud density; the number of layers of ResNet50 can be tailored according to computing resources; the number of time steps T in the conditional diffusion process can be set according to accuracy requirements. This flexible design enables the system to achieve a good balance between accuracy and efficiency. At the same time, considering the various situations that may be encountered in practical applications, the system has also designed an exception handling mechanism that can effectively handle abnormal situations such as noisy point clouds and blurred images.

[0083] And optionally, S3.5: before performing feature extraction, construct a feature pyramid structure including a bottom layer, a middle layer, and a top layer, wherein: Before feature guidance, this application first constructs a feature pyramid structure consisting of the bottom layer, middle layer, and top layer. The pyramid structure uses layer-by-layer downsampling to capture multi-scale information of images at different resolutions. This hierarchical design enables the system to focus on local details and global structures at the same time, thereby providing a more comprehensive feature representation. It should be noted that at each layer of the pyramid, channel alignment is performed through adaptive pooling and 1×1 convolution to ensure the consistency of feature dimensions.

[0084] S3.5.1: The bottom layer maintains the original resolution to preserve the basic features of the image data.

[0085] At the bottom level of the pyramid, features are kept at the original resolution to preserve the basic features of the image data.

[0086] This layer focuses on the detailed information of the object surface, such as the texture and edges of the vehicle body surface. For example, for a vehicle body image, the underlying features can accurately capture subtle features such as scratches, bumps, and seams on the surface. In order to ensure the integrity of the features, the underlying layer uses a dense feature map, and each pixel position retains complete feature information. In addition, an attention mechanism is introduced in this layer to enhance the feature extraction capability of key local areas.

[0087] S3.5.2: The middle layer uses 2 times downsampling of the original resolution to preserve the component-level features of the image data.

[0088] The mid-level features are downsampled to 2 times the original resolution, mainly used to preserve component-level features.

[0089] The features at this level can capture the medium-scale structure of the object, such as the shape and layout information of the car body parts such as doors, hoods, and lights. The downsampling process uses a maximum pooling operation with a step size of 2, which not only reduces the spatial dimension of the feature map, but also retains important structural information. In order to enhance the expressiveness of the features, a multi-scale convolution module is also introduced at this layer to capture the multi-scale features of the parts through receptive fields of different sizes. At the same time, residual connections are used to ensure the effective transmission of information and avoid feature degradation.

[0090] S3.5.3: The top layer adopts 4 times downsampling of the original resolution to preserve the overall level features of the image data.

[0091] At the top level of the pyramid, 4 times downsampling of the original resolution is used to preserve the overall level features.

[0092] The features of this layer mainly focus on the overall outline and large-scale structural relationships of the object, such as the overall shape of the car body and the relative position relationship between various components. Through further downsampling, the system can obtain a larger receptive field and thus better understand the global structure of the object. In this layer, in addition to the conventional convolution operation, a global context module is added to enhance the global expression ability of the features by modeling long-range dependencies.

[0093] In order to make the feature pyramid structure more efficient and accurate, this application designs a feature interaction mechanism between each layer. First, a bidirectional connection is established between adjacent layers to allow feature information to flow up and down. For example, the detailed features of the bottom layer can be passed to the middle layer through the upsampling path to help better locate the component boundaries; at the same time, the semantic information of the middle layer can also guide the extraction of the bottom layer features through the downsampling path. In addition, in the feature extraction process of each layer, an adaptive feature recalibration module is added to dynamically adjust the weight of the feature according to its importance.

[0094] In practical applications, the specific parameters of the pyramid structure can be flexibly adjusted according to demand. For example, for high-resolution images, the number of pyramid layers can be increased; for scenarios with limited computing resources, the number of feature channels can be appropriately reduced. At the same time, the downsampling ratio between different layers can also be adjusted according to specific tasks to achieve the best balance between accuracy and efficiency. In addition, considering the real-time requirements, this solution also optimizes the feature extraction process, such as using separable convolution to reduce the amount of calculation and adopting feature clipping strategies to remove redundant information.

[0095] S3.6: At different stages of the diffusion process, feature guidance information of corresponding levels is introduced, including: S3.6.1: Introduce top-level features in the early stage of diffusion.

[0096] At different stages of the conditional diffusion process, this application designs a hierarchical feature guidance mechanism to guide feature generation by dynamically introducing feature information of corresponding levels at different time steps.

[0097] First, in the early stage of diffusion, when t is close to T, the top-level features of the pyramid are mainly introduced to provide global guidance. At this stage, the system focuses on the overall structure and large-scale features of the object, such as the basic outline of the car body and the rough layout of the main components. Since the features contain more noise at this time, the use of high-level semantic features can provide more reliable structural constraints and prevent the generation process from deviating from the correct direction.

[0098] S3.6.2: Introduce both mid-level and top-level features during mid-diffusion.

[0099] In the middle stage of diffusion, the system introduces both middle-level and top-level features for guidance.

[0100] The features at this stage already have a certain structure, but still need to be refined. The middle-level features focus on component-level details, such as the shape of the door, the position of the headlights, etc., while the top-level features continue to provide global constraints. The coordinated use of the two-level features can gradually restore the detailed features of the component level while maintaining the accuracy of the overall structure. In order to achieve effective feature fusion, the system uses an attention mechanism to calculate the correlation between the current noise features and the guiding features of each layer, and controls the intensity of feature injection through the gating unit.

[0101] S3.6.3: In the later stage of diffusion, focus on introducing the underlying features.

[0102] In the late stage of diffusion, when t is close to 0, the system focuses on introducing underlying features to restore fine structures. At this time, the noise in the features is already small, which is more suitable for recovering local detail information. The underlying features contain rich local texture and edge information, which is crucial for recovering subtle features on the surface of objects. For example, in the vehicle body positioning scenario, this stage can accurately restore the texture details, edge contours and other features of the vehicle body surface. In order to ensure the accuracy of detail recovery, the system uses a smaller denoising step size at this stage and increases the injection weight of the underlying features.

[0103] S3.7: Residual connections are used to maintain the details of the underlying features and output feature representations that integrate multi-level information, including: S3.7.1: Add lateral connections at each level of the feature pyramid.

[0104] During the entire process of feature guidance, this application maintains the detailed information of the underlying features through a residual connection mechanism.

[0105] Specifically, lateral connections are added to each layer of the feature pyramid to fuse the original features of the corresponding resolution with the currently generated features. These lateral connections form a shortcut for information, allowing the detailed features of the underlying layer to be directly passed to the generation process, avoiding the loss of information during the multi-layer transmission process. For example, when processing car body images, this connection can maintain the clarity of subtle features such as the surface texture and welding lines of the car body.

[0106] S3.7.2: Based on the adaptive weighted sum fusion method, the original features of the corresponding resolution are fused with the currently generated features; Feature fusion adopts an adaptive weighted summation method, and the weight coefficient is dynamically adjusted through learnable parameters. The system designs a feature correlation evaluation module to determine the fusion weight based on the content similarity of the original features and the generated features. When the original features contain important structural information, the system automatically increases their weight; when the quality of the original features is poor, it relies more on the generated features. This adaptive fusion mechanism ensures that the final features can take into account both the fidelity of the original information and the quality of the generated features.

[0107] In order to further improve the quality of feature generation, this application also introduces a feature enhancement module in the residual connection. This module contains multiple residual units, each of which is equipped with channel attention and spatial attention mechanisms. Channel attention is used to enhance important feature channels, and spatial attention is used to highlight key spatial positions. Through this fine-grained feature enhancement, the system can better maintain and restore feature information in key areas. For example, in the vehicle body posture recognition task, the system automatically enhances the feature expression of the area around key feature points (such as vehicle corners, feature lines, etc.).

[0108] It should be noted that the entire feature guidance and fusion process is end-to-end trainable. During the training phase, the system simultaneously optimizes the parameters of the feature extraction, feature guidance, and feature fusion modules through back propagation. In order to improve the training efficiency, a progressive training strategy is adopted: first pre-train the model at a smaller time step, and then gradually increase the time step for fine-tuning. In addition, the system also designs a multi-scale discriminator to evaluate the quality of the generated features, and further improves the authenticity and detail performance of the features through adversarial training.

[0109] Step S4: Input the feature representation of the optimized image data into the differential transformer network to obtain the position and posture prediction results of the object.

[0110] In step S4, Figure 5 As shown in the figure, the differential transformer network adopts a multi-head self-attention structure with three attention heads, focusing on the overall contour, local details and spatial position relationship of the object respectively. The feature transformation is performed through a two-layer fully connected feedforward network. The first layer uses the ReLU activation function to expand the feature dimension, and the second layer maps the feature back to the original dimension. At the same time, a learnable position encoding is attached to each feature vector to inject spatial position information into the feature representation.

[0111] In the feature processing process, the output features of the differential transformer are processed at multiple scales based on the constructed feature pyramid structure. Multiple transformer modules are run in parallel at different scales. The larger scale transformer focuses on the global structure to determine the approximate position of the vehicle, and the smaller scale transformer focuses on local details to accurately locate key points. Then, an adaptive weight fusion mechanism is used to dynamically adjust the fusion weight according to the confidence of the prediction at each scale.

[0112] In the training phase, a multi-task loss function based on position positioning and posture recognition is constructed. Position positioning uses smooth L1 loss to reduce the influence of outliers, and posture recognition uses a weighted combination of cross entropy loss and regression loss. A phased training strategy is adopted. In the first phase, a larger learning rate (such as 0.001) is used to train only the position positioning branch for rapid convergence. In the second phase, a smaller learning rate (such as 0.0001) is used to train both the position positioning and posture recognition branches for fine-tuning. During the training process, gradient clipping is used to prevent gradient explosion.

[0113] S4 specifically includes: S4.1: Based on the constructed feature pyramid structure, the output features of the differential transformer in the differential transformer network are processed at multi-scales, and multiple transformer modules are run in parallel at different scales to capture spatial information of different granularities of the same object, wherein: large-scale features are used to determine the position of the object; small-scale features are used to locate the key points of the object.

[0114] The differential transformer network adopts a multi-scale feature processing architecture and performs hierarchical processing on the output features based on the constructed feature pyramid structure. Multiple transformer modules are run in parallel at different scales to capture the spatial information of the same object at different granularities. Larger-scale transformer modules focus on the overall structure and global features of the object, while smaller-scale transformer modules focus on the precise characterization of local details. For example, in the vehicle posture recognition task, large-scale features are used to determine the approximate position and main posture angles of the entire vehicle, while small-scale features are used to accurately locate key feature points on the vehicle body, such as wheel centers, door handles, etc.

[0115] In order to achieve efficient multi-scale feature processing, the system designs a hierarchical transformer structure. At the largest scale, a downsampling step of 4 is used to obtain a low-resolution feature map. The transformer module mainly processes the overall contour information of the vehicle body at this scale. At the medium scale, a downsampling step of 2 is used, and the corresponding transformer module is responsible for processing the spatial relationship of the main components of the vehicle body. At the smallest scale, the original resolution is maintained, and the transformer module focuses on processing local fine features. This multi-scale parallel processing method not only improves computational efficiency, but also makes full use of complementary information at different scales.

[0116] S4.2: Based on the adaptive weight fusion mechanism, the fusion weights are dynamically adjusted according to the confidence of the predictions at each scale, and based on the spatial attention module, feature representations of important areas are generated, and the original feature information is retained through residual connections.

[0117] In each scale transformer module, a specific attention mechanism is designed to enhance feature expression. For large-scale features, a global self-attention mechanism is used so that each position can obtain global context information. For medium-scale features, local window attention is used to establish feature associations within a fixed-size window. For small-scale features, a deformable attention mechanism is used to adaptively adjust the attention range according to the feature content. This hierarchical attention design ensures that the corresponding feature information can be efficiently captured at each scale.

[0118] In the feature fusion stage, this application designs an adaptive weight fusion mechanism, which dynamically adjusts the fusion weights of features at different scales based on the confidence of the prediction results at each scale.

[0119] Specifically, the system first calculates a confidence score for the prediction results of each scale, which reflects the reliability of the prediction at that scale. Then, these confidence scores are converted into fusion weights through the softmax function to ensure that the sum of the weights is 1. The larger the weight value, the greater the contribution of the prediction results of that scale in the final fusion. This adaptive weighting method can effectively balance the importance of prediction results at different scales.

[0120] In order to further improve the effect of feature fusion, the system introduces a spatial attention module. This module can adaptively identify and highlight the feature representation of important areas. In specific implementation, the spatial attention map of the feature map is first calculated to represent the importance of each spatial position. Then, the attention map is weighted multiplied with the original features to enhance the feature strength of important areas. This mechanism ensures that the system can pay more attention to areas that are critical to positioning and posture recognition, such as the characteristic edges of the car body, key connection points, etc.

[0121] In addition, the present application retains the original feature information through residual connections. On the feature processing path of each scale, a jump connection is added to pass the input feature directly to the output. This design not only facilitates the back propagation of gradients, but also maintains the integrity of the features. At the same time, in order to deal with the size differences between features of different scales, the system uses learnable upsampling and downsampling modules to ensure size alignment during feature fusion. Experimental verification shows that this multi-scale feature extraction and fusion scheme can significantly improve the accuracy of position positioning and posture recognition.

[0122] It should be noted that in actual applications, the system will dynamically adjust the parameters of each module according to the needs of specific tasks. For example, for scenarios with high accuracy requirements, the number of small-scale transformers can be increased; for scenarios with high real-time requirements, the number of processing scales can be appropriately reduced. In addition, the system also provides a feature clipping function, which can selectively retain the most important feature channels according to the limitations of computing resources, improving processing efficiency while ensuring performance.

[0123] S4.3: A position prediction regression branch based on a hybrid structure of a classification branch and a regression branch is used to predict the coordinate value of the object in three-dimensional space, wherein the classification branch is used to predict the range of the posture of the object, and the regression branch is used to fine-tune the posture angle of the object.

[0124] In the position and posture prediction link, this application designs a hybrid structure prediction network based on classification branches and regression branches. The network adopts the architecture of shared encoder and dual-path decoder, in which the classification branch is responsible for predicting the rough range of the object's posture, and the regression branch is responsible for accurately predicting the three-dimensional space coordinate value and posture angle. The design of this hybrid structure makes full use of the complementarity of the two tasks of classification and regression, and can provide accurate positioning results while ensuring the robustness of prediction. For example, in the vehicle body posture recognition task, the classification branch first divides the posture into several intervals (such as one interval every 30 degrees), predicts the approximate orientation of the vehicle body, and then the regression branch fine-tunes on this basis to obtain the accurate posture angle.

[0125] For position prediction, the regression branch uses a multi-layer perceptron structure to directly output the coordinate values ​​(x, y, z) of the object in three-dimensional space. In order to improve the accuracy of the prediction, the system introduces an adaptive weight mechanism in the regression branch to dynamically adjust the loss weight according to the uncertainty of different coordinate components. For example, when the prediction uncertainty in the depth direction (z-axis) is large, the system will automatically reduce the loss weight in this direction to avoid the negative impact of unreliable depth information on the overall prediction. In addition, the regression branch also outputs an estimate of the uncertainty of the prediction, which is of great significance for subsequent filtering and tracking.

[0126] In terms of attitude prediction, the classification branch first divides the Euler angle space into discrete intervals. The value range of each direction (pitch angle, roll angle, yaw angle) is evenly divided into multiple categories, and the system predicts the probability of the object's attitude falling into each interval through a softmax classifier. This discretization process can effectively deal with the periodicity and ambiguity problems in attitude representation. On this basis, the regression branch further outputs the angle offset relative to the center of the prediction interval to obtain the precise attitude angle. This strategy of combining coarse classification and fine regression significantly improves the accuracy and stability of attitude prediction.

[0127] S4.4: The steps for generating a differential transformer network include: A multi-head self-attention module with three attention heads is constructed, wherein: the first attention head focuses on the overall outline of the object, the second attention head focuses on the local details of the object, and the third attention head focuses on the spatial position relationship of the object.

[0128] In order to improve the robustness of the prediction, this application constructs a multi-head self-attention module with three attention heads in the differential transformer network. The first attention head focuses on the overall outline of the object and captures the main shape features of the object through the global receptive field. The second attention head focuses on the local details of the object, such as key areas such as feature points and edges. The third attention head is responsible for modeling the spatial position relationship between different parts of the object. The outputs of these three attention heads are weighted fused to obtain the final feature representation, where the fusion weights are adaptively determined by learnable parameters.

[0129] In the specific implementation, each attention head adopts the scaled dot product attention mechanism. First, the query vector (Q), key vector (K), and value vector (V) are generated through three sets of linear projections. Then, the dot product between Q and K is calculated to get the attention score, and the normalized attention weight is obtained through the scaling factor and softmax operation. Finally, the attention weight is multiplied by the value vector to get the weighted feature. In order to enhance the expressive power of the attention mechanism, the system also introduces relative position encoding so that the attention calculation can consider the spatial relationship between features.

[0130] A feed-forward network with two fully connected layers is constructed. The first layer uses the ReLU activation function to expand the feature dimension, and the second layer maps the features back to the original dimension and ensures the stability of the feature distribution through layer normalization.

[0131] In terms of feature transformation, this application designs a feedforward network with two fully connected layers. The first layer uses the ReLU activation function to map the features to a higher-dimensional space, enhancing the nonlinear expression ability of the network. The second layer maps the features back to the original dimension and ensures the stability of the feature distribution through layer normalization. This design not only helps the nonlinear transformation of features, but also effectively prevents the problems of gradient vanishing and gradient explosion. In addition, the system adds a dropout layer between the two fully connected layers to enhance the generalization ability of the model by randomly inactivating some neurons.

[0132] A learnable position encoding is attached to each feature vector, and the spatial position information is injected into the feature representation to obtain a feature representation with position information.

[0133] In order to make full use of spatial position information, the system attaches a learnable position encoding to each feature vector. These position encodings use a combination of absolute position encoding and relative position encoding to express both the absolute spatial position of the feature and the relative position relationship between features. During training, the parameters of the position encoding are updated simultaneously with the rest of the network, allowing the system to adaptively learn the optimal position representation. This position-aware feature representation is essential for accurately understanding the spatial structure and posture information of an object.

[0134] It should be noted that the specific parameters of each of the above modules can be flexibly adjusted according to application requirements. For example, the number of attention heads can be increased or decreased according to the complexity of the task, the dimension of the fully connected layer can be scaled according to the computing resources, and the position encoding method can also choose different implementation schemes according to the specific scenario. At the same time, the system provides a model compression interface, which supports the optimization of the model's computing efficiency through technologies such as knowledge distillation and channel pruning, so that it can better adapt to different application scenarios.

[0135] Next, a multi-task loss function based on position positioning and posture recognition is constructed, where position positioning is based on smoothing loss and posture recognition adopts a weighted combination of cross entropy loss and regression loss, where cross entropy loss is used to classify posture and regression loss is used to predict angle.

[0136] For the two core tasks of position positioning and posture recognition, this application designs a loss function based on a multi-task learning framework. This loss function comprehensively considers the accuracy of position positioning and the accuracy of posture recognition, and achieves balanced optimization of the two tasks through reasonable weight distribution. Among them, position positioning uses smooth L1 loss to reduce the influence of outliers, and posture recognition combines cross entropy loss and regression loss to achieve progressive optimization from rough classification to precise regression. For example, in the vehicle body positioning task, the system first learns the approximate posture range through cross entropy loss, and then accurately predicts the angle value through regression loss.

[0137] The smooth L1 loss function of the position localization branch is improved on the basis of the standard L1 loss. When the prediction error is less than the preset threshold β (for example, β=1.0), the loss is calculated in the form of a quadratic function, which can provide a smoother gradient near the target; when the error is greater than the threshold, the traditional L1 norm is used to reduce the influence of outliers. This design ensures sensitivity to small errors and improves the robustness of the model to abnormal samples. The mathematical expression of the loss function is: when |x|<β, loss=0.5x² / β; when |x|≥β, loss=|x|-0.5β, where x represents the difference between the predicted value and the true value.

[0138] For posture recognition tasks, cross entropy loss is mainly used to optimize the classification branch. The system first divides the posture space into multiple discrete intervals, for example, 360 degrees are evenly divided into 12 intervals, each interval is 30 degrees. Then the softmax function is used to calculate the probability that the predicted value falls into each interval, and the cross entropy loss is used to measure the difference between the predicted probability and the true label. This classification-based processing method can effectively deal with the periodicity problem in posture representation and avoid the training instability that may be caused by direct regression.

[0139] In the regression branch, the system uses a weighted smooth L1 loss to optimize angle prediction. Taking into account that the importance of different attitude parameters (pitch angle, roll angle, yaw angle) may be different, the system assigns different weights to each parameter. These weights can be adjusted according to specific application scenarios. For example, in some tasks, you may pay more attention to the accuracy of the yaw angle, so you can increase its loss weight appropriately. In addition, the system also takes into account the periodic characteristics of the attitude and selects the smallest angle difference when calculating the loss. That is, when the angle difference exceeds 180 degrees, use 360 ​​degrees minus the difference.

[0140] Finally, a phased training method is adopted to train the preset learning model, wherein the first stage is based on a first learning rate to train the position positioning branch of the learning model, and the second stage is based on a second learning rate to train the position positioning and posture recognition branches at the same time, and during the training process, the gradient introduces a gradient clipping method, wherein the first learning rate is greater than the second learning rate.

[0141] In terms of training strategy, this application adopts a phased training approach. The first phase mainly trains the location positioning branch, using a larger learning rate (such as 0.001) to quickly optimize the location prediction capability. The training focus of this phase is to enable the model to accurately locate the spatial position of the target object. During the training process, the position error on the validation set is dynamically monitored. When the error decreases gradually, the second phase of training begins.

[0142] The second stage trains the position localization and posture recognition branches simultaneously, using a smaller learning rate (such as 0.0001) for fine optimization. In this stage, the system balances the training difficulty of the two tasks by adjusting the weight coefficients in the multi-task loss. Specifically, when it is found that the training loss of a task is significantly higher than that of other tasks, the system will automatically increase the loss weight of the task to ensure that each task can be fully optimized.

[0143] In order to improve the stability of training, the system introduces a gradient clipping mechanism during the training process. When the calculated gradient exceeds the preset threshold (for example, the threshold is set to 10.0), the system will proportionally reduce the gradient value to avoid training instability caused by excessive gradients. In addition, a learning rate decay strategy is also adopted to gradually reduce the learning rate in the later stages of training to help the model converge to a better local minimum.

[0144] It should be noted that batch normalization technology is used throughout the training process to accelerate training convergence and improve model generalization. In each mini-batch, the system calculates the mean and variance of the features and performs normalization. At the same time, in order to prevent overfitting, regularization techniques such as weight decay and early stopping are also introduced. The combination of these training techniques ensures that the model can have good generalization performance while ensuring accuracy.

[0145] Step S5: Output the precise spatial position and posture information of the object based on the position and posture prediction results of the object.

[0146] In step S5, firstly, a sliding window averaging method is used to perform temporal smoothing on the prediction results of multiple consecutive frames.

[0147] For example, for the prediction results of 5 consecutive frames, a weighted average is used to obtain a more stable position estimate, where the weight of the most recent frame is the largest and decreases frame by frame. Then, the prediction results in the camera coordinate system are converted to the world coordinate system of the industrial robot through the pre-calibrated coordinate system conversion matrix.

[0148] Finally, the processed position and posture information is encapsulated into a standard data structure, including: the three-dimensional coordinates of the center point of the vehicle body (accurate to the millimeter level), the posture angle (accurate to 0.1 degrees), the spatial coordinates of the key feature points, and the confidence score of the prediction results. At the same time, a visualization result is generated for display in the human-computer interaction interface, with the detection box, posture arrows, and key point markers superimposed on the original image, and different colors are used to indicate the confidence level.

[0149] In a specific application scenario of the present application, the method can be used for body positioning and posture recognition on automobile manufacturing production lines. The system collects point cloud data and image data of the body through a 3D camera. After the above processing flow, the spatial position and posture information of the body can be accurately obtained, providing accurate positioning basis for subsequent automated assembly and quality inspection. In actual tests, this method can still maintain a high positioning accuracy in complex industrial environments, with an average position error of less than 1mm and a posture angle error of less than 0.5 degrees.

[0150] Among them, in S5, the precise spatial position and posture information of the object is output, including: S5.1: Use a sliding window averaging method to perform time-series smoothing on the prediction results of multiple consecutive frames, and obtain the smoothed prediction results by weighted averaging, wherein the weights are reduced frame by frame according to the input time sequence of the frames; In the final result output stage, the present application first uses the sliding window averaging method to perform time series smoothing on the prediction results of multiple consecutive frames. This method uses an exponentially decaying weighted average strategy to assign different weights to the prediction results at different times. Specifically, for a sliding window containing N frames (for example, N=5), the weight of the latest frame is the largest, and the weight gradually decreases as time goes by. For example, if the prediction results of 5 consecutive frames show that the coordinates of the center point of the vehicle body are ( , , ), , , )...( , , ), the final smoothing result can be expressed as: p = + +...+ , in > >...> And Σ =1.

[0151] This smoothing strategy can effectively suppress the noise and jitter of single-frame prediction and provide more stable output results.

[0152] In order to better handle sudden changes in time series smoothing, the system also designs an anomaly detection mechanism. When the deviation between the prediction result of a certain frame and the historical prediction exceeds the preset threshold, the system automatically adjusts the weight of the frame. If there are large deviations for multiple consecutive frames, it is considered that the target has indeed moved rapidly. At this time, the weight of the historical frame will be reduced so that the output can respond to the target's movement changes more quickly. In addition, the system also records the confidence score of each frame prediction, and uses the confidence as an adjustment factor for the weight when taking the weighted average to increase the contribution of reliable prediction results.

[0153] S5.2: transforming the smoothed prediction result from the camera coordinate system to the target coordinate system through a pre-calibrated coordinate system transformation matrix to obtain the position and attitude angle of the object in the target coordinate system; After obtaining the smoothed prediction result, it is necessary to convert it from the camera coordinate system to the target coordinate system (such as the robot world coordinate system). This step is achieved through a pre-calibrated coordinate system transformation matrix, including the rotation matrix R and the translation vector t. Specifically, if the position of the target in the camera coordinate system is p_cam=(x, y, z), and the attitude angle is (α, β, γ), then the position in the world coordinate system can be obtained through rigid body transformation p_world=Rp_cam+t. At the same time, the attitude angle also needs to be converted accordingly according to the relationship between the two coordinate systems to obtain the Euler angle (θ, , ψ).

[0154] Special attention was paid to accuracy during the coordinate system conversion process. First, the system uses a high-precision camera calibration algorithm to obtain internal and external parameters, and reduces errors by taking the average value through multiple calibrations. Secondly, in actual applications, online calibration is performed regularly, and the transformation matrix is ​​updated through calibration objects with known positions to compensate for calibration errors caused by environmental changes. In addition, the system also implements precise conversion based on hand-eye calibration to ensure that the converted coordinates can meet industrial-grade accuracy requirements.

[0155] The last step is to encapsulate the processing results into a standard data structure and generate visualization results. The data structure contains the following key information: the three-dimensional coordinates of the target center point (accurate to the millimeter level), the posture angle (accurate to 0.1 degrees), the spatial coordinates of key feature points (such as the four corners of the vehicle body, the endpoints of the feature lines, etc.), and the confidence scores of each prediction result. Each data field is accompanied by timestamp information to facilitate subsequent timing analysis and trajectory tracking. At the same time, the system will mark outliers. For example, when the confidence of a prediction result is lower than the threshold, a corresponding warning mark will be added to the data.

[0156] S5.3: Encapsulate the converted position and attitude angle into a standard data structure containing three-dimensional coordinates, attitude angles, key feature point coordinates and confidence scores, and generate visualization results.

[0157] In terms of visualization, the system overlays multiple layers of information on the original image. First, the detection box is drawn, using different colors to indicate the confidence level: green for high confidence (>0.9), yellow for medium confidence (0.7-0.9), and red for low confidence (<0.7). Then the posture arrows are displayed, with three orthogonal arrows representing the posture angles of the object. Key points are marked with markers of different shapes, and related feature points are connected by line segments to intuitively display the spatial structure of the object.

[0158] In addition, the system is designed with complete data recording and analysis functions. Each positioning result will be stored in the database, including the original image, intermediate results of the processing process, and final output information. These data can be used for subsequent offline analysis, such as accuracy evaluation, performance optimization, etc. The system also provides a data export interface, which supports exporting results to common data formats (such as JSON, CSV, etc.), which is convenient for data exchange and integration with other systems.

[0159] It should be noted that the design of the entire output module fully considers the actual application needs. The system supports a variety of data interface protocols (such as TCP / IP, industrial fieldbus, etc.), and can push the processing results to the downstream control system in real time. At the same time, considering the special needs of the industrial site, the system also provides a fault diagnosis and exception handling mechanism, which can promptly detect and report abnormal conditions of the positioning system to ensure the reliable operation of the entire system.

[0160] The embodiment of the present invention also provides an object positioning and posture recognition device based on 3D vision, such as Figure 6 As shown, it includes a processor and a memory, the memory is used to store a computer program, and when the processor executes the computer program, the object positioning and posture recognition method as mentioned above is implemented.

[0161] The system of the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0162] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

Claims

1. A method for object positioning and posture recognition based on 3D vision, characterized in that: include: Collect 3D point cloud data and image data corresponding to the 3D point cloud data; Preprocessing the image data to obtain a quality assessment result, and performing image restoration processing based on the quality assessment result to generate optimized image data; Inputting the optimized image data into a pre-built conditional diffusion model guided by hierarchical features to generate a feature representation of the optimized image data; Inputting the feature representation of the optimized image data into a differential transformer network to obtain a position and posture prediction result of the object; According to the position and posture prediction results of the object, the precise spatial position and posture information of the object is output.

2. The object positioning and posture recognition method according to claim 1, characterized in that: The preprocessing of the image data to obtain a quality assessment result includes: Based on the constructed two-stream neural network, low-level features and high-level features of the image data are extracted respectively to obtain dual representation features including the low-level features and the high-level features, wherein the low-level features include edge and texture information of the image data, and the high-level features include semantic information of the image data; Based on a pre-designed interaction module, the dual representation features are fused, and the relative weights of the low-level features and the high-level features are dynamically adjusted through an attention mechanism to obtain fused features; Based on the fused features, an image quality score is determined as the quality assessment result.

3. The object positioning and posture recognition method according to claim 2, characterized in that: The constructed two-stream neural network extracts low-level features and high-level features of the image data respectively, including: For the low-level features, a multi-scale convolution layer is used to extract basic features including edges and textures, wherein the multi-scale convolution layer includes at least one of 3×3, 5×5 and 7×7 convolution kernels; For the high-level features, a deep convolutional network with an attention mechanism is used to extract semantic level features; Images of the same area from different perspectives are constructed as positive sample pairs, and images of different areas are constructed as negative sample pairs. The feature extraction parameters of the two-stream neural network are optimized by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs.

4. The object positioning and posture recognition method according to claim 1, characterized in that: The performing of image restoration processing based on the quality assessment result to generate optimized image data comprises: For image areas with quality scores lower than a preset threshold, the denoising parameters are adaptively adjusted according to the type and degree of noise to obtain a denoised image; Dividing the denoised image into blocks, and using a local adaptive histogram equalization technique to process the image of each block to obtain an enhanced image; Through a multi-scale fusion strategy, the enhanced image is decomposed into pyramid structures of different scales, and the images at each scale are enhanced respectively, and the images at each scale are fused in a weighted manner to obtain the optimized image data, wherein the weight value of the image at each scale is determined based on the quality score.

5. The object positioning and posture recognition method according to claim 1, characterized in that: The step of inputting the optimized image data into a pre-built conditional diffusion model guided by hierarchical features to generate a feature representation of the optimized image data includes: Using a feature extraction network based on the PointNet++ architecture to extract geometric features of the 3D point cloud data; A backbone network extraction network based on ResNet50 is used to extract visual features of the image data, and multi-level visual features are obtained through residual connections; Based on a conditional diffusion process involving T time steps, the noise component in the feature is gradually removed through a denoising network with a U-Net structure at each time step to obtain a preliminary feature representation; The physical constraints of the object are encoded into a conditional vector, and the conditional vector is injected into the denoising network at each time step to ensure that the generated features are consistent with the physical properties of the input data, thereby obtaining the feature representation of the optimized image data.

6. The object positioning and posture recognition method according to claim 5, characterized in that: Before inputting the optimized image data into a pre-built conditional diffusion model guided by hierarchical features, the method further includes: Constructing a feature pyramid structure including a bottom layer, a middle layer, and a top layer, wherein the bottom layer maintains the original resolution for storing the basic features of the image data, the middle layer adopts 2 times downsampling of the original resolution for storing the component-level features of the image data, and the top layer adopts 4 times downsampling of 4 times the original resolution for storing the overall-level features of the image data; At different stages of the diffusion process, the corresponding levels of feature guidance information are introduced respectively. The top-level features are introduced in the early stage of diffusion, the middle and top-level features are introduced at the same time in the middle stage of diffusion, and the bottom-level features are introduced in the late stage of diffusion. The detailed information of the underlying features is maintained through residual connections, and the feature representation that integrates multi-level information is output, including: adding lateral connections at each layer of the feature pyramid; and fusing the original features of the corresponding resolution with the currently generated features based on an adaptive weighted summation fusion method.

7. The object positioning and posture recognition method according to claim 1, characterized in that: The step of inputting the feature representation of the optimized image data into a differential transformer network to obtain the position and posture prediction results of the object includes: Based on the constructed feature pyramid structure, multi-scale processing is performed on the output features of the differential transformer in the differential transformer network, wherein multiple transformer modules are run in parallel at different scales to capture spatial information of different granularities of the same object, wherein large-scale features are used to determine the position of the object, and small-scale features are used to locate key points of the object; Based on the adaptive weight fusion mechanism, the fusion weight is dynamically adjusted according to the confidence of the prediction at each scale, and based on the spatial attention module, the feature representation of the important area is generated, and the original feature information is retained through the residual connection; A position prediction regression branch based on a hybrid structure of a classification branch and a regression branch is used to predict the coordinate value of the object in three-dimensional space, wherein the classification branch is used to predict the range of the posture of the object, and the regression branch is used to fine-tune the posture angle of the object.

8. The object positioning and posture recognition method according to claim 1 or 7, characterized in that: Generate the differential transformer network as follows: Construct a multi-head self-attention module with three attention heads, where the first attention head focuses on the overall outline of the object, the second attention head focuses on the local details of the object, and the third attention head focuses on the spatial position relationship of the object; Construct a two-layer fully connected feedforward network. The first layer uses the ReLU activation function to expand the feature dimension. The second layer maps the features back to the original dimension and ensures the stability of feature distribution through layer normalization. Attach a learnable position code to each feature vector, inject the spatial position information into the feature representation, and obtain a feature representation with position information; Construct a multi-task loss function based on position positioning and posture recognition, where position positioning is based on smoothing loss, and posture recognition adopts a weighted combination of cross entropy loss and regression loss, where cross entropy loss is used to classify posture and regression loss is used to predict angle; A phased training method is adopted to train the preset learning model, wherein the first phase trains the position positioning branch of the learning model based on a first learning rate, and the second phase trains the position positioning and posture recognition branches simultaneously based on a second learning rate, and during the training process, the gradient introduces a gradient clipping method, wherein the first learning rate is greater than the second learning rate.

9. The object positioning and posture recognition method according to claim 1, characterized in that: Outputting the accurate spatial position and posture information of the object according to the position and posture prediction results of the object includes: The sliding window averaging method is used to smooth the prediction results of multiple consecutive frames in time sequence, and the smoothed prediction results are obtained by weighted averaging, where the weights are reduced frame by frame according to the input time sequence of the frames; The smoothed prediction result is converted from the camera coordinate system to the target coordinate system through a pre-calibrated coordinate system conversion matrix to obtain the position and attitude angle of the object in the target coordinate system; The converted position and attitude angle are encapsulated into a standard data structure containing three-dimensional coordinates, attitude angles, key feature point coordinates and confidence scores to generate visualization results.

Images