Image-free cross-species attitude estimation system and method based on single-pixel imaging

Abstract

A system and method for non-image cross-species attitude estimation based on single-pixel imaging belong to the field of single-pixel imaging and attitude estimation, the system comprises a single-pixel imaging module and an attitude estimation module, the attitude estimation module comprises an attitude estimation network model and is used for receiving a single-pixel detection value and directly using the single-pixel detection value as the input of the attitude estimation network model, and the attitude estimation network model is used for estimating the attitude of the single-pixel detection value. And extracting and decoding features through the attitude estimation network model, and outputting key point coordinates of the target. The method comprises the steps of collecting a single-pixel detection value of a target scene; the single-pixel detection value is directly input into a pre-trained attitude estimation network model, and image reconstruction is not carried out; according to the method, feature extraction and decoding are carried out on the single-pixel detection value through the attitude estimation network model, and the key point coordinates of the target are directly output. And the complex process of image reconstruction is avoided. According to the system, the model is deployed on edge equipment, and a single-pixel detection value is used as network input, so that the storage space required by data is greatly compressed, and the development of image-free attitude estimation is promoted.

Description

Technical Field

[0001] This invention belongs to the field of single-pixel imaging and pose estimation, specifically, it relates to an image-free cross-species pose estimation system and method based on single-pixel imaging. Background Technology

[0002] Correlation imaging (also known as ghost imaging or single-pixel imaging) is a novel imaging technique that differs from traditional optical imaging methods. Traditional imaging relies on point-to-point mapping from object space to image space to acquire image information, while correlation imaging utilizes the high-order correlation properties of the light field. It collects spatially modulated light field signals through a single-pixel detector and combines this with compressed sensing, deep learning, and other methods to reconstruct the spectral, spatial, and phase information of the target scene. This technology not only overcomes the Shannon limit in traditional imaging, improving information utilization efficiency, but also has applicability across a wide spectral range (such as visible light, infrared, and terahertz bands), demonstrating application potential in fields such as biomedical imaging (e.g., knee and brain MRI reconstruction), industrial internal defect detection, and military non-line-of-sight reconnaissance.

[0003] Pose estimation, a crucial task in computer vision, aims to locate joints or keypoints of targets in images or videos to reflect their static and dynamic states, providing a foundation for higher-level tasks such as action recognition, target tracking, 3D reconstruction, image generation, and virtual reality. Current deep learning-based pose estimation methods mainly include single-stage, bottom-up, and top-down approaches: single-stage methods predict both target pose and bounding box; bottom-up methods detect all keypoints first and then group them based on association information; and top-down methods locate the target bounding box first and then estimate the pose within the box. However, existing research largely focuses on improving model accuracy, leading to a significant increase in model parameters and computational complexity, limiting deployment on edge devices, especially in harsh outdoor environments (such as forests), facing multiple constraints such as battery life, storage space, and computing power.

[0004] In addition, image-free pose estimation methods use non-visual media such as motion sensors, auditory sensors, infrared, lidar, radio frequency systems or wireless networks for pose analysis. However, such systems still have high requirements in terms of bandwidth, storage and computing resources, which further restricts their application and development in real-world scenarios. Summary of the Invention

[0005] Given the high demands of conventional imaging methods on bandwidth, information storage, and centralized data processing, the purpose of this invention is to propose an image-free cross-species pose estimation system and method based on single-pixel imaging. This system combines a pose estimation network model with a single-pixel imaging module, using the single-pixel detection values ​​of the target object collected by the single-pixel imaging module directly as network input. A network encoder extracts shallow and deep features, and an orthogonal decomposition decoder analyzes and outputs the target pose, avoiding the complex process of image reconstruction. The system deploys the pose estimation network model on an edge device, using single-pixel detection values ​​as network input, significantly reducing the required data storage space and promoting the development of image-free pose estimation.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] According to a first aspect of the present invention, an image-free cross-species pose estimation system based on single-pixel imaging is provided, comprising:

[0008] A single-pixel imaging module is used to acquire single-pixel detection values ​​of a target scene. The single-pixel imaging module includes a spatial light modulator for modulating a preset light field pattern and a single-pixel detector for collecting the modulated light signal.

[0009] The pose estimation module includes a pose estimation network model. The pose estimation module is communicatively connected to the single-pixel imaging module. It receives the single-pixel detection value and directly uses the single-pixel detection value as the input of the pose estimation network model. The pose estimation network model extracts and decodes features and outputs the key point coordinates of the target without performing image reconstruction.

[0010] Furthermore, the spatial light modulator is a digital micromirror device (DMD), and the preset light field pattern is a Hadamard-based coded pattern; the single-pixel imaging module also includes a camera lens for collecting scene light signals, a reflector for reflecting light signals, and a focusing lens for focusing light signals.

[0011] Furthermore, the attitude estimation module employs the SPCPose model as its attitude estimation network, which includes:

[0012] The visual Transformer encoding module is used to perform position encoding and deep feature extraction on single-pixel detection values ​​that have been reshaped into two-dimensional form to obtain hidden feature representations.

[0013] The deconvolution module, connected to the output of the visual Transformer encoding module, is used to upsample the hidden feature representation to enhance the feature map resolution.

[0014] The orthogonal decomposition decoder, connected to the output of the deconvolution module, is used to orthogonally decompose the upsampled feature map in the horizontal and vertical directions to obtain the probability distribution of key points in the horizontal and vertical directions, and estimate the coordinates of key points based on the probability distribution.

[0015] Furthermore, the orthogonal decomposition decoder uses KL divergence to estimate the final coordinates of key points.

[0016] Furthermore, the attitude estimation module employs the ASPIPose model as its attitude estimation network, which includes:

[0017] A multi-scale feature extraction backbone network is used to compress the input single-pixel detection value and extract multi-scale features to generate at least two feature maps with different resolutions.

[0018] The feature fusion module is used to fuse feature maps of different resolutions output by the multi-scale feature extraction backbone network.

[0019] The multi-branch decoder is used to independently decode the fused feature maps of different resolutions, generate corresponding prediction vectors, and concatenate all prediction vectors into a unified prediction matrix to output the key point coordinates.

[0020] Furthermore, the multi-scale feature extraction backbone network includes multiple consecutive feature extraction stages, each stage comprising a convolutional module ConvBlock and a multi-scale feature extraction block MSFEBlock; the multi-scale feature extraction block MSFEBlock includes:

[0021] The splitting unit is used to split the input feature map into high-frequency components and low-frequency components;

[0022] The bottleneck layer, consisting of a convolutional module and a depthwise separable convolutional module, is used for feature extraction of the high-frequency components;

[0023] The splicing unit is used to fuse the processed high-frequency component with the low-frequency component through feature map splicing operations.

[0024] According to a second aspect of the present invention, an image-free cross-species pose estimation method based on single-pixel imaging is provided. This method utilizes the aforementioned system for pose estimation and specifically includes the following steps:

[0025] The process of acquiring single-pixel detection values ​​of the target scene involves using a digital micromirror device (DMD) to modulate a series of preset light field patterns based on Hadamard to encode the scene light signal, and then having the encoded light signal received by a single-pixel detector.

[0026] The single-pixel detection value is directly input into the pre-trained pose estimation network model without image reconstruction;

[0027] The pose estimation network model extracts and decodes features from the single-pixel detection value, and directly outputs the key point coordinates of the target.

[0028] Furthermore, the pose estimation network model is the SPCPose model. The single-pixel detection value is input into the SPCPose model for processing, including:

[0029] The single pixel detection value B M Remodeled into two-dimensional form B H×W , where H and W represent the height and width of the original image, respectively;

[0030] For B H×W Position encoding is performed to obtain the initial feature F. 0 ;

[0031] The initial feature F 0 The input is fed into a visual Transformer encoding module containing L layers for deep feature extraction to obtain the hidden features F. out It should be noted that the visual Transformer coding module contains L layers, each of which consists of a multi-head self-attention mechanism (MHSA) and a multilayer perceptron (MLP).

[0032] For the hidden feature F out Perform N deconvolution operations and then process them with 1×1 convolution to obtain the enhanced feature map K. Deconv ;

[0033] For the feature map K Deconv Perform orthogonal decomposition and calculate the probability distribution K of the key points in the horizontal direction. H and the probability distribution K along the vertical direction V ;

[0034] Based on the aforementioned horizontal probability distribution K H and longitudinal probability distribution K V The final keypoint coordinates are obtained using KL divergence estimation.

[0035] Furthermore, the processing procedure of the visual Transformer encoding module satisfies the following formula:

[0036]

[0037] Among them, Z L-1 This represents the feature map input to the visual Transformer. Z represents the feature map after multi-head self-attention processing. LThis represents the feature map after processing by the visual Transformer, where L represents the number of layers in the visual transformer, Norm represents layer normalization, MHSA represents multi-head self-attention mechanism, and MLP represents multilayer perceptron.

[0038] Furthermore, the pose estimation network model is the ASPIPose model. The single-pixel detection value is input into the ASPIPose model for processing, including:

[0039] The STEM module, composed of multiple convolutional modules, is used to perform initial compression and feature extraction on the input single-pixel detection values ​​to obtain the initial feature map I. STEM ;

[0040] The initial feature map I STEM The input is fed into a backbone network consisting of at least two feature extraction stages for multi-scale feature extraction. Each feature extraction stage includes a convolutional module ConvBlock and a multi-scale feature extraction block MSFEBlock, resulting in feature maps at different scales.

[0041] Multi-scale feature fusion is performed on the feature maps at different scales to generate fused feature maps of high, medium, and low resolutions.

[0042] The high, medium, and low resolution fused feature maps are decoded respectively to obtain the corresponding prediction vectors P1, P2, and P3;

[0043] The prediction vectors P1, P2, and P3 are concatenated to obtain a unified keypoint prediction matrix P, from which the keypoint coordinates are extracted.

[0044] Through the above design scheme, the present invention can bring the following beneficial effects:

[0045] 1) Ultra-high precision and ultra-low sampling rate pose estimation: This method is the first to achieve cross-species pose estimation without images or explicit or implicit reconstruction, without reducing the pose estimation accuracy. The amount of detection data required is four orders of magnitude lower than that of traditional image methods.

[0046] 2) Cross-species adaptability: SPCPose exhibits strong cross-species adaptability, significantly expanding its application scope. The SPCPose model is applicable to human and animal posture estimation, becoming a general tool suitable for diverse research and practical applications;

[0047] 3) Novel Cross-Species Pose Estimation Framework: ASPIPose is a novel cross-species pose estimation framework based on single-pixel imaging with an ultra-low compression ratio array light source. It fundamentally reshapes traditional pose estimation methods by directly estimating pose from single-pixel imaging data. It can extract multi-scale features and effectively integrate high-frequency and low-frequency information from the encoded signal, thereby comprehensively capturing the global and local structural details required for cross-species pose estimation. This approach forms a robust and resource-efficient solution, achieving accurate pose estimation while minimizing data usage, making it particularly suitable for resource-constrained scenarios such as remote monitoring, behavior analysis, and wildlife conservation.

[0048] 4) Robustness in real-world scenarios: In actual tests, both SPCPose and ASPIPose demonstrated excellent robustness. They can accurately decode object poses under harsh conditions such as complex outdoor backgrounds, showcasing their reliability and effectiveness in real-world environments. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a single-pixel imaging module;

[0050] Figure 2 The flowchart shows an image-free cross-species pose estimation method based on single-pixel imaging.

[0051] Figure 3 This is a diagram of the SPCPose algorithm network structure.

[0052] Figure 4 Here is the network structure diagram of the ASPIPose algorithm;

[0053] Figure 5 A schematic diagram illustrating the pose estimation results using the SPCPose algorithm on the Tri-Mouse dataset;

[0054] Figure 6 A schematic diagram illustrating the pose estimation results of the SPCPose algorithm on the Horse10 dataset under the condition of natural sequence extraction.

[0055] Figure 7 A schematic diagram illustrating the pose estimation results using the SPCPose algorithm on the Horse10 dataset, considering the effect of reverse sequence extraction.

[0056] Figure 8 A schematic diagram illustrating the pose estimation results using the SPCPose algorithm on the Horse10 dataset, considering the performance of random sequence extraction.

[0057] Figure 9 This is a schematic diagram showing the pose estimation results of single-pixel imaging under different undersampling conditions on the Tri-Mouse dataset;

[0058] Figure 10 This is a schematic diagram showing the pose estimation results of single-pixel imaging under different undersampling conditions on the Fly dataset.

[0059] Figure 11 This is a schematic diagram showing the pose estimation results of single-pixel imaging under different undersampling conditions on the Indoor Human dataset.

[0060] Figure 12 This is a schematic diagram of the experimental results of SPCPose in a real-world scenario;

[0061] Figure 13 This is a schematic diagram illustrating the experimental results of SPCPose in a real-world scenario under natural sequences.

[0062] Figure 14 This is a schematic diagram illustrating the experimental results of SPCPose in a real-world scenario under reverse sequencing.

[0063] Figure 15 This is a schematic diagram illustrating the experimental results of SPCPose in a real-world scenario under random sequences.

[0064] Figure 16 This is a schematic diagram of the experimental results of ASPIPose in a real-world scenario when the number of samples is 16384.

[0065] Figure 17 This is a schematic diagram illustrating the experimental results of ASPIPose in a real-world scenario when the number of samples is 4096.

[0066] Figure 18 This is a schematic diagram illustrating the experimental results of ASPIPose in a real-world scenario with 1024 sampling times.

[0067] Figure 19 This is a schematic diagram of the experimental results of ASPIPose in a real-world scenario when the number of samples is 256.

[0068] Figure 20 This is a schematic diagram showing the experimental results of SPCPose at different sampling rates on the Indoor Human dataset;

[0069] Figure 21 This is a schematic diagram illustrating the impact of the extraction method on the parsing performance of SPCPose on the 256-sample-point Tri-Mouse dataset.

[0070] Figure 22 This diagram illustrates the impact of the extraction method on the performance of SPCPose in parsing object poses on the Horse10 dataset with 256 sampling points.

[0071] Figure 23This diagram illustrates the impact of the extraction method on the performance of SPCPose in parsing object poses on the 256-sample Indoor Human dataset. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0073] like Figure 1 and Figure 2 As shown, the image-free cross-species pose estimation method based on single-pixel imaging proposed in this invention includes:

[0074] Step 1: Single-pixel detection value acquisition. The scene light signal generated by light illuminating an object enters the camera lens. After passing through the camera lens, the scene light signal enters the digital micromirror device (DMD). The DMD modulates a series of preset light fields based on Hadamard and encodes the light signal. The encoded light signal is refracted at a specific angle (+12°) and reflected by a mirror. After being focused by a condenser lens, it is received by a single-pixel detector. The optical signal collected by the single-pixel detector is represented as:

[0075] B(m)=∫∫H m (x,y)·I(x,y)dxdy

[0076] Where I(x,y) represents the target object, H (m) (x,y) represents the preset optical field modulation code in the DMD memory, and m represents the number of preset optical field codes; B (m) Optical signals collected for a single-pixel detector;

[0077] Step 2: Pose estimation. Step 2 consists of two algorithms. The first algorithm is an image-free pose estimation algorithm proposed for the first time in this invention, named SPCPose. First, the optical signal B collected by the single-pixel detector... (m) The single-pixel detection value B is fed into the pose estimation network model as input. M Its reshaping transformation into two-dimensional form B H×W Where H and W represent the height and width of the original image, respectively, position encoding is performed, and the encoded data F 0 The input is fed into the L-layer visual Transformer encoding module to obtain hidden features. Deconvolution is used to enhance the resolution of the feature map and analyze the latent representations.

[0078] F i+1 =ViTEncoderBlock(F i )

[0079] K Deconv =Conv1×1 [DeconvBlockN(F out )]

[0080] in, (Right now (a matrix of dimensions), where i represents the i-th layer transformer, and the output of the L-th layer is... This represents the implicit representation of information embedded in a single pixel detection value, where d is the downsampling rate (default 16), C is the channel dimension, and K... Deconv Size is K Deconv This represents the deconvolution module DeconvBlock, where N is the number of repetitions of the deconvolution module DeconvBlock;

[0081] To adapt to cross-species and complex scenes, the feature map is orthogonally decomposed, the probability distribution of key points in the horizontal and vertical directions is calculated, and finally, KL divergence is used to estimate the coordinates of key points.

[0082] The second algorithm, ASPIPose, is the second image-free pose estimation algorithm proposed in this invention. It inputs a single-pixel imaging instance into the pose estimation network model, uses three convolutional modules to preprocess the input single-pixel detection values, compressing the input values ​​to reduce the space required for data storage. Then, the compressed single-pixel detection values ​​are input into four consecutive feature extraction modules. Each feature extraction module consists of one ConvBlock and one MSFEBlock convolutional module. j represents the j-th stage. Represents the set of positive numbers. This represents the feature map of the j-th stage; then, for the input features of each stage... Figure X , (of which features) Figure X Size C in ×H in ×W in C in H in W in Let represent the number of input channels, the height, and the width of the input feature map, respectively. A 1×1 convolutional module is used for processing, followed by n-order depthwise separable convolutions and skip connections to extract high-frequency features. Low-frequency and high-frequency features are then fused. During fusion, high-resolution features are spatially aligned, and the aligned features are integrated from a channel perspective to generate three independent fusion channels, each representing a different spatial resolution. The high-resolution feature map is defined as... Define the medium-resolution feature map as Define low-resolution feature maps as Feature maps of different resolutions contain different feature information. To fully utilize both deep and shallow features, a decoding module is used to decode three feature maps of different resolutions to obtain the prediction vector P. q The resulting vectors are concatenated into a unified prediction matrix P.

[0083] Step 3: Output the results. Display the inferred keypoint coordinates on the original image for use in subsequent tasks.

[0084] This invention proposes for the first time an image-free cross-species pose estimation method, introducing the SPCPose and ASPIPose models. These models combine a single-pixel imaging module with an intelligent perception network (pose estimation network model), using single-pixel detection values ​​directly as network input and outputting pose estimation results. SPIPose employs an encoder-decoder structure with a Vision Transformer backbone, fully extracting both shallow and deep features. During decoding, features are orthogonally decomposed to fully explore the horizontal and vertical distribution probabilities of keypoints across different species, enabling the network to adapt to multiple species and addressing the issue of keypoint distribution differences among species. ASPIPose utilizes a multi-scale feature extraction backbone, combining the subsequent stages (excluding the first stage) pairwise to fuse features at different scales, achieving information complementarity. Furthermore, in the final decoding process, three feature maps at different resolutions are decoded, and the final keypoint prediction results are obtained through three different prediction vectors.

[0085] refer to Figure 1 As shown, the single-pixel imaging module includes: a camera lens, a reflector, a digital micromirror array, a condenser lens, and a single-pixel detector; its specific functions are as follows:

[0086] Camera lens: Used to collect light signals from the target scene;

[0087] Digital micromirror array: used to modulate a series of Hadamard substrate light fields to encode the target light information and transmit the encoded information at an angle of +12°;

[0088] Reflector: Reflects the light signal emitted by the digital micromirror array to the focusing lens;

[0089] Condensing lens: focuses the light signal and sends it to the single-pixel detector;

[0090] Single-pixel detector: collects light signals and transmits them to the host computer for the next step of attitude estimation.

[0091] refer to Figure 3As shown, SPCPose is the first image-free pose estimation algorithm proposed in this invention. The network consists of a single-pixel detection feature extraction module based on visual transformer and a feature map orthogonal decomposition module. The single-pixel detection feature extraction module based on visual transformer is used to extract and encode the shallow and deep features contained in the input image. The feature map orthogonal decomposition module is used to decode and orthogonally decompose the encoded features, thereby enhancing the network's ability to analyze cross-species poses.

[0092] The specific implementation steps are explained below:

[0093] 1) Given a single pixel detection value B M Reshape it into B H×W It contains N k There are several key points, among which N k This represents the number of keypoints in the SPCPose algorithm.

[0094] 2) First, perform position encoding, and then use formula F to encode the input data. 0 =PatchEmbed(B H×W The data is encoded using PatchEmbed, which embeds patches to adapt the input data to the visual transformer, facilitating feature extraction by the model; the data after position encoding is F. 0 After entering the visual Transformer encoding module, the feature map F is obtained through layer normalization, multi-head self-attention mechanism, layer normalization and multilayer perceptron. i Furthermore, skip connections are used after the multi-head self-attention mechanism and the multilayer perceptron to mitigate information loss caused by various operations.

[0095]

[0096] Among them, Z L-1 This represents the feature map input to the visual Transformer encoding module. Z represents the feature map after multi-head self-attention processing. L This represents the feature map after processing by the visual transformer, where L represents the number of layers in the visual transformer, Norm represents layer normalization, MHSA represents multi-head self-attention mechanism, and MLP represents multilayer perceptron.

[0097] 3) Encode the data F 0 The input is fed into the L-layer visual Transformer encoding module to obtain hidden features:

[0098] F i+1 =ViTEncoderBlock(F i )

[0099] in, i represents the i-th layer of the visual transformer; C is the channel dimension. It should be noted that PatchEmbed is a technique for processing image data in deep learning, particularly in the Visual Transformer (ViT) model. The core component of ViT (Vision Transformer) is the Encoder Block. PatchEmbed and ViTEncoderBlock are existing technologies.

[0100] 4) Enhance feature map resolution and analyze latent representations through deconvolution:

[0101] K Deconv =Conv 1×1 [DeconvBlockN(F out )]

[0102] The output of the Lth layer is d represents the downsampling rate, which defaults to 16, and K... Deconv Feature map size is N is the number of repetitions of the deconvolution module DeconvBlock, and H and W represent the height and width of the input image, respectively;

[0103] 5) To adapt to cross-species and complex scenes, the feature map is orthogonally decomposed, and the probability distribution of key points in the horizontal and vertical directions is calculated;

[0104] K Flatten =Flatten(K) Deconv )

[0105] K H =Linear(K) Flatten )

[0106] K V =Linear(K) Flatten )

[0107] Among them, K Deconv This represents the feature map after deconvolution. This represents the probability distribution of key points in the horizontal direction. Represents the probability distribution of key points in the vertical direction. The parameter γ is used to control the dimension of the output feature vector. Flatten indicates feature map unfolding, and Linear indicates linear operation.

[0108] 6) Finally, KL divergence is used to estimate the coordinates of key points.

[0109] refer to Figure 4As shown, this invention presents the pose estimation algorithm ASPIPose, which is proposed for the first time. The backbone network consists of four stages of feature extraction. Each stage consists of a convolutional block (ConvBlock) and a multi-scale feature extraction block (MSFEBlock) for extracting multi-scale features. The subsequent decoding process mainly consists of upsampling, skip connections, MSFEBlock, convolutional blocks, and a decoding module. Upsampling expands the feature map size to fit the input size of the next module. Skip connections enable information complementarity and reduce information loss caused by convolution operations. MSFEBlock is used for feature extraction. The convolutional blocks and the decoding module are used for local feature extraction.

[0110] The specific implementation steps are as follows:

[0111] 1) Given a single-pixel imaging example based on an array light source First, a STEM consisting of three ConvBlock convolutional modules is used to process the input image in the initial stage of the model, facilitating subsequent feature extraction operations. It's worth noting that in the initial stage of the model, the STEM, composed of three ConvBlocks, performs preliminary processing of the input image to facilitate subsequent feature extraction and object detection. This design is very common in many modern deep learning models and is considered existing technology. Each ConvBlock convolutional module has three parameters: k, s, and p. These three parameters represent the kernel size, the stride of the kernel across the input, and the extra bounding box value padded on each side of the input, respectively. The three ConvBlock convolutional modules that make up the STEM have the same parameters: k, s, and p are 3, 2, and 1, respectively. Using the STEM... Compress to Where H and W represent the height and width of the original image, respectively, then... Input to 4 consecutive feature extraction stages S 1, S 2, S 3, S 4， Each feature extraction

[0112] Each of the four feature extraction stages consists of a convolutional module ConvBlock and an MSFEBlock. The k, s, and p parameters of the convolutional module ConvBlock in these four feature extraction stages are 3, 2, and 1, respectively. Figure 2The MSFEBlock shown consists of a ConvBlock convolutional module with a kernel size k and stride s of 1, n bottleneck layers, and another ConvBlock convolutional module with a kernel size k and stride s of 1 stacked together. MSFEBlock has two parameters: n and add = True / False. Here, n represents the number of bottleneck layers, and add indicates whether to perform a feature map concatenation operation (Concat). Figure 2 If a feature map processed by MSFEBlock has two destinations (two arrows pointing to other modules), then add = True; if a feature map has only one destination (one arrow pointing to other modules), then add = False.

[0113]

[0114] Where, j∈N + ∩[1,4], The feature map representing the j-th stage, C j =64·2 j ·w, ConvBlock represents the convolution operation module, and MSFEBlock represents the multi-scale feature extraction block;

[0115] 2) Input features at each stage Figure X (Feature map size is C) in ×H in ×W in First, it is processed by a ConvBlock convolutional module with a kernel size of 1 and a stride of 1 to generate X. h X l X h The feature map is input into the bottleneck layer of MSFEBlock, which consists of convolutional blocks and depthwise separable convolutional blocks. The k, s, and p parameters of the convolutional block ConvBlock are 3, 1, and 1, respectively, while the k, s, and p parameters of the depthwise separable convolutional block DWConvBlock are 5, 1, and 2, respectively. The parameter of the bottleneck layer Bottleneck is set to add = True / False, where add = True indicates... Figure 2 In the schematic diagram of the bottleneck layer (Bottleneck), the residual connections are represented by dashed lines. `add=False` indicates that the residual connections represented by dashed lines are not used. High-frequency features Y are extracted using n-order depthwise separable convolutions and skip connections. h ,feature Figure X l For low-frequency features, use the feature map concatenation operation Concat to combine the low-frequency features X l and high-frequency features Yh Integration, merging multi-scale features to achieve information complementarity:

[0116] X h ,X l =Split(ConvBlock) 1×1 (X))

[0117] Y h =Bottleneck (n) (X h )

[0118] Y f =Concat(X) l ,X h )

[0119] Among them, X h ,X l The feature map size is Among them, C out Y represents the number of output channels. h This represents the feature map processed by the bottleneck layer, with a size of [size missing]. Y f Representation of features Figure X l and Y h The stitched feature map has a size of C. out ×H in ×W in Bottleneck (n) This indicates processing through n bottleneck layers; Concat represents the feature map concatenation operation.

[0120] 3) Then, a bottom-up approach is adopted to upsample low-resolution features, spatially aligning them with high-resolution features. The aligned features are then integrated from a channel perspective. The DecodeBlock module consists of two ConvBlock modules with kernel size k = 3 and stride S = 1, a 2D convolution operation with kernel size 1, and a Reshape operation for feature map reshaping. The three feature maps are processed by the three DecodeBlock modules to generate three independent fusion channels, each representing a different spatial resolution. The high-resolution feature map is defined as... Define the medium-resolution feature map as Define low-resolution feature maps as Finally, the three fused channels are independently decoded through a convolutional layer to generate the final predicted feature map. The predicted feature map is then input into the decoder module to obtain the predicted vector P. q The resulting vectors are concatenated into a unified prediction matrix P;

[0121]

[0122] P = Concat([P1; P2; P3])

[0123] Where P q This represents the prediction vector for the q-th resolution level obtained after processing by the decoding module, with a size of... P represents the prediction matrix obtained by concatenating the three prediction vectors P1, P2, and P3 sequentially, with a matrix size of . DecodeBlock represents the decoding operation module, n k The number of key points in the ASPIPose algorithm, 3·n k Represents n k The predicted x-coordinates, predicted y-coordinates, and total number of keypoint confidence scores.

[0124] refer to Figure 5 , Figure 6 , Figure 7 and Figure 8 The figure shows the results of image-free pose estimation achieved by this invention on the Tri-Mouse and Horse 10 datasets. The numbers in the figure represent the sampling rate, with 1.000 representing full sampling. Figure 5 and Figure 6 The first image in the first column is the original image of the dataset, and the remaining images are schematic diagrams of single-pixel imaging. Figure 5 , Figure 6 The second and third columns of images show the experimental results of using the image-free pose estimation algorithm. Figure 7 and Figure 8 Divided into 4 partitions by dashed lines, except for the first partition in the upper left corner which is the pose estimation result of the original dataset images, the other three partitions are the results of using the no-image pose estimation algorithm at different sampling rates.

[0125] refer to Figure 6 , Figure 7 and Figure 8 The images show the results of natural sequence extraction, reverse sequence extraction, and random sequence extraction at different sampling rates, respectively. Figure 5 , Figure 6 , Figure 7 , Figure 8 It can be concluded that the image-free pose estimation algorithm exhibits excellent cross-species adaptability, accurately estimating pose features under different imaging modes and sampling rates. Even in multi-target scenes and complex background environments, its performance is comparable to that of image-based pose estimation network models. This indicates that the present invention performs well under simple experimental conditions and provides reliable results in more complex practical applications.

[0126] refer to Figure 9 , Figure 10and Figure 11 As shown, to verify the performance of the second algorithm, ASPIPose, in this invention, four sets of natural sequences of 16384-order Hadamard matrices (optical field size 128×128) were used to construct a 256×256 array optical field in a 2×2 format. Undersampling imaging and cross-species pose estimation tests were then performed on three species. The Hadamard matrix used for image reconstruction calculations during the imaging process corresponds to... Figure 9 The extracted sub-images of the aliased reconstructed image are shown, namely matrices of 16384×16384, 4096×4096, 1024×1024, and 256×256. Figure 9 This paper presents the corresponding results of undersampled single-pixel imaging in cross-species pose estimation. These results demonstrate the application value of single-pixel imaging in cross-species pose estimation, showcasing its ability to accurately estimate the poses of diverse species under various experimental conditions. References Figure 9 , Figure 10 and Figure 11 The results highlight the robustness and adaptability of the proposed method when extracting pose information directly from raw imaging data, without the need for computationally intensive image reconstruction. This method significantly reduces latency and resource requirements, making it highly efficient in real-time applications.

[0127] refer to Figure 12 , Figure 13 , Figure 14 and Figure 15 As shown, the manufactured single-pixel imaging module was placed in a nearby building and photographed crowds in three different scenes near the track and field stadium. Figure 12 (a), Figure 12 (b), Figure 13 , Figure 14 and Figure 15 The shooting distances for the first, second, and third rows are 118 meters, 122 meters, and 146 meters, respectively. Figure 12 (a) presents the actual scene as seen in images taken by a traditional camera, while also demonstrating the imaging capabilities of the single-pixel system and its visualization effect on key detection points. Figure 12 (b) The first column displays the single-pixel imaging results, the second column presents the pose estimation results obtained using the single-pixel imaging module and image-based method of the present invention, and the third column maps the pose estimation results to the reconstructed image for intuitive display. Figure 13 , Figure 14 and Figure 15 The results obtained by applying SPCPose to three different single-pixel detection value extraction methods are shown respectively. Figure 13 The results are representations of single-pixel probe values ​​and pose estimations processed from natural sequences. Figure 14 The results are the single-pixel detector value images and pose estimation results after reverse processing. Figure 15This section presents the image representation and pose estimation results of single-pixel detection values ​​processed for random sequences. The first column graphically presents the single-pixel detection values, a crucial step in understanding the original data acquisition method. The second column shows the results after processing by this invention, transforming the single-pixel detection values ​​into pose estimation information. The third column maps the keypoint locations onto the reconstructed image, visually presenting the pose estimation results.

[0128] refer to Figure 16 , Figure 17 , Figure 18 and Figure 19 As shown, Figure 16 , Figure 17 , Figure 18 and Figure 19 The first column of images contains the original images of the pose estimation target. Figure 16 (b) Figure 17 (b) Figure 18 (b) and Figure 19 (b) Reconstructed images with pose and labeling information, sampled 16384, 4096, 1024, and 256 times, respectively. The experimental results demonstrate the practical value of single-pixel imaging technology in analyzing complex real-world scenes, highlighting its ability to capture fine information in diverse environments. This approach vividly illustrates the integration of advanced imaging technology with real-world contexts, thereby extracting insights with practical significance.

[0129] refer to Figure 20 As shown, a detailed analysis is performed on the Indoor Human dataset. All experiments used a 128×128 resolution light field pattern for object detection, obtaining 16,384 single-pixel detection values ​​under full sampling. Subsequently, 256 detection values ​​were extracted to evaluate the impact of different extraction methods on pose decoding. M1, M2, and M3 represent natural sequence extraction, reverse sequence extraction, and random sequence extraction, respectively. Figure 21 The subjective visual effects of the present invention under different extraction methods are demonstrated. For unobstructed objects, the performance differences among the various extraction methods are negligible. Although all methods can detect lateral objects, the decoding result generated by M3 is closest to the true annotation.

[0130] refer to Figure 21 , Figure 22 and Figure 23 As shown, from an objective evaluation perspective, Figure 21 , Figure 22 and Figure 23Quantitative results are provided. It should be noted that for the pose estimation task, mAP@50-95, PCK@0.05, AUC, and EPE are commonly used evaluation metrics and are existing technologies, so they will not be described separately here. In algorithm evaluation, M3 performs excellently in both subjective and objective metrics, especially adept at resolving complex poses or lateral objects. M2 is second best, still providing reliable results when aliasing imaging is not feasible. M3 is generally not recommended due to the potential for hallucinations, but it can still be considered in specific scenarios where initial "outliers" can be effectively handled. Furthermore, random sampling can improve diversity and help reduce model bias, especially when other methods struggle to distinguish cases.

Claims

1. An image-free cross-species pose estimation system based on single-pixel imaging, characterized in that, include: A single-pixel imaging module is used to acquire single-pixel detection values ​​of a target scene. The single-pixel imaging module includes a spatial light modulator for modulating a preset light field pattern and a single-pixel detector for collecting the modulated light signal. The pose estimation module includes a pose estimation network model. The pose estimation module is communicatively connected to the single-pixel imaging module. It receives the single-pixel detection value and directly uses the single-pixel detection value as the input of the pose estimation network model. The pose estimation network model extracts and decodes features and outputs the key point coordinates of the target without performing image reconstruction.

2. The image-free cross-species pose estimation system based on single-pixel imaging according to claim 1, characterized in that, The spatial light modulator is a digital micromirror device (DMD), and the preset light field pattern is a Hadamard-based coded pattern. The single-pixel imaging module also includes a camera lens for collecting scene light signals, a reflector for reflecting light signals, and a focusing lens for focusing light signals.

3. The image-free cross-species pose estimation system based on single-pixel imaging according to claim 1, characterized in that, The attitude estimation module uses the SPCPose model as its attitude estimation network model, which includes: The visual Transformer encoding module is used to perform position encoding and deep feature extraction on single-pixel detection values ​​that have been reshaped into two-dimensional form to obtain hidden feature representations. The deconvolution module, connected to the output of the visual Transformer encoding module, is used to upsample the hidden feature representation to enhance the feature map resolution. The orthogonal decomposition decoder, connected to the output of the deconvolution module, is used to orthogonally decompose the upsampled feature map in the horizontal and vertical directions to obtain the probability distribution of key points in the horizontal and vertical directions, and estimate the coordinates of key points based on the probability distribution.

4. The image-free cross-species pose estimation system based on single-pixel imaging according to claim 3, characterized in that, The orthogonal decomposition decoder uses KL divergence to estimate the final coordinates of key points.

5. The image-free cross-species pose estimation system based on single-pixel imaging according to claim 1, characterized in that, The attitude estimation module uses the ASPIPose model as its attitude estimation network, which includes: A multi-scale feature extraction backbone network is used to compress the input single-pixel detection value and extract multi-scale features to generate at least two feature maps with different resolutions. The feature fusion module is used to fuse feature maps of different resolutions output by the multi-scale feature extraction backbone network. The multi-branch decoder is used to independently decode the fused feature maps of different resolutions, generate corresponding prediction vectors, and concatenate all prediction vectors into a unified prediction matrix to output the key point coordinates.

6. The image-free cross-species pose estimation system based on single-pixel imaging according to claim 5, characterized in that, The multi-scale feature extraction backbone network includes multiple consecutive feature extraction stages, each stage comprising a convolutional module ConvBlock and a multi-scale feature extraction block MSFEBlock; the multi-scale feature extraction block MSFEBlock includes: The splitting unit is used to split the input feature map into high-frequency components and low-frequency components; The bottleneck layer, consisting of a convolutional module and a depthwise separable convolutional module, is used for feature extraction of the high-frequency components; The splicing unit is used to fuse the processed high-frequency component with the low-frequency component through feature map splicing operations.

7. A cross-species pose estimation method without image based on single-pixel imaging, characterized in that, This method uses the system described in any one of claims 1-6 for attitude estimation, and specifically includes the following steps: The process of acquiring single-pixel detection values ​​of the target scene involves using a digital micromirror device (DMD) to modulate a series of preset light field patterns based on Hadamard to encode the scene light signal, and then having the encoded light signal received by a single-pixel detector. The single-pixel detection value is directly input into the pre-trained pose estimation network model without image reconstruction; The pose estimation network model extracts and decodes features from the single-pixel detection value, and directly outputs the key point coordinates of the target.

8. The image-free cross-species pose estimation method based on single-pixel imaging according to claim 7, characterized in that, The pose estimation network model is the SPCPose model. The single-pixel detection value is input into the SPCPose model for processing, including: The single pixel detection value B M Remodeled into two-dimensional form B H×W , where H and W represent the height and width of the original image, respectively; For B H×W Position encoding is performed to obtain the initial feature F. 0 ; The initial feature F 0 The input is fed into a visual Transformer encoding module containing L layers for deep feature extraction to obtain the hidden features F. out ; For the hidden feature F out Perform N deconvolution operations and then process them with 1×1 convolution to obtain the enhanced feature map K. Deconv ; For the feature map K Deconv Perform orthogonal decomposition and calculate the probability distribution K of the key points in the horizontal direction. H and the probability distribution K along the vertical direction V ; Based on the aforementioned horizontal probability distribution K H and longitudinal probability distribution K V The final keypoint coordinates are obtained using KL divergence estimation.

9. The image-free cross-species pose estimation method based on single-pixel imaging according to claim 8, characterized in that, The processing procedure of the visual Transformer encoding module satisfies the following formula: Among them, Z L-1 This represents the feature map input to the visual Transformer. Z represents the feature map after multi-head self-attention processing. L This represents the feature map after processing by the visual Transformer, where L represents the number of layers in the visual transformer, Norm represents layer normalization, MHSA represents multi-head self-attention mechanism, and MLP represents multilayer perceptron.

10. The image-free cross-species pose estimation method based on single-pixel imaging according to claim 7, characterized in that, The pose estimation network model is the ASPIPose model. The single-pixel detection value is input into the ASPIPose model for processing, including: The STEM module, composed of multiple convolutional modules, is used to perform initial compression and feature extraction on the input single-pixel detection values ​​to obtain the initial feature map I. STEM ; The initial feature map I STEM The input is fed into a backbone network consisting of at least two feature extraction stages for multi-scale feature extraction. Each feature extraction stage includes a convolutional module and an MSFEBlock module to obtain feature maps at different scales. Multi-scale feature fusion is performed on the feature maps at different scales to generate fused feature maps of high, medium, and low resolutions. The high, medium, and low resolution fused feature maps are decoded respectively to obtain the corresponding prediction vectors P1, P2, and P3; The prediction vectors P1, P2, and P3 are concatenated to obtain a unified keypoint prediction matrix P, from which the keypoint coordinates are extracted.

Images