Space station astronaut visual human body part detection and tracking method and device

Abstract

The application discloses a space station astronaut visual human body part detection and tracking method and device, comprising: through direction invariant convolution, rotating feature extraction is carried out on an input image to obtain a first feature map; through space dimension automatic alignment, space dimension alignment and feature fusion are carried out on a plurality of first feature maps to obtain a second feature map; based on YOLOv5, a target detection image is obtained through a detection network; a third feature map is obtained through a feature extraction network; local and global feature vectors are obtained through component positioning and a transformer; feature fusion is carried out on the local and global feature vectors to obtain a fourth feature map; a target trajectory is obtained through an adaptive memory window type Kalman filter; the fourth feature map and the target trajectory are input into a hierarchical matching strategy, the target trajectory is evaluated, and a target tracking result is obtained. The application can improve detection accuracy and tracking performance, and can be widely applied in the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a space station astronaut visual human body part detection and tracking method and device. BACKGROUND

[0002] With the development of manned space technology, the space station has become the core platform for long-term manned space missions, and the behavior monitoring and management of astronauts in the microgravity environment has become the key to ensuring mission safety. However, the traditional monitoring method relying on wearable sensors or manual recording has low efficiency and poor real-time performance, which cannot meet the intelligent needs. At present, computer vision technology has injected new vitality into the intelligent development of the aerospace field, but the randomness of astronaut posture, frequent occlusion and complex lighting conditions in the microgravity environment have led to low detection accuracy and robustness of existing target detection and tracking models. SUMMARY

[0003] Therefore, the embodiments of the present application mainly aim to provide a space station astronaut visual human body part detection and tracking method and device, so as to solve at least one of the problems in the prior art, and improve the detection accuracy and tracking performance.

[0004] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a space station astronaut visual human body part detection and tracking method, comprising:

[0005] performing a rotation feature extraction operation on the input image through a direction invariant convolution module to obtain a first feature map;

[0006] performing spatial dimension alignment and feature fusion operations on a plurality of first feature maps through a spatial dimension automatic alignment module to obtain a second feature map;

[0007] based on a YOLOv5 network, according to the first feature map output by the direction invariant convolution module and the second feature map output by the spatial dimension automatic alignment module, through a detection network, a target detection image is obtained;

[0008] according to the target detection image, a third feature map is obtained through a feature extraction network;

[0009] according to the third feature map, a local feature vector and a global feature vector are obtained through a component positioning module and a Transformer module;

[0010] performing feature fusion operations on the local feature vector and the global feature vector to obtain a fourth feature map;

[0011] according to the target position information in the target detection image, a target trajectory is obtained through an adaptive memory window type Kalman filter;

[0012] inputting the fourth feature map and the target trajectory into a hierarchical matching strategy, evaluating the target trajectory, and obtaining a target tracking result.

[0013] In some embodiments, the rotation feature extraction operation on the input image by the direction-invariant convolution module includes the following steps:

[0014] obtaining a target distance from a pixel point of the input image to a target direction line;

[0015] constructing a first convolution kernel of a direction-sensitive branch based on a Gaussian function according to the target distance;

[0016] performing zero-mean processing and adding random disturbance on the first convolution kernel to obtain a second convolution kernel of the direction-sensitive branch;

[0017] performing convolution operation on the input image according to the second convolution kernel of the direction-sensitive branch to obtain a fifth feature map;

[0018] performing convolution operation on the input image according to a third convolution kernel of a main branch to obtain a sixth feature map;

[0019] The first feature map includes the fifth feature map and the sixth feature map, and the direction-invariant convolution module includes the direction-sensitive branch and the main branch.

[0020] In some embodiments, the target distance from the pixel point of the input image to the target direction line includes the following steps:

[0021] establishing a centralized coordinate system and obtaining coordinate values of each pixel point in the coordinate system;

[0022] obtaining a first distance from the pixel point to a diagonal line according to the coordinate values;

[0023] obtaining a second distance of the pixel point in the horizontal direction according to the coordinate values;

[0024] The target distance includes the first distance and the second distance.

[0025] In some embodiments, the spatial dimension automatic alignment module includes the following steps:

[0026] obtaining a minimum height of the plurality of first feature maps;

[0027] obtaining a minimum width of the plurality of first feature maps;

[0028] According to the minimum height and the minimum width, a minimum common spatial dimension is obtained;

[0029] According to the minimum common spatial dimension, spatial scales of a plurality of the first feature maps are aligned and adjusted to obtain a plurality of seventh feature maps;

[0030] The plurality of seventh feature maps are subjected to feature fusion operation to obtain the second feature map.

[0031] In some embodiments, the obtaining, according to the third feature map, of local feature vectors and global feature vectors by means of a part positioning module and a Transformer module comprises the following steps:

[0032] The part positioning module based on attention mechanism is used to position each human body part in the third feature map to generate a plurality of attention maps;

[0033] The feature information of the corresponding human body part in the plurality of attention maps is input into the Transformer module corresponding to the human body part to obtain a plurality of local feature vectors;

[0034] The global Transformer module is used to obtain the global feature vector according to the third feature map.

[0035] In some embodiments, the adaptive memory window Kalman filter comprises:

[0036] A dynamic speed smoothing mechanism, an appearance feature similarity calculation mechanism, a trajectory quality evaluation mechanism, an adaptive Mahalanobis distance and appearance distance fusion mechanism, and an adaptive survival threshold mechanism.

[0037] In some embodiments, the inputting of the fourth feature map and the target trajectory into a hierarchical matching strategy to evaluate the target trajectory to obtain a target tracking result comprises the following steps:

[0038] The target trajectory is subjected to hierarchical operation to obtain a high-confidence stable trajectory, an ordinary trajectory, and a low-confidence unstable trajectory;

[0039] The fourth feature map is used to perform time cascade matching on the high-confidence stable trajectory to obtain a matched trajectory set and an unmatched detection list;

[0040] The fourth feature map is used to perform time cascade matching on the ordinary trajectory to update the matched trajectory set and the unmatched detection list;

[0041] The fourth feature map is used to perform time cascade matching on the low-confidence unstable track, update the matched track set and the unmatched detection list, count the matched track set and the unmatched detection list, and obtain the target tracking result.

[0042] To achieve the above object, another aspect of the embodiment of the application provides a device for detecting and tracking visual human body parts of astronauts in space stations, which comprises:

[0043] A first module is configured to perform a rotation feature extraction operation on an input image by a direction-invariant convolution module to obtain a first feature map;

[0044] A second module is configured to perform a spatial dimension automatic alignment operation and a feature fusion operation on a plurality of first feature maps by a spatial dimension automatic alignment module to obtain a second feature map;

[0045] A third module is configured to obtain a target detection image by a detection network based on a YOLOv5 network according to the first feature map output by the direction-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module;

[0046] A fourth module is configured to obtain a third feature map by a feature extraction network according to the target detection image;

[0047] A fifth module is configured to obtain a local feature vector and a global feature vector by a part positioning module and a Transformer module according to the third feature map;

[0048] A sixth module is configured to perform a feature fusion operation on the local feature vector and the global feature vector to obtain a fourth feature map;

[0049] A seventh module is configured to obtain a target track by an adaptive memory window Kalman filter according to target position information in the target detection image;

[0050] An eighth module is configured to input the fourth feature map and the target track into a hierarchical matching strategy, evaluate the target track, and obtain a target tracking result.

[0051] To achieve the above object, another aspect of the embodiment of the application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned space station astronaut visual human body part detection and tracking method when executing the computer program.

[0052] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the aforementioned space station astronaut visual human body part detection and tracking method.

[0053] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the aforementioned space station astronaut visual human body part detection and tracking method.

[0054] The embodiment of the present application at least has the following beneficial effects: the present application provides a space station astronaut visual human body part detection and tracking method and device, which performs rotation feature extraction operation on an input image through a direction invariant convolution module to obtain a first feature map; performs spatial dimension automatic alignment and feature fusion operation on a plurality of first feature maps through a spatial dimension automatic alignment module to obtain a second feature map; based on a YOLOv5 network, according to the first feature map output by the direction invariant convolution module and the second feature map output by the spatial dimension automatic alignment module, a target detection image is obtained through a detection network; according to the target detection image, a third feature map is obtained through a feature extraction network; according to the third feature map, a local feature vector and a global feature vector are obtained through a component positioning module and a Transformer module; a fourth feature map is obtained by performing feature fusion operation on the local feature vector and the global feature vector; according to the target position information in the target detection image, a target trajectory is obtained through an adaptive memory window type Kalman filter; the fourth feature map and the target trajectory are input into a hierarchical matching strategy, the target trajectory is evaluated, and a target tracking result is obtained. Through direction invariant convolution and spatial dimension automatic alignment, the detection capability of human body parts in any direction under microgravity environment is enhanced, thereby improving the detection accuracy; through the local feature vector and the global feature vector, the component alignment failure problem caused by the arbitrariness of human body posture under microgravity is solved; through the adaptive memory window type Kalman filter, the tracking robustness under occlusion and complex scenes is improved; through the hierarchical matching strategy, the stability of high-quality trajectories is ensured, thereby improving the tracking performance. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.

[0056] Figure 1 is a flow chart of the space station astronaut visual human body part detection and tracking method provided by the embodiments of the present application;

[0057] Figure 2 is a schematic diagram of the direction-invariant convolution module network structure provided by the embodiments of the present application;

[0058] Figure 3 is a schematic diagram of the spatial dimension automatic alignment module network structure provided by the embodiments of the present application;

[0059] Figure 4 is a schematic diagram of the space station astronaut human body part detection flow provided by the embodiments of the present application;

[0060] Figure 5 is a schematic diagram of the space station astronaut human body part tracking flow provided by the embodiments of the present application;

[0061] Figure 6 is a schematic diagram of the global and component-based Transformer feature extraction network structure provided by the embodiments of the present application;

[0062] Figure 7 is a flow chart of assigning a given position component with a Transformer encoder provided by the embodiments of the present application;

[0063] Figure 8 is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0065] It is to be understood that even though a function module is divided in the system schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system, or the order in the flowchart. The terms "first / S100", "second / S200" in the description and claims and the above-mentioned drawings can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the embodiments of the present application. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0066] The terms "at least one", "multiple", "each", "any" and the like used herein include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0068] Existing target detection technology is mainly designed for the earth gravity environment, and has poor adaptability and detection ability in the space station microgravity scene. The current target detection has poor adaptability to the dynamic posture of astronauts (such as floating, rotating), and cannot effectively process targets in any direction; it is difficult to cope with complex and changeable lighting conditions (such as strong light, shadow, reflection), and the feature extraction ability is limited; it lacks an effective spatial relationship capturing mechanism for highly random postures; and the detection performance of small targets (such as hands and feet) is weak, and key details are easy to miss. These factors together lead to a decrease in detection accuracy and robustness.

[0069] In the tracking method in the space station microgravity scene, the traditional pedestrian re-identification and tracking method adopts the fixed grid division method, which is difficult to adapt to the non-standard posture and view angle change of astronauts floating or rotating, resulting in semantic component alignment failure; under complex scenes (such as occlusion or target intersection), the robustness of the existing prediction method is insufficient, which is easy to cause ID switching or tracking failure; the resistance to environmental noise such as light change is weak, and there is a lack of effective use of historical information, which is easy to produce misjudgment or trajectory drift; the traditional cascade matching is only based on the time continuity assumption, ignoring the difference in trajectory quality, and the matching stability is poor; in addition, the standard Kalman filter method only relies on the state at the last moment, and does not maintain long-term historical records, which is difficult to maintain reasonable motion trajectory prediction when there is long-time occlusion or detection failure. These factors together limit the adaptability and stability of the existing technology in the microgravity environment for astronaut human body part detection and tracking.

[0070] Therefore, as shown in the Figure 1 embodiments of the present application provide a space station astronaut visual human body part detection and tracking method, which can include but is not limited to steps S100 to S800:

[0071] Step S100, performing a rotation feature extraction operation on an input image through a direction-invariant convolution module to obtain a first feature map;

[0072] Step S200, performing spatial dimension alignment and feature fusion operations on a plurality of first feature maps through a spatial dimension automatic alignment module to obtain a second feature map;

[0073] Step S300, based on the YOLOv5 network, according to the first feature map output by the direction-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module, through a detection network, a target detection image is obtained;

[0074] Step S400, according to the target detection image, through a feature extraction network, a third feature map is obtained;

[0075] Step S500, according to the third feature map, through a component positioning module and a Transformer module, a local feature vector and a global feature vector are obtained;

[0076] Step S600, performing a feature fusion operation on the local feature vector and the global feature vector to obtain a fourth feature map;

[0077] Step S700, according to the target position information in the target detection image, through an adaptive memory window type Kalman filter, a target trajectory is obtained;

[0078] Step S800, input the fourth feature map and the target trajectory into a hierarchical matching strategy, evaluate the target trajectory, and obtain a target tracking result.

[0079] In steps S100 to S800 of some embodiments, the modeling of the rotating target is enhanced by a direction-invariant convolution and a spatial dimension automatic alignment module, thereby improving the detection accuracy. Through the component positioning module and the Transformer feature extraction method, fine-grained local features and global context information are captured. At the same time, through the adaptive memory window type Kalman filter and the hierarchical matching strategy, the tracking performance in special environments is enhanced.

[0080] In some embodiments, step S100 can include but is not limited to steps S110 to S150:

[0081] Step S110, obtaining a target distance from a pixel point of an input image to a target direction line;

[0082] Step S120, constructing a first convolution kernel of a direction-sensitive branch based on a Gaussian function according to the target distance;

[0083] Step S130, performing zero-mean processing and adding random disturbance on the first convolution kernel to obtain a second convolution kernel of the direction-sensitive branch;

[0084] Step S140, performing convolution operation on the input image according to the second convolution kernel of the direction-sensitive branch to obtain a fifth feature map;

[0085] Step S150, performing convolution operation on the input image according to a third convolution kernel of a main branch to obtain a sixth feature map;

[0086] Wherein, the first feature map includes the fifth feature map and the sixth feature map; the direction-invariant convolution module includes the direction-sensitive branch and the main branch.

[0087] In steps S110 to S150 of some embodiments, as Figure 2As shown, the efficient extraction of image rotation features is achieved by the multi-branch architecture of the direction-invariant convolution. The three parallel feature extraction branches in the direction-invariant convolution include a main branch occupying 50% of the output channels and two direction-sensitive branches each occupying 25% of the output channels. The main branch uses standard convolution operation to maintain the ability to extract general features, and the two direction-sensitive branches model the features in the 45-degree and 90-degree directions through Gaussian response-based convolution kernels. Through the direction-invariant convolution of the multi-branch architecture, the network can capture feature information in different directions at the same time, and automatically adjust the contribution of each branch according to the characteristics of the input image, significantly improving the detection ability of the model for rotated targets while maintaining a low computational overhead.

[0088] Among them, through the specially designed Gaussian response generating convolution kernel, the convolution operation can produce a strong response to the image features in a specific direction. In the subsequent processing process, the features of the three branches are aligned in the spatial dimension after being spliced in the channel dimension, and then pass through the batch normalization layer and the activation function to realize adaptive fusion and form a unified feature representation.

[0089] In step S110 of some embodiments, the direction-invariant convolution module implements two direction-sensitive convolution kernel initialization methods, which can optionally include 45-degree and 90-degree directions. These convolution kernels are based on Gaussian functions, and the weights are generated by calculating the distance of the pixel point to the target direction line, and zero-mean processing is performed to enhance the edge response. Exemplarily, the 45-degree convolution kernel assigns weights by calculating the distance of the point to the diagonal line, while the 90-degree convolution kernel only considers the horizontal distance to achieve sensitivity to vertical features, so that the convolution kernel can effectively detect and extract edge and texture features in a specific direction in the image.

[0090] In some embodiments, step S110 can include but is not limited to steps S111 to S113:

[0091] Step S111, establishing a centralized coordinate system and obtaining the coordinate value of each pixel point in the coordinate system;

[0092] Step S112, according to the coordinate value, obtaining the first distance of the pixel point to the diagonal line;

[0093] Step S113, according to the coordinate value, obtaining the second distance of the pixel point in the horizontal direction;

[0094] Among them, the target distance includes the first distance and the second distance.

[0095] In step S111 of some embodiments, a network coordinate system is established. Exemplarily, for a convolution kernel with a size of k by k, a centralized coordinate system is first established:

[0096] X,Y∈R k×k

[0097] wherein the coordinate value of each pixel point (i,j) is:

[0098]

[0099] wherein (x ij ,y ij ) is the coordinate of pixel point (i,j).

[0100] In steps S112-S113 of some embodiments, according to the coordinate value, the 45-degree direction kernel is weighted by calculating the first distance of the pixel point to the diagonal line, and the 90-degree direction kernel is weighted by obtaining the second distance of the pixel point in the horizontal direction to achieve the sensitivity to the vertical feature.

[0101] Exemplarily, the 45-degree direction kernel (K 45 ):

[0102]

[0103] The 90-degree direction kernel (K 90 ):

[0104]

[0105] wherein D 45 (i,j) is the first distance; and D 90 (i,j) is the second distance.

[0106] In steps S120-S130 of some embodiments, the kernel of the direction-sensitive branch is constructed based on the Gaussian function, and zero-mean processing is performed to enhance the edge response. Exemplarily, the Gaussian function is applied to construct the first kernel of the direction-sensitive branch, including the initial 45-degree direction kernel K 45 (i,j) and the initial 90-degree direction kernel K 90 (i,j), and the following calculation formula is obtained:

[0107]

[0108] wherein σ is a parameter for controlling the response range.

[0109] Zero-mean processing:

[0110]

[0111]

[0112] 45-degree direction convolution kernel W added with random disturbance 45 (i,j) and 90-degree direction convolution kernel W added with random disturbance 90 (i,j):

[0113]

[0114] wherein the random disturbance ∈ ij obeys the following normal distribution:

[0115]

[0116] The second convolution kernel W of the direction-sensitive branch can finally be obtained 45 (i,j) and W 90 (i,j).

[0117] In steps S140 to S150 of some embodiments, the input image is respectively convolved by the second convolution kernel of the direction-sensitive branch, the third convolution kernel of the main branch, and the feature map output by the direction-sensitive branch and the main branch can be obtained. Exemplarily, as shown in FIG. 5, the input image is input into the direction-invariant convolution module, and the feature maps of each branch are output after passing through the 45-degree direction-sensitive branch occupying 25% of the output channels, the 90-degree direction-sensitive branch occupying 25% of the output channels, and the main branch occupying 50% of the output channels. In the subsequent processing process, the features of the three branches are aligned in the spatial dimension, then spliced in the channel dimension, and then passed through the batch normalization layer and the activation function to realize adaptive fusion and form a unified feature representation. Figure 2

[0118] In some embodiments, step S200 can include but is not limited to steps S210 to S250:

[0119] Step S210, obtaining the minimum height in the plurality of first feature maps;

[0120] Step S220, obtaining the minimum width in the plurality of first feature maps;

[0121] Step S230, obtaining the minimum common spatial dimension according to the minimum height and the minimum width;

[0122] Step S240, aligning and adjusting the spatial scales of the plurality of first feature maps according to the minimum common spatial dimension to obtain a plurality of seventh feature maps;

[0123] Step S250, performing feature fusion operation on the plurality of seventh feature maps to obtain the second feature map.

[0124] ​In steps S210 to S250 of some embodiments, the feature fusion is implemented by the spatial dimension automatic alignment module using an intelligent feature alignment mechanism. For example, as shown in Figure 3 firstly, a downsampling method is used to obtain the minimum height and minimum width in all input first feature maps as a unified scale, and then a nearest neighbor interpolation method is used to align and adjust all first feature maps to the same spatial scale, i.e., the minimum common spatial dimension. This adaptive scale adjustment ensures the spatial consistency of feature fusion.

[0125] In step S300 of some embodiments, the direction-invariant convolution module and the spatial dimension automatic alignment module are embedded in the YOLOv5 network, and according to the first feature map output by the direction-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module, multi-scale detection is performed by the detection network (Detect) of the YOLOv5 network to predict bounding boxes and classes, etc., to obtain a target detection image.

[0126] For example, taking the YOLOv5 network embedded with the direction-invariant convolution module and the spatial dimension automatic alignment module as an example, the structure parameters of the YOLOv5 network are shown in Table 1.

[0127] Table 1

[0128]

[0129]

[0130] Referring to Table 1 and Figure 4 For example, taking the YOLOv5 network embedded with the direction-invariant convolution module and the spatial dimension automatic alignment module as an example, the spatial station astronaut human body part detection processing flow can include the following steps:

[0131] Step 1: Perform data augmentation operation on the original input image. Optionally, flipping and other techniques can be applied to improve the diversity of the data set, thereby generating an enhanced input image.

[0132] Step 2: The network backbone part (Backbone) performs preliminary feature extraction and dimension reduction on the enhanced input image through the Focus layer, to obtain a reduced input image.

[0133] Step 3: Step by step, the deep features of the reduced dimension input image are extracted by alternately using the orientation invariant convolution (OriConv) module and the C3 module. In this process, down-sampling is achieved through convolution with a step size of 2, and a feature pyramid containing P3 / 8, P4 / 16, P5 / 32, and P6 / 64 four different scale feature maps can be generated. In the orientation invariant convolution, a multi-branch architecture is used: a Gaussian kernel is used to extract 90-degree directional features (25% of the output channels), a Gaussian kernel is used to extract 45-degree directional features (25% of the output channels), and a standard convolution is used to extract general features (50% of the output channels).

[0134] Step 4: The SPP module is used to capture the feature information of the different scale feature maps, and the feature diversity is enhanced through multi-scale max-pooling, which can further enhance the multi-scale target detection capability. The output is a 1024-channel feature map for subsequent fusion operation steps.

[0135] Step 5: In the detection head part (Head), a top-down feature fusion strategy is used. The multi-branch features of the orientation invariant convolution module are aligned and fused through the up-sampling (Upsample) and spatial dimension automatic alignment (OriConcat) modules. For example, high-level semantic features (such as P5) are spliced with low-level detail features (such as P3), and the multi-scale target detection capability is strengthened by combining different levels of C3 modules.

[0136] Step 6: The YOLOv5 network contains four detection layers, corresponding to P3 / 8-small (small target), P4 / 16-medium (medium target), P5 / 32-large (large target), and P6 / 64-large (super large target). Through the four detection layers, the feature maps are detected in multiple scales to predict the bounding box and the class (such as head, hand), etc. Optionally, the hierarchical relationship between the body part and the astronaut instance (such as the body part face_1 associated with the astronaut instance person_1) is established through association detection, and the final detection network outputs the prediction result.

[0137] In steps S400 to S800 of some embodiments, as shown in Figure 5 the matching loss process containing the component-based Transformer feature extraction method and the fusion prediction, the target is re-identified (ReID), and the identified result is input into the hierarchical matching strategy. The gate mechanism containing the adaptive memory window type Kalman filter is also used to track and predict each target, and the result is output to the hierarchical matching strategy.

[0138] In step S400 of some embodiments, based on the basic feature extraction network, the basic visual features of the target detection image are captured, and the third feature map can be obtained.

[0139] In step S500 of some embodiments, the individual semantic parts of the human body are located by the attention mechanism, and then the local and global feature representations are enhanced by the Transformer feature enhancement module, so that in the detection and tracking of the human body parts of the astronauts in the space station, not only the detection and tracking of the entire bounding box of the human body are focused on, but also the detection and tracking of each body part are focused on, thereby improving the detection accuracy and tracking performance of the human body parts of the astronauts in the microgravity environment of the space station.

[0140] In some embodiments, step S500 can include but is not limited to steps S510 to S530:

[0141] In step S510, the individual human body parts in the third feature map are located by the part positioning module based on the attention mechanism, and a plurality of attention maps are generated.

[0142] In step S520, the feature information of the corresponding human body parts in the plurality of attention maps is input into the Transformer module corresponding to the human body parts, and a plurality of local feature vectors are obtained.

[0143] In step S530, the global feature vector is obtained by the global Transformer module according to the third feature map.

[0144] In step S510 of some embodiments, the semantic parts of the human body are adaptively learned and located by the attention mechanism. For example, based on capturing the basic visual features of the target detection image, the part positioning module generates a plurality of attention maps, each of which highlights the region of the corresponding part and suppresses the regions of other parts. These attention maps are normalized to ensure that the attention distribution of each part is reasonable.

[0145] In step S520 of some embodiments, as shown in Figure 6 For each located body part, the feature information of the corresponding part is first extracted from the basic features using the generated attention map, and then the feature information is rearranged into a sequence form and input into the Transformer encoder dedicated to the part. For example, as shown in Figure 7 Each part is assigned a separate transformer encoder to extract the features of each part. Figure 7 Part (a) of FIG. 8 is the original image, and part (b) of FIG. 8 shows that the first encoder focuses on the head, part (c) of FIG. 8 shows that the second encoder focuses more evenly, and part (d) of FIG. 8 shows that the third encoder focuses on the legs. Figure 7 Figure 7 Figure 6 ​​By this adaptive part positioning, different poses, view angles and partial occlusions can be adapted, without being limited by fixed grid division, while the learning of features of each body part is strengthened. Through the self-attention mechanism of the Transformer module, long-distance dependencies within the part can be effectively captured, so that the model understands the complex correlation of the internal structure and texture of the part, and finally generates a vector representing the features of the part through a pooling operation.

[0146] In step S530 of some embodiments, as shown, for global information, the generated attention map is used to extract feature information of the corresponding part from the base feature, and these feature information is rearranged into a sequence form and input into a global Transformer encoder to process the feature map of the entire image, ensuring the global context information and overall appearance features across different parts. The advantages of Transformer in processing sequence data are fully utilized, so that the model can understand the hierarchical structure of the image. Figure 8

[0147] In step S600 of some embodiments, a multi-level classification feature fusioner is used to fuse the local feature vectors of each part with the global feature vectors to form a multi-granularity comprehensive feature representation, which contains not only fine-grained local body part features but also overall context information, providing detailed and comprehensive feature description for identity recognition matching.

[0148] In step S700 of some embodiments, an adaptive memory window type Kalman filter is used to improve the tracking robustness in occlusion and complex scenes by designing dynamic speed smoothing, appearance feature similarity weighting, trajectory quality assessment, adaptive survival threshold, and combining the fusion strategy of Mahalanobis distance and appearance distance.

[0149] In the dynamic speed smoothing mechanism, the adaptive memory window extracts the speed information of the last few frames from the history record, gives different weights (the higher the weight, the closer the record), fuses the weighted average speed of the history with the current predicted speed, and obtains a smoother speed estimate that is more consistent with the true motion trend of the target. The problem that the traditional system only uses the currently calculated speed when predicting the target position, which is easily affected by instantaneous noise, is solved. Exemplarily, for the trajectory q at time t, the history speed memory window is defined as where n is the window size. The smoothed speed is calculated as

[0150]

[0151] wherein is the smoothed speed; is the speed of the standard Kalman prediction; and a is an adjustable speed weight coefficient; w​p is the weight of the history velocity, satisfying and w p ≥ w p-1 , the closer the history velocity, the greater the weight; p is used to represent the index. The target can still maintain a reasonable motion trajectory through the smoothing mechanism when there is a temporary occlusion or detection failure, without deviating due to noise.

[0152] In appearance feature similarity calculation, in order to maintain the appearance feature history of the target, when new observation data arrives, the similarity between the new feature and the history feature is calculated, and the time decay factor is considered to calculate the weighted average similarity as the matching basis. Exemplarily, for the appearance feature history set of the trajectory q and the newly detected feature a new , the cosine similarity between the appearance feature history and the newly detected feature is calculated as follows:

[0153]

[0154] where ∈ is a very small constant (such as e -6 ) to prevent division by zero errors.

[0155] The weighted average similarity is calculated as follows:

[0156]

[0157] Through the appearance feature similarity calculation mechanism, the system can rely on appearance similarity to maintain correct association when motion prediction is inaccurate, which is particularly suitable for target crossing, sudden turning and the like.

[0158] In the trajectory quality evaluation (dynamic evolution of trajectory quality) mechanism, each tracking target has a dynamically updated quality score, continuous successful tracking will increase the score, detection of occlusion will slightly reduce the score, and long time without update will significantly reduce the score. The quality score in turn affects the way the system handles the trajectory: high-quality trajectories enjoy a longer survival period, have priority in association, and their history data are also more valued. This quality-driven method enables the system to prioritize stable tracking of important targets. Exemplarily, the quality score of the trajectory q at time t is defined as its evolution satisfies:

[0159]

[0160] where ΔQ + is the quality increase step; γ ∈ (0, 1) is the quality decay coefficient.

[0161] In the adaptive Mahalanobis distance and appearance distance fusion strategy, the fusion distance concept is introduced through the adaptive memory window, combining the Mahalanobis distance (reflecting the position matching degree) and the appearance distance (reflecting the appearance similarity degree), and dynamically adjusting the weights of the two according to the specific situation. High-quality trajectories tend to rely more on Mahalanobis distance, and long-time non-updated trajectories rely more on appearance distance. This adaptive strategy enables the system to flexibly cope with various tracking challenges. Exemplarily, for the state vector x q and its covariance P q of the trajectory q, and the measurement vector z, the Mahalanobis distance d M (x q ,z) is defined as:

[0162]

[0163] where H is the measurement matrix; R is the observation noise covariance matrix.

[0164] The appearance distance d A (a q ,a z ) is defined as:

[0165]

[0166] By fusing the Mahalanobis distance and the appearance distance through adaptive weights, the fusion distance d C (x q ,z) can be obtained:

[0167]

[0168] where β(Q q ,τ) is an adaptive weight function based on the trajectory quality Q q and the target disappearance time τ, and then:

[0169] β(Q q ,τ)=max(0.1,min(0.9,1-β0-λ·τ))

[0170] where β0 is the initial appearance weight, and λ is the time decay factor.

[0171] In the adaptive survival threshold mechanism, the adaptive memory window adopts dynamic decision-making, and the basic survival deadline is dynamically adjusted according to the trajectory quality, historical length, and appearance similarity. High-quality trajectories obtain additional survival time, trajectories with high appearance similarity are retained even if they are not updated for a long time, and long-term stable tracking trajectories have higher fault tolerance. This mechanism effectively handles long-time occlusion and temporary disappearance, reduces identity (ID) switching and trajectory fragmentation. Exemplarily, the survival threshold T q of the trajectory q is defined as:

[0172]

[0173] wherein, T base is a basic threshold; T Q is a quality reward coefficient; T S is a similarity reward coefficient; S thresh is a similarity threshold.

[0174] By Kalman filtering based on an adaptive memory window, the possible motion trajectory of a target can be predicted based on historical speed, the correlation is reconstructed by appearance features when the target reappears, a high-quality trajectory is provided with a longer "survival period" to survive the occlusion period, and the occlusion robustness is improved; ID switching can also be reduced, the system uses historical appearance features instead of single-frame matching, considers the trajectory quality and historical length, and uses smooth speed prediction to more accurately distinguish targets when the targets cross; in addition, long-term stability can also be improved, the resistance to noise is improved through historical information accumulation, as the tracking duration increases, the system has a more comprehensive understanding of the target, the trajectory quality scoring mechanism rewards long-term stable tracking, and the system performs intelligent memory management to prevent excessive historical data from affecting efficiency.

[0175] In step S800 of some embodiments, based on a hierarchical matching strategy of trajectory stability, a multi-dimensional trajectory quality evaluation and hierarchical matching mechanism is introduced, forming a two-dimensional matching structure of "stability hierarchy + time cascade". The evaluation of trajectory stability not only considers the confirmation state of the trajectory, but also pays attention to the cumulative hit number and continuous tracking state of the trajectory. Based on these comprehensive indicators, the trajectory is divided into three stability levels: high-confidence stable trajectory of long-term stable tracking, ordinary trajectory of medium stability, and low-confidence unstable trajectory of frequent loss. After dividing all trajectories into three layers according to trajectory stability, time cascade matching is performed for each layer of trajectory. Within each matching stage, the matching order of time cascade is still maintained. Finally, the matched trajectory set is maintained to ensure that low-priority trajectories do not compete with high-priority trajectories for the same trajectory matching step, preventing different trajectories from being assigned to the same detected target.

[0176] In some embodiments, step S800 can include but is not limited to steps S810 to S840:

[0177] Step S810, performing a hierarchical operation on the target trajectory to obtain a high-confidence stable trajectory, an ordinary trajectory, and a low-confidence unstable trajectory;

[0178] Step S820, performing time cascade matching on the high-confidence stable trajectory using the fourth feature map to obtain a matched trajectory set and an unmatched detection list;

[0179] Step S830, using the fourth feature map, performing time cascade matching on the common track, updating the matched track set and the unmatched detection list;

[0180] Step S840, using the fourth feature map, performing time cascade matching on the low confidence unstable track, updating the matched track set and the unmatched detection list, and counting the matched track set and the unmatched detection list to obtain the target tracking result.

[0181] In steps S810 to S840 of some embodiments, all target tracks are divided into three layers according to track stability, i.e., high confidence stable track, common track, and low confidence unstable track. Time cascade matching is performed for each layer of track. In the first stage matching, the fourth feature map is used to process the high confidence stable track in time cascade order, a loose matching threshold is used to calculate the matching cost matrix, and a matching algorithm is executed for optimal assignment to record the matched track and detection. In the second stage matching, the fourth feature map is used to process the common track (excluding the matched track in the first stage) in time cascade order, a standard matching threshold is used to calculate the matching cost, and a matching algorithm is executed to assign the detection to the common track, update the matched track set and the unmatched detection list. In the third stage matching, the fourth feature map is used to process the low quality track (excluding the matched track in the previous two stages) in time cascade order, a strict matching threshold is used to calculate the matching cost, and a matching algorithm is executed to assign the detection to the low quality track, requiring higher matching accuracy. Then the matched track set is maintained to ensure that low priority tracks do not compete with high priority tracks for the same detection, and the final matching result and the unmatched track / detection list are counted. Next, the track sets of all levels are combined to form the final target tracking result, and optionally, whether to continue tracking or terminate is determined for the track that fails to match according to its stability and confidence.

[0182] Exemplarily, the first stage matching is:

[0183] Step 1: Identify the high confidence stable track (confirmed, hit many times, recently continuously tracked track) from all target tracks.

[0184] Step 2: Process these tracks in time cascade order, first process the high confidence stable track missing only 1 frame, then process the high confidence stable track missing 2 frames, and so on until the maximum cascade depth is reached.

[0185] Step 3: Using the feature information in the fourth feature map, a loose matching threshold (usually 110% of the standard threshold) is used to calculate the matching cost matrix between the track and the detection.

[0186] Step 4: Execute matching algorithm (e.g. Hungarian algorithm) to optimally assign high-confidence trajectories to detections.

[0187] Step 5: Record matched trajectory IDs and assigned detections, leaving unmatched detections for later stages.

[0188] Second stage matching:

[0189] Step 1: Identify all normal trajectories (non-high-confidence and non-low-quality trajectories) and exclude those matched in the first stage.

[0190] Step 2: Process these normal trajectories in temporal concatenation order, prioritizing recently lost trajectories, then longer lost trajectories.

[0191] Step 3: Utilize feature information from the fourth feature map, combined with remaining unmatched detections, to calculate matching cost between trajectories and detections using a standard matching threshold (system default).

[0192] Step 4: Execute matching algorithm to assign most suitable detections to normal trajectories.

[0193] Step 5: Update matched trajectory set and unmatched detections list in preparation for the third stage.

[0194] Third stage matching:

[0195] Step 1: Identify all low-quality trajectories (confirmed short time or recently frequent loss trajectories) and exclude those matched in the first two stages.

[0196] Step 2: Process these low-quality trajectories in temporal concatenation order.

[0197] Step 3: Utilize feature information from the fourth feature map, combined with remaining unmatched detections, to calculate matching cost between trajectories and detections using a strict matching threshold (typically 80% of the standard threshold).

[0198] Step 4: Execute matching algorithm to assign most suitable detections to low-quality trajectories, requiring higher matching precision.

[0199] Step 5: After completing all three stages of matching, tally final matching results and unmatched trajectory / detection list. Output updated low-quality trajectory set and unmatched detections list.

[0200] Maintaining matching results:

[0201] A set of matched trajectories is maintained, ensuring that low-priority trajectories do not compete with high-priority trajectories for the same detection. Optionally, for unmatched trajectories, a decision is made whether to continue tracking or terminate based on their stability and confidence. A final set of matched trajectories and a list of unmatched detections are output.

[0202] The tracking results are output:

[0203] Based on the final set of matched trajectories and the list of unmatched detections, all levels of trajectory sets are merged to form a final target tracking result. Optionally, for unmatched trajectories, a decision is made whether to continue tracking or terminate based on their stability and confidence.

[0204] The embodiment of the present application also provides a space station astronaut visual human body part detection and tracking device, which can realize the space station astronaut visual human body part detection and tracking method described above, and the device comprises:

[0205] A first module is configured to perform a rotation feature extraction operation on an input image through a direction-invariant convolution module to obtain a first feature map;

[0206] A second module is configured to perform a spatial dimension automatic alignment and feature fusion operation on a plurality of first feature maps through a spatial dimension automatic alignment module to obtain a second feature map;

[0207] A third module is configured to obtain a target detection image through a detection network based on a YOLOv5 network according to the first feature map output by the direction-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module;

[0208] A fourth module is configured to obtain a third feature map through a feature extraction network according to the target detection image;

[0209] A fifth module is configured to obtain a local feature vector and a global feature vector through a part positioning module and a Transformer module according to the third feature map;

[0210] A sixth module is configured to perform a feature fusion operation on the local feature vector and the global feature vector to obtain a fourth feature map;

[0211] A seventh module is configured to obtain a target trajectory through an adaptive memory window Kalman filter according to target position information in the target detection image;

[0212] An eighth module is configured to input the fourth feature map and the target trajectory into a hierarchical matching strategy, evaluate the target trajectory, and obtain a target tracking result.

[0213] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0214] The embodiment of the present application further provides an electronic device, which comprises a processor and a memory, the memory stores a computer program, and the processor implements the above-mentioned space station astronaut visual human body part detection and tracking method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0215] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0216] Reference Figure 8 , ​ The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0217] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application;

[0218] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and called and executed by the processor 901 to implement the space station astronaut visual human body part detection and tracking method of the embodiments of the present application;

[0219] The input / output interface 903 is used to realize information input and output;

[0220] The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like);

[0221] A bus 905 transmits information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0222] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other within the device through the bus 905.

[0223] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the space station astronaut visual human body part detection and tracking method.

[0224] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0225] The embodiment of the present application also provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the space station astronaut visual human body part detection and tracking method.

[0226] In summary, the space station astronaut visual human body part detection and tracking method and device provided by the embodiment of the present application has the following advantages:

[0227] 1. In the detection aspect, the embodiment of the present application utilizes the direction-invariant convolution and spatial dimension automatic alignment module to enhance the modeling of the rotating target, significantly improving the rotation invariance and spatial adaptability of human body part detection in microgravity environment, solving the detection bottleneck of traditional methods for arbitrary pose, complex lighting and small target, thereby improving the detection accuracy; in the tracking aspect, the component-based Transformer feature extraction method is used to realize multi-granularity feature fusion and adaptive component positioning, which not only captures fine-grained local features, but also does not lose global context information, realizes more stable local-global feature association in occlusion or view angle change scene, reduces the semantic alignment failure problem caused by traditional fixed grid division; also uses adaptive memory window Kalman filter, through dynamic speed smoothing, appearance similarity weighting and trajectory quality evolution mechanism, still can maintain reasonable motion prediction in long-term occlusion or detection interruption, reduces the ID switching frequency; at the same time, based on the layered matching strategy of trajectory stability, through multi-dimensional quality evaluation and dynamic threshold allocation, preferentially guaranteeing the continuity of high-quality trajectories, effectively inhibiting the trajectory fragmentation phenomenon in complex scenes, enhancing the tracking performance in special environments.

[0228] 2. The embodiment of the present application is suitable for astronaut detection and tracking in microgravity environment, and overcomes the problems of insufficient adaptability, robustness and stability of the prior art in detection and tracking in microgravity environment, and provides an efficient and reliable visual technology scheme for intelligent management of space station tasks.

[0229] 3. The traditional pedestrian re-identification model usually adopts fixed grid division (such as horizontal strips), which mechanically divides the image into several regions. This method is easy to fail when the pedestrian posture changes or the camera view angle changes, because the semantic components are no longer aligned to the preset grid. The embodiment of the present application captures the basic visual features of the image through the basic feature extraction network, locates the semantic components of the human body through the attention mechanism, and adaptively learns to locate the semantic components of the pedestrian through the attention mechanism. Through this adaptive component positioning, it can adapt to different poses, angles and partial occlusions, is not limited by fixed grid division, reduces the semantic alignment failure problem caused by traditional fixed grid division, and at the same time strengthens the learning of features of each body part.

[0230] 4、Compared with the traditional pedestrian re-identification method, the space station astronaut human body part tracking part of the embodiment of the present application ingeniously combines the local feature extraction capability of CNN and the long-distance dependence modeling capability of Transformer in feature representation, provides more comprehensive and rich identity representation through multi-granularity feature fusion, captures fine-grained local features without losing global context information, and lays a foundation for accurate identity matching.In addition, the embodiment of the present application also brings good interpretability through the attention mechanism, which can intuitively understand the body parts focused by the model through visual attention map, which not only helps to understand the decision-making process of the model, but also provides a powerful tool for model debugging and performance optimization.

[0231] 5、The traditional cascading matching strategy mainly relies on time dimension for cascading processing, and matches in turn according to the length of track loss time, and the recently observed track is preferentially processed.This method is based on the assumption that target motion is usually time-continuous, and considers that the prediction error will increase with time, so the shorter the track loss time is, the more reliable the state estimation is.However, this single-dimensional cascading strategy ignores the difference in track quality, and cannot distinguish between high-quality tracks with long-term stable tracking and newly created tracks, which may lead to unstable tracking in complex scenes.The hierarchical matching strategy based on track stability of the embodiment of the present application introduces a multi-dimensional track quality evaluation mechanism, and increases track stability layering on the basis of time cascading, forming a two-dimensional matching structure of "stability layering + time cascading", which significantly enhances the robustness and stability of the multi-target tracking system.

[0232] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram.For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order.In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology.The disclosed method is not limited to the operations and logical flows presented herein.Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0233] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able, using ordinary skill, to practice the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0234] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer- readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0235] The logic and / or steps represented in the flowcharts and / or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by an instruction execution system, apparatus, or device such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0236] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0237] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used in the described embodiments: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0238] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0239] Although embodiments of the application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application. The scope of the application is limited only by the claims and the equivalents thereof.

[0240] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for visual detection and tracking of human body parts by astronauts on a space station, characterized in that, Includes the following steps: The first feature map is obtained by performing rotation feature extraction on the input image through an orientation-invariant convolution module. The spatial dimension automatic alignment module performs spatial dimension alignment and feature fusion operations on several first feature maps to obtain a second feature map. Based on the YOLOv5 network, the target detection image is obtained by passing the detection network according to the first feature map output by the orientation-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module. Based on the target detection image, a third feature map is obtained through a feature extraction network; Based on the third feature map, local feature vectors and global feature vectors are obtained through the component localization module and the Transformer module; A feature fusion operation is performed on the local feature vector and the global feature vector to obtain a fourth feature map; Based on the target location information in the target detection image, the target trajectory is obtained through an adaptive memory window Kalman filter; The fourth feature map and the target trajectory are input into the hierarchical matching strategy to evaluate the target trajectory and obtain the target tracking result.

2. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 1, characterized in that, The step of performing rotation feature extraction on the input image using an orientation-invariant convolution module to obtain a first feature map includes the following steps: Obtain the target distance from a pixel in the input image to the target direction line; Based on the target distance, construct the first convolution kernel with a direction-sensitive branch using a Gaussian function; The first convolutional kernel is zero-meaned and random perturbation is added to obtain the second convolutional kernel of the orientation-sensitive branch; The input image is convolved according to the second convolution kernel of the direction-sensitive branch to obtain the fifth feature map; Based on the third convolution kernel of the main branch, the input image is convolved to obtain the sixth feature map; The first feature map includes the fifth feature map and the sixth feature map; the orientation-invariant convolution module includes the orientation-sensitive branch and the main branch.

3. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 2, characterized in that, The process of obtaining the target distance from a pixel in the input image to the target direction line includes the following steps: Establish a centralized coordinate system and obtain the coordinate values ​​of each pixel in the coordinate system; Based on the coordinate values, obtain the first distance from the pixel to the diagonal; Based on the coordinate values, obtain the second distance of the pixel in the horizontal direction; The target distance includes the first distance and the second distance.

4. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 1, characterized in that, The step of performing spatial dimension alignment and feature fusion operations on several first feature maps through the automatic spatial dimension alignment module to obtain a second feature map includes the following steps: Obtain the minimum height from several of the first feature maps; Obtain the minimum width of several of the first feature maps; The minimum common space dimension is obtained based on the minimum height and the minimum width; Based on the minimum common space dimension, the spatial scales of several first feature maps are aligned and adjusted to obtain several seventh feature maps; The second feature map is obtained by performing a feature fusion operation on several of the seventh feature maps.

5. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 1, characterized in that, The step of obtaining local and global feature vectors based on the third feature map through the component localization module and the Transformer module includes the following steps: The component localization module based on the attention mechanism is used to locate each human body component in the third feature map and generate multiple attention maps. The feature information of the human body parts corresponding to the multiple attention maps is input into the Transformer module corresponding to the human body parts to obtain multiple local feature vectors; Based on the third feature map, the global feature vector is obtained through the global Transformer module.

6. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 1, characterized in that, The adaptive memory window type Kalman filter includes: Dynamic velocity smoothing mechanism, appearance feature similarity calculation mechanism, trajectory quality assessment mechanism, adaptive Mahalanobis distance and appearance distance fusion mechanism, and adaptive survival threshold mechanism.

7. The method for visual detection and tracking of human body parts by astronauts on a space station according to claim 1, characterized in that, The step of inputting the fourth feature map and the target trajectory into a hierarchical matching strategy, evaluating the target trajectory, and obtaining the target tracking result includes the following steps: The target trajectory is layered to obtain a high-confidence stable trajectory, a normal trajectory, and a low-confidence unstable trajectory. Using the fourth feature map, time-cascaded matching is performed on the high-confidence stable trajectory to obtain a set of matched trajectories and a list of unmatched detections; Using the fourth feature map, time-cascaded matching is performed on the ordinary trajectory to update the set of matched trajectories and the list of unmatched detections; Using the fourth feature map, time-cascaded matching is performed on the low-confidence unstable trajectory, the set of matched trajectories and the list of unmatched detections are updated, and the set of matched trajectories and the list of unmatched detections are statistically analyzed to obtain the target tracking result.

8. A visual human body part detection and tracking device for astronauts on a space station, characterized in that, include: The first module is used to perform rotation feature extraction on the input image through the orientation-invariant convolution module to obtain the first feature map; The second module is used to perform spatial dimension alignment and feature fusion operations on several first feature maps through the spatial dimension automatic alignment module to obtain a second feature map. The third module is used to obtain a target detection image based on the YOLOv5 network, using the first feature map output by the orientation-invariant convolution module and the second feature map output by the spatial dimension automatic alignment module, through a detection network. The fourth module is used to obtain a third feature map based on the target detection image through a feature extraction network; The fifth module is used to obtain local feature vectors and global feature vectors based on the third feature map through the component positioning module and the Transformer module; The sixth module is used to perform feature fusion operation on the local feature vector and the global feature vector to obtain the fourth feature map; The seventh module is used to obtain the target trajectory by means of an adaptive memory window Kalman filter based on the target position information in the target detection image; The eighth module is used to input the fourth feature map and the target trajectory into the hierarchical matching strategy, evaluate the target trajectory, and obtain the target tracking result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images