Key point detection method and system based on multi-source data set joint model training

Abstract

The invention provides a key point detection method and system based on multi-source data set joint model training, and relates to the technical field of computer vision, and the method comprises the steps: firstly starting a cross-data-set semantic cooperative training process, constructing a cross-data-set joint semantic reference, building a bidirectional semantic mapping relation, and generating a unique identification sequence of each data set key point; performing semantic alignment processing by adopting a dynamic semantic aggregation algorithm to generate a multi-source fusion data set with unified semantics; dividing a joint training set and a joint test set according to rules; inputting the joint training set into a preset neural network in batches, screening neurons through the unique identification sequence, generating a directional training feedback signal, and updating a neural network weight parameter; and finally, the training process is repeated, a training convergence state is judged in combination with multiple factors, the optimized key point detection model is output, and the generalization ability and the detection precision of the model are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a keypoint detection method and system based on joint model training of multi-source datasets. Background Technology

[0002] In the field of computer vision, keypoint detection is a fundamental and crucial task, widely applied in numerous scenarios such as face recognition, pose estimation, and object tracking. Currently, keypoint detection methods primarily rely on a single dataset for model training. However, single datasets often suffer from limitations such as limited data volume, uneven data distribution, and limited scene coverage. This results in poor generalization ability of the trained models, making it difficult to maintain stable and accurate detection performance across different scenarios and data distributions.

[0003] To address this issue, some studies have attempted to fuse multiple datasets for model training. However, different datasets exhibit significant differences in keypoint semantic descriptions, annotation methods, and data features. Direct fusion can lead to severe semantic biases and representational discrepancies, making it difficult for the model to effectively learn a unified keypoint feature representation, thereby affecting detection accuracy and generalization ability. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a keypoint detection method based on joint model training of multi-source datasets, the method comprising: Initiate a cross-dataset semantic collaborative training process, construct a cross-dataset joint semantic benchmark based on the keypoint semantic description of multi-source datasets, establish a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark, and generate a unique identifier sequence of keypoints from each dataset in the cross-dataset joint semantic benchmark; A dynamic semantic aggregation algorithm is used to perform semantic alignment on the key points of each dataset, eliminating semantic bias and representation differences between different datasets, and generating a semantically unified multi-source fusion dataset. The semantically unified multi-source fusion dataset is split into batches according to the same batch division rule of the same dataset, forming a joint training set and a joint test set. Each batch contains only training data or test data of a single dataset. The joint training set is input into the preset neural network in batches. The neurons corresponding to the output layer of the preset neural network are selected by the unique identifier sequence of each dataset. Based on the semantic difference between the prediction results of the selected neurons and the real semantic labels, a targeted training feedback signal is generated. The weight parameters of the neurons corresponding to the backbone layer and output layer of the preset neural network are updated by the targeted training feedback signal. Repeat the batch training and weight update process, combine the semantic matching effect of the joint test set with the training iteration rounds to determine the training convergence status, and output the optimized key point detection model when the convergence condition is met.

[0005] Furthermore, embodiments of the present invention also provide a keypoint detection system based on a joint model training of multi-source datasets, characterized in that it includes: A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the aforementioned keypoint detection method based on a joint model training of a multi-source dataset by executing the machine-executable instructions.

[0006] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including machine-executable instructions, the machine-executable instructions being stored in a computer-readable storage medium, the processor of the keypoint detection system based on a multi-source dataset joint model training reads the machine-executable instructions from the computer-readable storage medium, the processor executes the machine-executable instructions, causing the keypoint detection system based on a multi-source dataset joint model training to execute the aforementioned keypoint detection method based on a multi-source dataset joint model training.

[0007] Based on the above, by initiating a cross-dataset semantic collaborative training process, constructing a joint semantic benchmark across datasets and establishing a bidirectional semantic mapping relationship, and employing a dynamic semantic aggregation algorithm for semantic alignment processing, semantic biases and representational differences between different datasets can be accurately eliminated, generating a semantically unified multi-source fusion dataset. The joint training set and joint test set are divided according to specific rules, and neurons are selected using unique identifier sequences to generate targeted training feedback signals. This allows for targeted updates to the weight parameters of corresponding neurons in the backbone and output layers of the neural network, improving the efficiency and accuracy of model training. By repeating the training and update process and considering multiple factors to determine the training convergence state, the optimized keypoint detection model is finally output, significantly improving the model's generalization ability and detection accuracy under different scenarios and data distributions. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the execution flow of the key point detection method based on multi-source dataset joint model training provided in this embodiment of the invention.

[0009] Figure 2 This is a schematic diagram of exemplary hardware and software components of a keypoint detection system based on a joint model training of multi-source datasets provided in an embodiment of the present invention. Detailed Implementation

[0010] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1This is a flowchart illustrating a keypoint detection method based on a joint model training of multi-source datasets, provided in one embodiment of the present invention. The following is a detailed description of this keypoint detection method based on a joint model training of multi-source datasets.

[0011] Step S110: Start the cross-dataset semantic collaborative training process, construct a cross-dataset joint semantic benchmark based on the keypoint semantic description of the multi-source dataset, establish a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark, and generate a unique identifier sequence of keypoints of each dataset in the cross-dataset joint semantic benchmark.

[0012] In this embodiment, human pose keypoint detection is used as the specific application scenario. The multi-source datasets involved include dataset A, dataset B, dataset C, and dataset D. All of these datasets are used for human keypoint detection tasks, but they differ in the number of keypoints, their definitions, and annotation standards. After initiating the cross-dataset semantic collaborative training process, it is necessary to unify the semantics of the keypoints in the above datasets, construct a cross-dataset joint semantic benchmark, and establish mapping relationships to achieve subsequent joint training.

[0013] Step S111: Start the cross-dataset semantic collaborative training process, trigger the multi-source dataset semantic extraction instruction, extract all keypoint semantic descriptions of each dataset. The keypoint semantic descriptions include the anatomical definition, functional attribute description, spatial location association information and annotation specification of the keypoints, forming a set of keypoint semantic descriptions for each dataset.

[0014] In the human pose keypoint detection scenario, after triggering the multi-source dataset semantic extraction command, the semantic descriptions of keypoints in datasets A, B, C, and D are extracted one by one. Taking dataset A as an example, it contains M keypoints, and the semantic description of each keypoint is extracted in detail. For example, for the "left shoulder" keypoint (dataset A index P), its anatomical definition is "the connection between the left scapula and clavicle, which is the connection point between the upper limb and the trunk"; its functional attribute description is "used to characterize the starting position of the left arm movement and the posture changes of the shoulder"; its spatial location association information is "located slightly to the left below the neck, symmetrically distributed with the right shoulder keypoint, and connected to the left elbow keypoint below"; the annotation specification is "when annotating, take the most prominent point on the outer side of the acromion, and when the image resolution is a specific value, the coordinates are accurate to two decimal places." Similarly, for the "pelvis" keypoint in dataset B (dataset B index Q), its anatomical definition is "a ring-shaped bone structure formed by the sacrum, coccyx, and left and right hip bones and their ligaments," its functional attribute description is "the connecting hub between the human trunk and lower limbs, bearing the weight of the upper body and coordinating lower limb movement," its spatial location association information is "located below the thoracic cavity, connecting the left and right hips respectively," and its annotation specification is "taking the geometric center of the pelvic region as the annotation point." Similar extraction operations are performed on each keypoint in all datasets, ultimately forming their respective keypoint semantic description sets, such as the semantic description set for dataset A, the semantic description set for dataset B, etc.

[0015] Step S112: Perform semantic deduplication on the set of keypoint semantic descriptions of all datasets. Eliminate duplicate or highly similar semantic descriptions through semantic similarity analysis, and retain keypoint descriptions with unique semantic dimensions.

[0016] The semantic descriptions of keypoints from four datasets—Dataset A, Dataset B, Dataset C, and Dataset D—are aggregated. Then, all aggregated semantic descriptions are compared pairwise to calculate semantic similarity. Semantic similarity analysis is performed by converting the semantic description text into vector representations and then calculating the cosine similarity between the vectors. For example, comparing the semantic description of the "right shoulder" keypoint in Dataset C with that in Dataset A, if their core anatomical definitions, functional attributes, and spatial location association information are essentially the same, with only minor differences in annotation specifications (such as the requirement for one or two decimal places in coordinate precision), they are considered highly similar. In this case, the more detailed semantic description from one dataset is retained, while the other is discarded. By removing duplicate or highly similar semantic descriptions, this ensures that each basic unit in the subsequently constructed cross-dataset joint semantic benchmark has a unique semantic dimension.

[0017] Step S113: Perform semantic deduplication on the set of keypoint semantic descriptions of all datasets. Eliminate duplicate or highly similar semantic descriptions through semantic similarity analysis, and retain keypoint descriptions with unique semantic dimensions.

[0018] This step is identical to step S112, serving as a re-confirmation and execution of the semantic deduplication process to ensure the purity of the processed semantic description set. In practice, different similarity calculation parameters or algorithms may be used for cross-validation to improve the accuracy of deduplication. For example, the first step might use a bag-of-words model-based similarity calculation, followed by a sentence embedding similarity calculation based on a pre-trained language model. If both results indicate duplication or high similarity, the duplicated descriptions are discarded.

[0019] Step S114: Perform hierarchical clustering of the deduplicated key point semantic descriptions according to semantic relevance, and group the semantic descriptions of key points that are functionally related and spatially adjacent into the same semantic cluster, with each semantic cluster corresponding to a core semantic concept.

[0020] In human posture keypoint detection, functionally related keypoints typically form a kinematic chain, such as "left shoulder," "left elbow," and "left wrist" in dataset A. These keypoints collectively participate in the flexion and extension movements of the left arm and are functionally closely related. Spatially adjacent keypoints, such as "chest cavity" and "pelvis" in dataset B, and "spine 1," "spine 2," and "spine 3" in dataset D, are sequentially distributed along the central axis of the human torso and are spatially adjacent. By calculating the correlation between functional attributes and spatial distance in the semantic descriptions, the semantic descriptions of these keypoints are grouped into the same semantic cluster. For example, the semantic descriptions of "left shoulder," "left elbow," "left wrist," "right shoulder," "right elbow," and "right wrist" are grouped into the "limb joint keypoint cluster," with the core semantic concept being "a set of joints that move the human limbs"; the semantic descriptions of "chest cavity," "pelvis," "spine 1," "spine 2," and "spine 3" are grouped into the "trunk central axis keypoint cluster," with the core semantic concept being "a set of keypoints in the skeletal structure of the human trunk central axis." The semantic descriptions of keypoints within each semantic cluster are highly correlated functionally or spatially.

[0021] Step S115: Perform semantic integration and standardized description for each semantic cluster, determine the unified semantic definition, core attributes and association rules of key points within each semantic cluster, and form the basic unit of the cross-dataset joint semantic benchmark.

[0022] Taking the "limb joint key point cluster" as an example, the semantic descriptions of key points such as "left shoulder," "left elbow," "left wrist," "right shoulder," "right elbow," and "right wrist" within the cluster are integrated and standardized. Regarding unified semantic definitions, the standard definition of "shoulder joint" is clarified as "a ball-and-socket joint connecting the upper limb and trunk, allowing multi-directional movement of the upper limb," and left and right side identification rules are distinguished for "left shoulder" and "right shoulder." Regarding core attributes, it is stipulated that each joint key point must include core attribute entries such as "anatomical location," "major axis of motion," and "functional range of motion." Regarding association rules, it is clarified that "shoulder joint" connects to "elbow joint," and "elbow joint" connects to "wrist joint," forming a "shoulder-elbow-wrist" serial association rule, with symmetrical distribution of joints on both sides. After the above integration and standardization, each semantic cluster forms a well-structured and clearly defined cross-dataset joint semantic benchmark unit.

[0023] Step S116: Assign a unique semantic identifier and point number to each basic unit of the cross-dataset joint semantic benchmark, arrange the order of the basic units according to the association logic and spatial distribution rules of the semantic clusters, and construct the structured cross-dataset joint semantic benchmark.

[0024] Semantic identifiers, such as "LimbsJoints," are assigned to the "LimbsJoints" cluster, and point sequence numbers are assigned to "left shoulder," "right shoulder," "left elbow," "right elbow," "left wrist," and "right wrist" within each cluster. These basic units are arranged according to the spatial distribution of the human body from top to bottom and from left to right; for example, the trunk is arranged before the limbs, and within the limbs, the upper limbs are arranged before the lower limbs. After arranging the basic units of all semantic clusters according to this rule, a cross-dataset joint semantic benchmark is formed. This cross-dataset joint semantic benchmark covers all keypoints with unique semantics after deduplication.

[0025] Step S117: For each set of keypoint semantic descriptions in the original dataset, compare them one by one with the semantic descriptions of the basic units in the cross-dataset joint semantic benchmark to identify the semantic correspondence between the keypoints of each dataset and the basic units of the cross-dataset joint semantic benchmark.

[0026] Taking the "pelvis" keypoint in dataset B as an example, its semantic description is compared with the "pelvis" basic unit under the "torso midline keypoint cluster" in the cross-dataset joint semantic benchmark. By comparing their anatomical definitions, functional attributes, and spatial locations, it is confirmed that the "pelvis" keypoint in dataset B is semantically consistent with the "pelvis" basic unit in the cross-dataset joint semantic benchmark, thus establishing a correspondence. For the "virtual neck" keypoint in dataset C, its semantic description is compared with the "neck" basic unit in the cross-dataset joint semantic benchmark. If its definition differs somewhat from the "neck" basic unit but the core concept is consistent, a correspondence is also established.

[0027] Step S118: Establish a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark based on the semantic correspondence. The forward mapping records the association path from the key points of the dataset to the basic unit of the cross-dataset joint semantic benchmark, and the reverse mapping records the semantic representation of the basic unit of the cross-dataset joint semantic benchmark in each dataset.

[0028] In the forward mapping, for the "nose tip" keypoint in dataset A, its association path is recorded in the forward mapping table as "dataset A - keypoint X - cross-dataset joint semantic benchmark - facial keypoint cluster - nose tip (point order Y)". In the reverse mapping, for the "pelvis" basic unit (point order Z) in the cross-dataset joint semantic benchmark, its representation in dataset B is recorded as "pelvis (index Q)" and its representation in dataset D is "pelvis (index W)".

[0029] Step S119: Based on the bidirectional semantic mapping relationship, extract the cross-dataset joint semantic benchmark point sequence number corresponding to the key points of each dataset, organize the number sequence according to the key point arrangement order of the dataset itself, and generate a unique identifier sequence of the key points of each dataset in the cross-dataset joint semantic benchmark.

[0030] The keypoints of dataset A are arranged in the order from the first keypoint to the Mth keypoint, namely keypoint 0, keypoint 1, ..., keypoint M-1. Based on the bidirectional mapping relationship, the cross-dataset joint semantic benchmark point sequence number corresponding to each keypoint is extracted, resulting in the unique identifier sequence of dataset A as [Y0, Y1, Y2, ..., YM-1], where Y0 is the point sequence number of keypoint 0 in dataset A in the cross-dataset joint semantic benchmark, Y1 is the point sequence number of keypoint 1 in dataset A in the cross-dataset joint semantic benchmark, and so on. Similarly, the unique identifier sequence of dataset B is [Z0, Z1, Z2, ..., ZN-1], where N is the number of keypoints in dataset B, and Z0 is the point sequence number of keypoint 0 in dataset B in the cross-dataset joint semantic benchmark.

[0031] Step S1110: Link and store the cross-dataset joint semantic benchmark, bidirectional semantic mapping relationship and unique identifier sequence to construct a cross-dataset semantic association database. The cross-dataset semantic association database contains the semantic mapping details of each dataset, the correspondence of key point identifiers and the complete definition of the cross-dataset joint semantic benchmark.

[0032] The constructed cross-dataset joint semantic benchmark, the bidirectional semantic mapping relationship table for each dataset, and the unique identifier sequence are stored in the database. The database table structure includes a "joint semantic benchmark table" (storing the semantic identifier, point sequence number, standardized description, etc. of the basic unit), a "forward mapping table" (storing the mapping from the dataset key points to the benchmark basic unit), a "reverse mapping table" (storing the mapping from the benchmark basic unit to the dataset key points), and a "unique identifier sequence table" (storing the unique identifier sequence of each dataset). These tables are interconnected through key fields to form a complete cross-dataset semantic association database.

[0033] Step S1111: Perform semantic parsing on the keypoint semantic description set of each original dataset, and extract the core semantic elements of each keypoint. The core semantic elements include the functional definition, anatomical location, relationship with other keypoints, and annotation specifications of the keypoint.

[0034] Semantic parsing was performed on the "left hip" keypoint (index K) in dataset A, and its functional definition was extracted as "connecting the trunk and the left lower limb, supporting lower limb movement". Its anatomical location was "left side of the pelvis, connected to the left knee keypoint". Its relationship with other keypoints was "connecting to the pelvis above and the left knee below". The annotation standard was "take the outermost point of the greater trochanter of the femur".

[0035] Step S1112: Compare the core semantic elements of each key point with the standardized semantic definitions of each basic unit in the cross-dataset joint semantic benchmark, calculate the similarity score using a semantic similarity calculation algorithm, and select the basic unit with the highest similarity score as the matching result.

[0036] The core semantic elements of the "left hip" keypoint in dataset A are compared with the standardized semantic definitions of each basic unit in the cross-dataset joint semantic benchmark. Through semantic similarity calculation, it is found that the similarity score with the "left hip" basic unit (point order L) is the highest, so this basic unit is taken as the matching result.

[0037] Step S1113: Establish a positive semantic mapping table based on the matching results. The positive semantic mapping table records the cross-dataset joint semantic benchmark basic unit number and semantic similarity score corresponding to the key points of each original dataset.

[0038] In the positive semantic mapping table, the cross-dataset joint semantic benchmark base unit number L and semantic similarity score are recorded for the "left hip" key point (index K) of dataset A.

[0039] Step S1114: Construct a reverse semantic mapping table, which records the key point information of all original datasets corresponding to each basic unit in the cross-dataset joint semantic benchmark, including dataset identifier and key point name.

[0040] In the reverse semantic mapping table, the "left hip" basic unit (number L) of the cross-dataset joint semantic benchmark corresponds to the key point name "left hip" (index K) of dataset A, the key point name "left hip" (index P) of dataset B, etc.

[0041] Step S1115: Check the forward semantic mapping table to confirm that each key point in the original dataset has an associated cross-dataset joint semantic benchmark base unit number.

[0042] Traverse the positive semantic mapping table and check whether each of the M key points in dataset A has a corresponding cross-dataset joint semantic benchmark base unit number. If a key point is found to be missing a corresponding number, perform semantic parsing and comparison again to ensure that all key points can be associated with the base unit of the cross-dataset joint semantic benchmark.

[0043] Step S1116: Based on the positive semantic mapping table, extract the basic unit number of the cross-dataset joint semantic benchmark corresponding to each key point according to the order of the key points of the original dataset, and form a unique identifier sequence of the key points of the original dataset in the cross-dataset joint semantic benchmark.

[0044] Based on the arrangement order of keypoints 0 to M-1 in dataset A, the basic unit number corresponding to each keypoint is extracted from the positive semantic mapping table and arranged sequentially to form the unique identifier sequence of dataset A [Y0, Y1, Y2, ..., YM-1].

[0045] Step S1117: Add sequence identification information to the uniquely identified sequence. The sequence identification information includes the dataset name, sequence generation time, and cross-dataset joint semantic benchmark version number.

[0046] Add sequence identification information to the unique identifier sequence of dataset A, such as "Dataset name: dataset A, sequence generation time: specific timestamp, cross-dataset joint semantic benchmark version number: V1".

[0047] Step S1118: Link and store the forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence of each dataset to form a bidirectional semantic mapping database.

[0048] The forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence of each dataset are stored in the database and linked by fields such as dataset name and basic unit number to form a bidirectional semantic mapping database, which facilitates subsequent querying and use.

[0049] Step S1119: Update and maintain the bidirectional semantic mapping database. When a new dataset is added, perform semantic parsing and matching steps to update the forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence. When the semantic benchmark is updated across datasets, re-execute the semantic parsing and matching steps to update the forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence, and output the bidirectional semantic mapping database and the unique identifier sequence of each dataset.

[0050] If a new human pose dataset E is added, semantic parsing and matching with the cross-dataset joint semantic benchmark are performed on dataset E. The mapping relationship and unique identifier sequence of dataset E are then added to the bidirectional semantic mapping database. If the cross-dataset joint semantic benchmark is updated, such as by adding new keypoint basic units, semantic parsing and matching are re-performed on all original datasets, and the corresponding mapping tables and unique identifier sequences are updated.

[0051] Step S120: Use a dynamic semantic aggregation algorithm to perform semantic alignment processing on the key points of each dataset, eliminate semantic bias and representation differences between different datasets, and generate a semantically unified multi-source fusion dataset.

[0052] By using a dynamic semantic aggregation algorithm, the key points of datasets A, B, C, and D are unified in terms of semantics and coordinates, eliminating semantic bias and different representations caused by differences in datasets. This ensures that the key points of all datasets meet the requirements of a cross-dataset joint semantic benchmark, ultimately integrating them into a semantically unified multi-source fusion dataset.

[0053] Step S121: Retrieve the cross-dataset joint semantic benchmark, bidirectional semantic mapping relationship and key point raw data of each dataset from the cross-dataset semantic association database. The key point raw data includes key point coordinate information, semantic labels and dataset source identifiers.

[0054] Retrieve detailed definitions of the cross-dataset joint semantic benchmark from the cross-dataset semantic association database, bidirectional semantic mapping relationships between each dataset and the benchmark, and original key point data of datasets such as dataset A and dataset B, such as the coordinates (x, y) of each key point in each image in dataset A, the corresponding semantic labels (e.g., "left shoulder"), and the dataset source identifier "dataset A".

[0055] Step S122: For each basic unit in the cross-dataset joint semantic benchmark, select all original datasets containing the semantic description of the basic unit from the bidirectional semantic mapping relationship, and count the completeness of the semantic description and the consistency of the annotation of each dataset on the basic unit.

[0056] Taking the "neck" basic unit (point order U) in the cross-dataset joint semantic benchmark as an example, datasets B, C, and D containing the semantic description of this basic unit are selected from the bidirectional semantic mapping relationship. The completeness of the semantic description of dataset B on the "neck" basic unit is statistically analyzed, such as whether it contains all core semantic elements; in terms of annotation consistency, the standard deviation, coefficient of variation, and other indicators of the "neck" keypoint annotations in this dataset are statistically analyzed to evaluate the stability of annotations among different samples.

[0057] Step S123: Based on the statistical results, select the dataset with the most complete semantic description and the highest consistency of annotation as the semantic benchmark dataset of the basic unit, and extract the standard semantic labels, coordinate distribution patterns and related attributes of the corresponding key points in the dataset as benchmark references.

[0058] Statistical analysis revealed that dataset B possessed the most complete semantic description and the highest annotation consistency for the "neck" basic unit. Therefore, dataset B was selected as the semantic benchmark dataset for the "neck" basic unit. The standard semantic label "neck," coordinate distribution patterns (such as the relative position range in human images and the proportional relationship relative to other key points), and associated attributes (such as distance and angle relationships with key points in the chest cavity and top of the head) of the "neck" key points (index H) in dataset B were extracted as benchmark references.

[0059] Step S124: For each non-benchmark dataset, locate the key points corresponding to each basic unit of the cross-dataset joint semantic benchmark according to the bidirectional semantic mapping relationship, extract the original semantic labels and coordinate information of the key points, and form a set of data to be aligned for the non-benchmark dataset.

[0060] For dataset C (the benchmark dataset that is not the "neck" basic unit), the "virtual neck" keypoints (index I) corresponding to the "neck" basic units are located based on the bidirectional semantic mapping relationship. The original semantic labels "virtual neck" and coordinate information of these keypoints are extracted to form part of the dataset to be aligned. At the same time, similar processing is performed on other basic units of the cross-dataset joint semantic benchmark, such as "left shoulder" and "right shoulder," to extract the original semantic labels and coordinate information of the corresponding keypoints in dataset C. After integration, the complete dataset to be aligned for dataset C is formed.

[0061] Step S125: Use a dynamic semantic awareness network to perform semantic depth analysis on the dataset to be aligned from the non-benchmark dataset and the benchmark reference, mine semantic expression differences, coordinate distribution deviations and related attribute differences, and generate semantic difference feature data.

[0062] By using a dynamic semantic perception network, the "virtual neck" data to be aligned in dataset C and the "neck" reference in dataset B are analyzed to identify differences in semantic representation (such as the difference between the terms "virtual neck" and "neck"), deviations in coordinate distribution (such as the coordinate offset patterns of corresponding key points in different images), and differences in associated attributes (such as different relative positional relationships with other key points). The above differences are then integrated to generate semantic difference feature data.

[0063] Step S126: Perform word segmentation on the semantic tags in the data set to be aligned from the non-benchmark dataset, and split them into semantic core words, modifier words and related words to form a semantic vocabulary set.

[0064] The semantic labels of dataset C, “virtual neck”, are segmented into core words “neck” and modifiers “virtual”, forming a semantic vocabulary set {“virtual”, “neck”}. For semantic labels containing multiple modifiers, such as “left upper shoulder joint”, they are segmented into core words “shoulder joint” and modifiers “left” and “upper”.

[0065] Step S127: Perform the same word segmentation process on the standard semantic tags in the benchmark reference to obtain the benchmark semantic vocabulary set, and establish a one-to-one correspondence between the semantic vocabulary to be aligned and the benchmark semantic vocabulary.

[0066] The standard semantic labels for "neck" in dataset B are segmented to obtain a baseline semantic vocabulary set {"neck"}. A correspondence is established between the semantic vocabulary "neck" to be aligned and the baseline semantic vocabulary "neck". For the modifier "virtual", if there is no direct corresponding word in the baseline semantic vocabulary set, it is marked as a newly added modifier.

[0067] Step S128: Convert the semantic vocabulary set to be aligned and the benchmark semantic vocabulary set into semantic vectors using a word embedding algorithm, calculate the semantic vector similarity of each corresponding word pair, and mark word pairs with similarity below a preset threshold as semantic difference points.

[0068] The word "neck" in the semantic vocabulary set to be aligned and the word "neck" in the benchmark semantic vocabulary set are converted into semantic vectors using a word embedding algorithm (such as Word2Vec or GloVe), and the cosine similarity between the two is calculated. If the similarity is lower than a preset threshold (determined based on experience or experiments), "neck"-"neck" is marked as a semantic difference point. For "virtual" in the semantic vocabulary to be aligned, since there is no corresponding word in the benchmark semantic vocabulary set, it is directly marked as a semantic difference point.

[0069] Step S129: Extract the key point coordinate data from the dataset to be aligned and the benchmark reference, calculate the coordinate difference of key points corresponding to the same semantic in the X and Y axis directions, statistically analyze the distribution range and central tendency of the coordinate difference, and identify the coordinate distribution deviation characteristics.

[0070] Coordinate data of the "virtual neck" in dataset C and the "neck" in dataset B are extracted from multiple images. For semantically corresponding keypoints in each image, the coordinate difference Δx = x_C - x_B in the X-axis direction and the coordinate difference Δy = y_C - y_B in the Y-axis direction are calculated, where x_C and y_C are the coordinates of the "virtual neck" in dataset C, and x_B and y_B are the coordinates of the "neck" in dataset B. The distribution range of Δx and Δy (e.g., Δx is in the interval [a, b], and Δy is in the interval [c, d]) and central tendency (e.g., the mean μx and median mx of Δx, and the mean μy and median my of Δy) are statistically analyzed to identify coordinate distribution deviation characteristics, such as Δx being generally biased towards positive values ​​and Δy being relatively dispersed.

[0071] Step S1210: Analyze the association attributes of key points in the dataset to be aligned, including the distance, angle, and relative position relationship between key points, compare them with the association attributes in the benchmark reference, and explore the differences in association attributes.

[0072] Analyze the association attributes of the virtual neck and virtual head (index J) in dataset C, including distance (Euclidean distance, Manhattan distance), angle (the angle formed by the virtual neck, the virtual head, and another reference keypoint), and relative position (e.g., the virtual head is directly above or slightly to the left of the virtual neck). Compare these attributes with those of the neck and head (index K) in dataset B. Calculate the distance and angle differences to determine if the relative positional relationships are consistent and to uncover differences in association attributes. For example, the distance between the virtual neck and virtual head in dataset C is generally 10% shorter than the distance between the neck and head in dataset B.

[0073] Step S1211: Normalize the difference values ​​of semantic expression differences, coordinate distribution deviations, and related attribute differences to make the values ​​of different difference types comparable.

[0074] The degree values ​​of semantic expression differences (such as the complement of the word similarity score, 1 - similarity), coordinate distribution deviations (such as the absolute value of the coordinate difference divided by the length of the image diagonal), and associated attribute differences (such as the absolute value of the distance difference divided by the baseline distance) are normalized and mapped to the range [0, 1] for comprehensive comparison and analysis. For example, if the semantic similarity is 0.6, the degree value of semantic expression difference is 0.4; if the coordinate difference Δx is 5 pixels and the length of the image diagonal is L pixels, the degree value of coordinate distribution deviation is 5 / L.

[0075] Step S1212: Construct a semantic difference feature matrix. The row dimension of the matrix is ​​the number of key points in the dataset to be aligned, and the column dimension is the difference type. The difference types include semantic expression differences, coordinate distribution deviations, and related attribute differences. The matrix elements are the normalized difference values ​​of the corresponding key points under each difference type.

[0076] Assuming the dataset C contains Q keypoints for alignment, the semantic difference feature matrix is ​​a Q-row, 3-column matrix. The first column of the q-th row (corresponding to the q-th keypoint in the dataset) represents the normalization degree of semantic representation difference for that keypoint, the second column represents the normalization degree of coordinate distribution deviation, and the third column represents the normalization degree of association attribute difference. For example, the r-th row of the matrix might contain [d_sem_r, d_coord_r, d_rel_r], where d_sem_r is the normalization degree of semantic representation difference for the r-th keypoint, d_coord_r is the normalization degree of coordinate distribution deviation, and d_rel_r is the normalization degree of association attribute difference.

[0077] Step S1213: Construct semantic difference feature data using a graph neural network, treating each key point as a graph node, with the node attribute being the row vector of the semantic difference feature matrix of that key point, and the edge weights between nodes being the difference correlation strength between the corresponding key points.

[0078] Using Q keypoints in the dataset C to be aligned as graph nodes, each node's attributes are set as the row vector [d_sem_q, d_coord_q, d_rel_q] of the corresponding row in the semantic difference feature matrix. The edge weights between nodes are determined based on the strength of the difference association between the corresponding keypoints. This strength is obtained by calculating the cosine similarity of the semantic difference feature vectors of the two keypoints, the covariance of their coordinate differences, and other factors. For example, the edge weight w_rs between keypoint r and keypoint s represents the degree of correlation between them in terms of difference features; the stronger the correlation, the larger w_rs.

[0079] Step S1214: Extract features from semantic difference feature data through graph convolution operation to enhance the representation ability of key difference features and weaken the influence of secondary differences.

[0080] The constructed semantic difference feature graph data is processed using a graph convolutional neural network. Node features are aggregated and transformed through multiple graph convolutional layers. Each graph convolutional operation, based on the edge weights between nodes, weights and aggregates the features of adjacent nodes to the current node, updating the node feature representation. This strengthens the representation of key difference features (such as those significantly impacting semantic alignment) while weakening the influence of minor differences (such as negligible subtle differences).

[0081] Step S1215: Perform dimensionality compression and feature fusion on the extracted differential features to generate a comprehensive differential feature vector for each key point.

[0082] The high-dimensional difference features extracted by the graph convolution operation are subjected to dimensionality compression processing such as principal component analysis (PCA) or linear discriminant analysis (LDA) to reduce feature dimensionality and remove redundant information. Then, different types of difference features (semantic, coordinate, and association attributes) are fused by concatenation to generate a comprehensive difference feature vector for each keypoint. This comprehensive difference feature vector contains comprehensive difference information between the keypoint and the benchmark reference in terms of semantics, coordinates, and association attributes.

[0083] Step S1216: Project the key point coordinates of the non-benchmark dataset and the semantic benchmark dataset to the preset target center standard coordinate system. Eliminate the coordinate offset caused by differences in viewpoint, scale and shooting angle through coordinate system transformation to generate standard coordinate data.

[0084] A target center standardized coordinate system is pre-defined with the center of the human body as the origin. The center of the human body can be defined as the midpoint of the left and right hip key points or the pelvic key point. For the key point coordinates of each dataset, projection is performed using a coordinate transformation matrix. The transformation matrix is ​​calculated based on the bounding box information of the human body in the image and the pre-defined human body proportion parameters. It is used to eliminate coordinate offsets caused by shooting perspective (such as front, side, oblique), image scale (such as different image resolutions, human body size), and shooting angle (such as top view, bottom view), to obtain standardized coordinate data in the target center standardized coordinate system.

[0085] Step S1217: Based on semantic difference feature data and standard coordinate data, construct a semantic difference dataset of non-benchmark dataset and semantic benchmark dataset. The semantic difference dataset includes data to be aligned, benchmark data, and difference annotation information between the data to be aligned and the benchmark data.

[0086] A semantic difference dataset is constructed by integrating the data to be aligned (canonical coordinates, semantic labels, and comprehensive difference feature vectors) from a non-benchmark dataset (such as dataset C), the benchmark data (canonical coordinates and standard semantic labels) from a semantic benchmark dataset (such as dataset B), and the difference annotation information between the two (such as semantic difference points, coordinate differences, and degree of difference values ​​of related attributes). Each sample contains relevant information from the non-benchmark data, the corresponding benchmark data information, and the difference information between the two, which are used to train the semantic difference compensation model.

[0087] Step S1218: Select a deep convolutional neural network as the basic architecture of the semantic difference compensation model, input the semantic difference dataset into the semantic difference compensation model for training, so that the semantic difference compensation model learns the semantic mapping rules and coordinate compensation rules between the non-benchmark dataset and the semantic benchmark dataset.

[0088] A deep convolutional neural network with multiple convolutional, pooling, and fully connected layers is chosen as the basic architecture for the semantic discrepancy compensation model. Batch normalization layers and dropout layers can be added as needed to prevent overfitting. The semantic discrepancy dataset is divided into training, validation, and test sets, which are then input into the model for training. The model's input consists of the canonical coordinates and comprehensive discrepancy feature vectors of the data to be aligned, and its output consists of semantic correction parameters (such as semantic label mapping relationships and modifier weights) and coordinate compensation values ​​(Δx_comp, Δy_comp). Through training, the model learns how to generate appropriate semantic correction parameters and coordinate compensation values ​​based on the features of the data to be aligned to match the benchmark data. During training, loss functions (such as mean squared error loss and cross-entropy loss) are used to measure the difference between the predicted values ​​and the true discrepancy annotations, and the model parameters are optimized using the backpropagation algorithm.

[0089] Step S1219: Input the dataset to be aligned from the non-benchmark dataset into the trained semantic difference compensation model to obtain semantic correction parameters and coordinate compensation values. Adjust the semantic labels and coordinate information of the non-benchmark dataset based on the semantic correction parameters and coordinate compensation values ​​so that the adjusted semantics and coordinates meet the requirements of the cross-dataset joint semantic benchmark.

[0090] The dataset C to be aligned is input into a trained semantic difference compensation model. The model outputs semantic correction parameters (e.g., mapping rules to correct "virtual neck" to "neck", removing or adjusting the modifier "virtual") and coordinate compensation values ​​(Δx_comp, Δy_comp) for each keypoint. The semantic labels of dataset C are adjusted according to the semantic correction parameters to align with the standard semantic labels in the cross-dataset joint semantic benchmark. The canonical coordinates are adjusted according to the coordinate compensation values, with the adjusted coordinates being x_adj = x_norm + Δx_comp, y_adj = y_norm + Δy_comp, where x_norm and y_norm are the original canonical coordinates of the non-benchmark dataset in the target center canonical coordinate system. The adjusted semantics and coordinates must conform to the requirements of the corresponding basic unit in the cross-dataset joint semantic benchmark.

[0091] Step S1220: Integrate all adjusted non-benchmark datasets and semantic benchmark datasets to form a multi-source fusion dataset, in which each data sample contains semantic labels, canonical coordinates, and dataset source identifiers that conform to the cross-dataset joint semantic benchmark.

[0092] The adjusted non-benchmark datasets, such as datasets A, C, and D, are integrated with the semantic benchmark dataset, such as dataset B. Each data sample contains semantic labels (such as "left shoulder" and "neck") that conform to the joint semantic benchmark across datasets, canonical coordinates (adjusted coordinates in the canonical coordinate system of the target center), and dataset source identifiers (such as "dataset A" and "dataset B"), thus forming a semantically unified multi-source fused dataset.

[0093] Step S130: Split the semantically unified multi-source fusion dataset into batches according to the same batch division rule of the same dataset, forming a joint training set and a joint test set. Each batch contains only training data or test data of a single dataset.

[0094] Following the rule that each batch contains only one dataset, the multi-source fusion dataset is split into a joint training set for model training and a joint test set for model testing. This ensures the uniqueness of data batches during training and testing, and avoids model learning chaos caused by mixing data from different datasets in the same batch.

[0095] Step S131: Obtain semantically unified multi-source fusion datasets, extract dataset source identifiers from the datasets, classify the semantically unified multi-source fusion datasets according to the source identifiers, and obtain the classification datasets corresponding to each original dataset.

[0096] Extract the dataset source identifier of each data sample from the multi-source fusion dataset, and classify the data samples according to the identifier to obtain dataset A, dataset B, dataset C, and dataset D. Each classification dataset contains all data samples of the corresponding original dataset after semantic alignment.

[0097] Step S132: Perform data volume statistics for each classification dataset, and determine the amount of training data and test data for each classification dataset by combining the preset training data ratio and test data ratio.

[0098] Given a classification dataset A with TA data samples, a pre-defined training data ratio of α, and a test data ratio of 1-α (α is typically 0.7, 0.8, or 0.9), the training data size for classification dataset A is determined to be TA_train = round(TA_train / TA_train / TA_train). α), the test data size is TA_test = TA - TA_train. Similarly, for dataset B, the total number of data samples is TB, and the training data size is TB_train = round(TB). α), the amount of test data is TB_test = TB - TB_train. And so on, determine the amount of training data and test data for all classification datasets.

[0099] Step S133: Set a batch data volume standard. This batch data volume standard is determined based on the memory capacity of the preset neural network, the training efficiency requirements, and the complexity of the data samples, so that the data volume of each batch is consistent.

[0100] Considering the limitations of the pre-defined neural network's GPU memory capacity (such as the maximum number of samples that can be loaded in a single training session), training efficiency (such as keeping the training time of each batch within a certain range to ensure the smoothness of the training process), and the complexity of the data samples (such as samples containing complex backgrounds or a large number of keypoints will occupy more GPU memory), the batch data size standard is set to B_size data samples, meaning each batch contains keypoint data from B_size images. For example, if the GPU memory capacity allows and the sample complexity is moderate, B_size can be set to 16, 32, 64, etc.

[0101] Step S134: Divide the training data portion of each classification dataset into batches, select data continuously in the order of data sample collection time to form multiple training batches, each training batch contains only the training data of that classification dataset, and the data volume meets the batch data volume standard.

[0102] The training data for dataset A (classification dataset) consists of TA_train samples. B_size samples are selected sequentially from the beginning of the data collection time to form the first training batch, followed by another B_size samples to form the second training batch, and so on. If the number of remaining samples is less than B_size, a strategy can be used to decide whether to discard them or merge them with previous batches (e.g., randomly discarding data from the last insufficient batch or padding). Each training batch contains only the training data from dataset A (classification dataset).

[0103] Step S135: Perform the same batch division on the test data portion of each classification dataset to generate multiple test batches. Each test batch contains only the test data of that classification dataset, and the amount of data is consistent with that of the training batches.

[0104] Similar to the batch partitioning of the training data, the test data portion of dataset A (classification dataset) is divided into batches of B_size samples each, generating multiple test batches. Each test batch contains only the test data from dataset A (classification dataset). The test data portions of other classification datasets are partitioned in the same way.

[0105] Step S136: Design batch identification rules. Each batch identifier includes the dataset source identifier, batch type, and batch sequence number. The batch type is used to distinguish between training batches and test batches. Batch identifiers are generated through unique coding rules to achieve traceability of batch information.

[0106] The batch identification rule is designed as "Dataset Source Identifier - Batch Type - Batch Sequence Number". For example, the training batch 1 of dataset A is identified as "Dataset A-Train-001", where "Dataset A" is the dataset source identifier, "Train" is the batch type (indicating the training batch), and "001" is the batch sequence number; the test batch 3 of dataset B is identified as "Dataset B-Test-003", where "Test" is the batch type (indicating the test batch), and "003" is the batch sequence number. Through this unique encoding, the source, type, and order of each batch can be traced.

[0107] Step S137: Develop batch arrangement rules for the joint training set, and arrange the training batches of different classification datasets in an alternating order to avoid multiple consecutive batches coming from the same dataset.

[0108] The rule is to alternate the training batches in the order of dataset A, dataset B, dataset C, and dataset D. That is, the first batch is the training batch of dataset A, the second is the training batch of dataset B, the third is the training batch of dataset C, the fourth is the training batch of dataset D, the fifth returns to the training batch of dataset A, and so on. This avoids the occurrence of two or more consecutive training batches from the same dataset, so as to ensure that the model learns the features of different datasets.

[0109] Step S138: Count the number of training batches for all classification datasets and mark the total number of training batches for each classification dataset.

[0110] The statistics show that dataset A (classification dataset) has NA training batches, dataset B (classification dataset) has NB training batches, dataset C (classification dataset) has NC training batches, and dataset D (classification dataset) has ND training batches. The total number of training batches for each dataset is then labeled.

[0111] Step S139: Construct an initial framework for the batch permutation sequence, with the number of classification datasets as the period, and arrange a training batch of each classification dataset in each period.

[0112] The dataset is divided into four categorical datasets, with an initial frame of [dataset A batch 1, dataset B batch 1, dataset C batch 1, dataset D batch 1, dataset A batch 2, dataset B batch 2, dataset C batch 2, dataset D batch 2, ...]. Each period contains four batches, each from a different categorical dataset.

[0113] Step S1310: When the number of training batches for different classification datasets is inconsistent, calculate the difference between the number of training batches for each classification dataset and the minimum number of training batches, and evenly insert the excess training batches corresponding to the difference into the initial frame.

[0114] Assume the minimum number of training batches is N_min = min(NA, NB, NC, ND). The difference between the number of training batches in dataset A and N_min is ΔA = NA - N_min, the difference in dataset B is ΔB = NB - N_min, the difference in dataset C is ΔC = NC - N_min, and the difference in dataset D is ΔD = ND - N_min. The ΔA excess batches from dataset A are evenly inserted into the ΔA different periods of the initial frame, one batch per period. Similarly, the ΔB excess batches from dataset B are evenly inserted into the ΔB different periods, and so on, to maintain the uniformity of batch arrangement and avoid excessive batch concentration in any particular dataset.

[0115] Step S1311: Set batch insertion priority and process the classification datasets in ascending order of the number of training batches, first arranging the training batches of the classification datasets with fewer training batches.

[0116] The classification datasets are sorted in ascending order of the number of training batches, assuming the sorted result is dataset C (NC), dataset A (NA), dataset B (NB), and dataset D (ND). When inserting extra batches, dataset C with fewer training batches is considered first, followed by dataset A, then dataset B, and finally dataset D. This ensures that datasets with fewer training batches are sorted first, avoiding excessive batch dispersion.

[0117] Step S1312: Calculate the number of basic units covered by the cross-dataset joint semantic benchmark unique identifier sequence corresponding to each batch of data, and sort the batches in descending order of the number of basic units.

[0118] For each training batch, the number of basic units covered by the unique identifier sequence of the cross-dataset joint semantic benchmark corresponding to its data samples is calculated, that is, the total number of different basic units contained in the batch of data. For example, training batch of dataset A covers PA basic units, and training batch of dataset C covers PC basic units. Training batches of the same dataset are sorted from largest to smallest in terms of coverage, with priority given to batches covering more basic units to increase the diversity of model learning.

[0119] Step S1313: Formulate batch arrangement verification rules, traverse the batch sequence of the joint training set after arrangement, count the number of consecutive occurrences of the same training batch in the same dataset, and adjust the position of the relevant batch when the number of consecutive occurrences exceeds the preset threshold.

[0120] The maximum allowed number of consecutive occurrences of training batches from the same dataset is preset to T_max (usually set to 1 or 2). When traversing the batch sequence, if T_max+1 consecutive training batches from dataset A are found, it is determined that the validation rules are not met, and the positions of the relevant batches need to be adjusted by moving one of the batches after other dataset batches in the sequence.

[0121] Step S1314: Set up a batch arrangement adjustment mechanism. When the number of consecutive occurrences of the same training batch in the same dataset exceeds a preset threshold, move one batch in the consecutive batches after other dataset batches in the sequence.

[0122] When T_max+1 consecutive training batches of dataset B appear, the last batch in the consecutive batches is selected and moved to the position after the next non-dataset B batch. For example, if the original sequence is [dataset B batch a, dataset B batch b, dataset B batch c, dataset C batch d], and T_max=2, and three consecutive dataset B batches appear, exceeding the threshold, the adjusted sequence is [dataset B batch a, dataset B batch b, dataset C batch d, dataset B batch c].

[0123] Step S1315: Record the adjustment log during the batch arrangement process. The adjustment log includes the batch sequence before adjustment, the reason for adjustment, the batch sequence after adjustment, and the distribution statistics after adjustment.

[0124] Record the batch sequence segment before a certain adjustment as [batch x of dataset A, batch y of dataset A, batch z of dataset A, batch m of dataset B]. The reason for the adjustment is that dataset A appears 3 times consecutively, exceeding the threshold T_max=2. The sequence segment after the adjustment is [batch x of dataset A, batch y of dataset A, batch m of dataset B, batch z of dataset A]. Calculate the distribution statistics of each dataset batch in the sequence after the adjustment, such as the distribution ratio and the maximum number of consecutive occurrences.

[0125] Step S1316: Perform data distribution statistics on the arranged joint training set batch sequence, calculate the proportion of each dataset's training batch in the total batch, and output the final joint training set batch arrangement rule and the arranged batch sequence.

[0126] After statistical permutation, the total number of batches in the joint training set is NT = NA + NB + NC + ND, where the batch percentage for training data set A is RA = NA / NT, for dataset B it is RB = NB / NT, for dataset C it is RC = NC / NT, and for dataset D it is RD = ND / NT. Output the batch permutation rules (such as alternating order, insertion method for redundant batches, priority setting, etc.) and the permuted batch sequence [batch 1, batch 2, ..., batch NT].

[0127] Step S1317: Sort the test batches according to the same arrangement rules to form a joint test set. The batch arrangement order of the joint test set is consistent with the corresponding batches of the joint training set.

[0128] The test batches of each classification dataset are sorted using the same rules as the training batches, including alternating arrangement, insertion of redundant batches, and priority sorting, to form a joint test set. The batch order of the joint test set corresponds to that of the joint training set. For example, the first batch of the joint training set is the training batch of dataset A, and the first batch of the joint test set is the test batch of dataset A; the second batch of the joint training set is the training batch of dataset B, and the second batch of the joint test set is the test batch of dataset B, and so on, to ensure consistency between training and testing.

[0129] Step S1318: Establish a batch index table for the joint training set and the joint test set. The batch index table includes the batch identifier, dataset source, data volume, data range, and the corresponding cross-dataset joint semantic benchmark unique identifier sequence.

[0130] In the batch index table, each entry corresponds to a batch of the joint training set or joint test set, including the batch identifier (e.g., “dataset A-Train-001”), dataset source (“dataset A”), data size (B_size), data range (the starting and ending index values ​​of the batch data samples in the classification dataset), and the unique identifier sequence of the cross-dataset joint semantic benchmark corresponding to the batch data (e.g., the unique identifier sequence of dataset A [Y0, Y1, ..., YM-1]).

[0131] Step S1319: Link and store the joint training set, joint test set, and batch index table to generate training and test data archives.

[0132] All batch data of the joint training set, all batch data of the joint test set, and the batch index table are stored in a database or file system and linked by key fields such as batch identifiers to form a complete training and testing data archive, which facilitates batch-based data retrieval and management during model training.

[0133] Step S140: Input the joint training set into the preset neural network in batches, filter the neurons corresponding to the output layer of the preset neural network through the unique identifier sequence of each dataset, generate a directional training feedback signal based on the semantic difference between the prediction results of the filtered neurons and the real semantic labels, and update the weight parameters of the neurons corresponding to the backbone layer and output layer of the preset neural network through the directional training feedback signal.

[0134] The batch data from the joint training set are sequentially input into the preset neural network. The unique identifier sequence of the dataset is used to determine the neurons that need to be focused on in the output layer. The difference between the prediction results of these neurons and the true labels is compared. Based on this, a targeted training feedback signal is generated to update the weight parameters of the relevant neurons in the network, thereby achieving targeted optimization of the model and enabling the model to accurately predict the key points of each dataset.

[0135] Step S141: Construct a preset neural network, which includes a backbone feature extraction network, a feature fusion layer and an output layer. The backbone feature extraction network adopts the Transformer architecture and contains multiple encoder modules for extracting deep semantic features and spatial features of the data.

[0136] The pre-defined neural network structure is as follows: The input layer receives image data of a fixed size (e.g., H×W×3, where H is the height, W is the width, and 3 is the number of channels). The input layer connects to a backbone feature extraction network, which uses a Transformer architecture and contains E encoder modules. Each encoder module consists of a multi-head self-attention layer and a feedforward neural network layer. The multi-head self-attention layer captures long-distance dependencies between keypoints in the image, while the feedforward neural network layer performs non-linear transformations and feature mapping on the output of the attention layer, thereby extracting deep semantic features (such as semantic category information of keypoints) and spatial features (such as the location and shape information of keypoints) from the image. The backbone network output connects to a feature fusion layer, which integrates the features extracted by different encoder modules, fusing features at different levels using skip connections or attention mechanisms, and finally connects to the output layer.

[0137] Step S142: Add a dynamic feature aggregation unit after each encoder module of the backbone feature extraction network. The dynamic feature aggregation unit strengthens the representation of key point-related features through an attention mechanism and suppresses the interference of irrelevant background features.

[0138] A dynamic feature aggregation unit is added to the output of each encoder module. This dynamic feature aggregation unit receives the feature map output by the encoder module, calculates the attention weight of each pixel in the feature map, and performs weighted processing on the feature map to enhance the features of key point-related regions (such as human joints) and suppress the features of background regions (such as complex backgrounds and irrelevant objects). The processed feature map is then passed to the next encoder module or feature fusion layer.

[0139] Step S143: The dynamic feature aggregation unit receives the feature map output by the backbone feature extraction network encoder module. The feature map contains the spatial and semantic features of the data, and the number of channels in the feature map is consistent with the number of output channels of the encoder module.

[0140] The first encoder module outputs a feature map with C1 channels. The dynamic feature aggregation unit receives this feature map, and each channel of the feature map contains spatial features (such as edges, textures, and contours) and semantic features (such as parts and categories) of the image at different levels of abstraction. The height and width of the feature map are H1 and W1, respectively, consistent with the output size of the encoder module.

[0141] Step S144: Perform channel-dimensional feature splitting on the feature map, dividing the feature map into multiple feature sub-maps, each feature sub-map corresponding to a feature channel, for analyzing the feature importance of each channel individually.

[0142] The feature map of channel C1 is split into C1 single-channel feature sub-maps, each with a size of H1×W1. Each feature sub-map corresponds to one channel of the original feature map and represents a specific feature mapping, used to analyze the distribution and importance of the features of that channel in the image individually.

[0143] Step S145: The attention weight of each feature sub-image is calculated using a self-attention mechanism. The calculation of the attention weight is based on the semantic and spatial correlation of the pixels in the feature sub-image. Pixels with strong semantic correlation and key spatial location are assigned corresponding weights.

[0144] For each single-channel feature sub-image, an attention weight for each pixel is calculated using a self-attention mechanism. First, each pixel in the feature sub-image is treated as a vector, and the similarity between pixels (e.g., dot product similarity) is calculated to obtain an attention score matrix. Then, softmax normalization is applied to each row of attention scores to obtain an attention weight matrix. Pixels with strong semantic relevance (e.g., belonging to the same keypoint region) and key spatial location (e.g., the center region of a keypoint) receive higher attention weights, while background regions or semantically unrelated pixels receive lower weights.

[0145] Step S146: Multiply each feature sub-map with its corresponding attention weight to obtain a weighted feature sub-map, which strengthens the representation of key features and weakens the influence of irrelevant features.

[0146] Each pixel value in each feature sub-image is multiplied by the corresponding weight in the attention weight matrix to obtain a weighted feature sub-image. After weighting, the pixel values ​​in the key point-related region are enhanced (when the weight is greater than 1), while the pixel values ​​in the background region are weakened (when the weight is less than 1), thereby highlighting the key features.

[0147] Step S147: Perform channel fusion on all weighted feature sub-images. By splicing, multiple feature sub-images are integrated into a complete feature map. The number of channels in the fused feature map remains the same as before splitting.

[0148] The C1 weighted single-channel feature sub-images are concatenated in the original channel order and reassembled into a C1-channel feature map. This feature map retains the channel structure of the original feature map, but the features of each channel have undergone attention weighting, and the key features are enhanced.

[0149] Step S148: Add a feature enhancement module to perform nonlinear transformation on the fused feature map, and improve the expressive power of the feature map through activation function to highlight the difference between key point-related features and background features.

[0150] The feature enhancement module consists of convolutional layers and activation function layers. The fused feature map first undergoes channel adjustment and feature transformation through a 1×1 convolutional layer, and then undergoes non-linear processing through activation functions such as ReLU or GELU to further enhance the feature map's ability to express key point-related features, making the difference between key point features and background features more obvious and enhancing the discriminative power of the features.

[0151] Step S149: Design a feature normalization unit to perform instance normalization on the enhanced feature map to eliminate feature offset caused by differences in data distribution.

[0152] The feature normalization unit performs instance normalization on the enhanced feature map. For each sample in the feature map (i.e., the feature map corresponding to each image), for each channel, the mean μ and standard deviation σ of all pixels in that channel are calculated. Then, μ is subtracted from the pixel value of that channel and divided by σ to obtain the normalized feature value. Instance normalization eliminates feature bias caused by differences in data distribution between different samples, making the features more stable during training and accelerating model convergence.

[0153] Step S1410: Based on a preset feature threshold, filter out feature pixels with values ​​higher than the threshold in the enhanced feature map, and retain the feature information corresponding to the feature pixels.

[0154] Set a feature threshold τ (which can be determined by statistically analyzing the feature value distribution of the training data, such as taking the upper quartile of all feature values). For the enhanced and normalized feature map, select pixels with values ​​higher than τ; these pixels are considered important feature pixels related to keypoints. Retain the feature information of these pixels, and set the feature values ​​of pixels lower than τ to 0 or other small values ​​to further weaken the background features with low thresholds.

[0155] Step S1411: Perform residual connection between the filtered feature map and the original feature map output by the encoder module, retain the effective information in the original features, and incorporate the enhanced key features to generate the output feature map of the dynamic feature aggregation unit. The output feature map contains enhanced key point related features and suppressed background features.

[0156] The filtered feature map is element-wise added to the original feature map output by the encoder module, and then residual concatenated. The original feature map retains the basic feature information extracted by the encoder module, while the filtered feature map highlights key features. The addition of the two retains the effective information from the original features while incorporating the enhanced and filtered key features, ultimately generating the output feature map of the dynamic feature aggregation unit. This output feature map enhances key-point related features and suppresses background features, improving feature quality and discriminative ability.

[0157] Step S1412: Design the number of neurons in the output layer to be consistent with the number of basic units in the cross-dataset joint semantic benchmark. Each neuron corresponds one-to-one with a basic unit in the cross-dataset joint semantic benchmark. The output of the neuron is the predicted probability and coordinate prediction value of the corresponding key point.

[0158] The joint semantic benchmark across datasets has U basic units, therefore the output layer has U neurons, with neuron 0 corresponding to basic unit 0, neuron 1 corresponding to basic unit 1, and so on, with each neuron corresponding to one basic unit. Each neuron outputs two values: one is the predicted probability of the keypoint's existence (obtained by mapping the output to the [0, 1] interval using the sigmoid activation function), representing the model's confidence in predicting the keypoint's existence in the image; the other is the predicted coordinates of the keypoint (x_pred, y_pred), representing the model's predicted location coordinates of the keypoint in the image.

[0159] Step S1413: Obtain the joint training set, batch index table, and unique identifier sequences of each dataset in the cross-dataset joint semantic benchmark. Extract the identifier information of the first training batch from the batch index table to determine the original dataset to which the batch belongs.

[0160] Retrieve the joint training set, batch index table, and unique identifier sequence for each dataset from the training and testing data archive. Examine the first entry in the batch index table; its batch identifier is "Dataset A-Train-001," thus confirming that the original dataset to which this batch belongs is dataset A.

[0161] Step S1414: Based on the identification information of the original dataset, retrieve the unique identifier sequence corresponding to the dataset from the cross-dataset semantic association database. The unique identifier sequence contains the basic unit number of all key points of the dataset in the cross-dataset joint semantic benchmark.

[0162] Based on the identifier “Dataset A”, retrieve the unique identifier sequence SA=[Y0, ​​Y1, Y2, ..., YM-1] corresponding to Dataset A from the cross-dataset semantic association database. The elements in this sequence are the basic unit numbers of all key points of Dataset A in the cross-dataset joint semantic benchmark.

[0163] Step S1415: Input the dataset of the first training batch into a preset neural network. Extract the deep features of the data through the backbone feature extraction network. After feature integration by the feature fusion layer, the prediction results of all neurons are output by the output layer. The prediction results include prediction probabilities and coordinate prediction values.

[0164] Input B_size data samples of the "Dataset A - Train - 001" batch into the preset neural network. The data samples first enter the backbone feature extraction network and are processed by E encoder modules with dynamic feature aggregation units. Each encoder module gradually abstracts and enhances the features to extract deep semantic and spatial features. The deep features are passed to the feature fusion layer, and the feature fusion layer integrates the features output by different encoder modules through operations such as concatenate or weighted summation to obtain the integrated comprehensive features. The comprehensive features are input to the output layer, and U neurons respectively output the prediction probabilities and coordinate prediction values corresponding to the basic units, forming U pairs of prediction results (p_u, (x_u, y_u)), where u is the neuron number (0 ≤ u < U), p_u is the prediction probability, and (x_u, y_u) are the coordinate prediction values.

[0165] Step S1416: Based on the unique identifier sequence of the dataset, filter out the neurons in the output layer corresponding to the basic unit numbers in the sequence, retain the prediction results of these neurons, and mask the prediction results of other neurons.

[0166] The unique identifier sequence of Dataset A is SA = [Y0, Y1,..., YM - 1], and the elements in the sequence are basic unit numbers. Based on this sequence, filter out the neurons numbered Y0, Y1,..., YM - 1 from the U neurons in the output layer and retain the prediction results of these neurons (p_Y0, (x_Y0, y_Y0)), (p_Y1, (x_Y1, y_Y1)),..., (p_Y(M - 1), (x_Y(M - 1), y_Y(M - 1))). Mask the prediction results of other neurons in the output layer (i.e., neurons whose numbers are not in SA), which can be achieved by setting their prediction probabilities to 0 or ignoring the outputs of these neurons.

[0167] Step S1417: Extract the true semantic labels and true coordinate information of each key point in the data of the first training batch, compare the filtered neuron prediction results with the corresponding true information, and calculate the semantic matching error and coordinate prediction error.

[0168] Extract the true semantic labels and true coordinate information for each data sample in the "Dataset A-Train-001" batch. The true semantic labels are the presence identifiers of each keypoint in the sample (1 indicates presence, 0 indicates non-existence), corresponding to the base unit number in the unique identifier sequence SA; the true coordinate information is the true coordinates (x_true, y_true) of each keypoint in the normalized coordinate system. Compare the filtered neuron prediction results with the true information: for semantic matching errors, use the binary cross-entropy loss function to calculate the error between the predicted probability p_u and the true presence identifier t_u, i.e., L_sem = -(t_u). log(p_u)+(1-t_u) log(1-p_u)); For coordinate prediction error, the mean squared error loss function is used to calculate the error between the predicted coordinates (x_u, y_u) and the true coordinates (x_true, y_true), that is, L_coord=((x_u-x_true)^2+(y_u-y_true)^2).

[0169] Step S1418: Generate a directional training feedback signal based on the semantic matching error and coordinate prediction error. The directional training feedback signal contains information on the error magnitude, error location, and error type. The error location corresponds to the neuron number, and the feedback signal intensity corresponding to the unselected neuron is set to zero.

[0170] Based on the calculated semantic matching error L_sem and coordinate prediction error L_coord, a directional training feedback signal is generated. For output layer neurons Yu (u=0, 1, ..., M-1), the feedback signal includes the error magnitude (the specific values ​​of L_sem and L_coord), the error location (neuron number Yu), and the error type (semantic matching error or coordinate prediction error). For neurons that are not selected (neurons whose numbers are not in SA), the corresponding feedback signal strength is set to zero, meaning that the errors of these neurons do not participate in the subsequent weight update process.

[0171] Step S1419: The directional training feedback signal is fed back into the preset neural network, and the encoder module weights of the backbone feature extraction network, the attention weights of the dynamic feature aggregation unit, and the connection weights of the selected neurons in the output layer are updated through the backpropagation algorithm.

[0172] The targeted training feedback signal propagates backward from the output layer, sequentially passing through the feature fusion layer, the encoder module of the backbone feature extraction network, and the dynamic feature aggregation unit. During backpropagation, gradient descent algorithms (such as Adam and SGD) are used to calculate the gradients of the weight parameters of each layer based on the magnitude of the error in the feedback signal. For the encoder module of the backbone feature extraction network, the weight parameters of the multi-head self-attention layer and the feedforward neural network layer are updated; for the dynamic feature aggregation unit, the calculation parameters of the attention weights (such as the linear transformation matrix of the self-attention layer) are updated; for the output layer, only the connection weights between the selected neurons (Yu) and the previous layer are updated, while the connection weights of the unselected neurons remain unchanged. Through targeted updates, the model focuses on optimizing network parameters relevant to the key points of the current dataset.

[0173] Step S1420: Repeat the above steps of extracting training batches, retrieving unique identifier sequences, inputting network predictions, filtering neurons, calculating errors, generating feedback signals, and updating weights, and train all training batches in the joint training set in sequence to complete one round of collaborative iterative training.

[0174] Following the order of the batch index table, perform the same operations as the first batch on the second batch in the joint training set (e.g., "dataset B-Train-001"): extract the batch identifier from the batch index table to determine that the dataset belongs to dataset B; retrieve the unique identifier sequence SB of dataset B; input the batch data into the preset neural network for prediction; select output layer neurons based on SB; calculate the error between the prediction result and the true information; generate a targeted training feedback signal; and update the relevant weight parameters through backpropagation. After processing all training batches in the joint training set in sequence, one round of collaborative iterative training is completed.

[0175] Step S150: Repeat the batch training and weight update process, and judge the training convergence status by combining the semantic matching effect of the joint test set and the number of training iterations. When the convergence condition is met, output the optimized key point detection model.

[0176] The batch training and weight update process on the joint training set is repeated multiple times (i.e., multi-round collaborative iterative training). After each round of training, the semantic matching performance of the model is evaluated using the joint test set, and the convergence of the model is determined based on the number of training iterations. When the model meets the preset convergence condition, training stops, and the optimized keypoint detection model is output.

[0177] Step S151: After completing one round of collaborative iterative training on the joint training set, start the joint testing process and retrieve the joint test set and batch index table from the training and testing data archive.

[0178] After completing one round of collaborative iterative training, the joint testing process is automatically initiated. All batch data and batch index tables of the joint test set are read from the training and testing data archive to prepare for performance evaluation of the model in the current training state.

[0179] Step S152: Extract the dataset of each test batch in the order of batch arrangement of the joint test set, determine the original dataset to which the batch belongs, and retrieve the corresponding unique identifier sequence.

[0180] According to the batch arrangement order of the joint test set, the first test batch is "dataset A-Test-001", and its original dataset is determined to be dataset A. The unique identifier sequence SA corresponding to dataset A is retrieved from the cross-dataset semantic association database.

[0181] Step S153: Input the dataset of the test batch into the preset neural network in the current training state. After processing by the backbone feature extraction network and the feature fusion layer, the output layer outputs the prediction results of all neurons.

[0182] The dataset from batch "Dataset A-Test-001" is input into a pre-trained neural network. The data samples are processed by a backbone feature extraction network (containing dynamic feature aggregation units) to extract deep features. After the feature fusion layer integrates the features, the U neurons in the output layer output their respective prediction results (prediction probability and coordinate prediction value).

[0183] Step S154: Based on the unique identifier sequence, filter the neurons of the output layer to obtain the prediction results corresponding to the test batch, compare them with the real semantic labels and real coordinates in the test batch data, and calculate the semantic matching accuracy and coordinate prediction error of the batch.

[0184] Based on the unique identifier sequence SA of dataset A, the prediction results of output layer neurons Yu (u=0, 1, ..., M-1) are filtered. These prediction results are then compared with the true semantic labels (keypoint presence markers) and true coordinates in the test batch data. The semantic matching accuracy is calculated as follows: the number of keypoints with a predicted probability greater than a preset threshold (e.g., 0.5) and a true presence marker of 1 in all samples of the batch, plus the number of keypoints with a predicted probability less than or equal to the preset threshold and a true presence marker of 0, divided by the total number of keypoints in the batch (sample size × M). The coordinate prediction error is calculated as follows: for all correctly predicted keypoints, the Euclidean distance between the predicted coordinates and the true coordinates is calculated, and the average of all these distances is taken as the average coordinate prediction error for the batch.

[0185] Step S155: Calculate the semantic matching accuracy and coordinate prediction error of all test batches, and calculate the overall semantic matching accuracy and average coordinate prediction error of the joint test set.

[0186] Arithmetically average the semantic matching accuracies of all test batches in the joint test set to obtain the overall semantic matching accuracy Acc of the joint test set. Arithmetically average the average coordinate prediction errors of all test batches to obtain the average coordinate prediction error Err of the joint test set. Acc and Err are jointly used as indicators to evaluate the model performance.

[0187] Step S156: Determine whether the overall semantic matching accuracy reaches the preset performance threshold. If it reaches, it is determined that the model training meets the convergence condition; when it does not reach the performance threshold, determine whether the current training iteration number reaches the preset maximum iteration number. If it reaches, it is determined that the training meets the convergence condition.

[0188] The preset model performance threshold is Acc_threshold (such as 0.9 or 0.95), and the maximum iteration number is Max_epoch. If the overall semantic matching accuracy Acc ≥ Acc_threshold, it is determined that the model training meets the convergence condition and the performance reaches the expectation. If Acc < Acc_threshold, check whether the current training iteration number Epoch reaches Max_epoch. If Epoch ≥ Max_epoch, it is determined that the training meets the convergence condition regardless of whether the performance meets the standard, to avoid overtraining.

[0189] Step S157: When neither the performance threshold nor the maximum iteration number is reached, return to the collaborative iterative training step of the joint training set and continue the next round of training. Dynamically adjust the learning rate during each round of training.

[0190] If Acc < Acc_threshold and Epoch < Max_epoch, return to step S1420 and continue the next round of collaborative iterative training (Epoch = Epoch + 1). To improve the training efficiency and model performance, dynamically adjust the learning rate during each round of training and adaptively adjust the learning rate size according to the change of the model performance in the previous round.

[0191] For example, step S158: Before the start of each round of training, extract the current training iteration number, which increases sequentially from the start of the initial training.

[0192] Before the start of a new round of training (the Epoch-th round), extract the current training iteration number Epoch, which increases sequentially from the start of the initial training (Epoch = 1) and is incremented by 1 after each round of collaborative iterative training is completed.

[0193] Step S159: Retrieve the overall semantic matching accuracy and average coordinate prediction error of the joint test set in the previous round of training.

[0194] Retrieve the overall semantic matching accuracy Acc_prev=Acc(Epoch-1) and the average coordinate prediction error Err_prev=Err(Epoch-1) of the joint test set obtained after training in Epoch-1.

[0195] Step S1510: Calculate the difference between the overall semantic matching accuracy of the previous round and the previous round, and divide the difference by the accuracy of the previous round to obtain the accuracy change rate.

[0196] Calculate the difference in overall semantic matching accuracy between Epoch-1 and Epoch-2 as ΔAcc = Acc_prev - Acc(Epoch-2), where Acc(Epoch-2) is the overall semantic matching accuracy of Epoch-2. Divide the difference ΔAcc by the accuracy of the previous round (Epoch-2) to obtain the rate of change in accuracy, R_acc = ΔAcc / Acc(Epoch-2).

[0197] Step S1511: Calculate the difference between the average coordinate prediction error of the previous round and the previous round, and divide the difference by the error of the previous round to obtain the error change rate.

[0198] Calculate the difference in average coordinate prediction error between Epoch-1 and Epoch-2: ΔErr = Err_prev - Err(Epoch-2), where Err(Epoch-2) is the average coordinate prediction error of Epoch-2. Divide the difference ΔErr by the error of the previous Epoch (Epoch-2) to obtain the error change rate R_err = ΔErr / Err(Epoch-2).

[0199] Step S1512: Set the learning rate adjustment coefficient. When the rate of change of accuracy is greater than zero and the rate of change of error is less than zero, the learning rate adjustment coefficient is set to the first preset value; when the rate of change of accuracy is less than or equal to zero or the rate of change of error is greater than or equal to zero, the learning rate adjustment coefficient is set to the second preset value.

[0200] Set the learning rate adjustment coefficient γ. When the rate of change in accuracy R_acc > 0 and the rate of change in error R_err < 0, it indicates that the model performance has improved in the previous training round (accuracy increased and error decreased). In this case, the learning rate adjustment coefficient γ is set to the first preset value γ1 (γ1 > 1, such as 1.1, indicating a 10% increase in the learning rate). When the rate of change in accuracy R_acc ≤ 0 or the rate of change in error R_err ≥ 0, it indicates that the model performance improvement is not significant or has decreased. In this case, the learning rate adjustment coefficient γ is set to the second preset value γ2 (0 < γ2 < 1, such as 0.9, indicating a 10% decrease in the learning rate).

[0201] Step S1513: Multiply the learning rate adjustment coefficient by the initial learning rate of the current round to calculate the target learning rate of the current round.

[0202] The initial learning rate of the current round (the $i$-th epoch) is $LR_{prev}$ (the learning rate used in the $(i - 1)$-th epoch training). The target learning rate $LR_{curr}=LR_{prev} \cdot \gamma$. $\gamma$.

[0203] Step S1514: Set the upper and lower threshold values for learning rate adjustment. When the target learning rate is greater than the upper threshold value, set the target learning rate to the upper threshold value; when the target learning rate is less than the lower threshold value, set the target learning rate to the lower threshold value.

[0204] Set the upper threshold value $LR_{max}$ and the lower threshold value $LR_{min}$ of the learning rate ($LR_{min}<LR_{max}$). If the target learning rate $LR_{curr}>LR_{max}$, then adjust $LR_{curr}$ to $LR_{max}$; if $LR_{curr}<LR_{min}$, then adjust $LR_{curr}$ to $LR_{min}$; otherwise, keep $LR_{curr}$ unchanged. By setting the upper and lower limits, it is avoided that the learning rate is too large resulting in unstable training or too small resulting in slow convergence.

[0205] Step S1515: Update the initial learning rate of the current round to the target learning rate and apply it to the backpropagation weight update process of the current round.

[0206] Update the initial learning rate of the $i$-th epoch training to the adjusted target learning rate $LR_{curr}$. In the backpropagation process of this round of training, use $LR_{curr}$ to calculate the update amount of the weight parameter (gradient $\times LR_{curr}$) and apply it to the update of the network weights.

[0207] Step S1516: Record the learning rate adjustment situation of the current round, including the initial learning rate, adjustment coefficient, target learning rate and adjustment basis, and output the learning rate parameters of the current round for the collaborative iterative training of the current round.

[0208] Record the learning rate adjustment situation of the $i$-th epoch: the initial learning rate $LR_{prev}$, the adjustment coefficient $\gamma$, the target learning rate $LR_{curr}$ and the adjustment basis (the specific numerical ranges of $R_{acc}$ and $R_{err}$). Output the learning rate parameter $LR_{curr}$ as the learning rate of the backpropagation algorithm in the collaborative iterative training of this round.

[0209] Step S1517: When the training meets the convergence condition, stop the training process, save all weight parameters of the current neural network, including the weights of the backbone feature extraction network, dynamic feature aggregation unit, feature fusion layer and output layer, and integrate the weight parameters with the neural network structure file to generate the optimized key point detection model.

[0210] When training meets the convergence condition (Acc ≥ Acc_threshold or Epoch ≥ Max_epoch), the training process is stopped. All weight parameters of the current neural network are saved, including the weights of all encoder modules in the backbone feature extraction network, the attention weights of the dynamic feature aggregation unit, the weights of the feature fusion layer, and the neuron connection weights of the output layer. These weight parameters are then integrated with the neural network structure file (such as a configuration file containing information about the network layer types, number of neurons, and connection methods) to ensure that the weight parameters are correctly loaded into the corresponding network layers, generating the optimized keypoint detection model.

[0211] Step S1518: Encapsulate the optimized keypoint detection model and output the model file and model usage instructions. The model usage instructions include the input and output formats of the optimized keypoint detection model, the definition of the joint semantic benchmark across datasets, and the correspondence between keypoint identifiers.

[0212] The generated model weight parameters and network structure files are packaged into standard model file formats (such as ONNX, TensorFlow SavedModel, etc.) for easy deployment and use. A user manual is also written, including the input format of the optimized keypoint detection model (e.g., image size, number of channels, data type), output format (e.g., output order of predicted probabilities and coordinates, data type), a complete definition of the cross-dataset joint semantic benchmark (semantic description and point sequence number of each basic unit), and the correspondence between keypoint identifiers of each original dataset and the cross-dataset joint semantic benchmark (e.g., a mapping table between keypoint indices of dataset A and basic unit numbers of the joint benchmark). The final output is the packaged model file and the user manual.

[0213] During data collection, if privacy-sensitive data (such as human image data containing personally identifiable information) is involved, privacy protection technologies must be employed. For example, the collected image data can be anonymized to remove sensitive information such as facial features and identification marks; federated learning can be used for model training, where each dataset holder trains the model locally, sharing only model parameter updates and not the original data; and transmitted model parameters and intermediate data can be encrypted (using techniques such as homomorphic encryption and secure multi-party computation) to prevent data leakage during transmission. These technologies ensure the security of privacy-sensitive data and protect user privacy.

[0214] In one exemplary embodiment, a keypoint detection system based on a joint model trained from multiple source datasets is provided. This keypoint detection system can be a terminal, server, etc., and its internal structure diagram can be as follows: Figure 2 As shown, this keypoint detection system based on a multi-source dataset joint model training includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements a keypoint detection method based on a multi-source dataset joint model training. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the shell of a key point detection system based on a joint model training of multi-source datasets, or an external keyboard, touchpad, or mouse, etc.

[0215] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A keypoint detection method based on joint model training of multi-source datasets, characterized in that, The method includes: Initiate a cross-dataset semantic collaborative training process, construct a cross-dataset joint semantic benchmark based on the keypoint semantic description of multi-source datasets, establish a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark, and generate a unique identifier sequence of keypoints from each dataset in the cross-dataset joint semantic benchmark; A dynamic semantic aggregation algorithm is used to perform semantic alignment on the key points of each dataset, eliminating semantic bias and representation differences between different datasets, and generating a semantically unified multi-source fusion dataset. The semantically unified multi-source fusion dataset is split into batches according to the same batch division rule of the same dataset, forming a joint training set and a joint test set. Each batch contains only training data or test data of a single dataset. The joint training set is input into the preset neural network in batches. The neurons corresponding to the output layer of the preset neural network are selected by the unique identifier sequence of each dataset. Based on the semantic difference between the prediction results of the selected neurons and the real semantic labels, a targeted training feedback signal is generated. The weight parameters of the neurons corresponding to the backbone layer and output layer of the preset neural network are updated by the targeted training feedback signal. Repeat the batch training and weight update process, combine the semantic matching effect of the joint test set with the training iteration rounds to determine the training convergence status, and output the optimized key point detection model when the convergence condition is met.

2. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The process of initiating cross-dataset semantic collaborative training involves constructing a cross-dataset joint semantic benchmark based on the semantic descriptions of key points from multiple source datasets, establishing a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark, and generating a unique identifier sequence for key points from each dataset in the cross-dataset joint semantic benchmark, including: Start the cross-dataset semantic collaborative training process, trigger the multi-source dataset semantic extraction instruction, extract all keypoint semantic descriptions of each dataset, the keypoint semantic descriptions include the anatomical definition of the keypoints, functional attribute descriptions, spatial location association information and annotation specification descriptions, forming a set of keypoint semantic descriptions for each dataset; Semantic deduplication is performed on the keypoint semantic description set of all datasets. Duplicate or highly similar semantic descriptions are removed through semantic similarity analysis, while keypoint descriptions with unique semantic dimensions are retained. The deduplicated semantic descriptions of key points are hierarchically clustered according to semantic relevance. Key point semantic descriptions that are functionally related and spatially adjacent are grouped into the same semantic cluster, and each semantic cluster corresponds to a core semantic concept. Each semantic cluster is semantically integrated and standardized to determine the unified semantic definition, core attributes and association rules of key points within each semantic cluster, forming the basic unit of the cross-dataset joint semantic benchmark. Assign a unique semantic identifier and point number to each basic unit of the cross-dataset joint semantic benchmark, arrange the order of the basic units according to the association logic and spatial distribution rules of the semantic clusters, and construct a structural cross-dataset joint semantic benchmark. For each set of keypoint semantic descriptions in the original dataset, the semantic descriptions of the basic units in the cross-dataset joint semantic benchmark are compared one by one to identify the semantic correspondence between the keypoints of each dataset and the basic units of the cross-dataset joint semantic benchmark. Based on semantic correspondence, a bidirectional semantic mapping relationship is established between each dataset and the cross-dataset joint semantic benchmark. The forward mapping records the association path from the key points of the dataset to the basic unit of the cross-dataset joint semantic benchmark, while the reverse mapping records the semantic representation of the basic unit of the cross-dataset joint semantic benchmark in each dataset. Based on the bidirectional semantic mapping relationship, extract the cross-dataset joint semantic benchmark point sequence number corresponding to all key points of each dataset, organize the number sequence according to the key point arrangement order of the dataset itself, and generate a unique identifier sequence of key points of each dataset in the cross-dataset joint semantic benchmark. The cross-dataset joint semantic benchmark, bidirectional semantic mapping relationship and unique identifier sequence are associated and stored to construct a cross-dataset semantic association database. The cross-dataset semantic association database contains the semantic mapping details of each dataset, the correspondence of key point identifiers and the complete definition of the cross-dataset joint semantic benchmark.

3. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The process employs a dynamic semantic aggregation algorithm to perform semantic alignment processing on key points of each dataset, eliminating semantic biases and representational differences between different datasets, and generating a semantically unified multi-source fusion dataset, including: Retrieve the cross-dataset joint semantic benchmark, bidirectional semantic mapping relationship, and key point raw data of each dataset from the cross-dataset semantic association database. The key point raw data includes key point coordinate information, semantic labels, and dataset source identifiers. For each basic unit in the cross-dataset joint semantic benchmark, all original datasets containing the semantic description of the basic unit are selected from the bidirectional semantic mapping relationship, and the completeness of the semantic description and the consistency of the annotation of each dataset on the basic unit are statistically analyzed. Based on the statistical results, the dataset with the most complete semantic description and the highest consistency of annotation was selected as the semantic benchmark dataset for this basic unit. The standard semantic labels, coordinate distribution patterns and associated attributes of the corresponding key points in the dataset were extracted as benchmark references. For each non-benchmark dataset, the key points corresponding to each basic unit of the cross-dataset joint semantic benchmark are located according to the bidirectional semantic mapping relationship. The original semantic labels and coordinate information of the key points are extracted to form a set of data to be aligned for the non-benchmark dataset. A dynamic semantic awareness network is used to perform semantic depth analysis on the dataset to be aligned from a non-benchmark dataset and the benchmark reference, to mine differences in semantic representation, coordinate distribution deviation and related attribute differences, and generate semantic difference feature data. The keypoint coordinates of the non-benchmark dataset and the semantic benchmark dataset are uniformly projected onto the preset target center standard coordinate system. The coordinate system transformation is used to eliminate the coordinate offset caused by differences in viewpoint, scale and shooting angle, and generate standard coordinate data. Based on semantic difference feature data and standardized coordinate data, a semantic difference dataset is constructed between the non-benchmark dataset and the semantic benchmark dataset. The semantic difference dataset includes data to be aligned, benchmark data, and difference annotation information between the data to be aligned and the benchmark data. A deep convolutional neural network was selected as the basic architecture of the semantic difference compensation model. The semantic difference dataset was input into the semantic difference compensation model for training, so that the semantic difference compensation model could learn the semantic mapping rules and coordinate compensation rules between the non-benchmark dataset and the semantic benchmark dataset. Input the dataset to be aligned from the non-benchmark dataset into the trained semantic difference compensation model to obtain semantic correction parameters and coordinate compensation values. Adjust the semantic labels and coordinate information of the non-benchmark dataset based on the semantic correction parameters and coordinate compensation values ​​so that the adjusted semantics and coordinates meet the requirements of the cross-dataset joint semantic benchmark. All adjusted non-benchmark datasets and semantic benchmark datasets are integrated to form a multi-source fusion dataset, in which each data sample contains semantic labels, canonical coordinates, and dataset source identifiers that conform to the cross-dataset joint semantic benchmark.

4. The keypoint detection method based on multi-source dataset joint model training according to claim 3, characterized in that, The method employs a dynamic semantic awareness network to perform deep semantic analysis on the dataset to be aligned from a non-benchmark dataset and the benchmark reference, mining differences in semantic representation, coordinate distribution deviations, and differences in associated attributes to generate semantic difference feature data, including: The semantic tags in the dataset to be aligned from the non-benchmark dataset are segmented into semantic core words, modifier words and related words to form a semantic vocabulary set. Perform the same word segmentation process on the standard semantic tags in the benchmark reference to obtain the benchmark semantic vocabulary set, and establish a one-to-one correspondence between the semantic vocabulary to be aligned and the benchmark semantic vocabulary. The semantic vocabulary set to be aligned and the benchmark semantic vocabulary set are converted into semantic vectors through word embedding algorithm. The semantic vector similarity of each corresponding word pair is calculated. Word pairs with similarity below a preset threshold are marked as semantic difference points. Extract the key point coordinate data from the dataset to be aligned and the benchmark reference, calculate the coordinate difference of key points corresponding to the same semantic in the X and Y axis directions, statistically analyze the distribution range and central tendency of the coordinate difference, and identify the coordinate distribution deviation characteristics. Analyze the association attributes of key points in the dataset to be aligned, including the distance, angle, and relative position relationship between key points, and compare them with the association attributes in the benchmark reference to explore the differences in association attributes; Normalize the values ​​of differences in semantic representation, coordinate distribution deviation, and related attribute differences to make the values ​​of different difference types comparable. Construct a semantic difference feature matrix. The row dimension of the matrix is ​​the number of key points in the dataset to be aligned, and the column dimension is the difference type. The difference types include semantic expression difference, coordinate distribution deviation, and association attribute difference. The matrix elements are the normalized difference values ​​of the corresponding key points under each difference type. A graph neural network is used to construct semantic difference feature data. Each key point is used as a graph node, the node attribute is the row vector of the semantic difference feature matrix of that key point, and the edge weight between nodes is the difference correlation strength between the corresponding key points. By using graph convolution operations to extract features from semantic difference data, the representation ability of key difference features is strengthened, while the influence of secondary differences is weakened. The extracted differential features are subjected to dimensionality compression and feature fusion to generate a comprehensive differential feature vector for each key point; The comprehensive difference feature vectors of all key points are arranged in point order according to the joint semantic benchmark across the dataset to form semantic difference feature data. The semantic difference feature data includes node features, edge weights, and difference type annotation information.

5. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The process of splitting the semantically unified multi-source fusion dataset into batches according to the same batch partitioning rule to form a joint training set and a joint test set includes: Obtain semantically unified multi-source fusion datasets, extract dataset source identifiers from the datasets, classify the semantically unified multi-source fusion datasets according to the source identifiers, and obtain the classification datasets corresponding to each original dataset; For each classification dataset, the data volume is statistically analyzed, and the training data volume and test data volume for each classification dataset are determined by combining the preset training data ratio and test data ratio. Set a batch data size standard, which is determined based on the pre-defined neural network's memory capacity, training efficiency requirements, and data sample complexity, so that the data size of each batch is consistent. The training data portion of each classification dataset is divided into batches. Data is selected continuously in the order of data sample collection time to form multiple training batches. Each training batch contains only the training data of that classification dataset, and the data volume meets the batch data volume standard. The test data portion of each classification dataset is divided into the same batches to generate multiple test batches. Each test batch contains only the test data of that classification dataset, and the amount of data is consistent with that of the training batches. Design batch identification rules. Each batch identifier includes the dataset source identifier, batch type, and batch sequence number. The batch type is used to distinguish between training batches and test batches. Batch identifiers are generated through unique coding rules to achieve traceability of batch information. Establish batch arrangement rules for the joint training set, and arrange the training batches of different classification datasets in an alternating order to avoid multiple consecutive batches coming from the same dataset; The test batches are sorted according to the same arrangement rules to form a joint test set. The batch order of the joint test set is consistent with the corresponding batch order of the joint training set. A batch index table is established for the joint training set and the joint test set. The batch index table includes the batch identifier, dataset source, data volume, data range, and the corresponding unique identifier sequence of the cross-dataset joint semantic benchmark. The joint training set, joint test set, and batch index table are stored together to generate training and test data archives.

6. The keypoint detection method based on multi-source dataset joint model training according to claim 5, characterized in that, The step of establishing batch arrangement rules for the joint training set, which arranges training batches from different classification datasets in an alternating order to avoid multiple consecutive batches coming from the same dataset, includes: Count the number of training batches for all classification datasets and label the total number of training batches for each classification dataset; An initial framework for constructing the batch permutation sequence is established, with the number of classification datasets as the period, and a training batch of each classification dataset is arranged sequentially in each period. When the number of training batches for different classification datasets is inconsistent, calculate the difference between the number of training batches for each classification dataset and the minimum number of training batches, and evenly insert the excess training batches corresponding to the difference into the initial frame. Set batch insertion priority and process the classification datasets in ascending order of the number of training batches, first arranging the training batches of the classification datasets with fewer training batches. Calculate the number of basic units covered by the unique identifier sequence of the cross-dataset joint semantic benchmark corresponding to each batch of data, and sort the batches in descending order of the number of basic units; Establish batch arrangement verification rules, traverse the batch sequence of the joint training set after arrangement, count the number of consecutive occurrences of the same training batch in the same dataset, and adjust the position of the relevant batch when the number of consecutive occurrences exceeds the preset threshold. Set up a batch arrangement adjustment mechanism. When the number of consecutive training batches of the same dataset exceeds a preset threshold, move one batch from the consecutive batches to the end of other dataset batches in the sequence. Record the adjustment log during the batch sorting process. The adjustment log includes the batch sequence before adjustment, the reason for adjustment, the batch sequence after adjustment, and the distribution statistics after adjustment. Perform data distribution statistics on the sorted batch sequence of the joint training set, calculate the proportion of each dataset's training batch in the total batch, and output the final batch sorting rules and the sorted batch sequence.

7. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The process involves batch-inputting the joint training set into a pre-defined neural network, filtering neurons corresponding to the output layer of the pre-defined neural network using unique identifier sequences from each dataset, generating targeted training feedback signals based on the semantic differences between the prediction results of the filtered neurons and the true semantic labels, and updating the weight parameters of the corresponding neurons in the backbone and output layers of the pre-defined neural network using these targeted training feedback signals. A pre-defined neural network is constructed, which includes a backbone feature extraction network, a feature fusion layer, and an output layer. The backbone feature extraction network adopts the Transformer architecture and contains multiple encoder modules for extracting deep semantic and spatial features of the data. A dynamic feature aggregation unit is added after each encoder module of the backbone feature extraction network. This dynamic feature aggregation unit strengthens the representation of key point-related features through an attention mechanism and suppresses the interference of irrelevant background features. The number of neurons in the output layer is consistent with the number of basic units in the cross-dataset joint semantic benchmark. Each neuron corresponds one-to-one with a basic unit in the cross-dataset joint semantic benchmark. The output of the neuron is the predicted probability and coordinate prediction value of the corresponding key point. Obtain the joint training set, batch index table, and unique identifier sequences of each dataset in the cross-dataset joint semantic benchmark. Extract the identifier information of the first training batch from the batch index table to determine the original dataset to which the batch belongs. Based on the identification information of the original dataset, the unique identifier sequence corresponding to the dataset is retrieved from the cross-dataset semantic association database. This unique identifier sequence contains the basic unit number of all key points of the dataset in the cross-dataset joint semantic benchmark. The dataset of the first training batch is input into a preset neural network. The deep features of the data are extracted by the backbone feature extraction network. After feature integration by the feature fusion layer, the prediction results of all neurons are output by the output layer. The prediction results include prediction probabilities and coordinate prediction values. Based on the unique identifier sequence of the dataset, the neurons in the output layer that correspond to the basic unit number in the sequence are selected, the prediction results of the neurons are retained, and the prediction results of other neurons are masked. Extract the true semantic labels and true coordinate information of each key point in the first batch of training data, compare the filtered neuron prediction results with the corresponding true information, and calculate the semantic matching error and coordinate prediction error. Based on semantic matching error and coordinate prediction error, a directional training feedback signal is generated. This directional training feedback signal contains information on error magnitude, error location and error type. The error location corresponds to the neuron number, and the feedback signal intensity corresponding to the unselected neuron is set to zero. The directional training feedback signal is fed back into the preset neural network, and the encoder module weights of the backbone feature extraction network, the attention weights of the dynamic feature aggregation unit, and the connection weights of the selected neurons in the output layer are updated through the backpropagation algorithm. Repeat the above steps of extracting training batches, retrieving unique identifier sequences, inputting network predictions, filtering neurons, calculating errors, generating feedback signals, and updating weights, and train all training batches in the joint training set in sequence to complete one round of collaborative iterative training. The addition of a dynamic feature aggregation unit after each encoder module of the backbone feature extraction network, which strengthens the representation of keypoint-related features through an attention mechanism and suppresses interference from irrelevant background features, includes: The dynamic feature aggregation unit receives the feature map output by the backbone feature extraction network encoder module. This feature map contains the spatial and semantic features of the data, and the number of channels in the feature map is the same as the number of output channels of the encoder module. The feature map is split into multiple feature sub-maps along the channel dimension. Each feature sub-map corresponds to a feature channel, which is used to analyze the feature importance of each channel separately. A self-attention mechanism is used to calculate the attention weight of each feature sub-image. The calculation of the attention weight is based on the semantic and spatial correlation of pixels in the feature sub-image. Pixels with strong semantic correlation and key spatial location are assigned corresponding weights. Each feature subgraph is multiplied by its corresponding attention weight to obtain a weighted feature subgraph, which strengthens the representation of key features and weakens the influence of irrelevant features. Channel fusion is performed on all weighted feature sub-maps. Multiple feature sub-maps are integrated into a complete feature map through a splicing operation. The number of channels in the fused feature map remains the same as that before splitting. A feature enhancement module is added to perform nonlinear transformation on the fused feature map, and the expressive power of the feature map is improved by activation function to highlight the difference between key point-related features and background features; Design a feature normalization unit to perform instance normalization on the enhanced feature map and eliminate feature offset caused by differences in data distribution. Based on a preset feature threshold, feature pixels with values ​​higher than the threshold in the enhanced feature map are selected, and the feature information corresponding to the feature pixels is retained. The filtered feature map is residually connected to the original feature map output by the encoder module, retaining the effective information in the original features and incorporating the enhanced key features to generate the output feature map of the dynamic feature aggregation unit. This output feature map contains enhanced key-point related features and suppressed background features.

8. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The repeated batch training and weight update process, combined with the semantic matching effect on the joint test set and the number of training iterations, determines the training convergence status. When the convergence condition is met, the optimized keypoint detection model is output, including: After completing one round of collaborative iterative training on the joint training set, the joint testing process is initiated, retrieving the joint test set and batch index table from the training and testing data archive. According to the batch arrangement order of the joint test set, the dataset of each test batch is extracted in turn, the original dataset to which the batch belongs is determined, and the corresponding unique identifier sequence is retrieved. The dataset of the test batch is input into the preset neural network in the current training state. After being processed by the backbone feature extraction network and the feature fusion layer, the prediction results of all neurons are output by the output layer. The neurons in the output layer are selected based on the unique identifier sequence to obtain the prediction results corresponding to the test batch. These results are then compared with the real semantic labels and real coordinates in the test batch data to calculate the semantic matching accuracy and coordinate prediction error of the batch. The semantic matching accuracy and coordinate prediction error of all test batches are statistically analyzed, and the overall semantic matching accuracy and average coordinate prediction error of the joint test set are calculated. Determine whether the overall semantic matching accuracy has reached the preset performance threshold. If it has, the model training is deemed to have met the convergence condition. If it has not reached the performance threshold, determine whether the current training iteration round has reached the preset maximum iteration round. If it has, the training is deemed to have met the convergence condition. If neither the performance threshold nor the maximum number of iterations is reached, return to the collaborative iterative training steps of the joint training set and continue to the next round of training, dynamically adjusting the learning rate during each round of training. When the training meets the convergence condition, the training process is stopped, and all weight parameters of the current neural network are saved, including the weights of the backbone feature extraction network, dynamic feature aggregation unit, feature fusion layer and output layer. The weight parameters are then associated and integrated with the neural network structure file to generate an optimized key point detection model. The optimized keypoint detection model is encapsulated, and the output model file and model usage instructions are provided. The model usage instructions include the input and output formats of the optimized keypoint detection model, the definition of the joint semantic benchmark across datasets, and the correspondence between keypoint identifiers.

9. The keypoint detection method based on multi-source dataset joint model training according to claim 1, characterized in that, The process of establishing a bidirectional semantic mapping relationship between each dataset and the cross-dataset joint semantic benchmark, and generating a unique identifier sequence for key points of each dataset in the cross-dataset joint semantic benchmark, includes: Semantic parsing is performed on the set of keypoint semantic descriptions in each original dataset to extract the core semantic elements of each keypoint. The core semantic elements include the functional definition, anatomical location, relationship with other keypoints, and annotation specifications of the keypoint. The core semantic elements of each key point are compared with the standardized semantic definitions of each basic unit in the cross-dataset joint semantic benchmark. A semantic similarity calculation algorithm is used to calculate the similarity score, and the basic unit with the highest similarity score is selected as the matching result. A positive semantic mapping table is established based on the matching results. The positive semantic mapping table records the cross-dataset joint semantic benchmark basic unit number and semantic similarity score corresponding to the key points of each original dataset. Construct a reverse semantic mapping table, which records the key point information of all original datasets corresponding to each basic unit in the cross-dataset joint semantic benchmark, including dataset identifier and key point name; Examine the positive semantic mapping table to confirm that each key point in the original dataset has an associated cross-dataset joint semantic benchmark base unit number; Based on the positive semantic mapping table, according to the order of the key points in the original dataset, the basic unit number of the cross-dataset joint semantic benchmark corresponding to each key point is extracted to form a unique identifier sequence of the key points of the original dataset in the cross-dataset joint semantic benchmark. Add sequence identification information to the uniquely identified sequence. The sequence identification information includes the dataset name, the sequence generation time, and the cross-dataset joint semantic benchmark version number. The forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence of each dataset are associated and stored to form a bidirectional semantic mapping database. The bidirectional semantic mapping database is updated and maintained. When a new dataset is added, semantic parsing and matching steps are performed to update the forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence. When the semantic benchmark is updated across datasets, the semantic parsing and matching steps are re-executed to update the forward semantic mapping table, the reverse semantic mapping table, and the unique identifier sequence, and the bidirectional semantic mapping database and the unique identifier sequence of each dataset are output.

10. A keypoint detection system based on a joint model training of multi-source datasets, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the keypoint detection method based on multi-source dataset joint model training as described in any one of claims 1 to 9 by executing the machine-executable instructions.

Images