Multi-camera motion fusion determination method and system

Abstract

The application discloses a kind of multi-camera action fusion determination method and system, belong to action fusion determination technical field;The method specifically includes: obtaining the continuous human action video of multi-camera shooting, determining action state transition position, and dividing action attempt, obtaining single visual angle action atomic sequence, action attempt demarcation node set and action atomic observation confidence;Cross-camera action atomic part optimal transmission and retreat residual feedback correction are carried out, and retreat feedback correction transmission relationship is obtained;Action attempt attribution graph is constructed, and propagation neighborhood reconstruction and cross-view complementary attention fusion are carried out, to obtain fusion action atomic sequence and action atomic receiving relationship;Action prototype overall matching and semantic chain check are carried out, to obtain action determination result;The application improves the accuracy of action determination, enhances the stability of action determination.

Description

Technical Field

[0001] This invention belongs to the field of motion fusion determination, and specifically relates to a multi-camera motion fusion determination method and system. Background Technology

[0002] In the process of multi-camera motion fusion judgment, it is necessary to extract motion segments from continuous human motion videos captured by different cameras, establish cross-view motion correspondence, and give motion judgment results accordingly. However, when people perform actions, there may be interruptions, withdrawals, and re-executions. Different cameras are also affected by shooting angles, human body occlusion, and differences in acquisition time, which can easily lead to unclear and incomplete local motion segments. This makes it difficult to accurately distinguish the boundaries between different action attempts and to establish stable cross-view motion relationships.

[0003] In existing technologies, multi-camera motion fusion methods typically associate and fuse motion segments captured by different cameras based on temporal proximity, similarity of motion features, or single-view recognition confidence. However, when a person interrupts, withdraws, or resumes an action, short motion segments indicating pauses, withdrawals, and restarts are easily misidentified due to occlusion or unclear observation. This leads to incorrect association between previously incomplete actions and subsequently resumed actions, resulting in inconsistencies between the determinations of action completion status, number of actions, and action category and the actual action process. Consequently, the accuracy and stability of motion determination are poor. Summary of the Invention: The purpose of this invention is to provide a multi-camera action fusion determination method and system. By correcting cross-view transmission relationships through residual feedback retraction, and combining propagation neighborhood reconstruction and cross-view complementary attention fusion, it avoids the erroneous fusion of local action segments in different action attempts, thereby improving the accuracy and stability of action determination.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, the present invention provides a multi-camera motion fusion determination method, including: Acquire continuous human motion videos captured by multiple cameras, determine the transition points of motion states, and classify motion attempts to obtain single-view motion atom sequences, motion attempt boundary node sets, and motion atom observation confidence scores. Based on the single-view action atom sequence and action attempt boundary node set, optimal transmission of cross-camera action atom parts and correction of retraction residual feedback are performed to obtain the retraction feedback correction transmission relationship. Based on the action atom observation confidence and the pullback feedback to correct the transmission relationship, an action attempt attribution graph is constructed, and the propagation neighborhood is reconstructed and cross-perspective complementary attention is fused to obtain the fused action atom sequence and action atom inheritance relationship; Based on the fusion of action atomic sequences and action atomic succession relationships, overall matching of action prototypes and semantic chain verification are performed to obtain action determination results.

[0005] Specifically, the process involves acquiring continuous human motion videos captured by multiple cameras, determining the transition points of the motion states, and segmenting motion attempts to obtain a single-view motion atom sequence, a set of motion attempt boundary nodes, and the confidence level of motion atom observations, including: Acquire continuous human motion videos captured by multiple cameras, perform human keypoint detection, and arrange the detection results to obtain a temporal record of human keypoints and a keypoint detection confidence record. Calculate posture changes from the time-series recordings of key human body points to determine the transition points of movement states; Based on the transition position of the action state, set the action attempt boundary to obtain the action attempt boundary node set; Action attempts are divided according to the action attempt boundary node set, and attitude change segmentation and observation reliability calculation are performed in combination with the key point detection confidence record to obtain single-view action atom sequence and action atom observation confidence.

[0006] Specifically, based on the single-view action atom sequence and action attempt boundary node set, optimal transmission of cross-camera action atom parts and correction of retraction residual feedback are performed to obtain the retraction feedback correction transmission relationship, including: The single-view action atom sequence and action attempt boundary node set are associated, and the action process features of the action atoms in the corresponding action attempts of each camera are collected to obtain the action attempt feature record. The action process features include the order of participating limbs, the sequence of movement directions, and the local order. Based on the action attempt feature record, cross-camera candidate action attempts are filtered, the matching cost of each action atom in the cross-camera candidate action attempts is determined, and partial optimal transmission is performed to obtain the initial cross-view transmission relationship and untransmitted quality. Based on the untransmitted quality, residual attribute discrimination and atomic chain connection of the retraction action are performed to obtain the retraction residual chain; Based on the pullback residual chain and the initial cross-view transmission relationship, cross-collision transmission quality backoff and redistribution are performed to obtain the pullback feedback correction transmission relationship.

[0007] Specifically, based on the untransmitted quality, residual attribute discrimination and atomic chain connection of the rollback action are performed to obtain the rollback residual chain, including: Residual discrimination features are extracted from the action atoms corresponding to the untransmitted quality to obtain residual discrimination feature records. The residual discrimination features include the motion direction after direction reversal and the key point observation quality features. The residual score record is obtained by calculating the retraction residual score and occlusion residual score of each action atom using the residual discrimination feature record. Compare the retraction residual score and occlusion residual score of each action atom in the residual score record to determine the retraction residual mass, and extract the direction of motion after the direction of the corresponding action atom is reversed to obtain the retraction direction record; Based on the retraction residual mass and retraction direction records, action atoms with the same participating limbs, continuous local sequence, and consistent retraction direction are connected within the same action attempt corresponding to the same camera to obtain the retraction residual chain.

[0008] Specifically, based on the pullback residual chain and the initial cross-view transmission relationship, cross-collision transmission quality backoff and reallocation are performed to obtain the pullback feedback correction transmission relationship, including: Tracing the preceding action atoms corresponding to the retracted residual chain along the initial cross-view transmission relationship, and determining the preceding association features between the preceding action atoms and the retracted residual chain, the preceding influence feature record is obtained; The reverse influence ratio of the preceding action atom is determined by combining the preceding influence feature record, and the preceding transmission influence record is obtained; Based on the preceding transmission impact record, cross-period conflict determination and transmission quality backtracking are performed on the initial cross-view transmission relationship to obtain the backtracked transmission relationship and backtracked transmission quality. The rollback transmission quality is limited to the corresponding cross-camera candidate action attempts. Compatible action atoms are selected according to the transmission relationship after rollback, and partial optimal transmission is performed to obtain the rollback feedback correction transmission relationship.

[0009] Specifically, based on the action atom observation confidence and the pullback feedback to correct the transmission relationship, an action attempt attribution graph is constructed, and propagation neighborhood reconstruction and cross-perspective complementary attention fusion are performed to obtain the fused action atom sequence and action atom inheritance relationship, including: Using action atoms in a single-view action atom sequence as nodes, and based on the arrangement order of action atoms in the corresponding sequence, the action attempt boundary node set, and the pullback feedback correction transmission relationship, action continuation edges, action reset edges, and cross-view complementary edges are established respectively to construct an action attempt attribution graph. The initialization action attempts to assign the continuation state and reset state of each edge in the graph, and performs propagation neighborhood reconstruction and graph attention propagation to obtain the updated propagation neighborhoods of each layer; Based on the updated propagation neighborhood and action atom observation confidence of each layer, dual-channel attention calculation and cross-view complementary weight correction are performed to obtain the graph attention propagation results. Based on the graph attention propagation results, the action atom nodes and edges in the action attempt at belonging graph are merged and updated to obtain the merged action atom sequence and action atom inheritance relationship.

[0010] Specifically, the initialization action attempts to assign the continuation and reset states of each edge in the graph, and performs neighborhood reconstruction and graph attention propagation to obtain updated propagation neighborhoods at each layer, including: Determine the initial continuation state and initial reset state of the action continuation edge, action reset edge, and cross-view complementary edge in the action attempt at the graph, and obtain the initial edge state record; Based on the initial edge state record, the edges that are dominant in the continuation state are determined and included in the action continuation propagation neighborhood, and the edges that are dominant in the reset state are determined and included in the action reset propagation neighborhood, thus obtaining the first-layer propagation neighborhood. Graph attention propagation is performed on the first-layer propagation neighborhood to generate first-layer action reset information, and the continuation state and reset state of each edge are updated to obtain the second-layer edge state record. Starting with the edge state records of the second layer, the current layer's propagation neighborhood is reconstructed layer by layer using the current layer's edge state records. Graph attention propagation is then performed, and the next layer's edge state is updated based on the action reset information generated by the propagation, resulting in the updated propagation neighborhoods of each layer.

[0011] Specifically, based on the updated propagation neighborhood and action atom observation confidence of each layer, dual-channel attention calculation and cross-view complementary weight correction are performed to obtain the graph attention propagation results, including: Graph attention is calculated for the updated action continuation propagation neighborhood and action reset propagation neighborhood of each layer to obtain the continuation channel edge attention record and the reset channel edge attention record. Extract the participating limbs and posture changes of the action atoms at both ends of the cross-view complementary edge, and compare the cross-view differences of the corresponding features to obtain the degree of cross-view action complementarity. Determine the observation confidence of the action atoms at both ends of the cross-view complementary edge, and calculate the attention compensation weight by combining the degree of cross-view action complementation; Extract the initial attention weights corresponding to the cross-view complementary edges from the continuous channel edge attention records and the reset channel edge attention records; The initial attention weights corresponding to the cross-view complementary edges in the corresponding channels are corrected by adjusting the attention compensation weights, and the attention records of the continuation channel edges and the attention records of the reset channel edges are competitively aggregated to obtain the graph attention propagation results.

[0012] Specifically, based on the fusion of action atomic sequences and action atomic succession relationships, overall action prototype matching and semantic chain verification are performed to obtain action determination results, including: Obtain the action recognition prototype sequence library, extract the action prototype sequence corresponding to each action category, and determine the node order of the action prototype sequence; Calculate the matching cost between the fused action atomic sequence and the action prototype sequence corresponding to each action category to obtain candidate action matching records; Based on the node order and action atom succession relationship of the action prototype sequence, the candidate action matching records are matched as a whole and the semantic chain is verified to obtain the target action matching sequence. Based on the target action matching sequence, the action prototype matching results are statistically analyzed to obtain the action judgment result.

[0013] On the other hand, the present invention provides a multi-camera motion fusion determination system, comprising: The video acquisition module is used to acquire continuous human motion videos captured by multiple cameras, determine the transition points of the motion state, divide the motion attempts, and obtain the single-view motion atom sequence, the motion attempt boundary node set, and the motion atom observation confidence. The optimal transmission module, based on the single-view action atom sequence and action attempt boundary node set, performs optimal transmission of cross-camera action atom parts and correction of retraction residual feedback, and obtains the retraction feedback correction transmission relationship. The action fusion module constructs an action attempt attribution graph based on the action atom observation confidence and the pullback feedback to correct the transmission relationship. It then performs propagation neighborhood reconstruction and cross-perspective complementary attention fusion to obtain the fused action atom sequence and action atom inheritance relationship. The action determination module performs overall matching of action prototypes and semantic chain verification based on the fusion of action atomic sequences and action atomic succession relationships to obtain the action determination result.

[0014] Compared with the prior art, the beneficial effects of the present invention include: by identifying the retraction remnants in the partial optimal transmission and using the retraction remnant chain to perform quality backtracking and redistribution within the same action attempt on the established cross-view transmission relationship, the erroneous association between the previous incomplete action and the subsequent re-execution of the action is corrected in a timely manner, improving the accuracy of action completion status, action occurrence count and action category determination; by setting action continuation edges, action reset edges and cross-view complementary edges in the action attempt attribution graph, and reconstructing the propagation neighborhood layer by layer according to the state of each edge, and at the same time correcting the cross-view complementary edge weights by combining the action atom observation confidence and the cross-view action supplementation degree, the impact of occlusion, pause and restart on the action fusion result is reduced, and the stability of action determination is enhanced. Attached Figure Description

[0015] Figure 1 A flowchart of a multi-camera motion fusion determination method provided by the present invention; Figure 2 This is a schematic diagram of cross-collision transmission quality backoff and redistribution provided by the present invention; Figure 3 This invention provides a structural diagram of a multi-camera motion fusion determination system. Detailed Implementation

[0016] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0017] Example 1: See Figures 1-2 This embodiment provides a multi-camera motion fusion determination method, including the following specific steps: Step S1: Acquire continuous human motion videos captured by multiple cameras, determine the transition points of the motion states, and divide the motion attempts to obtain single-view motion atom sequences, motion attempt boundary node sets, and motion atom observation confidence.

[0018] The specific steps of step S1 are as follows: Step S101: Acquire continuous human motion videos captured by multiple cameras, perform human key point detection, and arrange the detection results to obtain human key point temporal records and key point detection confidence records.

[0019] In this embodiment, multiple cameras are deployed around the human movement area so that the shooting range of each camera covers the human movement area. A unified synchronous acquisition signal is sent to each camera so that each camera captures continuous human movement video at the same sampling frequency, and camera identifier, acquisition time, and frame sequence number are written into each video frame.

[0020] The time alignment tolerance is set according to the sampling period of each camera. The specific setting method is as follows: read the sampling period of each camera, determine half of the minimum sampling period as the time alignment tolerance, decode the continuous human motion video frame by frame, classify the video frames whose acquisition time difference is not greater than the time alignment tolerance into the same acquisition time, and arrange the acquisition times according to the acquisition time to obtain the multi-camera synchronized video frame sequence.

[0021] The size of each video frame in the multi-camera synchronized video frame sequence is unified and the pixels are normalized to obtain the human key point detection image. The input size of the human key point detection model is determined during the model training stage. The specific determination method is as follows: the human key point detection model is trained with different candidate input sizes, the key point localization error and single frame processing time corresponding to each candidate input size are statistically analyzed, and the smallest input size with smaller key point localization error and single frame processing time not greater than the camera sampling period is selected. Pixel normalization means dividing the pixel value of each pixel channel by the maximum pixel value of the corresponding channel so that the pixel value falls within the range of zero to one.

[0022] The human keypoint detection image is input into the human keypoint detection model, which includes an image feature extraction layer, a multi-scale feature fusion layer, a keypoint heatmap output layer, a keypoint offset output layer, and a keypoint visibility output layer. The image feature extraction layer is sequentially connected to the multi-scale feature fusion layer, which is then connected to the keypoint heatmap output layer, the keypoint offset output layer, and the keypoint visibility output layer. The image feature extraction layer extracts human contour, limb structure, and joint position features. The multi-scale feature fusion layer adjusts the size and performs channel stitching on features at different resolutions. The three output layers output the keypoint heatmap, keypoint offset, and keypoint visibility, respectively.

[0023] The parameters of the human keypoint detection model are determined using training images with human keypoint location and visibility annotations. The training images are input into the human keypoint detection model, and the average squared difference of each pixel between the predicted keypoint heatmap and the labeled heatmap is calculated to obtain the heatmap loss; the average absolute difference between the predicted keypoint offset and the labeled offset is calculated to obtain the offset loss; the cross-entropy between the predicted visibility and the labeled visibility is calculated to obtain the visibility loss; the three losses are averaged to obtain the model training loss; the model parameters are iteratively updated until the model training loss of the validation image no longer decreases for several consecutive training epochs, at which point training stops.

[0024] Extract the maximum response position of various human key points from the key point heatmap, add the corresponding key point offset to the maximum response position, and then perform coordinate conversion according to the ratio of the heatmap size to the original video frame size to obtain the human key point image coordinates; multiply the maximum response value by the key point visibility to obtain the key point detection confidence. For key points where no effective response was detected, retain the key point type, camera identifier, and acquisition time, and record the corresponding key point detection confidence as zero to obtain the human key point detection results.

[0025] The detection results of human key points are grouped according to the camera identification. The detection results of each group are arranged in ascending order according to the acquisition time and frame number. The key points in the same video frame are arranged according to the arrangement order of each output channel of the human key point detection model to obtain the human key point time sequence record. The key point detection confidence is arranged according to the same order to obtain the key point detection confidence record.

[0026] Step S102: Calculate the posture changes based on the timing records of key human body points to determine the turning points of the action state.

[0027] In this embodiment, based on the time-series recording of human body key points, the image coordinates of human body key points of each camera at each acquisition time are read, and the left and right shoulder key points and left and right hip key points that can be continuously observed are selected. The abscissa of the selected key points is averaged to obtain the abscissa of the human body position reference point, and the ordinate of the selected key points is averaged to obtain the ordinate of the human body position reference point.

[0028] Calculate the Euclidean distance between the key points on the left and right shoulders, and the Euclidean distance between the key points on the left and right hips. Average the two distances to obtain the human body scale benchmark. The specific calculation process for any distance is as follows: calculate the difference between the horizontal and vertical coordinates of the two key points respectively, square the two differences respectively, add the two squared values ​​and take the square root to obtain the Euclidean distance between the key points.

[0029] Subtract the x-coordinate of the human body position reference point from the x-coordinate of each human body key point, and subtract the y-coordinate of the human body position reference point from the y-coordinate. Then divide the resulting x-coordinate and y-coordinate by the human body scale reference to obtain the normalized human body key point time sequence record. If the human body scale reference cannot be formed, read the human body scale reference of the adjacent effective acquisition time. Perform linear weighting based on the time distance between the current acquisition time and the effective acquisition time before and after to obtain the current human body scale reference.

[0030] Based on the normalized human keypoint time-series records, the differences in the horizontal and vertical coordinates of similar human keypoints at adjacent acquisition times are calculated to obtain the keypoint displacement vectors. The horizontal component of each keypoint displacement vector is multiplied by the corresponding keypoint detection confidence and summed, then divided by the sum of the confidence scores of all keypoints to obtain the horizontal component of the human posture displacement vector. The vertical component is calculated in the same way to obtain the human posture displacement vector. When the sum of the confidence scores of all keypoints is zero, the human posture displacement vector from the previous valid acquisition time is used.

[0031] Square the horizontal and vertical components of the human posture displacement vector, add the two squared values ​​and take the square root to obtain the posture change intensity; divide the horizontal and vertical components of the human posture displacement vector by the posture change intensity to obtain the motion direction vector. When the posture change intensity is zero, the motion direction vector is recorded as the zero vector.

[0032] The standard action samples are pre-collected from action videos that are of the same type as the action to be judged and are executed completely. The width of the direction calculation window is set according to the shortest duration of the standard action sample and the camera sampling period. The specific setting method is as follows: count the duration of each standard action sample, divide the minimum duration by the camera sampling period to obtain the number of frames corresponding to the shortest action, and then determine the maximum odd number that is not greater than half of the number of frames as the width of the direction calculation window. If the obtained window width is less than 3 frames, take 3 frames to ensure that forward and backward motion directions can be formed within the window.

[0033] Within the direction calculation window before the current acquisition time, the corresponding components of each motion direction vector are averaged to obtain the forward average motion direction. Within the direction calculation window after the current acquisition time, the backward average motion direction is obtained in the same way. The lateral component of the forward average motion direction is multiplied by the lateral component of the backward average motion direction, the two longitudinal components are multiplied, and the two products are added to obtain the directional inner product.

[0034] When the directional dot product is less than zero, and the attitude change intensity at the current acquisition moment is less than the attitude change intensity at both the previous and next acquisition moments, the current acquisition moment is determined as the orientation reversal position. When the attitude change intensity at the current acquisition moment is less than the attitude change intensity at other acquisition moments within the orientation calculation window, and the attitude change intensity in the first half of the window decreases overall while the attitude change intensity in the second half of the window increases overall, the current acquisition moment is determined as the pause and turning point position. The orientation reversal positions and pause and turning points are arranged according to the acquisition time to obtain the action state turning point position.

[0035] Step S103: Based on the action state transition position, set the action attempt boundary to obtain the action attempt boundary node set.

[0036] In this embodiment, the transition positions of motion states in different cameras are aligned according to the acquisition time, and the transition positions of motion states with acquisition time differences not exceeding the time alignment tolerance are grouped into the same transition position group.

[0037] The number of cameras that can continuously output the time sequence record of human key points during the current acquisition phase is read to obtain the number of effective cameras. The number of cameras required for an effective turning position group is set according to the majority consensus rule. The specific setting method is as follows: divide the number of effective cameras by 2 and round down, then add 1 to get the required number of cameras. Count the number of cameras contained in each turning position group. When the required number of cameras is reached, the corresponding turning position group is determined as an effective turning position group.

[0038] Add up the acquisition times of each action state transition position within the effective transition position group, and then divide by the number of action state transition positions within the group to obtain a unified transition time. Subtract the unified transition time from the acquisition time of each video frame from the acquisition time of each camera and take the absolute value. Select the video frame with the smallest absolute value as the action attempt boundary node.

[0039] Calculate the acquisition time difference between adjacent action attempt boundary nodes to obtain the boundary interval. When the boundary interval is less than the shortest duration of the standard action sample, read the attitude change intensity corresponding to the two action attempt boundary nodes respectively, retain the node with the larger attitude change intensity, and arrange the retained action attempt boundary nodes according to the acquisition time to obtain the action attempt boundary node set.

[0040] Step S104: Divide the action attempts according to the action attempt boundary node set, and combine the key point detection confidence record to perform attitude change segmentation and observation reliability calculation to obtain the single-view action atom sequence and action atom observation confidence.

[0041] In this embodiment, based on the action attempt boundary node set, the action attempt boundary node is located in the human body key point time sequence record corresponding to each camera, and the human body key point time sequence record between adjacent action attempt boundary nodes is divided into one action attempt, thus obtaining the action attempt corresponding to each camera.

[0042] Within each action attempt, the width of the direction calculation window in step S102 is used as the local comparison range. When the attitude change intensity at the current acquisition time is greater than the attitude change intensity at other acquisition times within the local comparison range, the current acquisition time is determined as the local maximum position. When the attitude change intensity at the current acquisition time is less than the attitude change intensity at other acquisition times within the local comparison range, the current acquisition time is determined as the local minimum position. The inner product of the average motion directions before and after the local comparison range is calculated. When the inner product is not greater than zero, the current acquisition time is determined as the position of motion direction change.

[0043] The continuous human body key point records between adjacent local extreme positions or positions of change in movement direction are divided into an action atom. The action atom represents a local action segment in which the trend of limb, movement direction and posture change remains consistent within a continuous time range.

[0044] Read the key point detection confidence of the action atom within the same acquisition time, add up the key point detection confidence, and then divide by the number of key points to obtain the time observation confidence. Add up the time observation confidence of each time within the action atom, and then divide by the number of acquisition times contained in the action atom to obtain the basic observation reliability.

[0045] The number of records with a keypoint detection confidence of zero within an action atom is counted. This number is divided by the total number of keypoint records within the action atom to obtain the keypoint missing ratio. The result obtained by subtracting the keypoint missing ratio from the keypoint missing ratio is multiplied by the basic observation reliability to obtain the action atom observation confidence.

[0046] The motion atoms within the same camera are arranged according to the acquisition time, and the action attempt, start acquisition time, end acquisition time, human key point recording and movement direction of each motion atom are recorded to obtain a single-view motion atom sequence. The observation confidence of the motion atoms are arranged in the same order to obtain the observation confidence of the motion atoms.

[0047] Step S2: Based on the single-view action atom sequence and action attempt boundary node set, perform optimal transmission of cross-camera action atom parts and correction of retraction residual feedback to obtain the retraction feedback correction transmission relationship.

[0048] The specific steps of step S2 are as follows: Step S201: Associate the single-view action atom sequence and the action attempt boundary node set, collect the action process features of the action atoms within the corresponding action attempts of each camera, and obtain the action attempt feature record. The action process features include the order of participating limbs, the sequence of movement directions, and the local order.

[0049] In this embodiment, based on the single-view action atom sequence, the start and end acquisition times of each action atom are read; the action attempt interval in which the start and end acquisition times of each action atom are located is determined according to the action attempt boundary node set, and the action atom is assigned to the corresponding action attempt.

[0050] The key points of the human body are divided into the key point group of the left upper limb, the key point group of the right upper limb, the key point group of the left lower limb, the key point group of the right lower limb, and the key point group of the trunk. For any key point group, the length of the displacement vector of the same type of key point at adjacent acquisition time is calculated, and the length of the displacement vector within the duration of the action atom is summed to obtain the cumulative displacement of the key point group.

[0051] Add up the cumulative displacements of all key point groups, then divide by the number of key point groups to obtain the average cumulative displacement; key point groups whose cumulative displacements are not less than the average cumulative displacements are identified as participating limbs.

[0052] The motion direction vectors within the action atoms are arranged according to the acquisition time to obtain the motion direction record of the action atom; the motion direction record of each action atom is arranged according to the order of the action atoms in the corresponding action attempts to obtain the motion direction sequence.

[0053] The action atoms within the same action attempt are arranged according to the start acquisition time, and a local order is assigned sequentially starting from the first action atom. The participating limbs of each action atom are arranged according to the local order to obtain the order of participating limbs. The camera identifier, action attempt interval, order of participating limbs, movement direction sequence and local order are associated to obtain the action attempt feature record.

[0054] Step S202: Filter cross-camera candidate action attempts according to the action attempt feature record, determine the matching cost of each action atom in the cross-camera candidate action attempts, and perform partial optimal transmission to obtain the initial cross-view transmission relationship and untransmitted quality.

[0055] In this embodiment, based on the action attempt feature recording, the start and end times of each action attempt in different cameras are read; the larger of the two start times is selected as the overlap start point, and the smaller of the two end times is selected as the overlap end point. The overlap end point is subtracted from the overlap start point to obtain the time overlap length; when the overlap end point is earlier than the overlap start point, the time overlap length is recorded as zero.

[0056] Divide the time overlap length by the smaller of the durations of the two action attempts to obtain the degree of time overlap; for any action attempt, select the action attempt with the largest degree of time overlap from other cameras; when two action attempts are selected from each other, they are identified as cross-camera candidate action attempts; when there are two actions with the same maximum degree of time overlap, select the combination of the two action attempts with the smaller difference in the center acquisition time.

[0057] Based on cross-camera candidate action attempts, the differences in acquisition time, participating limbs, movement direction, posture change, and local sequence of any two action atoms are calculated; the absolute value of the acquisition time difference between the centers of the two action atoms is divided by the time overlap length of the candidate action attempts to obtain the acquisition time difference; when the time overlap length is zero, the corresponding action attempt is not identified as a cross-camera candidate action attempt.

[0058] Divide the number of intersections of the limbs involved in two action atoms by the number of unions, and then subtract the result from one to obtain the difference of the limbs involved; when the union of the limbs involved is empty, the difference of the limbs involved is recorded as zero.

[0059] Calculate the inner product of the average motion directions of the two moving atoms, restrict the inner product to the range of negative one to one, perform an inverse cosine operation on the inner product to obtain the angle between the motion directions; divide the angle between the motion directions by half a circle angle to obtain the difference in motion directions.

[0060] For two action atoms, correspondence is established for the same type of human key points that can be observed together; the difference in the horizontal and vertical coordinates of the corresponding key points is calculated separately, the two differences are squared, the two squared values ​​are added together and the square root is taken to obtain the distance between the corresponding key points; the distances of all corresponding key points are added together and then divided by the number of corresponding key points to obtain the difference in posture change; when there are no commonly observed key points, the difference in posture change is recorded as the maximum value of the difference in posture change that has been formed in the current candidate action attempt.

[0061] Subtract one from the local order of the action atom, and then divide by the number of action atoms in the action attempt minus one to obtain the normalized local order. When the action attempt contains only one action atom, the normalized local order is recorded as zero. Calculate the absolute value of the difference between the two normalized local orders to obtain the local order difference.

[0062] Within each action atom combination corresponding to the current candidate action attempt, perform minimum-maximum normalization on each difference; subtract the minimum value of the same type of difference from the current difference, and then divide by the difference between the maximum and minimum values ​​of the same type of difference to obtain the corresponding normalized difference; when the maximum and minimum values ​​are the same, the corresponding normalized difference is recorded as zero.

[0063] The weights of each difference are set based on standard cross-camera action samples with correct and incorrect matching labels; the camera placement, shooting range, sampling frequency and synchronization method of the standard cross-camera action samples are consistent with the actual action judgment stage; the main viewing direction of each camera points to the same human action area from different directions, and any target action process can be continuously captured by at least two cameras.

[0064] The standard cross-camera motion samples cover all target motion categories that the motion determination system needs to identify, and cover different human body orientations, motion speeds, and partial limb occlusion. Motion speed is divided according to the duration of pre-collected samples of the same motion category. Samples with a duration not greater than the lower quartile are identified as faster motion samples, samples with a duration not less than the upper quartile are identified as slower motion samples, and the remaining samples are identified as normal motion samples.

[0065] The initial size of the standard cross-camera action sample is set according to the stable calculation requirements of the mean and difference weights. The specific setting method is as follows: for each action category and each effective camera combination, no less than 30 sets of correctly matched action atomic pairs and 30 sets of incorrectly matched action atomic pairs are collected respectively; after reaching the initial size, the difference weights of each item are recalculated for each additional batch of samples, and the difference between the current weight and the previous weight is compared; after three consecutive additions of samples, when the absolute value of the difference of each weight is no greater than 0.02, the addition of samples is stopped. The 0.02 is set according to the value range of the normalized weight, and is used to limit the change of a single weight to within 2% of the full range of the weight.

[0066] The standard cross-camera motion samples are processed according to steps S101 to S104 to obtain the standard motion atom sequence corresponding to each camera; the annotation granularity of the motion atom is consistent with the motion atom division granularity in step S104; the position where the intensity of the posture change forms a local extremum or the direction of movement changes is determined as the boundary of the motion atom, and the continuous human key point record between two adjacent boundaries is determined as a motion atom.

[0067] For two action atoms from different cameras, if the two action atoms belong to the same actual action attempt, describe the same limb movement stage, and the time difference between the center acquisition is not greater than the time alignment tolerance in step S101, they are marked as correctly matched action atom pairs; if the two action atoms belong to different action attempts, describe different limb movement stages, or have a local sequence that does not correspond, they are marked as incorrectly matched action atom pairs.

[0068] Standard action atom annotation is performed by two annotators respectively; when the annotation results of the two annotators on the action atom boundary and matching attributes are consistent, the corresponding annotation is retained; when the boundary annotation time difference is not greater than the time alignment tolerance in step S101, the boundary annotation is determined to be consistent; when the annotation results are inconsistent, a third annotator reviews the results by combining continuous human action video, human key point time sequence records and action attempt boundary node set, and the review result is used as the final annotation result.

[0069] Calculate the average value of the normalized differences for each correctly matched action atom and the incorrectly matched action atom, and then calculate the absolute value of the difference between the two average values ​​to obtain the discrimination degree of the corresponding difference; divide each discrimination degree by the sum of all discrimination degrees to obtain the corresponding weight; when the sum of all discrimination degrees is zero, set the same weight for each difference.

[0070] Multiply each normalized difference by its corresponding weight, and then sum the products to obtain the matching cost between action atoms.

[0071] Divide the duration of the action atom by the duration of the action attempt to obtain the action atom duration ratio; multiply the action atom duration ratio by the action atom observation confidence level to obtain the action atom initial mass; sum the initial masses of action atoms within the same action attempt to obtain the initial mass sum; divide the initial mass of each action atom by the initial mass sum to obtain the mass to be transmitted; when the initial mass sum is zero, divide one by the number of action atoms to determine the mass to be transmitted for each action atom.

[0072] The smaller of the sum of the masses to be transmitted in the two candidate action attempts is multiplied by the degree of time overlap to obtain the target total transmission mass. The solution objective is to minimize the sum of the products of the matching cost and the corresponding transmission mass between each action atom. The masses to be transmitted of each action atom are used as row mass constraints and column mass constraints, and the target total transmission mass is used as the total transmission constraint to establish a partially optimal transmission problem.

[0073] The initial cross-view transmission relationship is obtained by solving the transmission quality corresponding to each action atom combination through linear programming; for any action atom, the untransmitted quality is obtained by subtracting the sum of all transmitted qualities associated with it from its quality to be transmitted.

[0074] Step S203: Based on the untransmitted quality, perform residual attribute discrimination and atomic chain connection of the retraction action to obtain the retraction residual chain.

[0075] The specific steps of step S203 are as follows: Step S2031: Extract residual discriminant features from the action atoms corresponding to the untransmitted mass to obtain a residual discriminant feature record. The residual discriminant features include the motion direction after direction reversal and the key point observation mass features.

[0076] In this embodiment, action atoms with untransmitted mass greater than zero are selected to obtain residual action atoms; the motion direction sequence within the action attempt to which the residual action atom belongs is read, the nearest direction reversal position is found from the local sequence of the residual action atom, and the motion direction vector after the direction reversal position is read; the corresponding components of each motion direction vector after the direction reversal position are averaged and then normalized to obtain the motion direction after direction reversal.

[0077] Calculate the sum of the keypoint detection confidence scores for each human body keypoint within the duration of the residual action atom, and then divide by the corresponding keypoint record number to obtain the average detection confidence score for each human body keypoint; average the average detection confidence scores of all human body keypoints to obtain the keypoint reliability.

[0078] The number of records with a confidence level greater than zero for keypoint detection corresponding to each participating limb in the residual action atom is counted, and the number is divided by the total number of keypoint records of the participating limb to obtain the visibility rate of the participating limb.

[0079] Based on the collection time, the records of similar human key points are arranged, and the maximum number of collection times with zero confidence in the detection of consecutive key points is counted to obtain the length of consecutive key point missing. The length of consecutive key point missing is divided by the total number of collection times contained in the residual action atoms to obtain the proportion of consecutive key point missing.

[0080] The reliability of key points, the visibility rate of participating limbs, and the proportion of missing consecutive key points are determined as key point observation quality characteristics; the movement direction after direction reversal is correlated with the key point observation quality characteristics to obtain residual discriminant feature records.

[0081] Step S2032: Calculate the retraction residual score and occlusion residual score of each action atom using the residual discrimination feature record to obtain the residual score record.

[0082] In this embodiment, conventional residual processing typically determines whether the untransmitted quality is caused by occlusion or missed detection based on the key point detection confidence level, without distinguishing between active retraction and observational omissions. When a person retracts in the opposite direction of the original movement, the key points may remain clear, but the retraction action cannot form a stable match with the forward movement atoms in other cameras. Using only the observation confidence level will prevent this type of untransmitted quality from forming evidence of retraction.

[0083] In this embodiment, the retraction residual score and the occlusion residual score are calculated respectively. The retraction residual score represents the degree to which the untransmitted quality is likely to be caused by active reverse motion. The occlusion residual score represents the degree to which the untransmitted quality is likely to be caused by key point misses or limb occlusion.

[0084] Read the average motion direction of the residual action atom before the direction reversal and the motion direction after the direction reversal. Multiply the horizontal components of the two unit motion direction vectors, multiply the two vertical components, and then add the two products to obtain the directional inner product. Subtract the directional inner product from one and divide the result by 2 to obtain the degree of direction reversal. When the two motion directions are the same, the degree of direction reversal is zero; when the two motion directions are opposite, the degree of direction reversal is one.

[0085] The reliability of key points and the visibility of participating limbs are added together and then divided by 2 to obtain the effective observation level; the reversal of direction and the effective observation level are added together and then divided by 2 to obtain the retraction residual score; this calculation requires the movement direction to be reversed while maintaining effective observation, which can reduce false retraction judgments caused by key point jumps.

[0086] The results obtained by subtracting the keypoint reliability from one and the visible rate of the participating limbs are calculated separately. The two results are added to the proportion of continuous keypoints missing, and then divided by 3 to obtain the occlusion residual score. The three parameters represent the degree of detection unreliability, the degree of limb invisibility, and the degree of continuous keypoint missing, respectively.

[0087] Record the residual scores for retraction and occlusion based on the order of the action atoms to obtain the residual score record.

[0088] For example, the degree of direction reversal of residual action atoms is 0.86, the keypoint reliability is 0.82, the visibility rate of participating limbs is 0.88, and the proportion of missing consecutive keypoints is 0.06; adding 0.82 and 0.88 and dividing by 2, we get the effective observation degree of 0.85; adding 0.86 and 0.85 and dividing by 2, we get the retraction residual score of 0.855; adding the result of subtracting 0.82 (0.18), the result of subtracting 0.88 (0.12), and 0.06 and dividing by 3, we get the occlusion residual score of 0.12.

[0089] Step S2033: Compare the retraction residual score and occlusion residual score of each action atom in the residual score record to determine the retraction residual mass, and extract the direction of motion after the direction is reversed for the corresponding action atom to obtain the retraction direction record.

[0090] In this embodiment, the retraction residual score and the occlusion residual score of each residual action atom are compared one by one; when the retraction residual score is greater than the occlusion residual score, the untransmitted mass of the corresponding action atom is determined as the retraction residual mass; when the occlusion residual score is greater than the retraction residual score, the untransmitted mass of the corresponding action atom is determined as the occlusion residual mass.

[0091] When two scores are the same, the degree of direction reversal and the proportion of missing consecutive keypoints are compared; if the degree of direction reversal is greater than the proportion of missing consecutive keypoints, the corresponding untransmitted quality is determined as the reverted residual quality; otherwise, it is determined as the occlusion residual quality.

[0092] The direction of motion of the corresponding action atom after the retraction residual mass is read and arranged according to the camera identifier, action attempt and local sequence to obtain the retraction direction record.

[0093] Step S2034: Based on the retraction residual mass and retraction direction records, connect the action atoms with the same participating limbs, continuous local sequence, and consistent retraction direction within the same action attempt corresponding to the same camera to obtain the retraction residual chain.

[0094] In this embodiment, conventional processing typically involves independently identifying each unmatched action segment. When a single pullback is divided into multiple action atoms, independent identification can easily lead to the separate processing of local segments within the same pullback process, resulting in incomplete pullback evidence.

[0095] The retraction action atoms are determined based on the retraction residual mass, and the retraction action atoms are grouped according to the camera identifier and action attempt identifier, and the retraction action atoms of each group are arranged according to the local order.

[0096] For two locally adjacent retraction action atoms, calculate the intersection of the limbs involved in the two action atoms; if the intersection is not empty, it is determined that the limbs involved in the two action atoms are the same.

[0097] Calculate the inner product of the two retreat directions, limit the inner product to the range of -1 to 1, and then perform an inverse cosine operation to obtain the angle between the retreat directions. When the angle between the retreat directions is less than a right angle, it is determined that the retreat directions are consistent. The right angle is a fixed geometric boundary used to distinguish between overall same-direction movement and overall opposite-direction movement, and no additional empirical threshold needs to be set.

[0098] Connect the atoms of the same limbs, with continuous local order and consistent withdrawal direction according to the local order; if any condition is not met, terminate the current connection and reconnect from the next withdrawal atom to obtain the withdrawal residual chain.

[0099] like Figure 2 As shown, step S204: Based on the pullback residual chain and the initial cross-view transmission relationship, perform cross-conflict transmission quality back-off and redistribution to obtain the pullback feedback correction transmission relationship.

[0100] The specific steps of step S204 are as follows: Step S2041: Trace the preceding action atom corresponding to the retracted residual chain along the initial cross-view transmission relationship, and determine the preceding association features between the preceding action atom and the retracted residual chain to obtain the preceding influence feature record.

[0101] In this embodiment, conventional cross-view matching typically forms a transmission relationship once based on the time, posture, and direction of motion of the current action atom. After the transmission relationship is formed, subsequent rollback information no longer participates in the correction of the preceding relationship. When a person interrupts an action and re-executes it, the preceding segment of the previous action and the following segment of the next action have high similarity and are easily connected across actions.

[0102] In this embodiment, the retraction residual chain is used as reverse evidence. The camera identifier, action attempt identifier, and local sequence of the first retraction action atom in the retraction residual chain are read. The action atom that has a transmission relationship with the retraction action atom is searched along the initial cross-view transmission relationship. Then, the action atom is searched one by one along the local sequence in the same action attempt to obtain the preceding action atom. When the tracing reaches the first action atom of the corresponding action attempt, the action attempt boundary node is encountered, or the previous action atom does not exist, the tracing stops.

[0103] The local sequence interval is obtained by subtracting the local sequence of the preceding action atom from the local sequence of the first pullback action atom.

[0104] Calculate the number of intersections and unions of the preceding action atoms and the retraction residual chain participating limbs. Divide the number of intersections by the number of unions to obtain the degree of overlap of the participating limbs. When the union of participating limbs is empty, the degree of overlap of participating limbs is recorded as zero.

[0105] The number of action atoms whose average motion direction and the average motion direction of the preceding action atom are greater than zero between the preceding action atom and the first retreat action atom is counted to obtain the number of action atoms advancing along the original motion direction; the number is then divided by the total number of action atoms in the corresponding action attempt to obtain the action progress ratio.

[0106] Calculate the inner product of the average motion direction and the retreat direction of the preceding action atom, subtract the inner product from 1 and divide the result by 2 to obtain the degree of direction reversal; multiply the degree of direction reversal by the action progress ratio to obtain the action progress loss.

[0107] Local sequence interval represents the distance between the preceding action atom and the retraction position; the degree of overlap of participating limbs represents whether the two belong to the same limb movement; action progress loss represents the degree to which the action progress that has been formed is canceled by the subsequent retraction. These three parameters are determined as preceding association features and arranged according to the initial cross-view transmission relationship to obtain the preceding influence feature record.

[0108] Step S2042: Combine the preceding influence feature record to determine the reverse influence ratio of the preceding action atom, and obtain the preceding transmission influence record.

[0109] In this embodiment, conventional error relationship correction typically employs a fixed rollback ratio or direct deletion of the transmission relationship; a fixed rollback ratio cannot distinguish the degree of influence of the rolled-back residual chain on different preceding action atoms, while direct deletion may result in the loss of still valid local correspondences.

[0110] The normalized local sequence interval is obtained by dividing the local sequence interval by the total number of action atoms in the corresponding action attempt; the result of subtracting the normalized local sequence interval from one is determined as the sequence proximity; the sequence proximity is taken as zero when it is less than zero and as one when it is greater than one.

[0111] The weights of each preceding associated feature in the reverse influence ratio are set based on historical cross-camera action samples with cross-conflict annotations; the camera layout, sampling frequency, synchronization method, and human action area range of the historical cross-camera action samples are consistent with those of the standard cross-camera action samples in step S202.

[0112] Historical cross-camera motion samples cover all target motion categories in the motion determination system, and collect motion processes for each motion category, including complete execution, mid-process stoppage, execution followed by retraction, and re-execution after retraction. For motion categories prone to human occlusion, motion processes in which at least one camera participates in partial limb occlusion while other cameras maintain effective observation are also collected.

[0113] The initial size of historical cross-camera action samples is set according to the stable calculation requirements of the mean of the preceding associated features. The specific setting method is as follows: for each action category and each effective camera combination, no less than 30 sets of correct transmission relationships and 30 sets of cross-transmission error transmission relationships are labeled respectively; after reaching the initial size, for each batch of interrupted, rolled back or re-executed action samples, the weights corresponding to the degree of sequence proximity, the degree of overlap of the participating limbs and the loss of action progress are recalculated; after three consecutive additions of samples, when the absolute value of the difference of each weight is no greater than 0.02, the addition of samples is stopped.

[0114] The historical cross-camera action samples are labeled with actual action attempts; the person starts the action from the starting position and executes it continuously until the action is completed normally, stops midway, or reverses and retreats. This continuous action process is labeled as an actual action attempt; when the person stops the previous action and starts the action again from the starting stage, the re-execution process is labeled as another actual action attempt; the boundary position of the actual action attempt is determined according to the action state turning point obtained in step S102, and is verified in combination with the original continuous human action video.

[0115] The historical cross-camera action samples are processed according to steps S101 to S202 to obtain the initial cross-view transmission relationship; for any initial cross-view transmission relationship, the actual actions corresponding to the action atoms at both ends of the edge are read and labeled, and the participating limbs and local order are attempted.

[0116] When the action atoms at both ends of an edge belong to the same actual action attempt, the participating limbs overlap, and the local sequence corresponds to the same action progress stage, the initial cross-view transmission relationship is marked as a correct transmission relationship; when the action atoms at both ends of an edge belong to different actual action attempts, or the local sequence paths of the two action atoms cross the boundary of the actual action attempts, the initial cross-view transmission relationship is marked as a cross-incidental transmission relationship.

[0117] When two action atoms belong to different cameras, but there is a deviation in the local sequence of action atoms due to a short-term missed detection by one camera, the participating limbs and movement directions of the adjacent action atoms before and after are used for verification; if the adjacent action atoms before and after still correspond to the same actual action attempt, the correspondence is retained as the correct transmission relationship.

[0118] Cross-period conflict annotation is completed by two annotators respectively; if the annotation results of the two annotators are consistent, the corresponding annotation is retained; if the annotation results are inconsistent, a third annotator reads the original continuous human motion video, the action attempt boundary node set, the retraction residual chain and the initial cross-view transmission relationship for verification, and the verification result is used as the final annotation result.

[0119] Calculate the average values ​​of correct transmission relationships and cross-transmission error relationships in terms of sequential proximity, degree of overlap of involved limbs, and loss of movement progress, respectively. Then calculate the absolute value of the difference between the two average values ​​to obtain the discriminative power of each preceding association feature. Divide the discriminative power of each feature by the sum of all discriminative powers to obtain the corresponding weight. When the sum of all discriminative powers is zero, set the same weight for the three preceding association features.

[0120] Multiply the degree of sequential proximity, the degree of overlap of involved limbs, and the loss of movement progress by their respective weights, and then add the three products together to obtain the reverse influence ratio of the preceding action atom; take zero when the reverse influence ratio is less than zero, and take one when it is greater than one.

[0121] Based on the initial cross-perspective transmission relationship, the original transmission mass, reverse influence ratio, and corresponding retraction residual chain of the preceding action atom are recorded to obtain the preceding transmission influence record.

[0122] For example, in historical cross-camera action samples, the weights corresponding to sequential proximity, degree of overlap of participating limbs, and action progress loss are 0.30, 0.20, and 0.50, respectively; the sequential proximity of a preceding action atom is 0.90, the degree of overlap of participating limbs is 0.80, and the action progress loss is 0.74; multiplying 0.90 by 0.30 gives 0.27, multiplying 0.80 by 0.20 gives 0.16, and multiplying 0.74 by 0.50 gives 0.37. Adding the three products gives the reverse influence ratio of 0.80.

[0123] Step S2043: Based on the preceding transmission impact record, perform cross-period conflict determination and transmission quality backtracking on the initial cross-view transmission relationship to obtain the backtracked transmission relationship and backtracked transmission quality.

[0124] In this embodiment, based on the preceding transmission impact record, the preceding action atom, the corresponding rollback residual chain, and the original transmission quality are read; when the two action atoms connected by the initial cross-view transmission relationship are located in different action attempts, or when there is an action attempt boundary node in the local sequence path corresponding to the two action atoms, the corresponding relationship is determined as a cross-transmission relationship.

[0125] Conventional cross-view matching mainly preserves the transmission relationship based on the matching cost; local action atoms in different action attempts may have similar postures and directions of motion, and it is difficult to determine whether two action atoms belong to the same action process based solely on the matching cost.

[0126] For cross-transmission relationships, the original transmission quality is multiplied by the reverse impact ratio to obtain the backoff transmission quality; the original transmission quality is subtracted from the backoff transmission quality to obtain the retained transmission quality.

[0127] When the retained transmission quality is greater than zero, the corresponding cross-view transmission relationship is retained, and the transmission quality is updated to the retained transmission quality; when the retained transmission quality is zero, the corresponding cross-view transmission relationship is deleted, all cross-transmission relationships are processed to obtain the rollback transmission relationship, and the rollback transmission quality is obtained based on the action attempt and action atom record of each rollback transmission quality.

[0128] By rolling back a portion of the transmission quality based on the degree of impact of the rollback, cross-error relationships can be weakened while preserving transmission quality that still has local correspondence significance, thus avoiding abrupt changes in transmission relationships caused by direct deletion.

[0129] Step S2044: Limit the rollback transmission quality to the corresponding cross-camera candidate action attempts, filter compatible action atoms according to the rollback transmission relationship, and perform partial optimal transmission to obtain the rollback feedback correction transmission relationship.

[0130] In this embodiment, conventional rematching typically involves globally allocating the fallback quality back to all action atoms. When a person repeatedly performs the same action, local action atoms in different action attempts may have lower matching costs, and the fallback quality can easily be re-entered into other action attempts.

[0131] Read the preceding action atom that generates the fallback transmission quality and the corresponding cross-camera candidate action attempt, add the fallback transmission quality back into the transmission quality of the preceding action atom, and limit the action atoms that can receive the fallback transmission quality to the cross-camera candidate action attempt.

[0132] Remove candidate action atoms that cross the action attempt boundary node with the preceding action atom from the cross-camera candidate action attempt; calculate the intersection of the participating limbs of the preceding action atom and the remaining candidate action atoms, and determine that the participating limbs overlap if the intersection is not empty.

[0133] Calculate the inner product of the average motion direction of the preceding action atom and the average motion direction of the candidate action atom; when the inner product is greater than zero, the angle between the two motion directions is less than a right angle, and the corresponding candidate action atom is retained to obtain the compatible action atom.

[0134] Following the matching cost calculation process in step S202, the matching cost between the preceding action atom and each compatible action atom is calculated; taking the backhaul transmission quality as the total quality to be retransmitted, the remaining acceptable quality of the compatible action atoms as the quality constraint, and minimizing the sum of the products of the matching cost and the redistributed quality as the solution objective, a partially optimal transmission problem is re-established.

[0135] Solve for the redistributed mass corresponding to each compatible action atom, and add the newly formed transport relationship to the rollback transport relationship to obtain the rollback feedback corrected transport relationship; this process only redistributes the transport mass affected by the rollback residual chain, and retains the other stable transport relationships.

[0136] For example, the original transmission quality of a cross-transmission relationship is 0.30, and the reverse influence ratio is 0.80. Multiplying 0.30 and 0.80 gives a fallback transmission quality of 0.24. Subtracting 0.24 from 0.30 gives a reserved transmission quality of 0.06. There are two compatible action atoms in the candidate action attempt, with remaining acceptable qualities of 0.15 and 0.12, and matching costs of 0.20 and 0.35, respectively. The matching cost of the first compatible action atom is smaller, so it is first assigned an acceptable quality of 0.15, leaving a remaining unassigned quality of 0.09. Then, 0.09 is assigned to the second compatible action atom. The sum of the two assigned qualities is 0.24, resulting in a fallback feedback correction transmission relationship.

[0137] exist Figure 2 In the diagram, the three dashed boxes, from left to right, represent the initial cross-view transmission relationship, cross-collision transmission quality backoff, and backoff transmission quality redistribution and corrected transmission relationship, respectively. Camera A and Camera B in each dashed box represent two different shooting perspectives. The hollow circles arranged vertically represent the action atoms in the single-view action atom sequence corresponding to each camera. The vertical solid arrow connecting adjacent hollow circles represents the local order of action atoms within the same camera. The vertical ellipsis symbol represents the omitted intermediate action atoms. The horizontal dashed line represents the action attempt boundary, with action atoms above and below the horizontal dashed line belonging to different action attempts.

[0138] The slanted dashed arrows in the dashed box on the left represent the initial cross-view transmission relationship formed based on the partial optimal transmission. The slanted dashed arrows that cross the action attempt boundary represent the cross-transmission relationship that connects action atoms in different action attempts. The circle filled with slanted lines represents the first action atom in the retraction residual chain. The first action atom is used to trace the preceding action atom affected by the retraction process along the initial cross-view transmission relationship.

[0139] The dashed box in the middle indicates cross-collision transmission quality backtracking based on the remnant chain; the cross mark on the diagonal dashed arrow indicates that the corresponding cross-transmission relationship is determined to be a conflict transmission relationship, and the transmission quality in the transmission relationship is backtracked according to the reverse impact ratio; cross-view transmission relationships without cross marks indicate transmission relationships that are not affected by the remnant chain or are still retained after backtracking.

[0140] The dashed box on the right indicates the redistribution of fallback transmission quality; the circle with diagonal fill corresponds to the preceding action atom that generates fallback transmission quality; the diagonal dashed arrow pointing to multiple action atoms in camera B indicates that the fallback transmission quality is limited to the corresponding cross-camera candidate action attempts, and the redistribution relationship is formed after selecting compatible action atoms based on the participating limbs and movement direction; after transmission quality fallback and redistribution, the erroneous transmission relationship that crosses the action attempt boundary is weakened or broken, and the fallback transmission quality is redistributed to compatible action atoms within the same candidate action attempt, resulting in a fallback feedback correction transmission relationship.

[0141] Step S3: Based on the action atom observation confidence and the pullback feedback to correct the transmission relationship, construct the action attempt attribution graph, and perform propagation neighborhood reconstruction and cross-perspective complementary attention fusion to obtain the fused action atom sequence and action atom inheritance relationship.

[0142] The specific steps of step S3 are as follows: Step S301: Using the action atoms in the single-view action atom sequence as nodes, based on the arrangement order of the action atoms in the corresponding sequence, the action attempt boundary node set, and the pullback feedback correction transmission relationship, establish action continuation edges, action reset edges, and cross-view complementary edges respectively to construct the action attempt attribution graph.

[0143] In this embodiment, action atom nodes are established for each action atom in the single-view action atom sequence. Normalized human key point records, participating limbs, average movement direction, local sequence, action attempts, camera identifiers, and action atom observation confidence are written into the corresponding nodes to obtain action atom node features.

[0144] For action atom nodes in the same camera that belong to the same action attempt and are locally adjacent, establish an action continuation edge; calculate the absolute value of the normalized local order difference between the two action atoms to obtain the local order difference; calculate the inner product of the two average motion directions, divide the result of the inner product by 2 to obtain the degree of consistency of motion directions; calculate the number of intersections of participating limbs divided by the number of unions to obtain the degree of overlap of participating limbs, and write the above parameters into the action continuation edge.

[0145] For the action atomic nodes on both sides of the action attempt boundary node, establish an action reset edge. For the preceding action atomic node that experienced transmission quality rollback in step S204 and the first rollback action atomic node in the corresponding rollback residual chain, establish an action reset edge. Write the action attempt boundary type, direction reversal degree and rollback residual quality into the action reset edge.

[0146] Based on the feedback correction of the transmission relationship, cross-view complementary edges are established between the action atomic nodes of different cameras, and the transmission quality, matching cost and corresponding candidate action attempts are written into the cross-view complementary edges to obtain the action attempt attribution graph.

[0147] Step S302: Initialize the action to attempt to assign the continuation state and reset state of each edge in the graph, and perform propagation neighborhood reconstruction and graph attention propagation to obtain the updated propagation neighborhood of each layer.

[0148] The specific steps of step S302 are as follows: Step S3021: Determine the initial continuation state and initial reset state of the action continuation edge, action reset edge, and cross-view complementary edge in the action attempt at the graph, and obtain the initial edge state record.

[0149] In this embodiment, the continuation state indicates the degree to which the action atoms at both ends of the edge belong to the same continuous action process, and the reset state indicates the degree to which the action atoms at both ends of the edge belong to different action attempts or have an action restart relationship.

[0150] For the action continuation edge, add the degree of consistency of movement direction and the degree of overlap of participating limbs and divide by 2 to obtain the initial continuation state; subtract the initial continuation state from 1 to determine the initial reset state.

[0151] For an action reset edge, the retraction residual mass is divided by the mass to be transferred of the corresponding retraction action atom to obtain the normalized retraction residual mass; when the mass to be transferred is zero, the normalized retraction residual mass is recorded as zero; when the normalized retraction residual mass is greater than one, it is taken as one.

[0152] When the action reset edge directly crosses the action attempt boundary node, the action attempt boundary indicator value is recorded as one; otherwise, it is recorded as zero. The initial reset state is obtained by adding the direction reversal degree, the normalized retreat residual mass, and the action attempt boundary indicator value and dividing by 3. The result of subtracting the initial reset state from one is determined as the initial continuation state.

[0153] For cross-view complementary edges, the transmission quality is divided by the smaller value of the transmission quality of the action atoms at both ends of the edge to obtain the transmission quality ratio. When the smaller value is zero, the transmission quality ratio is recorded as zero; when the transmission quality ratio is greater than one, it is recorded as one.

[0154] When the action atoms at both ends of a cross-perspective complementary edge belong to the corresponding candidate action attempts, the transmission quality ratio is determined as the initial continuation state, and the result obtained by subtracting the transmission quality ratio from one is determined as the initial reset state; when the action atoms at both ends of the edge belong to different action attempts, the transmission quality ratio is determined as the initial reset state, and the result obtained by subtracting the transmission quality ratio from one is determined as the initial continuation state.

[0155] The initial edge state record is obtained by arranging the initial continuation state and initial reset state of each edge according to the edge identifier.

[0156] Step S3022: Based on the initial edge state record, determine the edges that are dominant in the continuation state and include them in the action continuation propagation neighborhood, determine the edges that are dominant in the reset state and include them in the action reset propagation neighborhood, and obtain the first-layer propagation neighborhood.

[0157] In this embodiment, the initial continuation state and the initial reset state of each edge are compared one by one; when the initial continuation state is greater than the initial reset state, the corresponding edge is included in the action continuation propagation neighborhood; when the initial reset state is greater than the initial continuation state, the corresponding edge is included in the action reset propagation neighborhood.

[0158] When two states are the same, the action continuation edge is included in the action continuation propagation neighborhood, and the action reset edge is included in the action reset propagation neighborhood; when both ends of the cross-view complementary edge belong to the corresponding candidate action attempt, it is included in the action continuation propagation neighborhood, otherwise it is included in the action reset propagation neighborhood.

[0159] Based on the sum of the action continuation propagation edges and action reset propagation edges connected to each action atomic node, the first-level propagation neighborhood corresponding to each action atomic node is obtained.

[0160] Step S3023: Perform graph attention propagation on the first-layer propagation neighborhood to generate first-layer action reset information, and update the continuation state and reset state of each edge to obtain the second-layer edge state record.

[0161] In this embodiment, graph attention propagation adopts a dual-channel graph attention structure, which includes an action continuation feature transformation layer, an action reset feature transformation layer, a continuation edge state gating layer, a reset edge state gating layer, a continuation attention normalization layer, a reset attention normalization layer, a dual-channel node feature aggregation layer, and an edge state update layer.

[0162] The action continuation feature transformation layer is connected to the continuation edge state gating layer, and the continuation edge state gating layer is connected to the continuation attention normalization layer; the action reset feature transformation layer is connected to the reset edge state gating layer, and the reset edge state gating layer is connected to the reset attention normalization layer; the continuation attention normalization layer and the reset attention normalization layer are respectively connected to the dual-channel node feature aggregation layer, and the dual-channel node feature aggregation layer is connected to the edge state update layer.

[0163] Each propagation layer shares the model parameters of the action continuation feature transformation layer, action reset feature transformation layer, continuation edge state gating layer, reset edge state gating layer, continuation attention normalization layer, reset attention normalization layer, and edge state update layer; the action continuation channel and the action reset channel respectively use single-head attention calculation.

[0164] The action atomic node features, the first-layer propagation neighborhood, and the initial edge state record are the inputs to the dual-channel graph attention structure, while the action continuation information, action reset information, and updated edge state of each node are the outputs.

[0165] The action atomic node features are input into the action continuation feature transformation layer and the action reset feature transformation layer, respectively. They are transformed into the hidden feature space through linear mapping and then nonlinearly transformed through modified linear units to obtain the action continuation hidden features and the action reset hidden features.

[0166] The hidden feature dimension is set based on the action atomic node feature dimension and the validation loss. The specific setting method is as follows: 32, 64, 128 and 256 are determined as candidate hidden feature dimensions, and candidate values ​​less than half of the action atomic node feature dimension are eliminated; candidate models are trained using the same training samples, initial learning rate, batch size and training rounds, and the candidate hidden feature dimension with the lowest validation loss is selected.

[0167] The continuation edge state gating layer multiplies the continuation state of each edge in the action continuation propagation neighborhood with the action continuation hidden feature of the corresponding adjacent node component by component to obtain the continuation gating feature; the reset edge state gating layer multiplies the reset state of each edge in the action reset propagation neighborhood with the action reset hidden feature of the corresponding adjacent node component by component to obtain the reset gating feature.

[0168] The attention normalization layer and the attention normalization layer calculate the attention weights of each adjacent edge corresponding to the same target node according to the attention calculation method in step S3031. The dual-channel node feature aggregation layer multiplies the hidden features of the adjacent nodes with the corresponding attention weights, and then sums the feature components to obtain the action continuation information and the action reset information.

[0169] The edge state update layer concatenates the action continuation information, action reset information, original continuation state, original reset state, and corresponding edge features of the nodes at both ends of the edge. Then, it obtains the continuation state output value and the reset state output value through a fully connected mapping. The exponents of the two state output values ​​are taken respectively. The exponent value of the continuation state is divided by the sum of the two exponent values ​​to obtain the updated continuation state. The exponent value of the reset state is divided by the sum of the two exponent values ​​to obtain the updated reset state.

[0170] The parameters of the dual-channel graph attention structure are obtained by training historical action samples with action attempt boundary labels and cross-camera action atom correspondence labels. The historical action samples are stratified according to action category, action completion status and camera combination, and then divided into training samples and validation samples in a 4:1 ratio, so that samples formed by the same continuous human action video only enter one of the sample sets.

[0171] The training samples are processed according to steps S101 to S301 to obtain the action attempt attribution map for training; based on the actual action attempt annotation and cross-view corresponding annotation, the true continuation state and the true reset state are set for each side.

[0172] Calculate the negative logarithm of the predicted probability corresponding to the true state of each edge, and average it over all edges to obtain the edge state classification loss; for nodes with correct cross-view correspondence, calculate the difference of each component of the hidden features of the two nodes respectively, square each difference, and then average it over all feature components and all node pairs to obtain the cross-view consistency loss; add the edge state classification loss and the cross-view consistency loss together and divide by 2 to obtain the model training loss.

[0173] The model parameters are updated using the adaptive moment estimation method. The first-order moment decay coefficient, the second-order moment decay coefficient, and the numerical stability term are set according to the common parameters of the adaptive moment estimation method, taking values ​​of 0.9, 0.999, and 10 to the power of -8, respectively. The first-order moment decay coefficient is used to smooth the first moment of the gradient, the second-order moment decay coefficient is used to smooth the second moment of the squared gradient, and the numerical stability term is used to avoid the denominator being zero or too small during the parameter update process.

[0174] The initial learning rate is set based on the results of short-cycle pre-training. The specific setting method is as follows: 10 to the power of -5, 10 to the power of -4, and 10 to the power of -3 are determined as candidate initial learning rates, and 10 rounds of pre-training are performed for each. The decrease in validation loss and the fluctuation of loss corresponding to each candidate initial learning rate are compared, and the candidate initial learning rate with a larger decrease in validation loss and without a continuous increase for 3 rounds is selected.

[0175] The batch size is set based on the available storage space of the computing device and the number of training samples. The specific setting method is as follows: test the candidate batch sizes of 8, 16, 32, 64 and 128 in turn, and select the largest candidate batch size that does not cause storage overflow and can form no less than 4 training batches in each training round.

[0176] The number of waiting rounds is set according to the natural fluctuation of the validation loss. Specifically, the method is as follows: benchmark training is performed using the selected hidden feature dimension, initial learning rate and batch size. The length of consecutive rounds in which the validation loss rises briefly and then falls again is recorded. The maximum value of the consecutive round length is added by 1 to determine the number of waiting rounds. For example, in the embodiments of this application, the number of waiting rounds is preferably 5 to 10 rounds.

[0177] The maximum number of training rounds is set according to the convergence process in the benchmark training. The specific setting method is as follows: record the training rounds when the verification loss reaches the minimum value in the benchmark training, and determine the minimum number of rounds that is greater than the training rounds and can cover the stop waiting rounds as the maximum training rounds; for example, in the embodiments of this application, the maximum number of training rounds is preferably 100 to 300 rounds.

[0178] The dual-channel graph attention structure is trained iteratively. When the validation loss has not decreased after a certain number of consecutive waiting rounds, or when the number of training rounds reaches the maximum, training is stopped, and the model parameters corresponding to the training round with the lowest validation loss are retained.

[0179] The first-layer action continuation information is obtained by calculating the continuation channel attention based on the first-layer action continuation propagation neighborhood; the first-layer action reset information is obtained by calculating the reset channel attention based on the first-layer action reset propagation neighborhood.

[0180] The absolute values ​​of the corresponding components of the action continuation information of the nodes at both ends of the edge are subtracted. The average of all absolute values ​​is then calculated, and the average value is subtracted from one to obtain the continuation information association strength. The continuation information association strength is restricted to the range of zero to one. The original continuation state and the continuation information association strength are added together and then divided by 2 to obtain the updated continuation state.

[0181] Calculate the reset information association strength in the same way, add the original reset state and the reset information association strength, and divide by 2 to get the updated reset state; add the updated continuation state and the updated reset state to get the state sum, divide the two updated states by the state sum respectively, so that the sum of the two states is one, and get the second layer edge state record; when the state sum is zero, record both the updated continuation state and the reset state as one-half.

[0182] For example, when the feature dimension of the action atomic node is 64, 64, 128, and 256 are determined as candidate hidden feature dimensions; 10 rounds of pre-training are performed using 10 to the power of -5, 10 to the power of -4, and 10 to the power of -3, respectively. The validation loss corresponding to 10 to the power of -4 decreases significantly and does not increase for three consecutive rounds, so the initial learning rate is set to 10 to the power of -4; the computing device does not experience storage overflow when the batch size is 64, and can form no less than 4 training batches per round, so the batch size is set to 64.

[0183] Step S3024: Starting with the edge state record of the second layer, reconstruct the current layer propagation neighborhood layer by layer using the current layer edge state record, perform graph attention propagation, and update the next layer edge state according to the action reset information formed by propagation, so as to obtain the updated propagation neighborhood of each layer.

[0184] In this embodiment, conventional graph attention networks use fixed adjacency relationships in each propagation layer, which cannot change the propagation path according to the action reset information formed during the propagation process. This makes it easy for node features in different action attempts to propagate across boundaries.

[0185] Based on the second-layer edge state records, the continuation and reset states of each edge are re-compared, and the second-layer propagation neighborhood is reconstructed according to the rules in step S3022; graph attention propagation is performed on the second-layer propagation neighborhood to obtain the second-layer action continuation information and the second-layer action reset information; the third-layer edge state records are obtained according to the edge state update method in step S3023.

[0186] The edge state update, neighborhood reconstruction, and graph attention propagation are iterated layer by layer until the propagation layer number is reached, or the average change of all edge states in two adjacent layers is less than the edge state convergence threshold, at which point the iteration stops.

[0187] The propagation layer is set based on the longest local sequence distance in the action attempt attribution graph. The specific setting method is as follows: count the maximum number of edges traversed between the first action atom and the last action atom in the same action attempt, and determine the maximum number of edges as the propagation layer; the propagation layer is at least 1.

[0188] The specific calculation process for the average change in edge state is as follows: Calculate the absolute value of the difference between the continued state and the absolute value of the difference between the reset state of the same edge in the two adjacent layers, add the two absolute values ​​and divide by 2 to obtain the change in state of a single edge; add the changes in state of a single edge of all edges and divide by the number of edges to obtain the average change in state of all edges in the two adjacent layers.

[0189] The edge state convergence threshold is set based on the historical action samples that correctly form the propagation neighborhood. The specific setting method is as follows: the historical action samples are propagated layer by layer, the average change of the edge state after two consecutive layers of the propagation neighborhood remain unchanged is recorded, and all the average changes of the edge state are arranged in ascending order; the number of samples is multiplied by 0.75, and the result is rounded up to obtain the arrangement position corresponding to the upper quartile. The average change of the edge state corresponding to this position is determined as the edge state convergence threshold.

[0190] When the number of historical action samples is insufficient to stably calculate the upper quartile, the number of historical action samples continues to increase; after each increase in samples, the edge state convergence threshold is recalculated, and when the absolute value of the threshold difference obtained from three consecutive calculations is not greater than 0.005, the increase in samples is stopped; for example, in the embodiments of this application, the edge state convergence threshold is preferably 0.01 to 0.05.

[0191] The propagation neighborhood is redefined based on the edge states at each layer, so that edges with a dominant reset state stop propagating action continuation information; the propagation neighborhoods for action continuation and action reset at each layer are recorded to obtain the updated propagation neighborhoods at each layer.

[0192] Step S303: Based on the updated propagation neighborhood and action atom observation confidence of each layer, perform dual-channel attention calculation and cross-view complementary weight correction to obtain the graph attention propagation result.

[0193] The specific steps of step S303 are as follows: Step S3031: Perform graph attention calculations on the updated action continuation propagation neighborhood and action reset propagation neighborhood of each layer to obtain the continuation channel edge attention record and the reset channel edge attention record.

[0194] In this embodiment, for any side in the action continuation propagation neighborhood, the action atomic node features at both ends of the side are linearly transformed by the action continuation feature transformation layer; the transformed source node features, target node features and the continuation state of the corresponding side are concatenated to obtain the continuation edge concatenation feature.

[0195] Each component of the continuation edge splicing feature is multiplied by the corresponding component of the trainable continuation attention parameter, and all products are summed to obtain the continuation edge association value. The association values ​​of each continuation edge corresponding to the same target node are exponentially calculated, and each exponential value is divided by the sum of all exponential values ​​to obtain the continuation channel edge attention weight.

[0196] For any side in the action reset propagation neighborhood, the source node features, target node features, and reset state are concatenated in the same way, and the corresponding components are multiplied and summed with the trainable reset attention parameters to obtain the reset edge association value; the reset edge association values ​​corresponding to the same target node are exponentially normalized to obtain the reset channel edge attention weight.

[0197] The attention weights of each continuation channel are arranged according to the propagation level, target node identifier, and edge identifier to obtain the attention record of the continuation channel; the attention weights of the reset channel are arranged in the same way to obtain the attention record of the reset channel.

[0198] Step S3032: Extract the participating limbs and posture changes of the action atoms at both ends of the cross-view complementary edge, and compare the cross-view differences of the corresponding features to obtain the degree of cross-view action complementarity.

[0199] In this embodiment, the participating limbs of the action atoms at both ends of each cross-view complementary edge, the normalized human key point records, and the key point detection confidence are read.

[0200] The number of keypoint types in the first-end action atom with a keypoint detection confidence of zero and the corresponding keypoint detection confidence in the second-end action atom with a keypoint detection confidence of greater than zero is counted. The number of keypoint types that the second-end action atom can supplement from the first end is counted in the same way. The two counts are added together and then divided by the total number of keypoint types involved in the two-end action atom to obtain the keypoint complementarity ratio.

[0201] Calculate the number of intersections and unions of the participating limbs of the two action atoms. Subtract the number of intersections from the number of unions and divide by the number of unions to obtain the limb replenishment ratio. When the union of participating limbs is empty, the limb replenishment ratio is recorded as zero.

[0202] For key points of the same type that can be effectively observed in both action atoms, correspondence is established. For any corresponding key point, the normalized coordinates at the end of the action atom are subtracted from the normalized coordinates at the start of the action atom to obtain the attitude change vector. The attitude change vectors under the two perspectives are normalized, and then the inner product of the two unit vectors is calculated. The result of adding the inner product is divided by 2 to obtain the attitude change consistency of a single key point. The attitude change consistency of all corresponding key points is averaged to obtain the cross-perspective attitude consistency. When there are no commonly observed key points, the cross-perspective attitude consistency is recorded as zero.

[0203] The degree of cross-view movement complementarity is obtained by adding the key point complementarity ratio, the limb complementarity ratio, and the cross-view posture consistency ratio together and then dividing by 3.

[0204] Step S3033: Determine the observation confidence of the action atoms at both ends of the cross-view complementary edge, and calculate the attention compensation weight by combining the degree of cross-view action complementation.

[0205] In this embodiment, conventional cross-view attention fusion typically sets edge weights directly based on node similarity or observation confidence. When a camera is occluded, the observation confidence of the corresponding action atom is low, and directly reducing the edge weight will prevent effective information from other cameras from supplementing the action atom.

[0206] Read the observation confidence of the action atoms at both ends of the cross-view complementary edge, determine the larger value as high observation confidence, and the smaller value as low observation confidence; subtract the low observation confidence from the high observation confidence to obtain the observation confidence difference.

[0207] The initial attention compensation weight is obtained by multiplying the difference in observation confidence by the high observation confidence, and then multiplying the resulting product by the degree of cross-view action supplementation. The three parameters are multiplied continuously so that the attention compensation only increases when there is a difference in the observation quality between the two views, and when the high-quality view has reliable observations and can provide missing action information.

[0208] The initial attention compensation weight is set to zero when it is less than zero and to one when it is greater than one, thus obtaining the attention compensation weight.

[0209] Step S3034: Extract the initial attention weights corresponding to the cross-view complementary edges from the continuous channel edge attention records and the reset channel edge attention records.

[0210] In this embodiment, the corresponding record is searched in the continuous channel side attention record and the reset channel side attention record based on the cross-view complementary edge identifier.

[0211] When a cross-view complementary edge is located in the neighborhood of action continuation propagation, the corresponding weight is read from the attention record of the continuation channel edge; when a cross-view complementary edge is located in the neighborhood of action reset propagation, the corresponding weight is read from the attention record of the reset channel edge.

[0212] The read weights are determined as the initial attention weights of the cross-view complementary edges, and arranged according to the propagation level and edge identifier to obtain the record of the initial attention weights of the cross-view complementary edges.

[0213] Step S3035: Correct the initial attention weights corresponding to the cross-view complementary edges in the corresponding channels by adjusting the attention compensation weights, and perform competitive aggregation on the attention records of the continued channel edges and the attention records of the reset channel edges to obtain the graph attention propagation results.

[0214] In this embodiment, conventional dual-channel fusion usually involves directly averaging the two channels or using fixed weights. The requirements for the two channels differ between the continuous action phase and the action reset phase, and fixed weights tend to retain too much continuity information at the action restart position.

[0215] Identify the action atom nodes with low observation confidence in the cross-view complementary edge, and multiply the result obtained by adding an attention compensation weight with the initial attention weight of the cross-view complementary edge to obtain the corrected attention weight pointing to the action atom nodes with low observation confidence.

[0216] For the same target node, the corrected attention weights and the remaining uncorrected attention weights are added together to obtain the total channel weights; each corrected attention weight and each uncorrected attention weight are divided by the total channel weights to obtain the renormalized edge attention weights; the continued channel edge attention records and the reset channel edge attention records are updated based on the normalization results.

[0217] Based on the corrected continuation channel edge attention record, the features of each adjacent node are multiplied by the corresponding attention weight, and the resulting feature vector is summed component by component to obtain the continuation channel aggregation result; the reset channel aggregation result is obtained in the same way.

[0218] Calculate the vector lengths of the continuation channel aggregation result and the reset channel aggregation result respectively. The calculation process of the vector length is as follows: square each component of the vector, add all the square values ​​together and then find the square root.

[0219] Divide the vector length of the continuation channel aggregation result by the sum of the lengths of the two channel vectors to obtain the continuation channel competition weight; divide the vector length of the reset channel aggregation result by the sum of the lengths of the two channel vectors to obtain the reset channel competition weight; when the sum of the lengths of the two channel vectors is zero, set the competition weights of both channels to one-half.

[0220] Multiply each component of the continuation channel aggregation result by the continuation channel competition weight, multiply each component of the reset channel aggregation result by the reset channel competition weight, and then sum the corresponding components of the two channels to obtain the graph attention propagation result.

[0221] For example, the initial attention weight of a cross-view complementary edge is 0.40, the observation confidence of the action atoms at both ends is 0.85 and 0.45 respectively, and the cross-view action complementation degree is 0.75. Subtracting 0.45 from 0.85 gives an observation confidence difference of 0.40. Multiplying 0.40 by 0.85 and then by 0.75 gives an attention compensation weight of 0.255. Multiplying the result of adding 0.255 to 1.255 by 0.40 gives a corrected attention weight of 0.502.

[0222] The attention weights of the other two edges of the same target node are 0.35 and 0.25, respectively. Adding 0.502, 0.35 and 0.25 together gives a total channel weight of 1.102. Dividing the three weights by 1.102 respectively gives the renormalized weights of 0.4555, 0.3176 and 0.2269, and the sum of the three weights is 1.

[0223] Step S304: Based on the graph attention propagation results, merge and update the action atom nodes and edges in the action attempt at belonging graph to obtain the merged action atom sequence and action atom inheritance relationship.

[0224] In this embodiment, the graph attention propagation results corresponding to each action atom node are read, the action atom observation confidence is determined as the original node feature weight, and the result obtained by subtracting the action atom observation confidence is determined as the propagation result weight.

[0225] Each component of the original node feature is multiplied by its weight, and the corresponding component of the graph attention propagation result is multiplied by its weight. The two products are then added together to obtain the updated action atom node features.

[0226] Based on the feedback correction of the transmission relationship, the action atomic nodes with transmission relationship in different cameras are grouped into the same fusion node group; the confidence scores of each action atomic observation in the group are added together to obtain the total confidence score, and the confidence scores of each action atomic observation are divided by the total confidence score to obtain the node fusion weight; when the total confidence score is zero, the same node fusion weight is set according to the number of nodes in the group.

[0227] The updated action atom node features are multiplied by the corresponding node fusion weights, and then the corresponding components of each node feature are summed to obtain the fused action atom features.

[0228] The fused action atoms are arranged according to the action attempt identifier, local order, and acquisition time. For adjacent fused action atoms, the continuation state and reset state corresponding to the edge connecting the two are read. When the continuation state is greater than the reset state, the action atom inheritance relationship is established. When the reset state is not less than the continuation state, the inheritance relationship is broken, and the next fused action atom is determined as the new action attempt starting point. The fused action atoms are connected according to the action atom inheritance relationship to obtain the fused action atom sequence and the action atom inheritance relationship.

[0229] Step S4: Based on the fused action atom sequence and action atom succession relationship, perform overall matching of action prototypes and semantic chain verification to obtain the action determination result.

[0230] The specific steps of step S4 are as follows: Step S401: Obtain the action recognition prototype sequence library, extract the action prototype sequence corresponding to each action category, and determine the node order of the action prototype sequence.

[0231] In this embodiment, the action recognition prototype sequence library is established by standard action samples with action category labels and action start and end position labels. The action categories covered by the action recognition prototype sequence library are determined according to the target action category set of the action determination system. The specific determination method is as follows: read all action categories that the action determination system needs to output, add each action category to the target action category set one by one, and establish a corresponding action prototype sequence for each target action category.

[0232] The standard action samples are collected within the same human action area as the actual action determination stage; the camera placement, shooting range, sampling frequency, synchronization method, and time alignment method are consistent with the camera acquisition conditions in steps S101 and S202; each standard action sample is executed continuously from the starting posture of the corresponding action category to the ending posture, without interruption, retraction, or re-execution during the action.

[0233] The standard motion samples for each motion category cover different motion speeds and different human orientations. The motion speed is set according to the duration of the pre-collected samples of the same motion category. The specific setting method is as follows: the duration is arranged in ascending order, and the lower quartile and upper quartile are calculated. Samples with a duration not greater than the lower quartile are determined as faster motion samples, samples with a duration not less than the upper quartile are determined as slower motion samples, and the remaining samples are determined as normal motion samples.

[0234] Action category labeling and action start and end position labeling are performed by two labelers respectively; the first acquisition time when the action begins to leave the corresponding category's starting posture is determined as the action start position, and the last acquisition time before the human body reaches the corresponding category's ending posture and remains stable is determined as the action end position; when the action category labels of the two labelers are consistent and the difference between the start and end positions is not greater than the time alignment tolerance in step S101, the corresponding standard action sample is retained; when the labeling results are inconsistent, a third labeler reviews them by combining continuous human action videos and human key point timing records.

[0235] Each standard action sample is processed according to steps S101 to S104 to obtain a standard action atomic sequence, and grouped according to the action category; the number of action atomic nodes contained in each standard action atomic sequence in the same action category is counted, and the number of nodes is arranged in ascending order; when the number of samples is odd, the number of nodes in the middle position is read; when the number of samples is even, the number of the two middle nodes is added together and divided by 2, and the nearest integer of the result is taken to obtain the reference number of nodes for that action category.

[0236] The number of reference nodes is determined for different action categories, and different action categories are not required to have the same number of nodes. The action prototype sequence of the same action category is preferentially selected from the standard action atomic sequence with the number of nodes equal to the number of reference nodes. If there is no standard action atomic sequence with the same number of nodes, the standard action atomic sequence with the smallest absolute value of the difference between the number of nodes and the number of reference nodes is selected.

[0237] The initial size of the standard action samples is set based on the number of reference nodes and the stability of the prototype sequence. The specific setting method is as follows: for each action category, at least 20 complete standard action samples are collected first, and it is ensured that normal actions, fast actions, and slow actions all have corresponding samples. After reaching the initial size, for each additional batch of standard action samples, the candidate standard action atomic sequence with the minimum number of reference nodes and average matching cost is recalculated. After three consecutive additions of samples, when the number of reference nodes and the selected candidate standard action atomic sequences remain unchanged, the addition of samples is stopped.

[0238] For any two standard action atom sequences in the same action category, the matching cost between action atoms is determined according to the differences in participating limbs, movement direction, posture change, and local sequence; the calculation method and weight setting method of each difference are the same as in step S202.

[0239] Dynamic sequence alignment is performed based on the action atom order of two standard action atom sequences. For any alignment result, the matching costs between each matching action atom are added to obtain the total matching cost. Then, the total matching cost is divided by the number of matching action atoms to obtain the sequence matching cost between the two standard action atom sequences. When the number of matching action atoms is zero, the corresponding sequence combination is excluded.

[0240] For any candidate standard action atomic sequence in the same action category, add its sequence matching cost with other candidate standard action atomic sequences in the same group, and then divide by the number of other candidate standard action atomic sequences in the same group to obtain the average matching cost.

[0241] Among the candidate standard action atomic sequences whose number of nodes meets the reference node number requirement for the action category, the standard action atomic sequence with the smallest average matching cost is determined as the action prototype sequence for the corresponding action category; when there are the same minimum average matching costs, the average of the observation confidence of all action atoms in each candidate standard action atomic sequence is calculated, and the standard action atomic sequence with the higher average value is selected.

[0242] Based on the local order of each action atom in the action prototype sequence, the node order of the action prototype nodes is determined sequentially from the start position to the end position of the action; the action category, action prototype sequence, number of reference nodes, node order, participating limbs and direction of movement are associated to obtain the action recognition prototype sequence library.

[0243] Step S402: Calculate the matching cost between the fused action atomic sequence and the action prototype sequence corresponding to each action category to obtain the candidate action matching record.

[0244] In this embodiment, the participating limbs, movement direction, posture change features, duration, and local order of the fused action atom are read, and the same type of features corresponding to the action prototype node are also read.

[0245] The differences in participating limbs, movement direction, and posture change are determined according to the calculation method in step S202; the absolute value of the difference between the duration of the fused action atom and the duration of the action prototype node is calculated, and then divided by the larger of the two durations to obtain the duration difference; when both durations are zero, the duration difference is recorded as zero.

[0246] The local order of the fusion action atoms and the order of the action prototype nodes are normalized from zero to one, and then the absolute value of the difference between the two normalized orders is calculated to obtain the order difference.

[0247] The weights of each difference are set based on standard action samples with correct and incorrect category labels. The specific setting method is as follows: calculate the average value of correct and incorrect category matches for each difference, calculate the absolute value of the difference between the two average values ​​to obtain the category discrimination of each difference; divide each category discrimination by the sum of all category discriminations to obtain the corresponding weight; when the sum of all category discriminations is zero, set the same weight for each difference.

[0248] Multiply each difference by its corresponding weight, and then sum all the products to obtain the matching cost between the fused action atom and the action prototype node. Arrange the matching costs according to the order of the fused action atom and the order of the action prototype node to form a matching cost matrix, and obtain the candidate action matching record.

[0249] Step S403: Based on the node order and action atom connection relationship of the action prototype sequence, perform overall matching and semantic chain verification on the candidate action matching records to obtain the target action matching sequence.

[0250] In this embodiment, based on the candidate action matching record, dynamic sequence alignment is performed on the fused action atoms and action prototype nodes according to the node order of the action prototype sequence; the sequence alignment includes action atom matching, fused action atom skipping, and action prototype node skipping.

[0251] The cost of action atom matching is the corresponding matching cost in the candidate action matching record; the fusion action atom skip cost is the observation confidence of the corresponding action atom; the action atom observation confidence is used as the skip cost, so that action atoms with lower observation confidence have a smaller skip cost, and action atoms with higher observation confidence have a larger skip cost.

[0252] The cost of skipping an action prototype node is set based on the proportion of the node's occurrence in standard action samples of the same category. The specific setting method is as follows: count the number of standard action samples that contain the local action corresponding to the action prototype node, divide the number by the total number of standard action samples of the same category to obtain the node's occurrence proportion, and determine the node's occurrence proportion as the cost of skipping the action prototype node; the higher the occurrence proportion of the prototype node, the less likely it is to be skipped easily.

[0253] Starting from the first node of the fused action atom sequence and action prototype sequence, calculate the sum of the previous cumulative cost and the current cost for the three alignment methods: action atom matching, fused action atom skipping, and action prototype node skipping. Select the result with the smallest value as the cumulative cost of the current position and record the corresponding previous position. Process all fused action atoms and action prototype nodes one by one, and then trace back in reverse according to the recorded previous position to obtain the candidate overall matching sequence.

[0254] Semantic chain verification is performed based on the action atom succession relationship; when adjacent fused action atoms in the candidate overall matching sequence have an action atom succession relationship and the node order of the corresponding action prototype node increases, the corresponding matching relationship is retained; when adjacent fused action atoms do not have a succession relationship, the candidate overall matching sequence is broken between the two fused action atoms, and a new matching sequence is formed from the latter fused action atom.

[0255] For any validated candidate overall matching sequence, the cumulative matching cost is divided by the number of alignment operations to obtain the average matching cost; the result of subtracting the average matching cost from one is restricted to the range of zero to one to obtain the overall matching similarity.

[0256] Divide the number of matched action prototype nodes by the total number of action prototype nodes to obtain the action prototype node coverage; count the number of adjacent matched action atoms that satisfy the action atom inheritance relationship, and divide the number by the total number of adjacent matched action atoms to obtain the action atom inheritance completeness; when the total number of adjacent matched action atoms is zero, the action atom inheritance completeness is recorded as zero.

[0257] The overall matching similarity, action prototype node coverage, and action atom continuity completeness are added together and then divided by 3 to obtain the candidate overall matching score; the matching sequence with the highest candidate overall matching score is selected to obtain the target action matching sequence.

[0258] Step S404: Based on the target action matching sequence, perform statistical analysis of the action prototype matching results to obtain the action determination result.

[0259] In this embodiment, the action category, fused action atom, action prototype node, candidate overall matching score, and action prototype node coverage corresponding to the target action matching sequence are read.

[0260] The action completion threshold is set based on the coverage of action prototype nodes for complete action samples and incomplete action samples. The complete action sample is a labeled sample that has fully executed the corresponding action prototype sequence, and the incomplete action sample is a labeled sample that has been interrupted or rolled back before the end of the action prototype sequence. The specific setting method is as follows: the coverage of action prototype nodes for complete action samples and incomplete action samples is calculated separately, the two sets of coverage are arranged in ascending order, the lower quartile of the coverage of complete action samples and the upper quartile of the coverage of incomplete action samples are read, the two values ​​are added together and divided by 2 to obtain the action completion threshold.

[0261] If the coverage of the action prototype node of the target action matching sequence is not less than the action completion threshold, and the target action matching sequence covers the end node in the action prototype sequence, the action completion status is determined to be completed; otherwise, it is determined to be incomplete.

[0262] Based on the action atom succession relationship, adjacent target action matching sequences that do not have a succession relationship are divided into different action occurrence processes. The number of completed target action matching sequences corresponding to each action category is counted to obtain the number of action occurrences.

[0263] When the same action occurrence process forms a target action matching sequence with multiple action categories, the candidate overall matching scores of each target action matching sequence are compared, and the category corresponding to the action prototype sequence with the highest score is determined as the action category; the action completion state, the number of times the action occurs, and the action category are associated to obtain the action judgment result.

[0264] Example 2: See Figure 3 This embodiment provides a multi-camera motion fusion and determination system, including: a video acquisition module, an optimal transmission module, a motion fusion module, and a motion determination module.

[0265] The video acquisition module is used to acquire continuous human motion videos captured by multiple cameras, determine the transition positions of the motion states, divide the motion attempts, and obtain single-view motion atom sequences, motion attempt boundary node sets, and motion atom observation confidence levels.

[0266] The optimal transmission module performs optimal transmission of cross-camera action atom sequences and retraction residual feedback correction based on single-view action atom sequences and action attempt boundary node sets, thereby obtaining the retraction feedback correction transmission relationship.

[0267] The action fusion module constructs an action attempt attribution graph based on the action atom observation confidence and the pullback feedback correction transmission relationship, and performs propagation neighborhood reconstruction and cross-perspective complementary attention fusion to obtain the fused action atom sequence and action atom inheritance relationship.

[0268] The action determination module performs overall matching of action prototypes and semantic chain verification based on the fusion of action atomic sequences and action atomic succession relationships to obtain the action determination result.

[0269] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0270] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0271] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining motion fusion from multiple cameras, characterized in that, include: Acquire continuous human motion videos captured by multiple cameras, determine the transition points of motion states, and classify motion attempts to obtain single-view motion atom sequences, motion attempt boundary node sets, and motion atom observation confidence scores. Based on the single-view action atom sequence and action attempt boundary node set, optimal transmission of cross-camera action atom parts and correction of retraction residual feedback are performed to obtain the retraction feedback correction transmission relationship. Based on the action atom observation confidence and the pullback feedback to correct the transmission relationship, an action attempt attribution graph is constructed, and the propagation neighborhood is reconstructed and cross-perspective complementary attention is fused to obtain the fused action atom sequence and action atom inheritance relationship; Based on the fusion of action atomic sequences and action atomic succession relationships, overall matching of action prototypes and semantic chain verification are performed to obtain action determination results.

2. The multi-camera motion fusion determination method according to claim 1, characterized in that, The process of acquiring continuous human motion videos captured by multiple cameras, determining the transition points of motion states, and dividing motion attempts into single-view motion atom sequences, motion attempt boundary node sets, and motion atom observation confidence scores includes: Acquire continuous human motion videos captured by multiple cameras, perform human keypoint detection, and arrange the detection results to obtain a temporal record of human keypoints and a keypoint detection confidence record. Calculate posture changes from the time-series recordings of key human body points to determine the transition points of movement states; Based on the transition position of the action state, set the action attempt boundary to obtain the action attempt boundary node set; Action attempts are divided according to the action attempt boundary node set, and attitude change segmentation and observation reliability calculation are performed in combination with the key point detection confidence record to obtain single-view action atom sequence and action atom observation confidence.

3. The multi-camera motion fusion determination method according to claim 1, characterized in that, The method involves performing optimal transmission of cross-camera action atom sequences and action attempt boundary node sets based on single-view action atom sequences and correcting residual feedback during retraction, resulting in a retraction feedback correction transmission relationship, including: The single-view action atom sequence and action attempt boundary node set are associated, and the action process features of the action atoms in the corresponding action attempts of each camera are collected to obtain the action attempt feature record. The action process features include the order of participating limbs, the sequence of movement directions, and the local order. Based on the action attempt feature record, cross-camera candidate action attempts are filtered, the matching cost of each action atom in the cross-camera candidate action attempts is determined, and partial optimal transmission is performed to obtain the initial cross-view transmission relationship and untransmitted quality. Based on the untransmitted quality, residual attribute discrimination and atomic chain connection of the retraction action are performed to obtain the retraction residual chain; Based on the pullback residual chain and the initial cross-view transmission relationship, cross-collision transmission quality backoff and redistribution are performed to obtain the pullback feedback correction transmission relationship.

4. The multi-camera motion fusion determination method according to claim 3, characterized in that, The step of performing residual attribute discrimination and atomic chain connection of the retraction action based on the untransmitted quality to obtain the retraction residual chain includes: Residual discrimination features are extracted from the action atoms corresponding to the untransmitted quality to obtain residual discrimination feature records. The residual discrimination features include the motion direction after direction reversal and the key point observation quality features. The residual score record is obtained by calculating the retraction residual score and occlusion residual score of each action atom using the residual discrimination feature record. Compare the retraction residual score and occlusion residual score of each action atom in the residual score record to determine the retraction residual mass, and extract the direction of motion after the direction of the corresponding action atom is reversed to obtain the retraction direction record; Based on the retraction residual mass and retraction direction records, action atoms with the same participating limbs, continuous local sequence, and consistent retraction direction are connected within the same action attempt corresponding to the same camera to obtain the retraction residual chain.

5. The multi-camera motion fusion determination method according to claim 3, characterized in that, The process of performing cross-collision transmission quality backoff and reallocation based on the pullback residual chain and the initial cross-view transmission relationship to obtain the pullback feedback correction transmission relationship includes: Tracing the preceding action atoms corresponding to the retracted residual chain along the initial cross-view transmission relationship, and determining the preceding association features between the preceding action atoms and the retracted residual chain, the preceding influence feature record is obtained; The reverse influence ratio of the preceding action atom is determined by combining the preceding influence feature record, and the preceding transmission influence record is obtained; Based on the preceding transmission impact record, cross-period conflict determination and transmission quality backtracking are performed on the initial cross-view transmission relationship to obtain the backtracked transmission relationship and backtracked transmission quality. The rollback transmission quality is limited to the corresponding cross-camera candidate action attempts. Compatible action atoms are selected according to the transmission relationship after rollback, and partial optimal transmission is performed to obtain the rollback feedback correction transmission relationship.

6. The multi-camera motion fusion determination method according to claim 1, characterized in that, The process involves constructing an action attempt attribution graph based on action atom observation confidence and pullback feedback to correct transmission relationships, followed by propagation neighborhood reconstruction and cross-perspective complementary attention fusion to obtain a fused action atom sequence and action atom inheritance relationships, including: Using action atoms in a single-view action atom sequence as nodes, and based on the arrangement order of action atoms in the corresponding sequence, the action attempt boundary node set, and the pullback feedback correction transmission relationship, action continuation edges, action reset edges, and cross-view complementary edges are established respectively to construct an action attempt attribution graph. The initialization action attempts to assign the continuation state and reset state of each edge in the graph, and performs propagation neighborhood reconstruction and graph attention propagation to obtain the updated propagation neighborhoods of each layer; Based on the updated propagation neighborhood and action atom observation confidence of each layer, dual-channel attention calculation and cross-view complementary weight correction are performed to obtain the graph attention propagation results. Based on the graph attention propagation results, the action atom nodes and edges in the action attempt at belonging graph are merged and updated to obtain the merged action atom sequence and action atom inheritance relationship.

7. The multi-camera motion fusion determination method according to claim 6, characterized in that, The initialization action attempts to assign the continuation and reset states of each edge in the graph, and performs propagation neighborhood reconstruction and graph attention propagation to obtain updated propagation neighborhoods at each layer, including: Determine the initial continuation state and initial reset state of the action continuation edge, action reset edge, and cross-view complementary edge in the action attempt at the graph, and obtain the initial edge state record; Based on the initial edge state record, the edges that are dominant in the continuation state are determined and included in the action continuation propagation neighborhood, and the edges that are dominant in the reset state are determined and included in the action reset propagation neighborhood, thus obtaining the first-layer propagation neighborhood. Graph attention propagation is performed on the first-layer propagation neighborhood to generate first-layer action reset information, and the continuation state and reset state of each edge are updated to obtain the second-layer edge state record. Starting with the edge state records of the second layer, the current layer's propagation neighborhood is reconstructed layer by layer using the current layer's edge state records. Graph attention propagation is then performed, and the next layer's edge state is updated based on the action reset information generated by the propagation, resulting in the updated propagation neighborhoods of each layer.

8. The multi-camera motion fusion determination method according to claim 6, characterized in that, The graph attention propagation results are obtained by performing dual-channel attention calculation and cross-view complementary weight correction based on the updated propagation neighborhood and action atom observation confidence of each layer, including: Graph attention is calculated for the updated action continuation propagation neighborhood and action reset propagation neighborhood of each layer to obtain the continuation channel edge attention record and the reset channel edge attention record. Extract the participating limbs and posture changes of the action atoms at both ends of the cross-view complementary edge, and compare the cross-view differences of the corresponding features to obtain the degree of cross-view action complementarity. Determine the observation confidence of the action atoms at both ends of the cross-view complementary edge, and calculate the attention compensation weight by combining the degree of cross-view action complementation; Extract the initial attention weights corresponding to the cross-view complementary edges from the continuous channel edge attention records and the reset channel edge attention records; The initial attention weights corresponding to the cross-view complementary edges in the corresponding channels are corrected by adjusting the attention compensation weights, and the attention records of the continuation channel edges and the attention records of the reset channel edges are competitively aggregated to obtain the graph attention propagation results.

9. The multi-camera motion fusion determination method according to claim 1, characterized in that, The process of performing overall matching of action prototypes and semantic chain verification based on the fused action atom sequence and action atom succession relationship to obtain action determination results includes: Obtain the action recognition prototype sequence library, extract the action prototype sequence corresponding to each action category, and determine the node order of the action prototype sequence; Calculate the matching cost between the fused action atomic sequence and the action prototype sequence corresponding to each action category to obtain candidate action matching records; Based on the node order and action atom succession relationship of the action prototype sequence, the candidate action matching records are matched as a whole and the semantic chain is verified to obtain the target action matching sequence. Based on the target action matching sequence, the action prototype matching results are statistically analyzed to obtain the action judgment result.

10. A multi-camera motion fusion determination system, used to implement the multi-camera motion fusion determination method according to any one of claims 1 to 9, characterized in that, include: The video acquisition module is used to acquire continuous human motion videos captured by multiple cameras, determine the transition points of the motion state, divide the motion attempts, and obtain the single-view motion atom sequence, the motion attempt boundary node set, and the motion atom observation confidence. The optimal transmission module, based on the single-view action atom sequence and action attempt boundary node set, performs optimal transmission of cross-camera action atom parts and correction of retraction residual feedback, and obtains the retraction feedback correction transmission relationship. The action fusion module constructs an action attempt attribution graph based on the action atom observation confidence and the pullback feedback to correct the transmission relationship. It then performs propagation neighborhood reconstruction and cross-perspective complementary attention fusion to obtain the fused action atom sequence and action atom inheritance relationship. The action determination module performs overall matching of action prototypes and semantic chain verification based on the fusion of action atomic sequences and action atomic succession relationships to obtain the action determination result.

Images