A new-born baby limb movement monitoring method based on a multi-task classification network

Abstract

The application relates to the technical field of image recognition, and discloses a newborn limb action monitoring method based on a multi-task classification network. In the application, event markers are first converted into candidate segment constraints, and then training samples are shunted according to visible limb movement evidence, background synchronous change and nursing residual change, so that event records are no longer directly equivalent to training labels of the multi-task classification network. Through branch training weight control of action classification, posture detection and amplitude measurement branches, the pollution of occluded frames, static frames and background disturbance frames to different task branches can be reduced, the model learning object is more concentrated on real observable limb action evidence, and meanwhile, the same gating logic is reused to generate output credibility in the real-time monitoring stage, low visual evidence results and suspected nursing residual results are limited in output and shunted for writing back, so that action boundary deviation caused by changes of ward camera visual angle, bed environment and labeling mode is reduced.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method for monitoring newborn limb movements based on a multi-task classification network. Background Technology

[0002] Neonatal limb movement monitoring typically relies on bedside cameras, incubator observation window cameras, or video-EEG synchronous acquisition devices to obtain continuous monitoring video. Compared with ordinary movement recognition scenarios, newborns' limbs are short, their distal movements are weaker, and the duration of their movements is shorter. The video acquisition process is also affected by nursing operations, changes in occlusion, and the way the equipment is labeled. For example, during a nighttime monitoring session, a newborn's lower limbs first exhibited short, sharp tremors. Then, a caregiver entered the frame to tidy the swaddle and secure the catheter. The event button was only triggered after the caregiver observed the abnormal reaction. In this case, the event time period recorded by the monitoring system may cover a long video range before and after the tremors occurred. The first part may only show slight visible movements, the middle part may show occlusion by the caregiver, and the latter part may have returned to stillness. For this type of video, although the event label can reflect the range of a monitoring event, not every frame contains limb movement evidence that can be learned by the image recognition model. There is a tendency for the start and end points of the actually visible movements to deviate from the event time period formed by the recording system.

[0003] Existing methods for monitoring neonatal movement typically employ video frame detection, keypoint tracking, posture estimation, inter-frame motion analysis, or classification networks to identify limb movements. Some methods also simultaneously output movement categories, abnormal postures, and movement amplitudes from the same video input to obtain multi-dimensional monitoring results. These methods can support movement recognition and monitoring prompts when video segment boundaries are clear and event markers are largely consistent with visible movements. However, in the process of processing real-world monitoring data, event markers may originate from nursing records, manual trigger times, or markers from synchronous monitoring devices. Their time range is not always completely consistent with the visible limb movements in the video. When this time range is used as the training label for a multi-task classification network without correction for visible movement evidence, the same annotation window may simultaneously contain frames of real movement, frames obscured by nursing care, frames with no obvious movement, and frames with slight background disturbances. In this case, the movement classification branch is easily affected by non-limb movement image features, the posture detection branch may be influenced by occlusion, and the movement amplitude quantification branch may weaken the boundaries of real movements due to the increased proportion of frames without movement. Consequently, after changing the ward camera, bed viewpoint, or annotation process, the model may exhibit abnormal movement boundary response shifts, affecting the stability of movement evidence output in continuous visual monitoring of newborns. Summary of the Invention

[0004] This application proposes a method for monitoring neonatal limb movements based on a multi-task classification network to address the problems mentioned in the background art.

[0005] To achieve the above objectives, this application adopts the following technical solution: a method for monitoring neonatal limb movements based on a multi-task classification network, comprising the following steps:

[0006] Acquire neonatal monitoring annotation data, perform time base correction on the event markers in the neonatal monitoring annotation data and historical monitoring video frames, and correct the corrected event markers according to the marker delay range corresponding to the event source, and generate a label correction record. The label correction record is used to limit the candidate segments in the historical monitoring video that correspond to the event markers.

[0007] Candidate segments are called, and a target detection network is used to extract candidate limb regions in the newborn's body region. Background motion suppression and continuity screening are performed on the inter-frame displacement of the candidate limb regions to generate visible action segments. Image changes that meet at least one of the following conditions are written into the excluded segment record.

[0008] Branch admission processing is used to allocate training windows for visible action segments and excluded segment records, generate branch training weights, and write candidate segments with event labels but no visible action segments into the low visual evidence sample set. The branch training weights are used to control the training contribution of the corresponding task branch in the multi-task classification network.

[0009] The loss function of the multi-task classification network is constrained by the branch training weights to train a newborn limb movement monitoring model. The gating rules corresponding to the branch admission process are reused in the scrolling window of the real-time monitoring video to generate output credibility. The newborn limb movement monitoring results are output based on the output credibility.

[0010] Furthermore, time base correction includes:

[0011] A video timeline is established based on the frame timestamps of historical monitoring video frames. A time base offset of the event timeline relative to the video timeline is generated using a synchronization reference point that simultaneously has event time records and video frame positioning information. The time position of the event marker is corrected based on the time base offset and mapped to an anchor frame.

[0012] Furthermore, generating label correction records includes:

[0013] According to the marked delay range corresponding to the event source, candidate segments are extracted based on the anchor frame, and a label correction record is generated based on the effective boundary of the candidate segment and the source confidence. The marked delay range is formed by the delay samples of historical events from the same source. When the delay samples do not meet the conditions for forming the delay range, the preset marked delay range of the corresponding event source is adopted and the source confidence is reduced.

[0014] Furthermore, the candidate limb regions to be extracted include:

[0015] A target detection network is run on historical monitoring video frames within the candidate segments to obtain the detection regions of the newborn's body and limbs. The detected regions are then mapped back to the original pixel coordinates of the historical monitoring video frames. Based on the tracking results of adjacent frames, candidate limb regions are generated within the newborn's body region using the original pixel coordinates.

[0016] Furthermore, generating visible action fragments includes:

[0017] Inter-frame displacement is generated for candidate limb regions that have been tracked and matched by adjacent frames. The inter-frame displacement is subtracted from the background global motion of the background reference region to obtain the residual displacement. Image changes that meet at least one of the background synchronization condition and nursing residual condition are written into the exclusion segment record. Visible motion segments are generated for consecutive frames that are not written into the exclusion segment record and whose residual displacement reaches the displacement threshold.

[0018] Furthermore, the training window allocation includes:

[0019] Based on the same candidate segment source index and video frame timestamp, visible action segments and excluded segment records are matched. A training window is generated using the start and end frames of the visible action segments. The frame-level intersection between the training window and the excluded segment records is calculated to obtain the exclusion coverage. The corresponding task branch admission judgment is then performed on the training window based on the exclusion coverage.

[0020] Furthermore, the weights for generating branch training include:

[0021] Based on the admission judgment results, source confidence, training evidence ratio and exclusion coverage of each task branch, the branch training weights of each task branch are generated; for candidate segments with event markers but no visible action segments, the candidate segments are written into the low visual evidence sample set, and the branch training weights of each task branch corresponding to the candidate segment are reset to 0.

[0022] Furthermore, the loss function for a multi-task classification network, constrained by branch training weights, includes:

[0023] Read the branch admission record formed by the branch admission process, input the training window into the multi-task classification network, and obtain the branch loss value of each task branch;

[0024] The branch loss value of the corresponding task branch is weighted according to the branch training weight in the branch admission record to generate a weighted loss value;

[0025] In this case, the training window with a branch training weight of 0 does not contribute to the loss of the corresponding task branch.

[0026] Furthermore, within the scrolling window of the real-time monitoring video, the reliability of the output generated by reusing the gating rules corresponding to the branch admission process includes:

[0027] A closed scrolling window is formed based on the received video frames. The closed scrolling window is then input into the neonatal limb movement monitoring model to obtain the model output results.

[0028] Within a closed scrolling window, the gating rules corresponding to the branch admission process are reused to perform gating calculations on the model output, the visibility of candidate limb regions, and the exclusion coverage, thereby generating output credibility.

[0029] When the exclusion coverage of the closed scrolling window reaches the exclusion coverage admission threshold, the output confidence is set to 0.

[0030] Furthermore, based on the output reliability, the monitoring results of newborn limb movements are output, including:

[0031] When the output confidence level reaches the confidence output threshold, the newborn limb movement monitoring results are output.

[0032] When the output confidence level does not reach the confidence output threshold, a low confidence warning is output.

[0033] When the output credibility reaches the credibility output threshold for automatic writing back of training samples and the low visual evidence label is not valid, the corresponding closed scrolling window is written into the training sample pool to be reviewed.

[0034] If any of the following conditions exist: the output credibility does not reach the credibility output threshold for automatic writing back of training samples, or the low visual evidence label is valid, the corresponding closed scrolling window will be written into the running record.

[0035] The beneficial effects of this invention are as follows:

[0036] This invention addresses the inconsistency between event markers and actual visible limb movement boundaries in neonatal monitoring videos. It first converts event markers into candidate segment constraints, then triages training samples based on visible limb movement evidence, synchronous background changes, and changes in nursing residue. This prevents event records from being directly equivalent to training labels in a multi-task classification network. By controlling the training contributions of the action classification, posture detection, and amplitude measurement branches through branch training weights, it reduces the contamination of different task branches by occluded frames, still frames, and background disturbance frames, allowing the model to focus more on real, observable limb movement evidence. Simultaneously, the same gating logic is reused during real-time monitoring to generate output credibility, restricting and triaging low-visual-evidence results and suspected nursing residue results, thereby reducing action boundary shifts caused by changes in ward camera perspective, bed environment, and annotation methods. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort:

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a schematic diagram of the event marker time base correction of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating the visible action segment extraction and exclusion segment recording of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] like Figure 1 , Figure 2 and Figure 3 As shown, this invention discloses a method for monitoring neonatal limb movements based on a multi-task classification network, including the following specific steps:

[0043] S1. Obtain neonatal monitoring annotation data, perform time base correction on the event markers in the neonatal monitoring annotation data and historical monitoring video frames, and correct the corrected event markers according to the marker delay range corresponding to the event source, and generate a label correction record.

[0044] In this embodiment, step S1 is used to pre-train the labeled data of historical monitoring videos, converting event markers into temporal boundary constraints that can be used in subsequent image processing. The neonatal monitoring labeled data includes historical monitoring videos, event markers, video frame timestamps, and event marker timestamps. The system acquires historical monitoring videos through at least one of a video acquisition device and a monitoring video storage system, obtains event markers and their timestamps through an event recording system, and reads video frame timestamps from the frame-level metadata of the historical monitoring videos. In this step, event markers are used as temporal constraint objects and are not directly used as positive samples of abnormal actions and negative samples of normal actions in the multi-task classification network.

[0045] The system establishes a video timeline based on the frame timestamps of historical monitoring video frames and performs time base correction between event markers and historical monitoring video frames. For the same event source, the system extracts synchronization reference points from the neonatal monitoring annotation data. Synchronization reference points must simultaneously possess event time records and video frame location information, and must point to the same recording object on both the event timeline and the video timeline. Synchronization reference points include synchronization pulses written by the synchronization monitoring device, acquisition start markers that simultaneously enter event recording and video recording, and nursing record switching points with video frame location information. Records with only unilateral time information are not included in time base correction.

[0046] The system generates a time base offset of the event timeline relative to the video timeline based on synchronization reference points. Specifically, the system calculates the difference between the frame timestamp of each synchronization reference point on the video timeline and its event time record on the event timeline to obtain the reference offset.

[0047] When the same event source has two valid synchronization reference points within the current recording segment, the system compares the difference between the two reference offsets. If the difference does not exceed the allowable residual range, the average of the two reference offsets is taken as the time base offset of the current recording segment within the maximum allowable time base offset range for the event source; if the difference exceeds the allowable residual range, no time base offset is generated for the current recording segment, and the system proceeds to the initial clock offset backoff process.

[0048] When there are three or more valid synchronization reference points, the system first takes the median value of the reference offset as the initial offset, then calculates the time residual of each synchronization reference point after the initial offset correction, and removes synchronization reference points whose time residuals exceed the allowable residual range. The system then regenerates the time base offset using the retained synchronization reference points. If the number of removed synchronization reference points is less than two, the system does not use the robust offset result and proceeds to the initial clock offset back-off process.

[0049] The permissible residual range is jointly limited by the video sampling time resolution and the event source recording error. The video sampling time resolution is determined by the time interval between adjacent frames of historical monitoring videos. The upper limit of the event source recording error is set according to the event source during initial deployment. Specifically, the upper limit of the recording error for synchronous monitoring devices is preferably 0.1 seconds to 1 second, the upper limit of the recording error for triggered recordings is preferably 1 second to 5 seconds, and the upper limit of the recording error for nursing records is preferably 5 seconds to 30 seconds. When the upper limit of the event source recording error is less than the time interval between adjacent frames, the system uses the time interval between adjacent frames as the criterion for judging the permissible residual range and reduces the source confidence of the corresponding event source.

[0050] The maximum time base offset range is the search limit range corresponding to the event source, and is calibrated jointly by the device synchronization level, historical synchronization reference point residuals, and event recording method. During initial deployment, for records from synchronous monitoring devices with a synchronized clock with the video acquisition device, the maximum time base offset range is preferably 0.5 seconds to 5 seconds; for triggered records, the maximum time base offset range is preferably 3 seconds to 60 seconds; and for nursing records, the maximum time base offset range is preferably 10 seconds to 180 seconds. If the time base offsets generated by adjacent recording segments are all close to the boundary of the corresponding search limit range, the system recalibrates the search limit range corresponding to the event source in subsequent training batches and reduces the source confidence of the event markers in that batch. If there are fewer than two valid synchronization reference points in the current recording segment, but there is an initial clock offset registered by the device, the system uses this initial clock offset as a backoff value and reduces the source confidence of the corresponding event source; if there are fewer than two valid synchronization reference points and no initial clock offset, the system writes the corresponding event marker into the invalid marker record.

[0051] After obtaining the time base offset corresponding to the recording segment, the system corrects the time position of the event marker based on the time base offset, mapping the event marker timestamp to the video timeline to obtain the corrected event marker time. The system retrieves the frame timestamp closest to the corrected event marker time from the historical monitoring video frames and uses the historical monitoring video frame corresponding to that timestamp as the anchor frame. The anchor frame represents the corrected position of the event marker in the historical monitoring video, but does not represent the start and end frames of the visible limb movement.

[0052] After generating anchor frames, the system corrects the event labels according to the label delay range corresponding to the event source. The label delay range is used to limit the search range for visible limb actions appearing earlier or continuing later than the anchor frames. The label delay range is formed by delayed samples of historical homogeneous events. Historical homogeneous events refer to historical events with the same event source as the current event label that have already undergone label verification. Delay samples include start offset and end offset; the start offset represents the offset of the start frame of the visible action in the historical homogeneous event relative to the corrected event label time, and the end offset represents the offset of the end frame of the visible action in the historical homogeneous event relative to the corrected event label time. Delay samples come from existing verification samples and historical batch samples before the current training batch, and do not call the processing results of the current candidate segment in S2.

[0053] When the number of valid delayed samples from historical events of the same origin reaches the conditions for forming a delay range, the system uses asymmetric boundary estimation to generate forward and backward delay boundaries. The conditions for forming the delay range are jointly limited by the number of valid delayed samples, the verification status, and the consistency of events of the same origin. Preferably, the number of valid delayed samples is no less than 5, and each valid delayed sample has a traceable visible action start frame and a visible action end frame. The system statistically analyzes the start offsets in the delayed samples according to the direction in which they appear earlier than the anchor frame, and selects the forward delay boundary according to the forward coverage ratio corresponding to the event source. Similarly, the system statistically analyzes the end offsets in the delayed samples according to the direction in which they continue later than the anchor frame, and selects the backward delay boundary according to the backward coverage ratio corresponding to the event source.

[0054] For trigger records, the forward coverage ratio is preferably 0.80 to 0.95, and the backward coverage ratio is preferably 0.60 to 0.85; for nursing records, both the forward and backward coverage ratios are preferably 0.80 to 0.95; for records from synchronous monitoring devices, both the forward and backward coverage ratios are preferably 0.60 to 0.85. The coverage ratio is a limiting parameter for the candidate fragment search boundary, calibrated by a verification delay sample of historical homologous events. If the valid delay samples of historical homologous events do not meet the delay range formation conditions, the system does not perform coverage ratio estimation, adopts the preset marked delay range of the corresponding event source, and reduces the source confidence.

[0055] The preset marker delay range is set as a forward preset boundary and a backward preset boundary according to the event source. During initial deployment, the forward preset boundary for synchronous monitoring device recording is preferably 1 to 5 seconds, and the backward preset boundary is preferably 1 to 10 seconds; the forward preset boundary for trigger recording is preferably 10 to 60 seconds, and the backward preset boundary is preferably 3 to 20 seconds; the forward preset boundary for nursing records is preferably 30 to 180 seconds, and the backward preset boundary is preferably 30 to 180 seconds. The preset marker delay range is used for candidate fragment search when historical homologous event samples are insufficient; it does not represent the probability of action occurrence and is calibrated as historical review samples accumulate.

[0056] After determining the anchor frame and the marked delay range, the system extracts candidate segments based on the anchor frame. The system prioritizes finding candidate segment boundaries using the frame timestamps of historical monitoring video frames. When the historical monitoring video has a fixed frame rate and no dropped frames, the system converts the marked delay range into frame offsets according to the acquisition frame rate. The starting frame of a candidate segment is jointly defined by the anchor frame and the forward delay boundary. If the starting frame is earlier than the first frame of the historical monitoring video, the first frame is used as the starting frame. The ending frame of a candidate segment is jointly defined by the anchor frame and the backward delay boundary. If the ending frame is later than the last frame of the historical monitoring video, the last frame is used as the ending frame. If the starting frame is later than the ending frame, the system does not generate a candidate segment and writes the corresponding event flag to the invalid flag record; if both the starting and ending frames are valid, the system writes the candidate segment to the tag correction record.

[0057] For candidate segments generated from nursing records, the system sets an upper limit for the width of the candidate segment based on the preset tag delay range corresponding to the nursing record. The upper limit for the width of the candidate segment is preferably between 60 seconds and 300 seconds. When the width of the candidate segment exceeds the upper limit, the system performs amplitude limiting and shrinking on the candidate segment based on the anchor frame, so that the width of the narrowed candidate segment does not exceed the upper limit, and reduces the source confidence, writing a low-confidence valid state in the tag correction record.

[0058] The system further generates source confidence. Source confidence characterizes the reliability of event markers as constraints in video evidence search, with a value ranging from 0 to 1. The source confidence uses the basic reliability of the event source as its initial value and is reduced based on temporal offset stability and candidate segment width. The basic reliability of the event source is determined by the proportion of verified homologous events in historical batches prior to the current training batch that can form valid candidate segments; when the number of verified homologous events is less than 5, the basic reliability of the event source adopts the lower value in the preset range for the corresponding event source. During initial deployment, the preferred basic reliability of the event source for synchronous monitoring device records is 0.60 to 0.80, the preferred basic reliability of the event source for trigger records is 0.40 to 0.70, and the preferred basic reliability of the event source for nursing records is 0.30 to 0.60.

[0059] When the time base offset is located near the boundary of the maximum time base offset range, when initial clock offset backoff is used, when the delayed sample does not meet the delay range formation conditions, or when the candidate segment is narrowed due to exceeding the upper limit of the candidate segment width, the system deducts the basic reliability of the event source for each of these events. The preferred deduction value for each event is 0.05 to 0.20. If the value after deduction is lower than 0, it is recorded as 0; if it is higher than 1, it is recorded as 1. The value after deduction is used as the source confidence level.

[0060] The system compares the source confidence score with the low confidence threshold for the corresponding event source. The low confidence threshold is formed by the lowest value within the preset range of the event source's basic reliability; the low confidence threshold for synchronous monitoring device records is 0.60, for trigger records it is 0.40, and for nursing records it is 0.30. When the source confidence score is lower than the low confidence threshold, the corresponding label correction record enters a low confidence valid state. The source confidence score is only used as a source constraint for subsequent branch training weights.

[0061] After generating candidate segments and source confidence scores, the system writes a label correction record using the event marker as an index. The label correction record records at least the event source, anchor frame, candidate segment, candidate segment start frame, candidate segment end frame, source confidence score, and valid status. Valid statuses are written according to the priority of invalid, low-confidence valid, and eligible for subsequent processing. If the candidate segment boundary is invalid, or there are insufficient synchronization reference points and no initial clock offset, the valid status is marked as invalid, and this label correction record does not enter the visible action segment extraction process in S2. If the candidate segment boundary is valid but the source confidence score is lower than the low-confidence judgment value, or the candidate segment is narrowed due to exceeding the upper limit of the candidate segment width, the valid status is marked as low-confidence valid, and this label correction record enters the visible action segment extraction process in S2, limiting the training contribution of the corresponding sample in S3. If the candidate segment boundary is valid, the synchronization reference meets the correction conditions, and the source confidence score is not lower than the low-confidence judgment value, the valid status is marked as eligible for subsequent processing.

[0062] In this step, event tags are first mapped to a unified video timeline, and then candidate segments are formed according to the tag delay range corresponding to the event source. The label correction record output by this step provides candidate segment boundaries for S2 and a source confidence basis for S3, thereby avoiding using the entire time period covered by the event tags as training labels for the multi-task classification network.

[0063] S2, call the candidate segments, use the object detection network to extract the candidate limb regions in the newborn's body region, perform background motion suppression and continuity screening on the inter-frame displacement of the candidate limb regions, generate visible action segments, and write the image changes that meet at least one of the background synchronization condition and the nursing residue condition into the excluded segment record.

[0064] In this embodiment, S2 takes over the label correction record generated by S1 and extracts video segments with observable limb evidence from candidate segments, isolating background motion and residual changes in nursing procedures from the training samples. The system call is valid for label correction records that can proceed to subsequent processing and those with low confidence. It reads the candidate segment, the candidate segment start frame, the candidate segment end frame, the event source, the anchor frame, and the source confidence. The candidate segment limits the video processing scope of this step, the anchor frame preserves the correction position of the event marker within the candidate segment, and the source confidence participates in the branch training weight formation in S3. Label correction records with an invalid valid state do not proceed to this step; label correction records with a low confidence valid state can proceed to this step, and their corresponding results are marked in the subsequent source state.

[0065] The system reads historical monitoring video frames according to the start and end frames of candidate segments, and uses an object detection network to detect the newborn's body regions. The input to the object detection network is the historical monitoring video frame within the candidate segment, and the output is the newborn's body region, body region detection confidence, and limb end detection region. The object detection network is implemented using at least one of a single-stage object detection network and an instance segmentation network to provide image localization results for the newborn's body region and limb end detection regions. When the object detection network performs at least one of scaling, padding, and cropping on the input frame, the system restores the newborn's body region and limb end detection regions to the original pixel coordinates of the historical monitoring video frame based on the detection network input size, the original size of the historical monitoring video frame, the scaling ratio, the padding offset, and the cropping start point. Subsequent candidate limb region generation, adjacent frame tracking, inter-frame displacement calculation, background reference region selection, and nursing intervention region formation are all performed at the original pixel coordinates.

[0066] The target detection network can be implemented using at least one of the following: a single-stage target detection network trained with labeled samples of newborn body regions and limb extremity detection regions, and an instance segmentation network. The newborn body region labels in the training samples are used to form the body region detection results, and the limb extremity detection region labels are used to form the initial localization results of candidate limb regions. The target detection network can be trained using at least one general supervised training method selected from cross-entropy loss, bounding box regression loss, and mask segmentation loss. After training, the network is used for image localization processing of historical monitoring video frames within candidate segments.

[0067] The threshold and time range in this step are preferred amplitude limiting parameters. During initial setup, the system selects corresponding parameters based on video resolution, acquisition frame rate, target detection stability, background reference region stability, and candidate limb region tracking stability. In subsequent training batches, the system calibrates the corresponding parameters based on the visible action segments and excluded segments that have already been verified in the current training batch. If the number of verified samples is less than 5, the system does not perform adaptive calibration and uses conservative values ​​within the initial amplitude limiting range. If a candidate segment passes the admission test and forms a visible action segment by using parameter fallback values, then that visible action segment is written into the low-confidence source state.

[0068] The stability of object detection is characterized by the proportion of successfully detected body regions in the reviewed candidate segments; the stability of the background reference region is characterized by the number of effective background feature points in the background reference region, the proportion of inliers after random consistency processing, and the number of background sub-regions covered; the stability of candidate limb region tracking is characterized by the proportion of candidate limb region matching interruption, the proportion of overlapping area between adjacent frames, and the proportion of local tracking points retained. These metrics are derived from the records of visible action segments and excluded segments that have been reviewed in previous training batches, and the processing results of the current candidate segment in subsequent steps are not used to reverse-adjust the current threshold.

[0069] The threshold for body region detection confidence is preferably between 0.50 and 0.80. This threshold is used to determine whether the target detection result enters the candidate limb region truncation. Initially, the system selects this threshold based on the detection stability of the newborn body region annotation samples; in subsequent training batches, the system is calibrated based on the proportion of successful body region detections in the reviewed candidate segments. If the continuous duration for which the body region detection confidence within the same candidate segment is below the threshold reaches the short-term missing time limit for the body region, and the tracking results of adjacent frames cannot maintain the body region position within this continuous duration, the system writes the corresponding interval to a body region unresolvable state. The short-term missing time limit for the body region is preferably the larger of 0.20 seconds and the time interval between two adjacent frames, used to adapt to short-term detection jitter under different acquisition frame rates. If the body region detection confidence is temporarily below the threshold but the tracking results of adjacent frames can maintain the body region position, the system marks the corresponding frame as a low-confidence detection frame and continues to perform candidate limb region truncation.

[0070] The system generates candidate limb regions based on the original pixel coordinates, combining the newborn's body region, limb end detection region, and tracking results from adjacent frames. Specifically, the system first divides the newborn's body region into trunk, upper limb side region, and lower limb side region according to the contour direction and boundary position. Then, it searches for limb end detection regions, elongated connected regions, and locally motion regions that continuously appear in adjacent frames within the newborn's body region. Regions that can maintain at least one of the following states in adjacent frames: positional continuity, contour continuity, and end detection continuity, are written into the candidate limb regions. The candidate limb regions include image regions corresponding to the proximal upper limb, distal upper limb, proximal lower limb, and distal lower limb. This division serves as the region type of the candidate limb region and is not used as an action category label. For frames with swaddling occlusion, incubator edge occlusion, or locally invisible limbs, the system only retains image regions where at least one of the limb contour, limb end detection region, or limb connected region can be located in the image.

[0071] To ensure that inter-frame displacement originates from the same candidate limb region, the system performs adjacent frame tracking matching on the candidate limb regions. The system extracts local tracking points within the candidate limb region and uses at least one of optical flow tracking and feature point matching to obtain the corresponding positions of these local tracking points in adjacent frames. It then determines whether the limb regions belong to the same candidate limb region based on the overlap ratio of the candidate limb regions in adjacent frames and the angle between the motion directions of the local tracking points. If either the overlap ratio of the candidate limb regions in adjacent frames is lower than the overlap threshold or the angle between the motion directions of the local tracking points exceeds the direction change threshold, the system marks the corresponding frame pair as limb matching interrupted and stops calculating the inter-frame displacement of that frame pair. The overlap threshold is preferably between 0.30 and 0.50, and the direction change threshold is preferably between 60° and 90°.

[0072] When the video image is stable and has minimal occlusion, the overlap admission threshold is set close to the upper limit of the range, and the directional change threshold is set close to the lower limit of the range. When the distal limb area is small and the boundary jitter is significant, the overlap admission threshold is set close to the lower limit of the range, and the directional change threshold is set close to the upper limit of the range. If the proportion of limb matching interruptions in the same candidate segment reaches the matching interruption proportion threshold, the system marks the result corresponding to that candidate segment as a low-confidence source. The matching interruption proportion threshold is preferably between 0.20 and 0.40, and is calibrated based on the tracking stability of the candidate limb region and the verified visible action segments.

[0073] For candidate limb regions matched by tracking in adjacent frames, the system generates inter-frame displacement. The inter-frame displacement is calculated using the same candidate limb region in adjacent historical monitoring video frames. The system calculates the pixel movement of local tracking points between adjacent frames and takes the median of the local tracking point movement as the original pixel displacement of the candidate limb region. The system removes the original pixel displacement by the diagonal length of the newborn's body region in the same frame to obtain a body-scale normalized displacement. The original pixel displacement is used to eliminate pixel-level jitter, while the body-scale normalized displacement is used to adapt to different video resolutions and shooting distances; both are used in the judgment process.

[0074] The displacement admission criteria for candidate limb regions employ a dual-condition judgment. The lower limit of the original pixel displacement is preferably no less than 1 pixel per frame, set based on image compression noise and subpixel edge flicker amplitude. The body-scale normalized displacement range is preferably 0.005 to 0.020. The system selects a body-scale normalized displacement threshold within this range and uses both the original pixel displacement lower limit and the body-scale normalized displacement threshold as the displacement admission value. When the video capture resolution is lower than 640×480, the body-scale normalized displacement threshold is taken as a value closer to the upper limit of the range; when the video capture resolution is higher than 1280×720 and the candidate limb region boundaries are continuous and stable, the body-scale normalized displacement threshold is taken as a value closer to the lower limit of the range. If the current frame lacks the diagonal length of the newborn's body region, the system uses the diagonal length of the newborn's body region from adjacent valid frames as a backoff value; if the diagonal length of the newborn's body region does not exist in adjacent valid frames, the system marks the corresponding frame as a frame with incalculable displacement. Frames with incalculable displacement do not participate in the formation of action candidate frames and low-confidence action candidate frames. If the continuous duration of the non-calculable displacement frame reaches the short-term missing time limit of the body region, the system will mark the corresponding interval as a low-confidence source state.

[0075] Before performing background motion inhibition, the system forms a nursing intervention region within the candidate fragment. The nursing intervention region is formed by inter-frame variations of non-neonatal occlusion areas within the neonatal body region's circumferential neighborhood, the edge of a thin catheter, the swaddling edge, and the edge of nursing instruments. The nursing intervention region must meet at least one of the following conditions: adjacency to the boundary of the neonatal body region, or entry into the neonatal body region's circumferential neighborhood. The nursing intervention region is used for background reference region exclusion and nursing residue condition determination, and is not considered as the neonatal limb movement region.

[0076] The system generates a global background motion value based on a background reference region. This global background motion value is represented by a background motion model. During actual deduction, the system uses the background motion model to calculate the predicted background displacement at the center position of the candidate limb region. The background reference region is selected from the outside of the newborn's body region, including the mattress edge, incubator frame, fixed bed rails, and stable texture areas in the image, excluding the newborn's body region, nursing intervention areas, and large reflective areas. The system extracts stable feature points from the background reference region, obtains background feature point displacements by matching feature points from adjacent frames, and removes abnormal matching points through random consistency processing. Subsequently, it fits a two-dimensional translation transformation or a two-dimensional affine transformation based on the retained background feature point displacements, and calculates the predicted background displacement within the same adjacent frame period at the center position of the candidate limb region. Preferably, the number of effective feature points in the background reference region is no less than 10, and these effective feature points are distributed across no less than 2 background sub-regions; if this condition is not met, the system marks the background suppression state as invalid.

[0077] When background suppression is effective, the system subtracts the predicted background displacement from the original pixel displacement of the candidate limb region to generate the residual displacement of the candidate limb region. The residual displacement is then removed and its value is calculated by the diagonal length of the newborn's body region in the same frame to obtain the body-scale normalized residual displacement. This body-scale normalized residual displacement is compared with a body-scale normalized displacement threshold. When background suppression is ineffective, the system retains the original pixel displacement of the candidate limb region as the displacement to be reviewed; this displacement is only used for low-confidence continuity screening and not for background synchronization condition determination.

[0078] The system determines background synchronization conditions based on the original pixel displacement, predicted background displacement, and normalized residual displacement of the body scale. After obtaining the original pixel displacement, the system first marks frames whose original pixel displacement reaches the lower limit and whose candidate limb regions are visible as initial image change frames. These initial image change frames are used to form local detection intervals and perform exclusion judgments. The system expands forward and backward by 0.20 seconds from the continuous initial image change frames to form local detection intervals. The local detection intervals must not exceed the range defined by the candidate segment start frame and the candidate segment end frame.

[0079] When the angle between the original pixel displacement direction of the candidate limb region within a local detection interval and the predicted background displacement direction is no greater than 20° to 45°, and the body-scale normalized residual displacement is lower than the body-scale normalized displacement threshold, and the proportion of frames satisfying this condition to the total number of frames in the local detection interval reaches 0.70 to 0.90, the system determines that the local detection interval meets the background synchronization condition. The direction angle threshold and the frame ratio threshold are the preferred limiting parameters for background synchronization exclusion, and are calibrated based on video resolution, background reference area stability, camera installation stability, and the already verified exclusion segment records. Image changes that meet the background synchronization condition are written into the exclusion segment record.

[0080] The system further determines the nursing residue condition. The system uses the frame where the nursing intervention area and the newborn's body area last intersect as the withdrawal frame, and calculates the time interval after the withdrawal frame based on the video frame timestamp. If an image change occurs within 0.3 to 3 seconds after the withdrawal frame, and the proportion of visible frames in the candidate limb area to the total number of frames in the local detection interval is less than 0.50, and the main boundary of the image change propagates along any one of the following directions: the edge of the sheet, the direction of the catheter, or the edge of the swaddle, the system determines that the image change meets the nursing residue condition. The main boundary of the image change is determined by the edge connectivity direction in the original pixel coordinates; if the angle between the edge connectivity direction and any one of the following directions (sheet edge, catheter direction, swaddle edge) is no greater than 30° to 60°, then the image change is determined to propagate along the corresponding direction.

[0081] The nursing residue time range of 0.3 seconds to 3 seconds is selected based on the rebound duration of sheets, catheters, and swaddles after the nursing procedure is removed, and calibrated according to the records of previously reviewed and excluded segments. When there are no fewer than 5 previously reviewed and excluded nursing residue segments in the current training batch, the system counts the duration of continuous propagation of the residue boundary after the removal frame, and selects an upper limit of 0.3 seconds to 3 seconds to cover most of the reviewed nursing residue segments; when there are fewer than 5 reviewed and excluded nursing residue segments, the system uses 3 seconds as a conservative upper limit. The nursing residue boundary direction angle threshold of 30° to 60° is selected based on the stability of the propagation direction of the sheet edge, catheter direction, and swaddle edge.

[0082] After determining background motion suppression, background synchronization conditions, and nursing residue conditions, the system performs continuity screening on the body-scale normalized residual displacement of candidate limb regions. Continuity screening takes body-scale normalized residual displacement, displacement to be verified, visibility of candidate limb regions, excluded segment records, and video frame timestamps as input. The system first marks frames with body-scale normalized residual displacement reaching a body-scale normalized displacement threshold and candidate limb regions in a visible state as action candidate frames. When background suppression is ineffective, original pixel displacement reaches the lower limit of original pixel displacement, body-scale normalized displacement reaches the body-scale normalized displacement threshold, and candidate limb regions are in a visible state, the system marks the corresponding frames as low-confidence action candidate frames. The system then deletes frames already written into the excluded segment record; the deleted action candidate frames and low-confidence action candidate frames are used to form action candidate intervals. Excluded segment records have higher priority than action candidate frames and low-confidence action candidate frames; once a frame is written into the excluded segment record, it does not participate in the generation of visible action segments.

[0083] For the remaining action candidate frames and low-confidence action candidate frames, the system merges them into action candidate intervals according to the frame order continuity. When the frame gap between adjacent action candidate frames is no more than 2 frames and the time gap is no more than 0.12 seconds, the system merges them into the same action candidate interval; otherwise, the system splits them into different action candidate intervals. When the duration of an action candidate interval is no less than 0.12 seconds and meets the valid frame count admission criteria, it enters the visible action segment generation process. When the candidate segment frame rate is no less than 15fps, the number of valid action candidate frames is no less than 3 frames; when the candidate segment frame rate is less than 15fps, the number of valid action candidate frames is no less than 2 frames. The action candidate interval must also meet the requirement that the visible frame ratio of the candidate limb region is no less than 0.60 and it is not covered by excluded segment records. Action candidate intervals that do not meet the above conditions do not generate visible action segments. The action candidate frame time gap of 0.12 seconds, the action candidate interval duration of 0.12 seconds, and the action candidate frame gap of no more than 2 frames are all set based on the minimum continuous observation time and short-term detection loss tolerance for neonatal short limb movements.

[0084] For action candidate intervals that meet the continuity screening criteria, the system generates visible action segments. Visible action segments are indexed by candidate segments and record the start frame, end frame, corresponding candidate limb region, residual displacement sequence, percentage of visible frames in the candidate limb region, excluded segment association results, and source status. Source status includes normal source status and low-confidence source status. Visible action segments generated from candidate segments called by low-confidence valid label correction records, candidate segments with invalid background suppression, action candidate intervals containing a proportion of low-confidence detection frames reaching the low-confidence ratio threshold, action candidate intervals containing low-confidence action candidate frames, and candidate segments with a candidate limb region matching interruption ratio exceeding the matching interruption ratio threshold are written into the low-confidence source status. The low-confidence ratio threshold is preferably between 0.20 and 0.40, calibrated based on target detection stability and verified visible action segments.

[0085] Excluded segment records are indexed by candidate segments and include the start frame, end frame, exclusion type, corresponding candidate limb region, background motion model, predicted background displacement, residual displacement of the candidate limb region, nursing intervention area, and exclusion criteria. Exclusion types include background synchronization and nursing residue. Excluded segment records are not used as negative samples for normal movements or positive samples for abnormal movements. Excluded segment records participate in branch admission processing in S3, ensuring that changes in excluded images do not contribute to the training of the movement classification branch and amplitude measurement branch.

[0086] In this step, candidate segments are parsed into visible motion segments and excluded segment records. Visible motion segments provide a training window with observable limb evidence, while excluded segment records isolate background synchronous motion and nursing residual image changes, and are passed to S3 for branch admission processing.

[0087] S3 uses branch admission processing to allocate training windows for visible action segments and excluded segment records, generates branch training weights, and writes candidate segments with event labels but no visible action segments into the low visual evidence sample set. The branch training weights are used to control the training contribution of the corresponding task branch in the multi-task classification network.

[0088] In this embodiment, S3 receives the label correction record output by S1 and the visible action fragment and excluded fragment records output by S2, and is used to establish the admission relationship between the training window and the task branch before training the multi-task classification network. The multi-task classification network includes an action classification branch, a pose detection branch, and an amplitude measurement branch. The action classification branch is used to learn the action category corresponding to the visible action fragment, the pose detection branch is used to learn the positional state of the candidate limb region in the visible action fragment, and the amplitude measurement branch is used to learn the motion amplitude of the candidate limb region in the visible action fragment.

[0089] This step processes the visible motion segments and excluded segment records generated by S2. The label correction record provides the event source, anchor frame, source confidence, and validity status; the visible motion segment provides the start frame, end frame, corresponding candidate limb region, residual displacement sequence, percentage of visible frames for the candidate limb region, excluded segment association results, and source status; the excluded segment record provides the start frame, end frame, exclusion type, corresponding candidate limb region, background motion model, background predicted displacement, residual displacement of the candidate limb region, nursing intervention area, and exclusion criteria. The matching criteria between the training window, visible motion segments, and excluded segment records are the same candidate segment source index and video frame timestamp.

[0090] The system generates training windows using the start and end frames of visible action segments as core boundaries. The training window must not exceed the start and end frames of the corresponding candidate segment. When multiple visible action segments exist within the same candidate segment, the system generates training windows separately according to chronological order. If adjacent visible action segments have the same corresponding candidate limb region, the time interval between them does not exceed the time gap of the action candidate frames in S2, and the interval is not covered by excluded segment records, the system merges them into a single training window. If there is at least one of the following conditions between adjacent visible action segments: background synchronization coverage, nursing residue coverage, unresolvable body region state, or uncalcifiable displacement frames, the system does not perform training window merging.

[0091] For any training window, the system calculates the frame-level intersection between the training window and the excluded segment records based on the same candidate segment source index and video frame timestamp. The number of intersection frames is divided by the total number of frames in the training window to obtain the exclusion coverage rate. The exclusion coverage rate characterizes the degree to which the training window is covered by at least one change, either background synchronization change or nursing residue change, and its value ranges from 0 to 1. When the total number of frames in the training window is 0, the system sets the admission judgment result for each task branch corresponding to that training window to 0 and records the branch training weight for each corresponding task branch as 0.

[0092] The preferred admission threshold for exclusion coverage is between 0.10 and 0.30, which is the limiting range in branch admission processing. Initially, the system selects this admission threshold based on the candidate segment acquisition frame rate, the shortest duration of the action candidate interval in S2, and the training window length. When either the total number of frames in the training window is no more than 5 frames or the training window duration is no more than 0.20 seconds, the admission threshold is set to a value close to 0.10; when the total number of frames in the training window is no less than 10 frames and the training window duration is no less than 0.40 seconds, the admission threshold is set to a value close to 0.30; otherwise, the admission threshold is set to 0.20 or a value close to 0.20.

[0093] When there are at least 5 historical excluded segment records that have been reviewed before the current training batch, the system calculates the proportion of historical excluded segment records that still need to be excluded from training contributions after review. If this proportion is not less than 0.70, the admission threshold for exclusion coverage is selected in the range of 0.10 to 0.20; if this proportion is less than 0.40, the admission threshold for exclusion coverage is selected in the range of 0.20 to 0.30; if this proportion is between 0.40 and 0.70, the initial value is used. When there are fewer than 5 historical excluded segment records, the system does not perform adaptive calibration and uses the initial value formed based on the total number of frames in the training window and the duration of the training window. When the source state of the training window is a low-confidence source state, the admission threshold for exclusion coverage is not higher than 0.20. Historical excluded segment records are only used to calibrate the admission threshold for exclusion coverage and do not change the source index relationship between the training window, visible action segments, and excluded segment records.

[0094] When the exclusion coverage of the training window reaches the exclusion coverage admission threshold, the system sets the admission judgment result of each task branch corresponding to the training window to 0, records the branch training weight of each task branch corresponding to the training window as 0, and records the exclusion reason and corresponding exclusion type of the training window; the training window no longer performs the calculation of training evidence ratio and exclusion decay coefficient. When the exclusion coverage of the training window is lower than the exclusion coverage admission threshold, the system removes the frames covered by the excluded segments from the valid training frames, and uses the frames that are not removed as valid training frames. The valid training frames are used for subsequent task branch admission judgments.

[0095] The system performs task branch admission judgments on valid training frames. The admission criteria for the action classification branch are: the training window originates from visible action segments, the number of valid training frames meets the valid frame count admission criteria for the action candidate interval in S2, and the training window is not completely covered by excluded segment records. The admission criteria for the pose detection branch are: the proportion of visible frames in the candidate limb region within the training window is not lower than the visible frame proportion admission threshold in S2, and the corresponding candidate limb region does not reach the matching interruption ratio threshold in S2. The admission criteria for the amplitude measurement branch are: a usable body-scale normalized residual displacement sequence exists within the training window, background suppression is effective, and the candidate limb region corresponding to the residual displacement sequence is consistent with the candidate limb region in the training window. When no valid training frames exist after removing frames covered by excluded segment records, the system sets the admission judgment results for each task branch corresponding to that training window to 0, and records the branch training weights for each task branch as 0.

[0096] After completing the task branch admission judgment, the system generates branch training weights. These weights are recorded separately for each task branch: action classification branch training weight, pose detection branch training weight, and amplitude measurement branch training weight, with values ​​ranging from 0 to 1. A branch training weight of 0 indicates that the corresponding training window does not contribute to the training of that task branch; a branch training weight greater than 0 indicates that the corresponding training window participates in the loss calculation of the corresponding task branch according to that weight.

[0097] Branch training weights are generated as follows: .

[0098] Where B represents the task branch, and the value of B is C, PZ, or A. C represents the action classification branch, PZ represents the pose detection branch, and A represents the amplitude measurement branch. This represents the training weight of task branch B, with a value ranging from 0 to 1; This indicates the admission decision result for task branch B, with a value of 0 or 1; L represents the source acceptance value, with a value range from 0 to 1. R represents the proportion of training evidence for task branch B, ranging from 0 to 1; R represents the exclusion coverage, ranging from 0 to 1; P represents the exclusion decay coefficient, preferably ranging from 1 to 3. All of the above quantities are dimensionless. This formula applies to training windows where the exclusion coverage is below the exclusion coverage admission threshold and valid training frames exist. If the training window meets the admission criteria for task branch B, then... The value is 1; if the training window does not meet the admission criteria for task branch B, then... The value is 0. When the exclusion coverage of the training window reaches the exclusion coverage threshold, the values ​​of each task branch are reduced to 0. All values ​​are 0, and the branch training weights for each task branch are all 0.

[0099] The source acceptance value is formed by the source confidence in S1 and the source status in S2. The upper limit for low-confidence sources is preferably between 0.50 and 0.70, with an initial value of 0.60. When there are at least 5 records in the current training batch that have been reviewed and have a source status of low confidence, including historical visible action segments and historical branch admission records, the system calculates the proportion of records in that record that can still be used as visible action segments for training after review. If this proportion is not less than 0.70, the upper limit for low-confidence sources is selected within the range of 0.60 to 0.70; if this proportion is less than 0.40, the upper limit for low-confidence sources is selected within the range of 0.50 to 0.60; if this proportion is between 0.40 and 0.70, the upper limit for low-confidence sources is 0.60. When there are fewer than 5 samples, the system does not increase the upper limit for low-confidence sources and uses 0.60 as the initial value. When the source status is normal, the source acceptance value is the source confidence level; when the source status is low confidence, the source acceptance value is the smaller of the source confidence level and the upper limit of low confidence; when the source confidence level is missing, the source acceptance value is 0.50, and the corresponding training window is marked as a low confidence source status.

[0100] The training evidence percentage for the action classification branch is the ratio of effective training frames to the total number of frames in the training window. The training evidence percentage for the pose detection branch is the smaller of the percentage of visible frames in the candidate limb region and the tracking continuity value. The tracking continuity value is 1 minus the percentage of candidate limb region matching interruptions within the training window; adjacent frame pairs where limb matching interruptions occur are counted as the next frame in the candidate limb region matching interruption count. The training evidence percentage for the amplitude measurement branch is the ratio of available frames in the body-scale normalized residual displacement sequence to the total number of frames in the training window. If the input sequence required for a corresponding task branch is unavailable, the training evidence percentage for that task branch is set to 0.

[0101] The exclusion attenuation coefficient controls the attenuation strength of the excluded segment record's contribution to training, and its value is preferably between 1 and 3. When the exclusion coverage is 0, the exclusion attenuation coefficient is 1, the exclusion attenuation term is 1, and the exclusion type proportion judgment is not performed. When the exclusion coverage is greater than 0 and there are exclusion intersection frames, if the number of nursing residual coverage frames exceeds half of the number of exclusion intersection frames, the exclusion attenuation coefficient is 2 to 3; if the number of background synchronization coverage frames exceeds half of the number of exclusion intersection frames, the exclusion attenuation coefficient is 1 to 2; if the number of nursing residual coverage frames is the same as the number of background synchronization coverage frames, the exclusion attenuation coefficient is 2.

[0102] If there are at least 5 similar exclusion type records that have been reviewed before the current training batch, the system calculates the proportion of records of the same exclusion type that still need to be excluded from training contribution after review. If this proportion is not less than 0.70, the exclusion attenuation coefficient is taken as a value close to the upper limit of the corresponding range; if this proportion is less than 0.40, the exclusion attenuation coefficient is taken as a value close to the lower limit of the corresponding range; if this proportion is between 0.40 and 0.70, the exclusion attenuation coefficient is taken as a value in the middle of the corresponding range. If there are fewer than 5 historical similar exclusion type records, the exclusion attenuation coefficient corresponding to nursing residue coverage is 3, and the exclusion attenuation coefficient corresponding to background synchronization coverage is 2.

[0103] The system generates branch training weights based on the admission judgment result, source acceptance value, training evidence ratio of the corresponding task branch, exclusion coverage, and exclusion decay coefficient for each task branch. When the admission judgment result is 0, the branch training weight of the corresponding task branch is 0; when the exclusion coverage is 0, the exclusion decay term does not reduce the branch training weight; when the exclusion coverage is greater than 0 but lower than the exclusion coverage admission threshold, the branch training weight decreases as the exclusion coverage increases. Therefore, at least one change, either background synchronization change or nursing residue change, will not be written into the training window with a high training contribution due to the presence of event markers.

[0104] The system further performs low-visual-evidence sample set writing processing on candidate segments that have event markers but have not formed visible action segments. The low-visual-evidence sample set is for candidate segments that have event markers in the label correction record, whose validity status is either "can proceed to subsequent processing" or "low-confidence valid," and for which S2 has not generated a corresponding visible action segment. When writing the low-visual-evidence sample set, the system records the event source, anchor frame, candidate segment start frame, candidate segment end frame, source confidence, reason for not forming a visible action segment, and associated excluded segment records. Reasons for not forming a visible action segment include at least one of the following: unresolvable body region, invisible candidate limb region, incalculable displacement, background synchronization coverage, nursing residue coverage, and failure to pass continuity screening.

[0105] For candidate fragments written into the low-visual evidence sample set, the system generates a low-visual evidence record and sets the branch training weights of each task branch in the low-visual evidence record to 0. This low-visual evidence record is written into the branch admission record for S4 to determine the gating relationship between the low-visual evidence label and the upper limit of output confidence. The low-visual evidence sample set is not used as a positive sample of abnormal actions, nor as a negative sample of normal actions, and the low-visual evidence record is not used as the basis for confirming the current event during the real-time monitoring phase.

[0106] After the branch training weights are generated, the system writes them into the branch admission record using the training window as the index. The branch admission record uses the candidate fragment source index and the training window boundary as the primary index, and includes the corresponding visible action fragment, associated excluded fragment record, exclusion coverage, source acceptance value, branch training weights for each task branch, and low visual evidence markers. For candidate fragments in the low visual evidence sample set, the branch admission record uses the candidate fragment source index and candidate fragment boundary as the primary index, and includes low visual evidence markers and a weight of 0 for each task branch. The branch admission record is passed to S4, allowing S4 to call upon the branch training weights during the training of the multi-task classification network, ensuring that each training window only contributes to training within task branches that meet the admission criteria.

[0107] This step organizes visible action segments, excluded segment records, and low-visual evidence sample sets into branch admission records that can be used for training the multi-task classification network. This processing allows the action classification branch, pose detection branch, and amplitude measurement branch to receive training windows that match their evidence conditions, while blocking background synchronization changes, nursing residue changes, and low-visual evidence candidate segments from contaminating the training labels.

[0108] S4. Based on the branch training weight constraint of the loss function of the multi-task classification network, a newborn limb movement monitoring model is trained. The corresponding gating rules of the branch admission process are reused in the scrolling window of the real-time monitoring video to generate the output credibility. The newborn limb movement monitoring results are output based on the output credibility.

[0109] In this implementation, S4 takes the branch admission record, branch training weights, and low visual evidence sample set output by S3, applies the branch admission relationship formed during the training phase to the loss calculation process of the multi-task classification network, and re-executes the gating calculation based on a closed rolling window during the real-time monitoring phase. Thus, the sample admission constraints during the training phase and the output credibility judgment during the real-time phase employ the same processing logic.

[0110] The system reads the branch admission record generated by the branch admission process. The branch admission record uses the candidate segment source index and the training window boundary as its primary index, recording the visible action segments corresponding to the training window, the associated excluded segment records, the exclusion coverage, the source inheritance value, the training weights of the action classification branch, the pose detection branch, the amplitude measurement branch, and the low visual evidence label. For candidate segments in the low visual evidence sample set, the branch admission record records the low visual evidence label and the 0 weight of each task branch. The system organizes the training samples according to the branch admission record, ensuring that the training window only contributes to the training of task branches with non-zero branch training weights.

[0111] The multi-task classification network includes a shared feature extraction layer, an action classification branch, a pose detection branch, and an amplitude measurement branch. The shared feature extraction layer extracts temporal image features from video frames within the training window; the action classification branch outputs the category of newborn limb movements; the pose detection branch outputs the positional state and visibility of candidate limb regions; and the amplitude measurement branch outputs the body-scale normalized motion amplitude corresponding to the candidate limb regions. The shared feature extraction layer and each task branch can be implemented using at least one network structure selected from convolutional neural networks, temporal convolutional networks, and lightweight visual Transformers.

[0112] During training, the system inputs the training window into the multi-task classification network. The training objective of the action classification branch is formed by the event markers corresponding to the action category in the label correction record. Visible action fragments are used to limit the training window and effective training frames for the action classification branch. The training objective of the pose detection branch is formed by the candidate limb region positions retained when the visible action fragments are generated by S2. The training objective of the amplitude measurement branch is formed by the body-scale normalized residual displacement sequence. Frames covered by excluded fragment records are not used as training target frames, and candidate fragments in the low visual evidence sample set do not form the positive or negative training objectives for the action classification branch, pose detection branch, and amplitude measurement branch.

[0113] For task branches with training weights greater than 0 and corresponding training objectives, the system calculates the branch loss value for that task branch. The branch loss value for the action classification branch is formed by the classification error between the action classification output and the action classification branch training objective; the classification error can be calculated using cross-entropy loss. The branch loss value for the pose detection branch is formed by the detection error between the pose detection output and the candidate limb region position; the detection error can be calculated using at least one of intersection-over-union loss and smoothing L1 loss. The branch loss value for the amplitude measurement branch is formed by the measurement error between the amplitude measurement output and the body-scale normalized residual displacement sequence; the measurement error can be calculated using at least one of mean absolute error and mean square error. If any of the following conditions are met: the branch training weight for the corresponding task branch is 0, the corresponding training objective does not exist, or the number of effective training frames is 0, the system does not calculate the branch loss value for that task branch, and treats both the branch training weight and the branch loss value for that task branch as 0 in the weighted loss value calculation. Before entering the weighted loss value calculation, the branch loss values ​​of each task branch are averaged according to the number of effective training frames for the corresponding task branch and normalized to a dimensionless non-negative loss value according to the output caliber of the corresponding task branch.

[0114] The system weights the branch loss values ​​of the corresponding task branches according to the branch training weights in the branch admission record, generating a weighted loss value. Training windows with a branch training weight of 0 do not contribute to the loss of the corresponding task branch; training windows with a branch training weight greater than 0 participate in the loss calculation and parameter update of the corresponding task branch according to the corresponding branch training weight.

[0115] The weighted loss value is generated as follows:

[0116] ;

[0117] Wherein, LT represents the weighted loss value of the multi-task classification network, and its value is a non-negative number; LC represents the action classification branch loss value, LP represents the pose detection branch loss value, and LA represents the amplitude measurement branch loss value. All three are dimensionless non-negative loss values ​​normalized according to the output caliber of their respective tasks; WC represents the training weights of the action classification branch entering the formula for calculation, WP represents the training weights of the pose detection branch entering the formula for calculation, and WA represents the training weights of the amplitude measurement branch entering the formula for calculation. All three are derived from the branch admission records of S3, and their values ​​range from 0 to 1; SUM represents the summation of the training windows involved in the loss calculation. The training window involved in the loss calculation is a training window where the branch training weights of at least one task branch are greater than 0 and there is a corresponding training objective; when the summation of the branch training weights involved in the loss calculation is 0, the system does not perform model parameter updates. This formula is used for training loss constraints in this step, and its symbolic meaning is limited to this step.

[0118] The system updates the model parameters of the shared feature extraction layer, action classification branch, pose detection branch, and amplitude measurement branch based on the weighted loss value. When the training weight of a branch corresponding to a certain task branch is 0, the training window does not contribute to the loss of that task branch; when the training weight of a branch corresponding to at least one task branch is greater than 0, the system only performs backpropagation according to the branch loss value corresponding to the non-zero branch training weight. After training, the system obtains the neonatal limb movement monitoring model and saves the gating rules and their gating parameters corresponding to the branch admission processing.

[0119] During the real-time monitoring phase, the system forms a closed scrolling window from the real-time monitoring video according to a preset time step. The closed scrolling window consists of already received video frames, and the system does not use video frames after the end time of the closed scrolling window in the output judgment of the current window. The length of the closed scrolling window is preferably 0.5 seconds to 3 seconds, and the time step is preferably 0.1 seconds to 1 second, and the time step is not greater than the length of the closed scrolling window.

[0120] During initial setup, the system selects the closed scrolling window length based on the real-time monitoring video frame rate, the duration of the newborn's brief limb movements, and the system inference latency requirements. When the real-time monitoring video frame rate is not less than 25fps and the inference latency meets the real-time output requirements, the closed scrolling window length is a value close to 0.5 seconds. When at least one of the following conditions is met: real-time monitoring video frame rate is less than 15fps, video transmission latency is greater than 1 second, or model inference latency is greater than 1 second, the closed scrolling window length is a value close to 3 seconds. In other cases, the closed scrolling window length is between 1 and 2 seconds. When any of the following conditions is met: missing real-time monitoring video frame rate or discontinuous video frame timestamps, the system uses the number of valid video frames within the closed scrolling window as the fallback criterion. When the number of valid video frames is less than 5, the system sets the output confidence level to 0.

[0121] The system inputs a closed scrolling window into the neonatal limb movement monitoring model and obtains the model output results. The model output results include movement classification output, posture detection output, and amplitude measurement output. The movement classification output provides candidate movement categories and their confidence values; the posture detection output provides the positional state and visibility of candidate limb regions; and the amplitude measurement output provides the body-scale normalized movement amplitude. These model output results serve as input for output confidence calculation, and after passing through output confidence gating, they enter the result output process.

[0122] Within a closed scrolling window, the system reuses the gating rules corresponding to the branch admission process, performing gating calculations on the model output, the visibility of candidate limb regions, and the exclusion coverage rate. Within the closed scrolling window, the system invokes the same background synchronization and nursing residue conditions as S2, marking at least one change in either background synchronization or nursing residue to form a real-time exclusion interval. Then, it calculates the frame-level intersection between the closed scrolling window and the real-time exclusion interval, dividing the number of intersection frames by the total number of frames in the closed scrolling window to obtain the exclusion coverage rate of the closed scrolling window.

[0123] Before generating output confidence, the system first performs a check to set the output confidence to 0. The system sets the output confidence to 0 and outputs a low confidence warning if any of the following conditions are met: the total number of frames in the closed scrolling window is 0; the number of effective video frames is less than 5; the model output is missing; the exclusion coverage of the closed scrolling window reaches the exclusion coverage threshold in S3; the body-scale normalized displacement threshold is missing; or the body-scale normalized displacement threshold does not meet the positive value condition. Missing model output includes any necessary output not being returned (action classification output, pose detection output, amplitude measurement output); or any value in the action category confidence value, candidate limb region visibility, or body-scale normalized motion amplitude being non-numerical; or the corresponding value exceeding the preset range and unable to be corrected by amplitude limiting. Closed scrolling windows meeting any of the above conditions are not included in the output confidence formula calculation.

[0124] Low-visual-evidence markers are regenerated in real-time according to the low-visual-evidence sample set writing logic in S3. When the real-time monitoring system accesses the same monitoring event source as in S1, and event markers exist within the closed scrolling window, but at least one of the following conditions is met: candidate limb regions are invisible, body-scale normalized residual displacement sequences are unavailable, or the proportion of visible frames for candidate limb regions is lower than the visible frame proportion threshold in S2, the system generates low-visual-evidence markers. When the real-time monitoring system does not access a monitoring event source, the system does not generate low-visual-evidence markers based on event markers, and reduces output credibility through candidate limb region visibility, effective value of motion amplitude, and exclusion coverage.

[0125] Historical low-visual evidence records, historical exclusion type records, and historical output records all originate from the training sample pool records and running records that have already undergone annotation and verification before the current closed scrolling window. The annotation and verification records must at least record the verification status, verification action category, verification exclusion type, verification credibility status, and verification time. The running records and training sample pool records corresponding to the current closed scrolling window do not participate in the calibration of their own low-visual evidence credibility upper limit, output attenuation coefficient, and credible output threshold.

[0126] When low-visual evidence is flagged, the output confidence level should not exceed the upper limit of low-visual evidence confidence level. The upper limit of low-visual evidence confidence level is preferably between 0.30 and 0.55, and lower than the confidence output threshold for anomaly alerts. The initial value is 0.50. If there are at least 5 historical low-visual evidence records that have been reviewed before the current training batch, the system calibrates the upper limit of low-visual evidence confidence level within the range of 0.30 to 0.55 based on the proportion of historical low-visual evidence records that are still confirmed to have visible action evidence after review. If this proportion is not less than 0.70, the upper limit of low-visual evidence confidence level is 0.50 to 0.55; if this proportion is less than 0.40, the upper limit of low-visual evidence confidence level is 0.30 to 0.40; if this proportion is between 0.40 and 0.70, the upper limit of low-visual evidence confidence level is 0.40 to 0.50. If there are fewer than 5 historical low-visual evidence records, the system uses 0.50 as the initial value. Historical low-visual evidence records are those that have been reviewed before the current closed scrolling window. The current closed scrolling window does not participate in its own low-visual evidence credibility limit calibration.

[0127] Within a closed scrolling window where the output confidence level is not set to 0, the system generates output confidence based on the action category confidence value, the visibility of candidate limb regions, the effective value of motion amplitude, the exclusion coverage of the closed scrolling window, and low visual evidence markers. The action category confidence value comes from the action classification branch; the visibility of candidate limb regions comes from the pose detection branch; and the effective value of motion amplitude is formed by comparing the body-scale normalized motion amplitude output from the amplitude measurement branch with the body-scale normalized displacement threshold in S2. The system divides the body-scale normalized motion amplitude by the body-scale normalized displacement threshold and limits the result to between 0 and 1, using this as the effective value of motion amplitude. When the low visual evidence marker is valid, the low visual evidence gating value is the upper limit of the low visual evidence confidence level; when the low visual evidence marker is invalid, the low visual evidence gating value is 1.

[0128] Output credibility is generated as follows: ;

[0129] in, This indicates the output confidence level, with a value ranging from 0 to 1. This represents the confidence value for the action category, ranging from 0 to 1. Indicates the visibility of the candidate limb region, with a value ranging from 0 to 1; This represents the effective value of the motion amplitude, ranging from 0 to 1. This represents the exclusion coverage of the closed scroll window, with a value ranging from 0 to 1; This represents the low visual evidence gating value, ranging from 0 to 1; This represents the output attenuation coefficient, with a preferred value range of 1 to 3.

[0130] This formula applies to situations where the closed scrolling window does not trigger the condition of setting the output confidence to 0. When the exclusion coverage is 0, the output attenuation term is 1; when the exclusion coverage is greater than 0 and there are real-time exclusion interval intersection frames, if the number of nursing residual coverage frames exceeds half of the number of real-time exclusion interval intersection frames, the output attenuation coefficient is 2 to 3; if the number of background synchronization coverage frames exceeds half of the number of real-time exclusion interval intersection frames, the output attenuation coefficient is 1 to 2; if the number of nursing residual coverage frames is the same as the number of background synchronization coverage frames, the output attenuation coefficient is 2. When there are no fewer than 5 records of the same exclusion type that have been reviewed before the current training batch, the system calculates the proportion of records of the same exclusion type that are still confirmed as evidence of contamination of actions after review; when this proportion is not less than 0.70, the output attenuation coefficient is a value close to the upper limit of the corresponding range; when this proportion is less than 0.40, the output attenuation coefficient is a value close to the lower limit of the corresponding range; when this proportion is between 0.40 and 0.70, the output attenuation coefficient is a value in the middle of the corresponding range. If there are fewer than 5 historical records of the same exclusion type, the output attenuation coefficient for nursing residue coverage is set to 3, and the output attenuation coefficient for background synchronization coverage is set to 2. Historical records of the same exclusion type are those that have already been reviewed before the current closed scrolling window; the current closed scrolling window does not participate in its own output attenuation coefficient calibration. This formula is used for output reliability calculation in this step, and its symbols are limited to this step.

[0131] The system outputs the newborn's limb movement monitoring results based on the output confidence level. When the output confidence level reaches the confidence output threshold for abnormal alerts, the system outputs the newborn's limb movement monitoring results, which include the candidate movement category, the corresponding candidate limb region, the amplitude of movement output, the closed scrolling window boundary, and the output confidence level. When the output confidence level does not reach the confidence output threshold for abnormal alerts, the system outputs a low confidence alert.

[0132] The preferred reliable output threshold for anomaly alerts is 0.60 to 0.75, and the preferred reliable output threshold for automatic training sample write-back is 0.75 to 0.85. Initially, this threshold is selected based on the real-time monitoring video frame rate, target miss tolerance, and false alarm tolerance. For anomaly alerts, the reliable output threshold should be close to 0.60 to 0.70; for automatic training sample write-back, the reliable output threshold should be close to 0.80 to 0.85. When there are at least 10 historical output records that have been reviewed before the current training batch, the system calibrates the credible output threshold for anomaly alerts and the credible output threshold for automatic rewrite of training samples within the corresponding limits based on the proportions of false positives and false negatives in the historical output records. When the proportion of false positives is not less than 0.20, the credible output threshold for anomaly alerts is a value close to 0.75; when the proportion of false negatives is not less than 0.20 and the proportion of false negatives is less than 0.20, the credible output threshold for anomaly alerts is a value close to 0.60; when the proportion of automatic rewrite records failing review is not less than 0.20, the credible output threshold for automatic rewrite of training samples is a value close to 0.85. When there are fewer than 10 historical output records, the credible output threshold for anomaly alerts is 0.65, and the credible output threshold for automatic rewrite of training samples is 0.80. The credible output threshold for automatic rewrite of training samples is not lower than the credible output threshold for anomaly alerts. Historical output records are those that have been reviewed before the current closed scrolling window; the current closed scrolling window does not participate in its own credible output threshold calibration.

[0133] The real-time monitoring result write-back is divided into training sample write-back and runtime record write-back. When the output confidence reaches the confidence output threshold for automatic training sample write-back and the low visual evidence label is not established, the system writes the corresponding closed scrolling window into the training sample pool to be reviewed, and writes the closed scrolling window boundary, model output result, output confidence, exclusion coverage, output confidence set to 0, low visual evidence label status, and exclusion coverage gating status. When there is any situation where the output confidence does not reach the confidence output threshold for automatic training sample write-back and the low visual evidence label is established, the system writes the corresponding closed scrolling window into the runtime record, and writes the closed scrolling window boundary, low confidence prompt, output confidence, low visual evidence label, and exclusion coverage. After subsequent review, the closed scrolling windows in the training sample pool to be reviewed are used as at least one of the historical visible action segments and historical excluded segment records to participate in the threshold calibration of subsequent training batches; the runtime records with established low visual evidence labels are only used as historical low visual evidence records to participate in the low visual evidence confidence upper limit calibration after review, and are not used as positive or negative training samples for the action classification branch, pose detection branch, and amplitude measurement branch.

[0134] Through this step, the branch training weights constrain the training loss of the multi-task classification network, and the visibility of candidate limb regions within the closed scrolling window, exclusion coverage, and low visual evidence markers jointly constrain the credibility of the real-time output, thus forming a continuous processing chain of training admission, real-time gating, result output, and write-back offloading.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring neonatal limb movements based on a multi-task classification network, characterized in that, Includes the following steps: Acquire neonatal monitoring annotation data, perform time base correction on the event markers in the neonatal monitoring annotation data and historical monitoring video frames, and correct the corrected event markers according to the marker delay range corresponding to the event source, and generate a label correction record. The label correction record is used to limit the candidate segments in the historical monitoring video that correspond to the event markers. Candidate segments are called, and a target detection network is used to extract candidate limb regions in the newborn's body region. Background motion suppression and continuity screening are performed on the inter-frame displacement of the candidate limb regions to generate visible action segments. Image changes that meet at least one of the following conditions are written into the excluded segment record. Branch admission processing is used to allocate training windows for visible action segments and excluded segment records, generate branch training weights, and write candidate segments with event labels but no visible action segments into the low visual evidence sample set. The branch training weights are used to control the training contribution of the corresponding task branch in the multi-task classification network. The loss function of the multi-task classification network is constrained by the branch training weights to train a newborn limb movement monitoring model. The gating rules corresponding to the branch admission process are reused in the scrolling window of the real-time monitoring video to generate output credibility. The newborn limb movement monitoring results are output based on the output credibility.

2. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 1, characterized in that, Time base correction includes: A video timeline is established based on the frame timestamps of historical monitoring video frames. A time base offset of the event timeline relative to the video timeline is generated using a synchronization reference point that simultaneously has event time records and video frame positioning information. The time position of the event marker is corrected based on the time base offset and mapped to an anchor frame.

3. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 2, characterized in that, The generated label correction record includes: According to the marked delay range corresponding to the event source, candidate segments are extracted based on the anchor frame, and a label correction record is generated based on the effective boundary of the candidate segment and the source confidence. The marked delay range is formed by the delay samples of historical events from the same source. When the delay samples do not meet the conditions for forming the delay range, the preset marked delay range of the corresponding event source is adopted and the source confidence is reduced.

4. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 1, characterized in that, The candidate limb regions to be extracted include: A target detection network is run on historical monitoring video frames within the candidate segments to obtain the detection regions of the newborn's body and limbs. The detected regions are then mapped back to the original pixel coordinates of the historical monitoring video frames. Based on the tracking results of adjacent frames, candidate limb regions are generated within the newborn's body region using the original pixel coordinates.

5. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 4, characterized in that, Generating visible action fragments includes: Inter-frame displacement is generated for candidate limb regions that have been tracked and matched by adjacent frames. The inter-frame displacement is subtracted from the background global motion of the background reference region to obtain the residual displacement. Image changes that meet at least one of the background synchronization condition and nursing residual condition are written into the exclusion segment record. Visible motion segments are generated for consecutive frames that are not written into the exclusion segment record and whose residual displacement reaches the displacement threshold.

6. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 1, characterized in that, Training window allocation includes: Based on the same candidate segment source index and video frame timestamp, visible action segments and excluded segment records are matched. A training window is generated using the start and end frames of the visible action segments. The frame-level intersection between the training window and the excluded segment records is calculated to obtain the exclusion coverage. The corresponding task branch admission judgment is then performed on the training window based on the exclusion coverage.

7. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 6, characterized in that, The weights for generating branches include: Based on the admission judgment results, source confidence, training evidence ratio and exclusion coverage of each task branch, the branch training weights of each task branch are generated; for candidate segments with event markers but no visible action segments, the candidate segments are written into the low visual evidence sample set, and the branch training weights of each task branch corresponding to the candidate segment are reset to 0.

8. The method for monitoring neonatal limb movements based on a multi-task classification network according to claim 1, characterized in that, The loss function for a multi-task classification network, which constrains the training weights of the branches, includes: Read the branch admission record formed by the branch admission process, input the training window into the multi-task classification network, and obtain the branch loss value of each task branch; The branch loss value of the corresponding task branch is weighted according to the branch training weight in the branch admission record to generate a weighted loss value; In this case, the training window with a branch training weight of 0 does not contribute to the loss of the corresponding task branch.

9. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 8, characterized in that, The reliability of the output generated by reusing the gating rules corresponding to the branch admission process within the scrolling window of the real-time monitoring video includes: A closed scrolling window is formed based on the received video frames. The closed scrolling window is then input into the neonatal limb movement monitoring model to obtain the model output results. Within a closed scrolling window, the gating rules corresponding to the branch admission process are reused to perform gating calculations on the model output, the visibility of candidate limb regions, and the exclusion coverage, thereby generating output credibility. When the exclusion coverage of the closed scrolling window reaches the exclusion coverage admission threshold, the output confidence is set to 0.

10. A method for monitoring neonatal limb movements based on a multi-task classification network according to claim 9, characterized in that, The output of neonatal limb movement monitoring results, based on output reliability, includes: When the output confidence level reaches the confidence output threshold, the newborn limb movement monitoring results are output. When the output confidence level does not reach the confidence output threshold, a low confidence warning is output. When the output credibility reaches the credibility output threshold for automatic writing back of training samples and the low visual evidence label is not valid, the corresponding closed scrolling window is written into the training sample pool to be reviewed. If any of the following conditions exist: the output credibility does not reach the credibility output threshold for automatic writing back of training samples, or the low visual evidence label is valid, the corresponding closed scrolling window will be written into the running record.

Images