Computer vision based online identification system for child abnormal behavior

Abstract

The present application relates to the technical field of computer vision, in particular to a kind of online identification system of children's abnormal behavior based on computer vision.The system respectively constructs the action feature sequence of each child;According to the action feature sequence of the child to be evaluated and the reference child, a similarity matrix is formed;The similarity matrix of the child to be evaluated and the reference child is time series matched, and the behavior similarity is determined;If the behavior similarity is greater than the preset threshold, the corresponding historical period is recorded as similar behavior period;Extract the environmental feature point of the influence period of the child to be evaluated, and determine the environmental influence index of each point;The environmental feature point of the child to be evaluated is matched with the environmental feature point of the influence period before similar behavior period, and the matching feature point pair is determined;And determine the environmental similarity;According to the environmental similarity and the number of similar behavior period, the behavior deviation degree of the child to be evaluated is determined;According to the behavior deviation degree and the preset condition, it is judged whether to trigger early warning, so as to accurately identify children's abnormal behavior.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to an online system for recognizing abnormal behavior in children based on computer vision. Background Technology

[0002] With the development of computer vision technology, child behavior analysis has been widely applied in the fields of safety monitoring and health management. Existing technologies mainly identify behavior through video capture and deep learning models, and compare it with a pre-set behavior sample database; or they compare it with the child's own historical behavior to determine whether the current behavior is abnormal.

[0003] However, this technology has significant shortcomings: First, children's behavior is highly variable and difficult to be fully covered by a predefined sample database. It is easily affected by individual behavioral differences and fluctuations in normal behavior, leading to misjudgments of abnormal behavior. Second, relying solely on its own historical data lacks reasonable references and cannot distinguish between normal fluctuations and potential abnormalities. Third, it ignores the influence of environmental factors on behavior. Ignoring the influence of environmental factors on children's behavior makes it impossible to accurately distinguish between behavioral changes caused by environmental changes and real abnormal behavior. Judging solely by the similarity of actions is prone to false alarms or missed alarms. Summary of the Invention

[0004] To address the technical problems of existing child behavior assessment methods that primarily rely on comparing children's historical behavioral data, making it difficult to determine whether a child's current behavior is normal when there are discrepancies between the current and historical behaviors, and neglecting environmental factors that lead to low accuracy and high false alarm rates, this invention provides an online child abnormal behavior identification system based on computer vision. The specific technical solution adopted is as follows: This invention proposes an online system for identifying abnormal behaviors in children based on computer vision. The system includes: The acquisition module is used to collect historical and current behavioral video data of the child to be evaluated and a reference child in a similar scenario; the behavioral video data is analyzed frame by frame to extract the action feature indicators of each child in each frame and form action feature sequences respectively. The behavior assessment module is used to determine the single-frame action similarity and form a similarity matrix based on the current action feature sequence of the child to be assessed and the historical action feature sequence of the reference child; to perform temporal matching of the similarity matrices of the child to be assessed and the reference child, determine the temporal matching distance and convert it into behavior similarity; if the behavior similarity is greater than a preset threshold, the corresponding historical time period is recorded as the similar behavior time period; The environmental assessment module is used to extract environmental feature points from the periods preceding the current time of the child to be assessed and to determine the environmental impact indicators for each point; it matches the environmental feature points of the child to be assessed with the environmental feature points from the periods preceding the time of similar behavior to determine matching feature point pairs; and it determines the environmental similarity based on the matching feature point pairs and their respective environmental impact indicators. The anomaly detection module is used to determine the degree of behavioral deviation of the child to be evaluated based on environmental similarity and the number of similar behavior time periods; based on the degree of behavioral deviation and preset conditions, it determines whether to trigger an early warning; if triggered, it sends an early warning notification.

[0005] Furthermore, the acquisition module continuously acquires each frame of image at a preset frame rate, and performs aggregate analysis on multiple consecutive frames of images within a preset first time window as a period. Based on the children's historical behavioral video data, the data is divided into multiple historical periods according to a preset first time window, and each historical period corresponds to a set of behavioral video data.

[0006] Furthermore, the frame-by-frame analysis of the behavioral image data extracts the action feature indicators of each child in each frame, forming action feature sequences, including: Identify the child's pose joints in each frame of behavioral video data; obtain the pixel coordinates of the pose joints in the image; A relative coordinate system is constructed with the preset torso reference center as the origin. For each posture joint, the difference between the pixel coordinates of the posture joint and the pixel coordinates of the torso reference center in each frame is calculated to obtain the relative position coordinates of the posture joint. For each pose joint, obtain the relative position coordinates of the same pose joint in two adjacent frames; calculate the difference between the relative position coordinates of the next frame and the relative position coordinates of the previous frame to obtain the motion vector of the pose joint in the next frame; wherein, the motion vector of the first frame is set to a preset initial value. The relative position coordinates of each pose joint point in the same frame are combined with the motion vector to form the motion feature index of the same frame. The motion feature indicators of each frame are arranged in chronological order to form a motion feature sequence.

[0007] Furthermore, the similarity matrix formation process includes: For each historical time period, any frame of the current time period of the child to be evaluated and any frame of the historical time period of the reference child are combined into a pair of frames; and the same pose key point in the pair of frames is combined into a pose key point pair. For each pair of pose joints in each set of frames, the cosine similarity of the motion vectors of each pose joint pair is calculated as an index of orientation consistency; the Euclidean distance of the relative position coordinates of each pose joint pair is calculated as an index of position proximity. The orientation consistency index and the position proximity index are normalized and mapped separately and then combined to obtain the similarity contribution value of the pose joint pairs; the arithmetic mean of the similarity contribution values ​​of all pose joint pairs in the frame is calculated as the single-frame action similarity of the frame. An initial matrix is ​​constructed using the number of frames in the current time period of the child to be evaluated as the number of rows and the number of frames in the historical time period of the reference child as the number of columns. The calculated single-frame action similarity of each pair of frames is then filled into the corresponding positions in the matrix to form a similarity matrix between the current time period of the child to be evaluated and the historical time period of the reference child.

[0008] Further, the step of performing temporal matching of the similarity matrices between the child to be evaluated and the reference child, determining the temporal matching distance, and converting it into behavioral similarity includes: For each historical period, obtain each element in the similarity matrix between the current period and the historical periods for the child to be evaluated; subtract the value of each element from one to obtain the distance value; arrange the distance values ​​according to the row and column positions of the similarity matrix to form a distance matrix; A dynamic time warping algorithm is used to search for the optimal alignment path in the distance matrix, and the cumulative distance corresponding to the optimal alignment path is used as the temporal matching distance. The average matching distance is obtained by dividing the temporal matching distance by the optimal alignment path length; the value of the exponential function with the natural constant as the base and the negative of the average matching distance as the exponent is calculated as the behavioral similarity.

[0009] Furthermore, the period of influence is a preset second time window preceding the period of behavior, and the period of behavior includes the current period and historical periods; The process for determining the environmental impact indicators includes: For each frame of the image within the affected period, the position of the child in the current frame is identified using a target detection algorithm to obtain the child detection box, and the geometric center of the child detection box is extracted as the center point of the child's position. The corner point, edge point, or texture change point in the current frame is identified using a feature point detection algorithm as feature points. The feature points belonging to the child's body area are removed by using the area formed by connecting the human pose joints, and the non-human feature points within the child detection box and the feature points outside the box are retained as environmental feature points. The scale-invariant feature transformation algorithm is used to generate feature description vectors for each environmental feature point. Obtain the pixel coordinates of environmental feature points in the current frame, and obtain the pixel coordinates of the child's center point in the current frame; calculate the Euclidean distance between the two pixel coordinates; convert the Euclidean distance into a distance factor using a preset monotonically decreasing mapping function; Record the first frame in which the environmental feature point appears during the period of influence; obtain the displacement distance of the environmental feature point between adjacent frames using optical flow; starting from the first frame in which the environmental feature point appears, accumulate the displacement distance frame by frame to obtain the cumulative displacement distance up to the current frame; convert the cumulative displacement distance into a cumulative displacement mapping value using a preset monotonically increasing mapping function; add the basic retention weight to the cumulative displacement mapping value to obtain the motion factor; Calculate the time interval from the first appearance frame of an environmental feature point to the current frame, and use it as the existence duration up to the current frame; convert the existence duration into a duration factor using a preset monotonically decreasing mapping function; Multiply the distance factor, motion factor, and duration factor to obtain the environmental impact index of environmental feature points in the current frame.

[0010] Furthermore, the system also includes: For each environmental feature point, the time interval between each frame of the image and the center of the behavior period is extracted and used as the nearest time. The reciprocal of the nearest time is taken as the temporal influence weight of the environmental feature point in each frame of the image during the period of influence. The product of the environmental impact index and the corresponding time impact weight of each frame of the environmental feature point during the period of influence is calculated as the environmental impact component; the sum of the environmental impact components of all frames of the environmental feature point during the period of influence is calculated as the total environmental impact. The sum of the temporal influence weights of environmental feature points across all frames within the influence period is calculated and used as the total temporal influence weight. The ratio of the total environmental impact to the total time impact weight is calculated and used as the average environmental impact index of environmental feature points during the impact period.

[0011] Furthermore, the step of matching the environmental feature points of the child to be evaluated with the environmental feature points of the period of influence preceding the similar behavior to determine the matching feature point pairs includes: Obtain the first set of environmental feature points for the child to be evaluated during the period of influence before the current time period, and the second set of environmental feature points for the child to be evaluated during the period of influence before the similar behavior time period; wherein, each environmental feature point in the first set of environmental feature points and the second set of environmental feature points corresponds to an environmental feature time sequence, and the environmental feature time sequence is composed of the feature description vectors of the environmental feature points in each frame during the period of influence arranged in chronological order; The dynamic time warping algorithm is used to calculate the first temporal matching distance between the environmental feature time series sequence of each environmental feature point in the first set of environmental feature points and the environmental feature time series sequence of each environmental feature point in the second set of environmental feature points; The nearest neighbor matching algorithm is used to find the environmental feature point with the smallest first temporal matching distance from the second environmental feature point set for each environmental feature point in the first environmental feature point set as a candidate matching point; If the minimum first temporal matching distance is less than the preset matching threshold, the candidate matching point and its related environmental feature points in the first set of environmental feature points are determined as successfully matched environmental feature point pairs and recorded as matching feature point pairs.

[0012] Furthermore, the environmental similarity determination process includes: Obtain successfully matched feature point pairs. Each matched feature point pair includes the first environmental feature point of the child to be evaluated, the second environmental feature point of the reference child, and the first temporal matching distance between the two. For each pair of matching feature points, the first temporal matching distance between the two is converted into environmental feature similarity through a preset monotonically decreasing mapping function; the product of the average environmental impact index of the first environmental feature point during its influence period and the average environmental impact index of the second environmental feature point during its influence period is calculated as the important weight of the matching feature point pair; the environmental feature similarity is multiplied by the important weight to obtain the environmental feature similarity contribution of the matching feature point pair. The contribution numerator is obtained by summing the environmental feature similarity contributions of all matching feature point pairs; the contribution denominator is obtained by summing the importance weights of all matching feature point pairs. Divide the contribution numerator by the contribution denominator to obtain the environmental similarity.

[0013] Furthermore, the process for determining the degree of behavioral deviation includes: Obtain the number of similar behavior time periods and the environmental similarity corresponding to each similar behavior time period; The number of similar behavior time periods is converted into a quantitative influence factor through a preset monotonically decreasing mapping function; Calculate the difference between the environmental similarity of a given time period and the environmental similarity of each similar behavior time period, and use this difference as the environmental dissimilarity of each similar behavior time period. Calculate the arithmetic mean of the environmental dissimilarity for all similar behavior time periods, and use it as the average environmental dissimilarity; The quantity impact factor is taken as the first term; the quantity impact factor is subtracted from one to obtain the environmental impact weight; the environmental impact weight is multiplied by the average environmental dissimilarity to obtain the second term; the first term and the second term are added together to obtain the behavioral deviation degree.

[0014] The present invention has the following beneficial effects: This invention uses children's behavioral image data as a comparison benchmark. If the current behavior of the child to be evaluated differs from their historical behavior, the behavioral patterns of the reference child group can be used to assist in the judgment, avoiding misjudgments caused by relying solely on their own historical data. By constructing a similarity matrix and performing temporal matching, the single-frame behavioral similarity is expanded to behavioral similarity in a temporal dimension, so as to accurately characterize the temporal correlation between the current behavior of the child to be evaluated and the historical behavior of the reference children, thereby avoiding the influence of subjective factors, achieving accurate quantification of behavioral difference comparison, and further improving the sensitivity of anomaly identification. Based on behavioral difference comparison, environmental feature points before the behavior occurred are further extracted. By analyzing the consistency between the environmental feature points of the current behavior of the child to be evaluated and the environmental feature points of similar behaviors of the reference children, the environmental similarity is determined, thereby judging whether the current behavior of the child to be evaluated belongs to the typical behavior in the past. This enables the system to effectively identify and avoid false alarms when the behavior is similar but the environmental factors are inconsistent. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of the structure of an online recognition system for abnormal behavior in children based on computer vision, provided in one embodiment of the present invention; Figure 2 This is an example diagram illustrating the process of determining environmental impact indicators according to an embodiment of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, details a specific solution for an online identification system for abnormal child behavior based on computer vision provided by this invention.

[0018] Please see Figure 1 The diagram illustrates a computer vision-based online identification system for abnormal behavior in children, according to an embodiment of the present invention. The system includes: The acquisition module 101 is used to acquire historical and current behavioral image data of the child to be evaluated and a reference child in a similar scenario; the behavioral image data is analyzed frame by frame to extract the action feature indicators of each child in each frame and form action feature sequences respectively.

[0019] In this setting, the reference child and the child to be assessed are in the same type of environment.

[0020] Homogeneous scenarios refer to scenarios with similar environmental characteristics such as scenario type, spatial layout, and activity content. For example, if the child to be assessed is playing in a kindergarten classroom, other children in the same class in the same kindergarten are selected as reference children; if the child to be assessed is playing in a family living room, reference children with similar home environments are selected.

[0021] It should be noted that this system aims to identify atypical isolated behaviors (such as stereotyped actions or stress responses in the absence of external stimuli) triggered by a lack of appropriate environmental stimulation in children.

[0022] In this embodiment, behavioral image data is acquired by image acquisition devices placed at preset indoor and outdoor locations, including behavioral image data of the child to be evaluated in historical and current time periods, as well as behavioral image data of the child as a reference in historical time periods.

[0023] The preset location refers to the pre-determined installation position of the image acquisition equipment based on the actual needs of the monitoring scenario; this is not the case in this embodiment. Generally speaking, the selection of the preset location follows these principles: first, the equipment layout should cover the child's main activity areas to avoid blind spots; second, multiple camera positions should be used to acquire more comprehensive behavioral image data from different angles; and third, appropriate installation positions and heights should be selected based on the characteristics of different indoor and outdoor scenarios.

[0024] It should be noted that the acquisition module 101 continuously acquires each frame of image at a preset frame rate, and performs aggregate analysis on multiple consecutive frames of images within a preset first time window as a period; referring to the child's historical behavioral image data, it is divided into multiple historical time periods according to the preset first time window, and each historical time period corresponds to a set of behavioral image data.

[0025] It should be noted that the specific value of the preset frame rate is usually consistent with the output frame rate of conventional camera equipment, and this embodiment does not impose a specific limitation. For example, this system uses 25 frames per second as a preferred example.

[0026] It should be noted that the specific value of the preset first time window is determined based on the characteristics of children's behavior, and this embodiment does not impose a specific limitation. For example, a child's single action is usually completed within a few seconds to tens of seconds, so choosing 30 seconds as the preset first time window value can fully encompass the start and end process of a child's behavior.

[0027] It can be understood that each acquisition moment corresponds to one frame of image, and each preset first time window corresponds to multiple consecutive frames of images.

[0028] Since each historical time period corresponds to a set of behavioral video data, the video data for each historical time period is analyzed frame by frame to extract motion feature indicators for each frame. These indicators are then arranged chronologically to form a motion feature sequence for that historical time period. In other words, each historical time period uniquely corresponds to one motion feature sequence. Similarly, behavioral video data for the children to be evaluated is also collected according to a preset first time window. After frame-by-frame analysis, motion feature sequences are formed in different windows, such as the motion feature sequence for the current time period.

[0029] It's important to understand that this system operates on a preset first time window cycle, with each first time window independently executing a complete behavior assessment process. Specifically, the system continuously collects behavior video data. After each first time window (e.g., 30 seconds) is completed, all frames within that first time window are analyzed and calculated, outputting the behavior assessment results for that first time window. Then, at the start of the next first time window, the system collects new behavior video data and independently performs a new round of analysis.

[0030] To accurately analyze behavioral video data frame by frame, extract the motion feature indicators of each child in each frame, and form motion feature sequences, the following steps are included: 1) Identify the child's pose joints in each frame of the behavioral image data; obtain the pixel coordinates of the pose joints in the image.

[0031] It should be noted that pose keypoint recognition can be implemented using mature pose estimation algorithms already available in the technology. Specifically, the system first locates the child's position in the image using a target detection algorithm (such as YOLO) and obtains the child's bounding box; then, it uses a pose estimation algorithm (such as YOLOv8-pose) to extract the child's pose keypoints within the bounding box and outputs the pixel coordinates of each pose keypoint. The above techniques are all well-known in the field of computer vision and will not be elaborated upon here.

[0032] Postural joints include at least the following: shoulder joints, hip joints, elbow joints, wrist joints, knee joints, and ankle joints.

[0033] 2) Construct a relative coordinate system with the preset torso reference center as the origin. For each posture joint, calculate the difference between the pixel coordinates of the posture joint and the pixel coordinates of the torso reference center in each frame to obtain the relative position coordinates of the posture joint.

[0034] It should be noted that the specific selection of the trunk reference center is determined based on clinical experience, and this embodiment does not impose specific limitations. For example, since the hip is the approximate location of the body's center of gravity, it is relatively stable in common postures such as standing, sitting, and walking in children. Using the midpoint between the left and right hip joints as the trunk reference center can effectively eliminate the influence of overall body movement on posture description.

[0035] Pixel coordinates are defined with the top-left corner of the image as the origin, the horizontal axis as the X-axis (positive to the right), and the vertical axis as the Y-axis (positive downwards). For example, if a frame has a resolution of 1920×1080, the pixel coordinates of the left shoulder joint are (800, 450), indicating that the left shoulder joint is located 800 pixels from the left boundary and 450 pixels from the top boundary in the image.

[0036] It can be understood that the relative position coordinates of the attitude joints include lateral relative coordinates and longitudinal relative coordinates. The process of determining the relative position coordinates is as follows: lateral relative coordinate = lateral pixel coordinate of attitude joint - lateral pixel coordinate of torso reference center; longitudinal relative coordinate = longitudinal pixel coordinate of attitude joint - longitudinal pixel coordinate of torso reference center.

[0037] Relative position coordinates reflect the positional relationship of a certain posture joint point relative to the torso reference center, so that subsequent motion descriptions are not affected by the child's movement in the picture.

[0038] 3) For each attitude joint, obtain the relative position coordinates of the same attitude joint in two adjacent frames; calculate the difference between the relative position coordinates of the next frame and the relative position coordinates of the previous frame to obtain the motion vector of the attitude joint in the next frame; wherein, the motion vector of the first frame is set to a preset initial value.

[0039] The motion vector includes a lateral displacement component and a longitudinal displacement component. The lateral displacement component = the lateral relative coordinate of the next frame - the lateral relative coordinate of the previous frame; the longitudinal displacement component = the longitudinal relative coordinate of the next frame - the longitudinal relative coordinate of the previous frame.

[0040] Since there is no previous frame as a reference for the first frame, the system sets the motion vectors of each pose joint in the first frame to preset initial values.

[0041] It should be noted that the specific values ​​of the preset initial values ​​are determined based on engineering experience, and this embodiment does not impose specific limitations. For example, the motion example of the first frame image is that the horizontal displacement component is zero and the vertical displacement component is zero.

[0042] 4) Combine the relative position coordinates of each pose joint point in the same frame with the motion vector to form the motion feature index of the same frame.

[0043] It should be noted that the data structure of the motion feature index is as follows: each frame image is a unit, and each frame corresponds to one motion feature index; each motion feature index contains feature units corresponding to each posture joint; each feature unit contains the relative position coordinates and motion vector of that posture joint.

[0044] 5) Arrange the motion feature indicators of each frame in chronological order to form a motion feature sequence.

[0045] The behavior assessment module 102 is used to determine the single-frame action similarity and form a similarity matrix based on the current action feature sequence of the child to be assessed and the historical action feature sequence of the reference child; to perform temporal matching of the similarity matrix of the child to be assessed and the reference child, determine the temporal matching distance and convert it into behavior similarity; if the behavior similarity is greater than a preset threshold, the corresponding historical time period is recorded as the similar behavior time period.

[0046] It's important to understand that children's behavioral patterns are highly variable. The same behavior may vary in speed and rhythm between different individuals or at different times, making it impossible to accurately determine behavioral similarity through direct frame-by-frame comparison. Therefore, this system first calculates the single-frame action similarity between each frame of the current time period and each frame of historical time periods, forming a similarity matrix. This similarity matrix provides the foundational data for subsequent temporal matching, enabling the system to flexibly align along the timeline using a dynamic time warping algorithm, thus effectively addressing the matching problem of the same behavior at different speeds and rhythms.

[0047] The process of forming a similarity matrix includes the following steps: 1) For each historical time period, any frame of the current time period of the child to be evaluated and any frame of the historical time period of the reference child are combined into a pair of frames; and the same attitude key point in the pair of frames is combined into a pair of attitude key points.

[0048] It's important to understand that the "reference child" in this approach doesn't refer to a specific child, but rather to a group of all reference children in similar scenarios to the child being assessed. Each reference child's historical data is divided into multiple independent historical periods within a pre-defined time window, with each period serving as an independent behavioral sample for matching. Therefore, the core comparison unit in this approach is the "historical period" rather than the "individual reference child." The system iterates through all historical periods of all reference children, selecting those periods that are similar to the current behavior of the child being assessed—the "similar behavior periods."

[0049] For example, if there are M frames in the current time period of the child to be evaluated, and N frames in the historical time period of the child, then a total of M×N pairs of frames are formed. For each pair of frames, the system further combines two attitude joints that belong to the same attitude joint point into an attitude joint point pair. Assuming there are G attitude joint points in total, each pair of frames contains G attitude joint point pairs.

[0050] 2) For each pair of attitude joints in each set of frames, calculate the cosine similarity of the motion vectors of each attitude joint pair as an indicator of orientation consistency; calculate the Euclidean distance of the relative position coordinates of each attitude joint pair as an indicator of position proximity.

[0051] It should be noted that the specific methods for calculating cosine similarity and Euclidean distance are common techniques in this field, and can be directly applied in this embodiment, so they will not be elaborated here.

[0052] It's important to note that when calculating the cosine similarity of two motion vectors, the dot product of the two vectors needs to be divided by the product of their respective magnitudes. If the magnitude of a motion vector is zero (i.e., the attitude joint is stationary in the frame), the denominator is zero, making cosine similarity calculation impossible. Therefore, this system adds a very small positive number (e.g., 10 to the power of -6) to the denominator, ensuring the denominator is always greater than zero and guaranteeing numerical stability. The introduction of this very small positive number does not affect the physical meaning of cosine similarity; that is, when the magnitude is significantly greater than the very small positive number, the calculation result is basically consistent with the standard cosine similarity. If the magnitude is zero, the cosine similarity approaches zero, which aligns with the physical expectation that the orientation consistency index is zero in a stationary state.

[0053] 3) After normalizing and mapping the orientation consistency index and the position proximity index respectively, combine them to obtain the similarity contribution value of the pose joint pair; calculate the arithmetic mean of the similarity contribution values ​​of all pose joint pairs in the frame as the single-frame action similarity of the frame.

[0054] Because the directional consistency index and the positional proximity index have different dimensions, the system performs normalization mapping on the directional consistency index and the positional proximity index respectively. Specifically, since the directional consistency index (i.e., cosine similarity) ranges from negative one to positive one, it can be converted into a directional similarity index within the range of zero to one.

[0055] Directional similarity = (directional consistency index + 1) ÷ 2.

[0056] It's important to understand that the closer the direction similarity index (cosine similarity) of a pair of attitude joints is to positive one, the more consistent the motion directions of the two attitude joints are; conversely, the closer the index is to negative one, the more opposite the motion directions are. Therefore, direction similarity is used to measure the degree of consistency in the motion directions of two attitude joints in a pair. A higher direction similarity indicates more consistent motion directions, while a lower direction similarity indicates greater differences, or even opposite, motion directions.

[0057] Since the proximity index (i.e., Euclidean distance) ranges from zero to infinity, to convert the proximity index into a similarity index within the range of zero to one, the smaller the Euclidean distance, the closer the similarity index is to one. Specifically, a pre-defined monotonically decreasing mapping function is used to map the proximity index to a similarity index.

[0058] It's important to understand that this solution uses a pre-defined monotonically decreasing mapping function in multiple calculations to convert input values ​​into output values ​​within the range of zero to one, with larger input values ​​resulting in smaller output values. Specifically, the pre-defined monotonically decreasing mapping function is used when converting Euclidean distance and duration of existence in environmental impact indicator calculations, and when converting temporal matching distance in behavioral similarity calculations. For example, the monotonically decreasing mapping function is specifically... or ;in, This represents an adjustment parameter greater than zero, used to control the decay rate of the mapping function.

[0059] It should be noted that the specific form of the preset monotonically decreasing mapping function is not limited to one type. It can be flexibly selected according to the application scenario. For example, it can be an exponential decay function, an inverse proportional function, or a piecewise linear function, as long as it satisfies the condition of monotonically decreasing and the output range is from zero to one.

[0060] in, The larger the value, the faster the monotonically decreasing mapping function decays; a tiny increase in the input value x can cause the output value y to rapidly approach zero. The smaller the value, the slower the monotonically decreasing mapping function decays, and the output value decreases gradually as the input value increases. This system preferably uses... Setting it to 0.1 ensures a smoother decay curve for input values ​​within a common range. It should be noted that... The specific value is not fixed and can be flexibly adjusted according to the actual application scenario and the distribution range of the input values.

[0061] For example, taking the calculation of position similarity as an example, the specific calculation process of the monotonically decreasing mapping function is explained: Assuming the Euclidean distance (i.e., the position proximity index) between the attitude joints and their respective relative position coordinates is d, the system uses an exponential decay function for mapping: Position similarity = exp(- ×d), where exp() represents an exponential function with the natural constant as the base.

[0062] It's important to understand that a smaller Euclidean distance (the index of spatial proximity between two attitude joints) indicates a closer spatial position between them. Therefore, positional similarity is used to measure the closeness of the spatial positions of two attitude joints within a pair. A higher positional similarity indicates a closer relative position, while a lower similarity indicates a greater difference in relative position.

[0063] Similarity contribution value = directional similarity × positional similarity.

[0064] It's important to understand that directional similarity and positional similarity together determine the overall similarity between the two posture joints in a given posture joint pair. If both the directional similarity and positional similarity of the posture joint pair are close to one, it indicates that the posture joint pair is similar in both posture and motion, meaning their movements are highly similar.

[0065] For example, if two children's wrists are in similar positions but move in opposite directions, it means that although their postures are similar, their intentions are different. Or, if two children's wrists move in the same direction but their positions are very different, it means that although their intentions are the same, their postures are different.

[0066] In this case, the greater the similarity of the single-frame actions in a pair of frames, the more similar the actions of the two children in that pair of frames are.

[0067] 4) Construct an initial matrix with the number of frames in the current time period of the child to be evaluated as the number of rows and the number of frames in the historical time period of the reference child as the number of columns; fill the calculated single-frame action similarity of each pair of frames into the corresponding positions in the matrix to form a similarity matrix between the current time period of the child to be evaluated and the historical time period of the reference child.

[0068] It's important to understand that children may exhibit differences in speed and time shifts when performing the same actions. For example, one child might wave quickly while another waves slowly. If a frame-by-frame comparison is used, the 5th frame of the fast wave might have a similar posture to the 10th frame of the slow wave, but they cannot be correctly matched. Therefore, the Dynamic Time Warping (VTW) algorithm allows for non-linear alignment on the time axis. Using VTW to calculate the temporal matching distance allows matching frames with similar postures but misaligned times. The temporal matching distance is then converted into behavioral similarity through a monotonically decreasing mapping, ensuring that the smaller the temporal matching distance, the closer the similarity is to one, and the larger the temporal matching distance, the closer the similarity is to zero. This provides an intuitive behavioral similarity value ranging from zero to one, facilitating subsequent judgment and comprehensive evaluation.

[0069] In this embodiment, for each historical time period, each element in the similarity matrix between the current time period and the historical time periods of the child to be evaluated is obtained; the value of each element is subtracted from one to obtain the distance value; the distance values ​​are arranged according to the row and column positions of the similarity matrix to form a distance matrix; the optimal alignment path is searched in the distance matrix using a dynamic time warping algorithm, and the cumulative distance corresponding to the optimal alignment path is taken as the temporal matching distance; the temporal matching distance is divided by the length of the optimal alignment path to obtain the average matching distance; the exponential function value with the natural constant as the base and the negative of the average matching distance as the exponent is calculated as the behavioral similarity.

[0070] It should be noted that the dynamic time warping algorithm is a common technique in this field, and this embodiment can be directly applied. Specifically, a cumulative distance matrix of the same size as the distance matrix is ​​constructed; the element in the first row and first column of the cumulative distance matrix is ​​set to the value of the corresponding element in the distance matrix; for other elements in the first row of the cumulative distance matrix, the cumulative distance of the previous column element is added to the value of the current column element in the distance matrix; for other elements in the first column of the cumulative distance matrix, the cumulative distance of the previous row element is added to the value of the current row element in the distance matrix; for other elements in the cumulative distance matrix, the minimum value among the cumulative distances of the element to its left, the element above, and the element to its upper left is calculated, and this minimum value is added to the value of the current distance matrix element to obtain the cumulative distance of the element; after traversing the entire matrix, the element in the lower right corner of the matrix is ​​the cumulative distance of the optimal alignment path, which is used as the temporal matching distance.

[0071] Temporal matching distance is the minimum cumulative distance between two action feature sequences calculated using a dynamic time warping algorithm. It reflects the overall difference between the two action feature sequences after optimal alignment on the time axis. A smaller temporal matching distance indicates that the two action feature sequences are closer after flexible alignment on the time axis, meaning the two children's actions are more similar. Conversely, a larger temporal matching distance indicates that the two action feature sequences are still significantly different even after optimal alignment, meaning the two children's actions are less similar.

[0072] Behavioral similarity directly reflects the degree of similarity between two action feature sequences. Specifically, the closer the behavioral similarity is to one, the more similar the two action feature sequences are, meaning that the current behavior of the child being evaluated is highly consistent with the historical behavior of the reference child during a certain historical period. Conversely, the closer the behavioral similarity is to zero, the less similar the two action feature sequences are, meaning that the current behavior of the child being evaluated differs significantly from the historical behavior of the reference child during a certain historical period.

[0073] It should be noted that the specific value of the preset threshold can be adjusted according to the strictness of the application scenario, and this embodiment does not impose a specific limitation. For example, in scenarios requiring higher accuracy, the preset threshold can be increased to 0.8, while in scenarios requiring higher recall, the threshold can be decreased to 0.6.

[0074] It's important to understand that the "reference child" in this solution doesn't refer to a specific child, but rather to a broad group of reference children in similar scenarios to the child being evaluated. Each reference child's historical data is divided into multiple independent historical time periods according to a preset first time window. Furthermore, when calculating behavioral similarity, the system iterates through all historical time periods of all reference children, performing independent matching for each time period. Therefore, a reference child may correspond to zero, one, or more similar behavioral time periods: if all historical time periods of a reference child are dissimilar to the current behavior of the child being evaluated, then it corresponds to zero; if only one historical time period is similar, then it corresponds to one; if multiple historical time periods are similar, then it corresponds to multiple.

[0075] The environmental assessment module 103 is used to extract environmental feature points from the period of influence preceding the current period of the child to be assessed, and to determine the environmental impact indicators of each point; to match the environmental feature points of the child to be assessed with the environmental feature points from the period of influence preceding the similar behavior, and to determine the matching feature point pairs; and to determine the environmental similarity based on the matching feature point pairs and their respective environmental impact indicators.

[0076] It is important to understand that, since children's behavior is often a response to recent environmental stimuli, the environmental characteristics at the moment the behavior occurs cannot fully explain the triggering cause. It is necessary to examine the environmental changes over a period preceding the behavior. Therefore, this system sets an influence period before the behavior occurs. By extracting environmental feature points and their change trajectories within the influence period, it quantifies the potential impact of environmental stimuli on children's behavior, thereby linking behavioral analysis with the environmental context and improving the accuracy and interpretability of abnormal behavior assessment.

[0077] As a preferred implementation, the period of influence is a preset second time window preceding the behavior period, and the behavior period includes the current period and historical periods.

[0078] It should be noted that during system operation, there may not be enough video data to construct a complete impact period before the first behavioral period of the child to be evaluated or the first historical period of the child to be referenced. In this case, the system uses the earliest actual time as the starting point of the impact period, that is, extracts environmental feature points as much as possible within the range of available data. If the available time is insufficient, a second time window is preset, and the analysis is based on the actual data, which does not affect the normal execution of the subsequent evaluation process.

[0079] The period of influence is used to extract environmental characteristics before the behavior occurs and to analyze the temporal correlation between environmental stimuli and behavioral responses.

[0080] It should be noted that the preset second time window is determined based on industry experience, and this embodiment does not impose specific limitations. For example, the preset second time window can be set to ten times the duration of the behavior period; for instance, if the behavior period is 30 seconds, the impact period can be set to 5 minutes.

[0081] The process of determining environmental impact indicators is as follows: Figure 2 As shown, it includes: S201: For each frame of the image within the affected time period, the position of the child in the current frame is identified by the object detection algorithm, the child detection box is obtained, and the geometric center of the child detection box is extracted as the center point of the child's position; the corner point, edge point or texture change point in the current frame is identified by the feature point detection algorithm and used as feature points; the feature points belonging to the child's body range are removed by using the area formed by connecting the human pose joints, and the non-human feature points within the child detection box and the feature points outside the box are retained as environmental feature points; and the scale-invariant feature transformation algorithm is used to generate feature description vectors for each environmental feature point.

[0082] It should be noted that the child detection bounding box is obtained using a mature object detection algorithm (such as YOLO) to identify the location of children in the image and output a rectangular box surrounding the child. Object detection algorithms are well-known techniques in the field of computer vision and will not be elaborated upon here.

[0083] The child detection box is a rectangular area surrounding the child, including the coordinates of the top left corner and the bottom right corner.

[0084] The horizontal coordinate of the child's center point is (the horizontal coordinate of the upper left corner + the horizontal coordinate of the lower right corner) ÷ 2, and its vertical coordinate is (the vertical coordinate of the upper left corner + the vertical coordinate of the lower right corner) ÷ 2.

[0085] It should be noted that feature point detection can employ mature corner detection algorithms already in use (such as the Scale Invariant Feature Transform (SIFT) algorithm) to identify corners, edge points, or texture change points in the image. Corner detection algorithms are well-known techniques in the field of computer vision and will not be elaborated upon here.

[0086] Since the child detection bounding box may contain objects the child is holding (such as toys or water cups) or the texture of clothing worn by the child, these feature points are actually part of the environmental stimuli and should not be simply deleted. Therefore, to accurately distinguish between environmental features and the child's own features, this system uses the area formed by connecting the joints of the child's posture to define the child's body range. Specifically, the system obtains the joints of the child's various postures (such as shoulders, hips, elbows, wrists, knees, and ankles) through a posture estimation algorithm, and connects these joints sequentially to form a closed polygon, which serves as the approximate contour of the child's body region. The system removes feature points located within this contour (i.e., feature points belonging to the child's own body), while retaining feature points located outside the contour within the child's detection bounding box (such as toys or water cups held by the child) and feature points outside the child's detection bounding box, which together serve as environmental feature points.

[0087] It should be noted that for each detected feature point, the system uses a scale-invariant feature transform algorithm to generate a feature description vector. This feature description vector describes the local image features surrounding the feature point and is scale-invariant and rotation-invariant, facilitating the matching of the same physical feature point across different frames. The scale-invariant feature transform algorithm is a well-known technique in computer vision and will not be elaborated upon here.

[0088] S202, obtain the pixel coordinates of environmental feature points in the current frame, and obtain the pixel coordinates of the child's position center point in the current frame; calculate the Euclidean distance between the two pixel coordinates; convert the Euclidean distance into a distance factor through a preset monotonically decreasing mapping function.

[0089] Specifically, the Euclidean distance is converted into a distance factor using a monotonically decreasing mapping function, such that the smaller the Euclidean distance, the closer the distance factor is to one; and the larger the Euclidean distance, the closer the distance factor is to zero. The specific form of the pre-defined monotonically decreasing mapping function has been uniformly explained in the preceding sections of this paper and will not be repeated here.

[0090] Children generally pay more attention to environmental stimuli that are closer to them and less to those that are farther away. Therefore, if the Euclidean distance between an environmental feature point and the child's center of location is zero, that feature point has the greatest impact on the child, and the distance factor is set to its maximum value of one. As the Euclidean distance increases, the impact gradually weakens, and the distance factor approaches zero. This monotonically decreasing mapping function reflects the moderating effect of spatial distance on the intensity of environmental influence.

[0091] S203: Record the first frame in which the environmental feature point appears during the period of influence; obtain the displacement distance between adjacent frames of the environmental feature point using optical flow; starting from the first frame in which the environmental feature point appears, accumulate the displacement distance frame by frame to obtain the cumulative displacement distance up to the current frame; convert the cumulative displacement distance into a cumulative displacement mapping value through a preset monotonically increasing mapping function; add the basic retention weight to the cumulative displacement mapping value to obtain the motion factor.

[0092] The displacement distance of environmental feature points between adjacent frames can be obtained using the well-established optical flow method, which can calculate the motion vector of feature points between consecutive frames. Optical flow is a well-known technique in computer vision and will not be elaborated upon here.

[0093] It is important to understand that this scheme uses a pre-defined monotonically increasing mapping function in the motion factor calculation. This function converts input values ​​into output values ​​within the range of zero to one, with larger input values ​​resulting in larger output values. Specifically, the pre-defined monotonically increasing mapping function is used when converting cumulative displacement distance into motion factors in the environmental impact index calculation.

[0094] It should be noted that the specific form of the preset monotonically increasing mapping is not limited to one type and can be flexibly selected according to the application scenario. For example, it can be a proportional function, an exponential function, or a piecewise linear function, as long as it satisfies monotonically increasing and the output range is from zero to one. For example, the preset monotonically increasing mapping function is specifically... or ;in, This represents an adjustment parameter greater than zero, used to control the decay rate of a preset monotonically increasing mapping function.

[0095] For example, taking the calculation of the motion factor as an example, the specific calculation process of the monotonically increasing mapping function is explained: Assuming the cumulative displacement distance of the environmental feature points is s, the system uses a proportional function for mapping, i.e., motion factor = s ÷ (s + β), where β represents the preset scale parameter. Assuming β = 10, if the cumulative displacement distance s = 0, the motion factor = 0; if s = 10, the motion factor = 0.5. This function ensures that the larger the cumulative displacement distance, the closer the motion factor is to one, and the smaller the cumulative displacement distance, the closer the motion factor is to zero, satisfying the requirement of monotonically increasing and an output range from zero to one.

[0096] It should be noted that the preset scale parameter is used to adjust the sensitivity of the monotonic mapping function, controlling the degree to which changes in the input value affect the output value. The specific value of the preset scale parameter can be adjusted according to the characteristics of the scene in practical applications; this embodiment does not impose specific limitations. For example, a smaller value can be used in scenes that need to highlight subtle movements, while a larger value can be used in scenes that need to filter out minor jitter. For instance, it is preferable to set β to 10, meaning that when the cumulative displacement distance reaches 10 pixels, the motion factor is 0.5, and when it reaches 90 pixels, it is 0.9.

[0097] Because this system considers that even if environmental feature points are completely stationary, they may still have some influence on children's behavior (such as a stationary but conspicuous toy), when calculating the motion factor, a basic retention weight is introduced into the motion factor. Specifically, the motion factor = basic retention weight + cumulative displacement mapping value, where the basic retention weight is a preset positive number (e.g., 0.1), and the cumulative displacement mapping value is the value obtained by transforming the cumulative displacement distance through a monotonically increasing mapping function. Therefore, if the cumulative displacement distance is zero, the motion factor equals the basic retention weight, ensuring that stationary feature points still have a minimum level of influence contribution; if the cumulative displacement distance increases, the motion factor increases linearly on the basis of the basic weight, approaching one.

[0098] Children generally pay more attention to moving objects than stationary ones, and the more vigorous the movement, the more attention it attracts. Therefore, if a certain environmental feature point is stationary with a cumulative displacement distance of zero, the motion factor reaches its minimum value of zero, indicating that the environmental feature point is unlikely to attract children's attention due to a lack of movement. As the cumulative displacement distance increases, the environmental feature point becomes more active. By using a pre-set monotonically increasing mapping function, the motion factor gradually increases and approaches one, reflecting the enhancing effect of the degree of movement on attention.

[0099] S204, calculate the time interval from the first occurrence frame of the environmental feature point to the current frame, and use it as the existence duration up to the current frame; convert the existence duration into a duration factor through a preset monotonically decreasing mapping function.

[0100] Children typically develop a habituation effect on long-term environmental stimuli, meaning that objects that have been present for a longer period are more easily ignored. Therefore, if a certain environmental feature appears for the first time, its duration is zero, and the duration factor reaches its maximum value of one, indicating that the newly appearing object is most likely to attract children's attention. As the duration increases, the habituation effect gradually strengthens. By using a pre-set monotonically decreasing mapping function, the duration factor gradually decreases and approaches zero, reflecting the attenuation of attention over time.

[0101] Among them, the larger the environmental impact index of a certain environmental feature point, the stronger the impact of that environmental feature point on children. That is, the closer the environmental feature point is to children, the more active it is, and the higher its novelty, the more likely it is to attract children's attention and induce subsequent behavioral responses. Conversely, the smaller the environmental impact index of a certain environmental feature point, the weaker the impact of that environmental feature point on children. That is, the farther the environmental feature point is from children, the less active it is, or the longer it has existed and the more it has become habitual, the less it triggers children's behavior.

[0102] S205 multiplies the distance factor, motion factor, and duration factor to obtain the environmental impact index of environmental feature points in the current frame.

[0103] It is important to understand that environmental impact indicators are used to quantify the potential influence of environmental features on children's behavior. Children's attention to their surroundings is primarily influenced by three factors: the distance between the environmental stimulus and the child (i.e., the closer the stimulus, the greater the impact), the activity level of the stimulus (i.e., the more vigorous the activity, the more likely it is to attract attention), and the novelty of the stimulus (i.e., the longer it exists, the more likely it is to be habituated and ignored). Therefore, this system multiplies the distance factor, the activity factor, and the duration factor to comprehensively reflect the intensity of the influence of environmental features on children, providing a quantitative basis for subsequent environment-behavior correlation analysis.

[0104] It's important to understand that because the impact period includes multiple frames, each environmental feature point has an instantaneous environmental impact index for each frame, reflecting the degree of its instantaneous impact on the child at a given moment. Directly using these instantaneous indices for subsequent analysis would be computationally burdensome due to the large amount of data, and the instantaneous fluctuations in a single frame might not represent the overall impact intensity of the environmental feature point throughout the entire impact period. Therefore, this system performs a weighted average of the instantaneous environmental impact indices for each frame to obtain the average environmental impact index for each environmental feature point within the impact period, facilitating subsequent calculation and comparison of environmental similarity.

[0105] In this embodiment, for each environmental feature point, the time interval from each frame of the image to the center of the behavior period is extracted as the nearest time. The reciprocal of the nearest time is taken as the temporal influence weight of the environmental feature point in each frame of the image during the influence period. The product of the environmental influence index and the corresponding temporal influence weight of the environmental feature point in each frame of the image during the influence period is calculated as the environmental influence component. The sum of the environmental influence components of the environmental feature point in all frames of the image during the influence period is calculated as the total environmental influence. The sum of the temporal influence weights of the environmental feature point in all frames of the image during the influence period is calculated as the total temporal influence weight. The ratio of the total environmental influence to the total temporal influence weight is calculated as the average environmental influence index of the environmental feature point in the influence period.

[0106] The center moment of a behavior segment refers to the middle moment of the behavior segment associated with the current period of influence. Specifically, it refers to the middle moment of the current time segment of the child being assessed or the middle moment of a similar time window for a reference child, for example, the center of a 30-second segment is the 15th second.

[0107] Because environmental features closer to the moment a behavior occurs have a stronger triggering effect on the behavior, and a weaker effect the farther away they are, this reflects the law that the influence of environmental stimuli on behavior decays over time. Therefore, frames closer to the center moment of the behavior period receive a larger temporal influence weight, while frames farther away receive a smaller temporal influence weight, meaning that environmental features closer to the moment the behavior occurs have a stronger triggering effect on the behavior.

[0108] It's important to note that the influence period is defined as the time window preceding the action period; all frames within the influence period occur before the start of the action period. Since the center time of the action period is located within it, the time difference between any frame within the influence period and the center time of the action period is always positive and cannot be zero. In other words, no frame within the influence period is exactly at the center time of the action period. Therefore, the proximity time is always greater than zero.

[0109] The environmental impact component reflects the intensity of the influence of environmental feature points in a certain frame on children's behavior, while also taking into account the importance of that frame on the time axis.

[0110] Among them, the larger the average environmental impact index of a certain environmental feature point, the closer the environmental feature point is to the children, the more active the movement, and the higher the novelty during the period of influence, and the stronger the potential triggering effect on children's behavior. Conversely, the smaller the average environmental impact index of a certain environmental feature point, the farther the environmental feature point is from the children, the less active the movement, or the longer it has existed and has become habitual, and the weaker the triggering effect on children's behavior.

[0111] In summary, the average environmental impact index is the weighted average of the environmental impact indices of each frame, with the time impact weight as the weighting coefficient. This is to highlight the impact of environmental feature points closer to the center of the behavior period on the behavior, while weakening the contribution of environmental feature points farther away from the behavior period.

[0112] It is important to understand that because the sets of environmental feature points extracted from the child being evaluated and the reference child differ in both quantity and location during their respective periods of influence, the same physical environmental feature (such as the same edge of a toy) may correspond to different feature point identifiers in the two sets, making direct point-by-point comparison impossible. Therefore, this system uses a feature point matching algorithm to match the environmental feature points of the child being evaluated with those of the reference child, identifying successfully matched feature point pairs. This ensures that only environmental feature points with similar visual semantics and texture characteristics can participate in subsequent environmental similarity calculations, guaranteeing the comparability of the compared objects.

[0113] To completely and accurately match the environmental feature points of the child to be evaluated with the environmental feature points of the period influencing the similar behavior, and to determine the matching feature point pairs, as a preferred implementation method, a first set of environmental feature points of the child to be evaluated during the period influencing the similar behavior, and a second set of environmental feature points of the child to be evaluated during the period influencing the similar behavior, are obtained. Each environmental feature point in the first and second sets corresponds to an environmental feature time sequence, which is composed of the feature description vectors of the environmental feature points in each frame of the period influencing the similar behavior, arranged chronologically. A dynamic time warping algorithm is used to calculate the first temporal matching distance between the environmental feature time sequences of each environmental feature point in the first set and the environmental feature time sequences of each environmental feature point in the second set. A nearest neighbor matching algorithm is used to find the environmental feature point with the smallest first temporal matching distance from the second set for each environmental feature point in the first set, as a candidate matching point. If the smallest first temporal matching distance is less than a preset matching threshold, the candidate matching point and its related environmental feature points in the first set are determined as a successfully matched environmental feature point pair, denoted as a matching feature point pair.

[0114] It is important to note that there may be multiple similar behavior time periods for the reference child group, and each similar behavior time period corresponds to an independent influence time period. Therefore, the second set of environmental feature points is not a large set of environmental feature points for all similar behavior time periods, but rather is extracted separately for each similar behavior time period. That is, the system iterates through each similar behavior time period one by one, and at each time takes the second set of environmental feature points within the influence time period corresponding to one of the similar behavior time periods, and matches it with the first set of environmental feature points for the child to be evaluated to obtain the matching feature point pair under that similar behavior time period. After all similar behavior time periods have been traversed, a set of matching feature point pairs will be generated for each similar behavior time period, which will be used for subsequent environmental similarity calculations for that similar behavior time period. It can be understood that the matching feature point pairs belong to each of the various similar behavior time periods, rather than being a cross-time period mixed match.

[0115] It should be noted that the Euclidean distance or cosine distance between two feature description vectors is used as the local distance metric for the Dynamic Time Warping (DTW) algorithm at each time step.

[0116] The first temporal matching distance reflects the similarity of the change trajectories of two environmental feature points on the time axis. The smaller the first temporal matching distance, the more consistent the visual feature change process of the two environmental feature points are, and the more likely they are environmental feature points with similar visual semantics and texture characteristics.

[0117] Candidate matching points are reference children's environmental feature points that are most similar to the children's environmental feature points to be evaluated in terms of visual change trajectory.

[0118] It should be noted that the specific value of the preset matching threshold can be adjusted according to the characteristics of the scene. For example, in scenes with dense environmental feature points, the threshold can be appropriately lowered to reduce false matches, while in scenes with sparse environmental feature points, the threshold can be appropriately increased to increase the number of matches. This embodiment does not impose specific limitations. For example, 0.6 is preferably used as the preset matching threshold.

[0119] It should be noted that if the minimum temporal matching distance is less than the preset matching threshold, it means that the visual change trajectories of the two environmental feature points are highly similar, and they can be determined to be environmental feature points with similar visual semantics and texture characteristics. At this time, the candidate matching point and its corresponding environmental feature point in the first set of environmental feature points are determined as a successfully matched pair of environmental feature points, and are recorded as a matched feature point pair.

[0120] The process of determining environmental similarity includes the following steps: 1) Obtain successfully matched feature point pairs. Each matched feature point pair includes the first environmental feature point of the child to be evaluated, the second environmental feature point of the reference child, and the first temporal matching distance between the two.

[0121] It should be noted that if the number of successfully matched feature point pairs is zero, the environmental similarity is directly set to zero.

[0122] 2) For each pair of matching feature points, the first temporal matching distance between the two is converted into environmental feature similarity through a preset monotonically decreasing mapping function; the product of the average environmental impact index of the first environmental feature point during its influence period and the average environmental impact index of the second environmental feature point during its influence period is calculated as the important weight of the matching feature point pair; the environmental feature similarity is multiplied by the important weight to obtain the environmental feature similarity contribution of the matching feature point pair.

[0123] In essence, for each pair of matching feature points, the system converts the first temporal matching distance between them into environmental feature similarity using a preset monotonically decreasing mapping function. This preset monotonically decreasing mapping function ensures that the smaller the temporal matching distance, the closer the environmental feature similarity is to one; conversely, the larger the temporal matching distance, the closer the environmental feature similarity is to zero. The specific form of the preset monotonically decreasing mapping function has been uniformly explained in the preceding sections of this paper and will not be repeated here.

[0124] Environmental feature similarity reflects the degree of similarity between two environmental feature points in a pair of matched feature points along their visual change trajectories. A higher environmental feature similarity indicates that the visual change processes of the two environmental feature points within the affected time period are more consistent, suggesting that they are more likely to have similar visual semantics and texture characteristics and have undergone similar change processes. Conversely, a closer environmental feature similarity to zero indicates a greater difference in the visual change trajectories of the two environmental feature points; even if they are successfully matched, their change processes are not similar.

[0125] The importance weight reflects the importance of environmental feature points in a pair of matching feature points to the children's behavior during their respective periods of influence: the higher the degree of influence of the two environmental feature points, the larger their product, and the greater their contribution to the overall environmental similarity; conversely, if the degree of influence of any environmental feature point is low, its importance weight is small, and its contribution to the overall similarity is correspondingly reduced.

[0126] The contribution of environmental feature similarity comprehensively reflects the combined contribution of a matching feature point in the two dimensions of "whether it is similar" and "whether it is important".

[0127] 3) Sum the environmental feature similarity contributions of all matching feature point pairs to obtain the contribution numerator; sum the importance weights of all matching feature point pairs to obtain the contribution denominator.

[0128] 4) Divide the contribution numerator by the contribution denominator to obtain the environmental similarity.

[0129] It should be noted that since the importance weight of each matched feature point pair is composed of the product of two average environmental impact indicators, and these average environmental impact indicators are all positive, the importance weight of each matched feature point pair is always greater than zero. When at least one environmental feature point pair is successfully matched, the contribution denominator is the sum of multiple positive numbers, which is necessarily greater than zero. If no successfully matched feature point pair exists, the environmental similarity calculation process does not proceed. Therefore, in the actual environmental similarity calculation, the contribution denominator is always greater than zero.

[0130] It is important to understand that the greater the environmental similarity, the more consistent the environmental triggers of the child being evaluated's current behavior are with those of similar behaviors in reference children, and the more typical the current behavior is. Conversely, the closer the environmental similarity is to zero, the greater the difference in environmental triggers, and the more likely the current behavior is to be unrelated to the scene and environment, and the more atypical the behavioral trigger pattern is.

[0131] The anomaly identification module 104 is used to determine the degree of behavioral deviation of the child to be evaluated based on environmental similarity and the number of similar behavior time periods; to determine whether to trigger an early warning based on the degree of behavioral deviation and preset conditions; and to send an early warning notification if triggered.

[0132] The process of determining behavioral deviation includes the following steps: 1) Obtain the number of similar behavior time periods and the environmental similarity corresponding to each similar behavior time period.

[0133] It should be noted that if the number of similar behavior time periods is zero, the environmental similarity calculation is skipped, and the behavior deviation is directly set to the maximum value of one.

[0134] 2) Convert the number of similar behavior periods into a quantity influence factor using a preset monotonically decreasing mapping function.

[0135] Since the number of similar behavior periods reflects the prevalence of the current behavior in a homogeneous setting, a small number of similar behavior periods indicates that the current behavior is rarely seen in the reference group of children, suggesting that the behavior itself is unique and worthy of attention even if environmental factors are consistent. Therefore, this system converts the number of similar behavior periods into a quantitative influence factor through a monotonically decreasing mapping. The fewer the number of similar behavior periods, the closer the quantitative influence factor is to one, directly contributing a higher degree of behavioral deviation; conversely, the more the number of similar behavior periods, the closer the quantitative influence factor is to zero, indicating the prevalence of the behavior.

[0136] It should be noted that the specific form of the pre-defined monotonically decreasing mapping function has been uniformly explained in the preceding sections of this article, and will not be repeated here.

[0137] 3) Calculate the difference between the environmental similarity of one and each similar behavior time period, and use it as the environmental dissimilarity corresponding to each similar behavior time period.

[0138] Specifically, if the environmental similarity of a similar behavior period is closer to one and the environmental dissimilarity is closer to zero, it means that the environmental triggers of the behavior within that similar behavior period are more consistent with the current behavior; if the environmental similarity of a similar behavior period is closer to zero and the environmental dissimilarity is closer to one, it means that the environmental triggers of the behavior within that similar behavior period are more different.

[0139] 4) Calculate the arithmetic mean of the environmental dissimilarity for all similar behavior time periods, and use it as the average environmental dissimilarity.

[0140] Average environmental dissimilarity reflects the average degree of difference in environmental factors among all similar behaviors. A higher average environmental dissimilarity indicates a greater overall difference between the environmental triggers of the similar behavior and the current behavior.

[0141] 5) Take the quantity impact factor as the first term; subtract the quantity impact factor from one to get the environmental impact weight; multiply the environmental impact weight by the average environmental dissimilarity to get the second term; add the first term and the second term to get the behavioral deviation degree.

[0142] Since the environmental impact weight reflects the proportion of environmental factors in the overall assessment—that is, the fewer similar behaviors, the smaller the environmental impact weight; and the more similar behaviors, the larger the environmental impact weight—it is used to adjust the contribution of environmental factors to behavioral deviation. If the number of similar behavior periods is small and the quantitative influence factor is large, then behavioral deviation is mainly determined by quantitative factors, and the reference value of environmental factors is limited. In this case, the environmental impact weight (minus the quantitative influence factor) is small, weakening the contribution of environmental factors. Conversely, if the number of similar behavior periods is large and the quantitative influence factor is small, it indicates that the behavior is prevalent in the group. In this case, the consistency of environmental factors becomes the key to judgment, so the environmental impact weight is large, strengthening the contribution of environmental factors.

[0143] It's important to understand that the formula for calculating behavioral deviation balances two dimensions: the number of similar behaviors and the consistency of environmental factors. If there are few similar behaviors, the deviation is primarily determined by the number of similar behaviors; if there are many similar behaviors, the deviation is primarily determined by the consistency of environmental factors. Therefore, a larger behavioral deviation indicates that the current behavior deviates more from the typical behavioral pattern and requires more attention; a smaller behavioral deviation indicates that the current behavior conforms more to the typical behavioral pattern, and the assessment result tends to be normal.

[0144] As one possible implementation method, the preset conditions include the following two conditions: Condition 1 is that if the behavioral deviation of the child to be evaluated in a consecutive preset number of first time windows is not less than the first threshold, it is determined that the continuous warning condition is met; if the increment of the behavioral deviation in adjacent first time windows (i.e. the absolute difference between the two behavioral deviations) is not less than the second threshold, it is determined that the sudden warning condition is met.

[0145] It should be noted that an early warning is triggered when either the continuous early warning condition or the sudden early warning condition is met; the two conditions are related by an OR logic.

[0146] It should be noted that the specific value of the preset quantity is determined based on industry experience, and this embodiment does not impose a specific limitation. For example, it is preferable to set the preset quantity to 2.

[0147] It should be noted that the first threshold is used to determine whether the behavioral deviation within a single time window reaches a level requiring attention, while the second threshold is used to determine whether the magnitude of change in the behavioral deviation reaches a level requiring attention. The specific values ​​of the first and second thresholds can be determined based on industry experience. In practical applications, the first and second thresholds can be adjusted according to the stringency of the application scenario. Generally, the thresholds can be lowered in scenarios requiring higher sensitivity and raised in scenarios requiring reduced false alarms. For example, it is preferable to set the first threshold to 0.5 and the second threshold to 0.5.

Claims

1. An online system for identifying abnormal behaviors in children based on computer vision, characterized in that, The system includes: The acquisition module is used to collect historical and current behavioral video data of the child to be evaluated and a reference child in a similar scenario; the behavioral video data is analyzed frame by frame to extract the action feature indicators of each child in each frame and form action feature sequences respectively. The behavior assessment module is used to determine the single-frame action similarity and form a similarity matrix based on the current action feature sequence of the child to be assessed and the historical action feature sequence of the reference child; to perform temporal matching of the similarity matrices of the child to be assessed and the reference child, determine the temporal matching distance and convert it into behavior similarity; if the behavior similarity is greater than a preset threshold, the corresponding historical time period is recorded as the similar behavior time period; The environmental assessment module is used to extract environmental feature points from the periods preceding the current time of the child to be assessed and to determine the environmental impact indicators for each point; it matches the environmental feature points of the child to be assessed with the environmental feature points from the periods preceding the time of similar behavior to determine matching feature point pairs; and it determines the environmental similarity based on the matching feature point pairs and their respective environmental impact indicators. The anomaly detection module is used to determine the degree of behavioral deviation of the child to be evaluated based on environmental similarity and the number of similar behavior time periods; based on the degree of behavioral deviation and preset conditions, it determines whether to trigger an early warning; if triggered, it sends an early warning notification.

2. The online identification system for abnormal child behavior based on computer vision according to claim 1, characterized in that, The acquisition module continuously acquires each frame of image at a preset frame rate, and performs aggregate analysis on multiple consecutive frames of images within a preset first time window as a period. Based on the children's historical behavioral video data, the data is divided into multiple historical periods according to a preset first time window, and each historical period corresponds to a set of behavioral video data.

3. The online identification system for abnormal child behavior based on computer vision according to claim 2, characterized in that, The frame-by-frame analysis of behavioral video data extracts the action feature indicators of each child in each frame, forming action feature sequences, including: Identify the child's pose joints in each frame of behavioral video data; obtain the pixel coordinates of the pose joints in the image; A relative coordinate system is constructed with the preset torso reference center as the origin. For each posture joint, the difference between the pixel coordinates of the posture joint and the pixel coordinates of the torso reference center in each frame is calculated to obtain the relative position coordinates of the posture joint. For each pose joint, obtain the relative position coordinates of the same pose joint in two adjacent frames; calculate the difference between the relative position coordinates of the next frame and the relative position coordinates of the previous frame to obtain the motion vector of the pose joint in the next frame; wherein, the motion vector of the first frame is set to a preset initial value. The relative position coordinates of each pose joint point in the same frame are combined with the motion vector to form the motion feature index of the same frame. The motion feature indicators of each frame are arranged in chronological order to form a motion feature sequence.

4. The online identification system for abnormal child behavior based on computer vision according to claim 3, characterized in that, The similarity matrix formation process includes: For each historical time period, any frame of the current time period of the child to be evaluated and any frame of the historical time period of the reference child are combined into a pair of frames; and the same pose key point in the pair of frames is combined into a pose key point pair. For each pair of pose joints in each set of frames, the cosine similarity of the motion vectors of each pose joint pair is calculated as an index of orientation consistency; the Euclidean distance of the relative position coordinates of each pose joint pair is calculated as an index of position proximity. The orientation consistency index and the position proximity index are normalized and mapped separately and then combined to obtain the similarity contribution value of the pose joint pairs; the arithmetic mean of the similarity contribution values ​​of all pose joint pairs in the frame is calculated as the single-frame action similarity of the frame. An initial matrix is ​​constructed using the number of frames in the current time period of the child to be evaluated as the number of rows and the number of frames in the historical time period of the reference child as the number of columns. The calculated single-frame action similarity of each pair of frames is then filled into the corresponding positions in the matrix to form a similarity matrix between the current time period of the child to be evaluated and the historical time period of the reference child.

5. The online identification system for abnormal child behavior based on computer vision according to claim 4, characterized in that, The step of performing temporal matching of the similarity matrices between the child to be evaluated and the reference child, determining the temporal matching distance, and converting it into behavioral similarity includes: For each historical period, obtain each element in the similarity matrix between the current period and the historical periods for the child to be evaluated; subtract the value of each element from one to obtain the distance value; arrange the distance values ​​according to the row and column positions of the similarity matrix to form a distance matrix; A dynamic time warping algorithm is used to search for the optimal alignment path in the distance matrix, and the cumulative distance corresponding to the optimal alignment path is used as the temporal matching distance. The average matching distance is obtained by dividing the temporal matching distance by the optimal alignment path length; the value of the exponential function with the natural constant as the base and the negative of the average matching distance as the exponent is calculated as the behavioral similarity.

6. The online identification system for abnormal child behavior based on computer vision according to claim 1, characterized in that, The period of influence is a preset second time window preceding the period of behavior, and the period of behavior includes the current period and historical periods. The process for determining the environmental impact indicators includes: For each frame of the image within the affected period, the position of the child in the current frame is identified using a target detection algorithm to obtain the child detection box, and the geometric center of the child detection box is extracted as the center point of the child's position. The corner point, edge point, or texture change point in the current frame is identified using a feature point detection algorithm as feature points. The feature points belonging to the child's body area are removed by using the area formed by connecting the human pose joints, and the non-human feature points within the child detection box and the feature points outside the box are retained as environmental feature points. The scale-invariant feature transformation algorithm is used to generate feature description vectors for each environmental feature point. Obtain the pixel coordinates of environmental feature points in the current frame, and obtain the pixel coordinates of the child's center point in the current frame; calculate the Euclidean distance between the two pixel coordinates; convert the Euclidean distance into a distance factor using a preset monotonically decreasing mapping function; Record the first frame in which the environmental feature point appears during the period of influence; obtain the displacement distance of the environmental feature point between adjacent frames using optical flow; starting from the first frame in which the environmental feature point appears, accumulate the displacement distance frame by frame to obtain the cumulative displacement distance up to the current frame; convert the cumulative displacement distance into a cumulative displacement mapping value using a preset monotonically increasing mapping function; add the basic retention weight to the cumulative displacement mapping value to obtain the motion factor; Calculate the time interval from the first appearance frame of an environmental feature point to the current frame, and use it as the existence duration up to the current frame; convert the existence duration into a duration factor using a preset monotonically decreasing mapping function; Multiply the distance factor, motion factor, and duration factor to obtain the environmental impact index of environmental feature points in the current frame.

7. The online identification system for abnormal child behavior based on computer vision according to claim 6, characterized in that, The system also includes: For each environmental feature point, the time interval between each frame of the image and the center of the behavior period is extracted and used as the nearest time. The reciprocal of the nearest time is taken as the temporal influence weight of the environmental feature point in each frame of the image during the period of influence. The product of the environmental impact index and the corresponding time impact weight of each frame of the environmental feature point during the period of influence is calculated as the environmental impact component; the sum of the environmental impact components of all frames of the environmental feature point during the period of influence is calculated as the total environmental impact. The sum of the temporal influence weights of environmental feature points across all frames within the influence period is calculated and used as the total temporal influence weight. The ratio of the total environmental impact to the total time impact weight is calculated and used as the average environmental impact index of environmental feature points during the impact period.

8. The online identification system for abnormal child behavior based on computer vision according to claim 7, characterized in that, The process of matching environmental feature points of the child to be evaluated with environmental feature points of the period of influence preceding similar behavior to determine matching feature point pairs includes: Obtain the first set of environmental feature points for the child to be evaluated during the period of influence before the current time period, and the second set of environmental feature points for the child to be evaluated during the period of influence before the similar behavior time period; wherein, each environmental feature point in the first set of environmental feature points and the second set of environmental feature points corresponds to an environmental feature time sequence, and the environmental feature time sequence is composed of the feature description vectors of the environmental feature points in each frame during the period of influence arranged in chronological order; The dynamic time warping algorithm is used to calculate the first temporal matching distance between the environmental feature time series sequence of each environmental feature point in the first set of environmental feature points and the environmental feature time series sequence of each environmental feature point in the second set of environmental feature points; The nearest neighbor matching algorithm is used to find the environmental feature point with the smallest first temporal matching distance from the second environmental feature point set for each environmental feature point in the first environmental feature point set as a candidate matching point; If the minimum first temporal matching distance is less than the preset matching threshold, the candidate matching point and its related environmental feature points in the first set of environmental feature points are determined as successfully matched environmental feature point pairs and recorded as matched feature point pairs.

9. The online identification system for abnormal child behavior based on computer vision according to claim 8, characterized in that, The environmental similarity determination process includes: Obtain successfully matched feature point pairs. Each matched feature point pair includes the first environmental feature point of the child to be evaluated, the second environmental feature point of the reference child, and the first temporal matching distance between the two. For each pair of matching feature points, the first temporal matching distance between the two is converted into environmental feature similarity through a preset monotonically decreasing mapping function; the product of the average environmental impact index of the first environmental feature point during its influence period and the average environmental impact index of the second environmental feature point during its influence period is calculated as the important weight of the matching feature point pair; the environmental feature similarity is multiplied by the important weight to obtain the environmental feature similarity contribution of the matching feature point pair. The contribution numerator is obtained by summing the environmental feature similarity contributions of all matching feature point pairs; the contribution denominator is obtained by summing the importance weights of all matching feature point pairs. Divide the contribution numerator by the contribution denominator to obtain the environmental similarity.

10. The online identification system for abnormal child behavior based on computer vision according to claim 1, characterized in that, The process for determining the degree of behavioral deviation includes: Obtain the number of similar behavior time periods and the environmental similarity corresponding to each similar behavior time period; The number of similar behavior time periods is converted into a quantitative influence factor through a preset monotonically decreasing mapping function; Calculate the difference between the environmental similarity of a given time period and the environmental similarity of each similar behavior time period, and use this difference as the environmental dissimilarity of each similar behavior time period. Calculate the arithmetic mean of the environmental dissimilarity for all similar behavior time periods, and use it as the average environmental dissimilarity; The quantity impact factor is taken as the first term; the quantity impact factor is subtracted from one to obtain the environmental impact weight; the environmental impact weight is multiplied by the average environmental dissimilarity to obtain the second term; the first term and the second term are added together to obtain the behavioral deviation degree.

Images