Growth file dynamic recording and tracking system for children with development disorder

Abstract

The invention discloses a growth file dynamic recording and tracking system for children with development disorder, which relates to the technical field of children development disorder evaluation and rehabilitation and comprises a data acquisition module, a feature extraction module, a feature fusion module, a semantic projection module and a semantic output module. The system extracts a behavior feature vector and an environment feature vector by collecting behavior text records, audio and video streams, course types, noise spectrums, interaction objects and other environment context data, and generates an environment-behavior fusion vector by using a cross attention mechanism; the fusion vector is projected to a semantic space to calculate semantic similarity, and the result confidence is judged according to the difference between the highest similarity and the second highest similarity; if not, re-extracting the environment features based on an extended time window, carrying out weighted fusion to generate an environment feature enhancement vector, carrying out fusion and semantic projection again, and outputting a semantic category; the method can ensure that the growth archive keeps context consistency and development coherence in long-term tracking.

Description

Technical Field

[0001] This invention relates to the field of assessment and rehabilitation technology for children with developmental disabilities, specifically a dynamic recording and tracking system for the growth profiles of children with developmental disabilities. Background Technology

[0002] In the rehabilitation training and daily observation of children with developmental disabilities, therapists are accustomed to recording the children's behavioral texts (such as screaming, clapping) and simultaneously collecting audio and video data at the time; through multimodal fusion, the text records and audio and video evidence can be automatically linked on the timeline to enhance the credibility of the records.

[0003] However, the same objective behavior (such as loud screaming or running fast) has drastically different meanings in different environmental contexts (such as noisy play classes or quiet individual training classes). Although existing solutions have established a consensus on the temporal correlation of multimodal data, they lack the ability to perceive and interpret environmental context. This may lead to the misidentification of a child's excited screaming in a play class and anxious screaming in an individual training class as the same "high-volume vocalization" behavior, resulting in completely wrong interpretations of the behavior and assessment conclusions that contradict the actual situation.

[0004] Therefore, there is an urgent need for a technical solution that can integrate intelligent perception and judgment of environmental context while associating behavioral and media data, thereby generating behavioral records that not only have time anchors but also contextualized semantic annotations, fundamentally improving the interpretability and usability of automated recording. Summary of the Invention

[0005] In view of the shortcomings of existing technologies, the present invention provides a dynamic recording and tracking system for the growth records of children with developmental disabilities.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention discloses a dynamic recording and tracking system for the growth profiles of children with developmental disabilities, comprising:

[0008] The data acquisition module is used to acquire behavioral text records of the target object, synchronously collected audio and video streams, and environmental context metadata; the environmental context metadata includes course type, environmental noise spectrum, and interactive objects;

[0009] The feature extraction module is used to extract behavioral feature vectors and environmental feature vectors from the audio and video streams and the environmental context metadata based on the timestamps of the behavioral text records;

[0010] The feature fusion module is used to fuse the environment feature vector as a query vector and the behavior feature vector as a key vector and value vector through a cross-attention fusion mechanism to obtain an environment-behavior fusion vector.

[0011] The semantic projection module is used to project the environment-behavior fusion vector onto a predefined semantic space and calculate its semantic similarity with the prototype vectors of each semantic category in the semantic space.

[0012] The semantic output module is used to determine whether the difference between the highest and second-highest semantic similarity scores among all semantic similarities exceeds a preset confidence threshold.

[0013] If yes, output the corresponding semantic category; otherwise, re-extract the environmental feature vector based on the preset extended time window and perform weighted fusion with the original environmental feature vector to generate an environmental feature enhancement vector.

[0014] Based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

[0015] Secondly, this invention discloses a method for dynamically recording and tracking the growth profiles of children with developmental disabilities, comprising the following steps:

[0016] The system acquires behavioral text records of the target object, synchronously collected audio and video streams, and environmental context metadata; the environmental context metadata includes course type, environmental noise spectrum, and interactive objects.

[0017] Based on the timestamps of the behavioral text records, behavioral feature vectors and environmental feature vectors are extracted from the audio and video streams and the environmental context metadata.

[0018] By using a cross-attention fusion mechanism, the environmental feature vector is used as the query vector, and the behavioral feature vector is used as the key vector and value vector for fusion processing to obtain the environment-behavior fusion vector;

[0019] The environment-behavior fusion vector is projected onto a predefined semantic space, and its semantic similarity with the prototype vectors of each semantic category in the semantic space is calculated.

[0020] Determine whether the difference between the highest and second-highest semantic similarity scores among all semantic similarities exceeds a preset confidence threshold:

[0021] If yes, output the corresponding semantic category; otherwise, re-extract the environmental feature vector based on the preset extended time window and perform weighted fusion with the original environmental feature vector to generate an environmental feature enhancement vector.

[0022] Based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. By introducing environmental context information such as course type, environmental noise spectrum and interactive objects into the feature fusion process, and using a cross-attention mechanism for environment-behavior fusion, it is possible to distinguish differences in behaviors that appear similar in different scenarios.

[0025] 2. By introducing an environmental pattern memory module and a temporal modeling module, the system can not only correct the current semantic distribution based on historical environmental patterns, but also extract behavioral development trends by combining long-term semantic category sequences. This avoids isolated analysis of single behavioral fragments and ensures that the growth portfolio maintains contextual consistency and developmental coherence in long-term tracking. This provides more realistic dynamic data support for rehabilitation assessment and personalized intervention. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an overall block diagram of the system according to Embodiment 1 of the present invention;

[0028] Figure 2 This is a system execution flowchart of Embodiment 1 of the present invention;

[0029] Figure 3 This is an overall block diagram of the method in Embodiment 2 of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In traditional multimodal behavior recording systems, while the time-synchronized association mechanism between behavioral text and audio / video data can achieve basic behavior anchoring, environmental context features are not involved in the multimodal fusion process. Because dynamic factors such as environmental noise spectrum, course type, and interactive objects are not encoded into computable feature vectors, the system cannot establish a mapping between behavioral semantics and environmental state, leading to biases in cross-scene behavior similarity calculations. This deficiency directly reduces the contextual consistency of semantic annotation, causing the system's output semantic categories to fail to reflect behavioral intent in real-world situations, thus affecting the dynamic tracking accuracy of the growth portfolio.

[0032] For example, in a rehabilitation training scenario, the system simultaneously collects audio and video streams of children's high-volume screaming behavior during play-based and individual training sessions, and extracts Mel spectrograms and optical flow features to generate behavioral feature vectors. Because the environmental feature submodule only maps course types to discrete labels and does not generate continuous vectors, the feature fusion stage cannot distinguish the differences in noise energy distribution between the two course environments through an attention mechanism. When the audio and video behavioral feature vectors are fused with metadata that does not carry environmental semantics, the computation process projecting them into the semantic space ignores the constraint of course type on behavior interpretation, leading to excited vocalizations in play-based sessions and anxious vocalizations in individual training sessions being incorrectly classified into the same semantic category.

[0033] If the aforementioned problems are not addressed, the temporal correlation of multimodal data will generate significant semantic annotation noise, preventing behavioral development trajectory analysis models from capturing the evolutionary patterns of environment-sensitive behaviors. Long-term accumulation of mislabeled data will distort children's ability assessment reports, causing intervention plans to deviate from actual needs. More seriously, unreliable semantic labels output by the system may trigger a crisis of trust in automated recording mechanisms among therapists, hindering the practical application of digital assessment tools in clinical settings.

[0034] To address the aforementioned challenges, this application first considers how to dynamically encode environmental context features into computable continuous vectors and establish a mapping between them and behavioral features. To overcome the semantic loss caused by the discretization of environmental metadata in existing technologies, this application attempts to map course types and interactive objects into continuous vectors through an embedding layer. Simultaneously, it extracts noise spectrum features through principal component analysis to form a multidimensional environmental feature vector. To resolve the attention distraction problem caused by simply concatenating behavioral and environmental features, a cross-attention mechanism is explored, using environmental features as query vectors to proactively guide the selection and reorganization of behavioral features. When the confidence level of the semantic projection result is insufficient, the application further considers expanding the time window to re-extract environmental features and enhancing the robustness of environmental representation through weighted fusion, thereby overcoming the limitations of environmental information in a single time segment.

[0035] Example 1:

[0036] like Figure 1-2 As shown, the dynamic recording and tracking system for the growth portfolios of children with developmental disabilities includes:

[0037] The data acquisition module is used to acquire behavioral text records of the target object, synchronously collected audio and video streams, and environmental context metadata; the environmental context metadata includes course type, environmental noise spectrum, and interactive objects;

[0038] The feature extraction module is used to extract behavioral feature vectors and environmental feature vectors from audio and video streams and environmental context metadata based on the timestamps of behavioral text records;

[0039] The feature fusion module is used to fuse environmental feature vectors as query vectors and behavioral feature vectors as key and value vectors through a cross-attention fusion mechanism to obtain an environment-behavior fusion vector.

[0040] The semantic projection module is used to project the environment-behavior fusion vector onto a predefined semantic space and calculate its semantic similarity with the prototype vectors of each semantic category in the semantic space.

[0041] The semantic output module is used to determine whether the difference between the highest and second-highest semantic similarity among all semantic similarities exceeds a preset confidence threshold: if yes, the corresponding semantic category is output; otherwise, the environmental feature vector is re-extracted based on a preset extended time window and weighted and fused with the original environmental feature vector to generate an environmental feature enhancement vector; based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

[0042] The data acquisition module refers to the component used to collect behavioral text records of the target object, synchronously collected audio and video streams, and environmental context metadata. Specifically, it can be implemented using sensors, cameras, microphones, and metadata annotation tools to acquire multi-dimensional data to support subsequent analysis.

[0043] The feature extraction module is a component that extracts behavioral feature vectors and environmental feature vectors from audio and video streams and environmental context metadata based on the timestamps of behavioral text records. Specifically, it can be implemented using techniques such as principal component analysis, convolutional neural networks, and embedding layers to transform raw data into structured features to support semantic analysis.

[0044] The feature fusion module is a component that uses a cross-attention mechanism to fuse environmental feature vectors as query vectors, behavioral feature vectors as key vectors and value vectors. Specifically, it can be implemented using a multi-head attention mechanism or a self-attention mechanism to dynamically associate environmental and behavioral features to enhance the accuracy of semantic expression.

[0045] The semantic projection module is a component that maps environment-behavior fusion vectors to a predefined semantic space and calculates the similarity with prototype vectors of each semantic category. Specifically, it can be implemented using cosine similarity or Euclidean distance calculation, and is used to match fused features with semantic categories to determine the meaning of behavior.

[0046] The semantic output module is a component that determines whether the time window needs to be expanded to re-extract environmental feature vectors based on the difference between the highest and second-highest semantic similarity. Specifically, it can be implemented using threshold comparison algorithms and weighted fusion methods to dynamically adjust the confidence of environmental features in order to improve the robustness of semantic discrimination.

[0047] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0048] The data acquisition module collects behavioral text records, audio and video streams, and environmental context metadata of children with developmental disabilities during rehabilitation training. Behavioral text records are entered in real-time by therapists and include behavioral descriptions and timestamps. Audio and video streams are captured by a camera and microphone array permanently installed in the training room. In the environmental context metadata, the course type is obtained from the rehabilitation center's curriculum scheduling system, the environmental noise spectrum is captured in real-time by the microphone array, and the interacting objects are identified through facial recognition technology in the video.

[0049] The feature extraction module first extracts audio and video segments corresponding to the time period from the audio and video streams based on the timestamps of the behavioral text records. Mel-spectrum features are extracted from the audio segments, and optical flow features are extracted from the video segments. These are then encoded into audio and video feature vectors respectively using a convolutional neural network. These two vectors are concatenated to form the behavioral feature vector. For environmental features, principal component analysis is performed on the environmental noise spectrum to extract noise feature vectors. An embedding layer maps the course type and interactive object into continuous vectors, and these three vectors are then concatenated to generate the environmental feature vector.

[0050] The feature fusion module uses a multi-head attention mechanism to achieve cross-attention fusion. The environment feature vector is used as the query vector, and the behavior feature vector is used as the key and value vectors. By calculating attention weights and performing a weighted sum, the environment-behavior fusion vector is obtained.

[0051] The semantic projection module uses a pre-trained semantic space model to project the environment-behavior fusion vector onto this semantic space through a fully connected layer. Then, it calculates the cosine similarity between the projection result and the prototype vectors of each semantic category in the semantic space to obtain the semantic similarity vector.

[0052] The semantic output module first identifies the maximum and second-largest values ​​in the semantic similarity vector and calculates their difference. If the difference exceeds a preset confidence threshold, the semantic category corresponding to the maximum value is directly output. Otherwise, the environmental feature vector is re-extracted based on a preset extended time window (e.g., 15 seconds before and after), and a weighted average is performed with the original environmental feature vector to generate an enhanced environmental feature vector. This enhanced vector is then used for cross-attention fusion and semantic projection, and finally, a new semantic category result is output.

[0053] Through the above-described approach, this application achieves precise semantic annotation of the behaviors of children with developmental disabilities. By incorporating environmental contextual information and performing cross-attention fusion with behavioral features, the system can distinguish semantic differences in similar behaviors under different environments, such as accurately identifying excited screams in play sessions and anxious screams in individual training sessions. When the confidence level of the semantic projection results is insufficient, the accuracy and robustness of semantic annotation are further improved by expanding the time window, re-extracting environmental features, and performing weighted fusion. This method effectively reduces mis-annotation, improves the reliability of developmental profiles, and provides more accurate data support for therapists to develop personalized rehabilitation plans. Simultaneously, the increased automation and interpretability of the system help enhance therapists' trust in digital assessment tools and promote their widespread application in clinical settings.

[0054] This application further proposes that the feature extraction module also includes an environmental feature subunit and a behavioral feature subunit.

[0055] The environmental feature subunit performs principal component analysis on the environmental noise spectrum to extract noise feature vectors. Through the embedding layer, the course type and interactive object are mapped into continuous vectors respectively. The noise feature vector, course type vector and interactive object vector are concatenated to generate the environmental feature vector.

[0056] The behavioral feature subunit extracts Mel spectrograms from audio segments and encodes them into audio feature vectors using a convolutional neural network. It also extracts optical flow features from video segments and encodes them into video feature vectors using a convolutional neural network. Finally, it concatenates the audio and video feature vectors to generate a behavioral feature vector.

[0057] Principal component analysis (PCA) calculates the eigenvectors of the covariance matrix, retaining the top k principal components with the largest variances as noise feature vectors. This removes redundant information from the environmental noise spectrum and extracts key frequency band distribution features. The embedding layer uses a pre-trained word vector model to map course types and interactive object category labels into low-dimensional dense vectors, enabling discrete metadata to participate in subsequent vector operations. Mel spectrograms are generated through pre-emphasis, framing, windowing, and fast Fourier transform. The convolutional neural network uses three convolutional layers and a max-pooling layer for feature encoding. Optical flow features are calculated using the Horn-Schunck algorithm to compute motion vector fields between adjacent frames, and then two convolutional layers are used to extract spatiotemporal motion patterns.

[0058] Specifically, during the execution of the environmental feature subunit, the environmental noise spectrum is first standardized, its covariance matrix is ​​calculated, and eigenvalue decomposition is performed. The first three principal components are selected to form a noise feature vector. The course type and interactive object are input into the embedding layer and mapped to 128-dimensional continuous vectors. The three vectors are concatenated in the concatenation layer to generate a 384-dimensional environmental feature vector. During the execution of the behavioral feature subunit, the audio stream is divided into 2-second segments according to the timestamp, generating an 80-dimensional Mel spectrogram, which is input into a convolutional neural network to output a 256-dimensional audio feature vector. The optical flow feature map of the video stream is extracted and input into another convolutional neural network to output a 256-dimensional video feature vector. The two are concatenated to generate a 512-dimensional behavioral feature vector. Through principal component analysis dimensionality reduction and embedding layer mapping, the frequency domain features of environmental noise and the semantic information of the course scene are jointly encoded. The parallel extraction of Mel spectrogram and optical flow features can simultaneously capture the frequency domain characteristics of sound and the motion patterns of behavior, providing highly discriminative feature inputs for subsequent cross-attention fusion.

[0059] Through the above technical solutions, this application achieves effective extraction of environmental and behavioral features. The environmental feature subunit transforms environmental factors such as noise, course type, and interactive objects into numerical vectors using principal component analysis and embedding techniques, effectively capturing environmental context information. The behavioral feature subunit utilizes a deep learning model to extract high-level semantic features from audio and video data, avoiding the limitations of manual feature engineering. This multimodal feature extraction method lays the foundation for subsequent feature fusion and semantic analysis, improving the system's ability to understand and analyze complex scenes.

[0060] This application further proposes the following formula for calculating the weighted fusion-generated environmental feature enhancement vector:

[0061]

[0062] in, This is the original environmental feature vector. For the re-extracted environmental feature vector, For environmental feature enhancement vectors, These are the weighting coefficients.

[0063] In practice, the first step is to obtain the original environmental feature vector. This vector contains information such as course type, environmental noise spectrum, and interactive objects. Then, the environmental feature vector is re-extracted based on a preset extended time window. Next, set the weighting coefficients. The initial value is 0.5. , and Substituting into the above formula, the environmental feature enhancement vector is calculated. Finally use Perform cross-attention fusion and semantic projection again to output the corresponding semantic category.

[0064] Through the above technical solution, this application achieves dynamic enhancement of environmental feature vectors. By weighted fusion of the original environmental feature vector and the re-extracted environmental feature vector, an enhanced environmental feature vector is generated, improving the accuracy and completeness of environmental context information. This dynamic enhancement mechanism can better capture environmental changes, enhance the system's adaptability to complex scenes, and thus generate more accurate semantic category output.

[0065] In some of the solutions described above in this application, when the difference between the highest semantic similarity and the second highest semantic similarity is small, using a fixed weight coefficient to perform weighted fusion of environmental feature vectors may result in the fused environmental feature enhancement vector failing to fully reflect the dynamic changes of the current environmental context, thereby affecting the accuracy of semantic category determination.

[0066] This application further proposes that the weighting coefficient can be adjusted according to the similarity difference, and the calculation formula is as follows:

[0067]

[0068] in, Scaling factor The similarity difference. This is the sigmoid function.

[0069] The sigmoid function maps input values ​​to a continuous interval between 0 and 1, and the scaling factor β controls the magnitude of the influence of the similarity difference on the weight coefficients. For example, when When the value is small, the output value of the sigmoid function approaches or falls below 0.5. At this point, the weight coefficient α decreases, and the re-extracted environmental feature vector occupies a higher proportion in the fusion process; when... When the value is large, the output of the sigmoid function approaches 1, the weight coefficient α increases, and more information is retained in the original environmental feature vector. The scaling factor β can be set from 3 to 10, and the specific value should be adjusted according to the actual data distribution.

[0070] Specifically, in scenarios where the semantic similarity difference does not exceed the confidence threshold, the system first calculates the difference between the current highest and second-highest semantic similarity. The data is then input into a sigmoid function for nonlinear transformation. By dynamically adjusting the weight coefficients, the system can adaptively balance the original environmental features with the new environmental features extracted through the extended time window based on the confidence differences in semantic judgment, thereby generating a more accurate environmental feature enhancement vector. This mechanism effectively solves the problem of rigid environmental feature fusion under fixed weights and improves the robustness of semantic category judgment in low-confidence scenarios.

[0071] Through the above technical solution, this application achieves adaptive fusion of environmental feature vectors. When the confidence level of semantic judgment is low, the system automatically increases the weight of the re-extracted environmental feature vectors, thereby improving the perception accuracy of environmental context. This dynamic adjustment mechanism effectively enhances the system's adaptability to complex and ever-changing environments, thereby improving the accuracy and reliability of behavioral semantic recognition.

[0072] In some of the solutions mentioned above in this application, environmental context awareness relies on the extraction and fusion of data at the current moment, without considering the consistency of historical environmental patterns. This may lead to a significant deviation between the current semantic distribution and long-term environmental patterns, and may cause the semantic classification results to be inconsistent with the real situation.

[0073] This application further proposes an environment pattern memory module: constructing an environment pattern memory bank containing historical environment feature vectors and timestamps; calculating the cosine similarity between the current environment feature vector and each historical environment feature vector in the memory bank; filtering out historical environment feature vectors whose cosine similarity exceeds a preset similarity threshold and the average frequency distribution of their corresponding semantic categories; calculating the KL divergence between the current semantic distribution and the average probability distribution, and initiating semantic correction when the KL divergence exceeds a preset difference threshold; the current semantic distribution is a vector composed of the semantic similarities of each semantic category calculated at the current time.

[0074] The environment pattern memory establishes temporal-dimensional associations by storing historical environment feature vectors and their corresponding timestamps. Cosine similarity calculation employs vector dot product and modulus normalization to filter historical data with similarity scores above a threshold of 0.85. KL divergence calculation compares the current semantic similarity distribution with the historical average distribution, with a difference threshold set to 1.2. Semantic correction is achieved by adjusting the weighting ratio between the current semantic similarity and the historical average distribution; for example, a dynamic weighting coefficient λ is used, whose value is dynamically adjusted based on the KL divergence. When the KL divergence increases, the value of λ decreases to enhance the weight of historical patterns.

[0075] Specifically, when the current environment feature vector matches the historical environment pattern memory, the system extracts the top 5 historical environment feature vectors with the highest similarity and calculates the average frequency distribution of their corresponding semantic categories. If the KL divergence between the current semantic distribution and the historical average distribution exceeds 1.2, it indicates that the current classification result deviates from the historical pattern, and a correction mechanism is activated. This mechanism makes the semantic classification result more consistent with long-term environmental patterns, avoiding misjudgments caused by instantaneous environmental noise or feature extraction errors. For example, in a game class environment, even if current noise interference causes the system to misjudge as anxious behavior, the distribution of frequently occurring excited behaviors in historical patterns can improve classification accuracy through the correction mechanism.

[0076] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0077] The environment pattern memory module constructs an environment pattern memory bank containing historical environment feature vectors and timestamps. This memory bank can be stored in the form of key-value pairs, where the key is the timestamp and the value is the corresponding environment feature vector.

[0078] The environment pattern memory module calculates the cosine similarity between the current environment feature vector and the feature vectors of each historical environment in the memory. Specifically, the cosine similarity can be calculated using the ratio of the dot product to the vector norm.

[0079] Furthermore, the environment pattern memory module filters out historical environment feature vectors with cosine similarity exceeding a preset similarity threshold and the average frequency distribution of their corresponding semantic categories. For example, a similarity threshold of 0.8 can be set to filter out historical environment feature vectors with cosine similarity greater than 0.8. For each filtered historical environment feature vector, the frequency of each semantic category near its corresponding timestamp is counted, and the average frequency distribution is calculated.

[0080] The environment pattern memory module calculates the KL divergence between the current semantic distribution and the average probability distribution. The current semantic distribution is a vector composed of the semantic similarities of each semantic category calculated so far. The KL divergence can be obtained by summation.

[0081] Therefore, when the KL divergence exceeds a preset difference threshold, the environment pattern memory module initiates semantic correction. The difference threshold can be adjusted according to the actual application scenario. Semantic correction can employ a weighted average method to fuse historical semantic distributions with the current semantic distribution.

[0082] Through the above technical solution, this application achieves semantic correction based on historical environmental patterns. The environmental pattern memory module can capture the semantic distribution patterns in similar environments and correct for anomalies in the current semantic distribution. This method improves the accuracy and stability of semantic recognition, reduces misjudgments caused by environmental changes, and thus enhances the system's ability to understand and track the behavior of children with developmental disabilities.

[0083] In some of the solutions described above in this application, the environment pattern memory module filters similar historical environments by calculating the cosine similarity between the current environment feature vector and the historical environment feature vector, and performs semantic correction based on the average frequency distribution. However, when multiple similar historical environments exist, if the similarity is not sorted and filtered, it may lead to interference from irrelevant historical data; at the same time, if a fixed weight is used in the correction process, it is difficult to adapt to the dynamic changes of different scenarios, affecting the accuracy of semantic similarity calculation.

[0084] This application further proposes a similarity ranking unit to sort cosine similarity from high to low, and selects the top m historical environment feature vectors with the highest cosine similarity as similar historical environments; the correction formula is:

[0085]

[0086] in, Let i be the semantic similarity after semantic correction for the i-th class. The semantic similarity currently calculated for the i-th semantic class is... Let λ represent the average semantic similarity of the i-th semantic class in the j-th similar historical environment, and let λ represent the adaptive weighting coefficient, which takes values ​​in the range [0,1].

[0087] The cosine similarity ranking unit retains only the top m historical environmental feature vectors with the highest similarity by setting ranking rules, thus avoiding noise from low-similarity data. The value of m is dynamically adjusted according to the actual scenario; for example, it is set to 5 in a classroom environment to balance computational efficiency and data coverage. The adaptive weighting coefficient λ in the correction formula is dynamically calculated based on the difference between the current semantic distribution and the historical distribution. When the difference is large, λ approaches 0 to enhance the reference role of historical data; when the difference is small, λ approaches 1 to retain the dominance of the current semantic similarity.

[0088] Specifically, in the semantic correction process, the cosine similarity between the current environment feature vector and the historical environment feature vectors in the memory is first calculated and sorted from highest to lowest. For example, when m=3, only the top three historical data with the highest similarity are selected for correction. Then, the current semantic similarity is adjusted according to the correction formula. Average semantic similarity with similar historical environments Weighted fusion is performed. The value range of λ is obtained through training with historical data. For example, in a classroom scenario, it is set to 0.7, meaning that the current semantic similarity accounts for 70% of the weight, and historical data accounts for 30%. By dynamically adjusting λ, overcorrection or undercorrection problems caused by fixed weights can be avoided, thereby improving the robustness of semantic similarity calculation. For example, when the current environmental noise spectrum differs significantly from historical data, λ automatically decreases to 0.5, increasing the average semantic similarity ratio of historical data to 50% to suppress noise interference.

[0089] Through the above technical solution, this application can utilize historical similarity environment information to correct the current semantic similarity, thereby improving the accuracy and stability of semantic recognition. By selecting the historical environment with the highest similarity for correction, the relevance of the correction information is ensured. The introduction of adaptive weighting coefficients achieves a dynamic balance between current and historical information, making the correction process more flexible and controllable. This correction mechanism effectively reduces the interference of environmental changes on semantic recognition, improving the system's adaptability and robustness in different environments.

[0090] In some of the solutions mentioned above in this application, the environmental feature vector only includes static environmental elements and does not consider the dynamic change patterns of interactive objects in the time dimension, resulting in insufficient characterization of the interactive context when the behavior occurs, which may lead to misclassification of behaviors with different interactive patterns under the same environmental type.

[0091] This application further proposes an interaction pattern analysis module, including obtaining a sequence of interactive objects within a preset time window, the time window being symmetrically expanded around the timestamp of the behavior occurrence; performing pattern analysis on the sequence of interactive objects, calculating the object switching frequency and the duration of the dominant object; the object switching frequency is obtained by dividing the number of times the interactive object changes within the time window by the length of the time window; the duration of the dominant object is obtained by calculating the duration of each interactive object within the time window and taking the maximum value; the object switching frequency and the duration of the dominant object are combined into a two-dimensional vector and normalized to generate an interaction pattern feature vector; the interaction pattern feature vector is concatenated with the original environmental feature vector to form an enhanced environmental feature vector for cross-attention fusion and semantic projection.

[0092] The time window can be symmetrically extended by 5 seconds before and after, forming a 10-second observation interval. The object switching frequency is calculated using a sliding window counting method; the counter increments by 1 each time a change in the interactive object identifier is detected, and the final count is divided by the total window length in seconds. The duration of the dominant object is determined by maintaining the start and end timestamps of each object within the window, calculating the difference, and taking the maximum value. Normalization uses a min-max scaling method, mapping the values ​​of each dimension of the two-dimensional vector to the 0-1 range. The concatenation operation uses vector concatenation, appending the dimension of the interaction mode feature vector to the end of the environment feature vector.

[0093] Specifically, when the system detects a timestamp of a behavior, it extends forward and backward by 5 seconds from that time point, capturing the sequence of interactive object identifiers within that time window. For example, within a 10-second window, the interactive objects are observed to be [therapist, teaching aid, therapist, parent], with 3 object switching times, resulting in a switching frequency of 3 / 10 = 0.3 times / second. The duration of each object is 6 seconds for the therapist, 2 seconds for the teaching aid, 2 seconds for the parent, and 6 seconds for the dominant object. The original two-dimensional vector is [0.3, 6], which becomes [0.3, 1.0] after normalization. This vector is concatenated with an environmental feature vector containing course type and noise characteristics, and the newly added interaction pattern dimension can characterize whether the child is in a highly dynamic situation of frequently switching interactive objects or a stable situation of interacting with a single object for a long time when the behavior occurs. During the semantic projection stage, the enhanced environmental feature vectors influence the fusion weights of behavioral features through a cross-attention mechanism, enabling high-frequency switching interaction patterns to trigger different semantic category matching under the same course type. For example, high-volume vocalizations in frequently switching object environments are identified as participatory interactions, while the same behavior in stable object environments is identified as an anxiety response.

[0094] Through the aforementioned technical solutions, this application can capture the dynamic characteristics of children's interactions with their environment, providing richer contextual information. This allows for a more accurate interpretation of the semantics of children's behavior, avoiding misclassification of similar behaviors in different environments as belonging to the same category. Furthermore, by integrating interaction pattern features, the system can distinguish the differences in children's performance across different course types, thereby generating more contextualized and personalized behavioral assessment results.

[0095] In some of the solutions mentioned above in this application, although the semantic category judgment at the current moment is achieved through the dynamic fusion of environmental features and behavioral features, the potential impact of historical semantic category sequences on the development of current behavior is not considered. This may lead to the association of behavioral features at isolated time points with environmental features, ignoring the continuous changes in long-term behavioral patterns, resulting in a lack of temporal consistency in semantic category judgment.

[0096] This application further proposes a temporal modeling module, which extracts the semantic category sequence output within a preset time period before the current time step, encodes the semantic category sequence using a temporal convolutional network or a long short-term memory network, and outputs a behavior development trend vector; concatenates the behavior development trend vector with the environment-behavior fusion vector of the current time step to form a temporal enhanced fusion vector; and projects the temporal enhanced fusion vector onto the semantic space to output the semantic category enhanced by long-term context.

[0097] Among them, the temporal convolutional network extracts the local dependency patterns of semantic category sequences through multi-layer dilated convolutional kernels, and the long short-term memory network models the long-term dependency relationship of the sequence through a gating mechanism; the concatenation operation of the behavior development trend vector and the current environment-behavior fusion vector adopts channel-dimensional concatenation and adjusts the dimension through a fully connected layer; during the projection of the temporal enhancement fusion vector into the semantic space, its weight matrix is ​​initialized by the transition probability of historical semantic categories and current semantic categories.

[0098] Specifically, in the semantic category sequence encoding stage, a preset time period of 30 minutes is used to extract historical semantic category sequences backward from the current time step. The temporal convolutional network adopts a three-layer dilated convolutional structure with dilation coefficients of 1, 2, and 4, and a kernel size of 3. The hidden layer dimension of the Long Short-Term Memory network is set to 128. After the behavior development trend vector is generated, it is merged with the current environment-behavior fusion vector through a concatenation layer to form a 512-dimensional vector, which is then compressed to 256 dimensions through a fully connected layer. When projected into the semantic space, the similarity calculation between the temporal enhanced fusion vector and the semantic category prototype vector introduces a time decay factor, and the contribution weight of historical semantic categories decays exponentially with the increase of time interval. For example, when the semantic category of "attention distraction" is detected for three consecutive time steps, the behavior development trend vector will strengthen the attention-related feature dimensions in the current environment-behavior fusion vector, thereby accurately identifying attention distraction behavior even in noisy environments and avoiding misjudgments caused by environmental noise interference at a single moment.

[0099] As a preferred embodiment, the solution of this application is implemented as follows: The temporal modeling module is configured to perform the following operations: the semantic category sequence of the 30 days prior to the current time step is extracted, and the sequence contains the behavioral semantic labels of daily records; a long short-term memory network is trained to encode the sequence, the hidden layer dimension of the network is set to 128, the time step size is 30, and after bidirectional LSTM processing, the hidden state of the last time step is taken as the behavioral development trend vector; this vector and the current environment-behavior fusion vector are unified in dimension through a fully connected layer, and then concatenated along the feature dimension to generate a temporal enhancement fusion vector; the fusion vector is input to a pre-trained semantic projection layer, and by calculating the cosine similarity with the semantic prototype vector, the semantic category label of the fused long-term behavioral trend is output.

[0100] Through the aforementioned technical solution, this application can effectively capture the long-term evolution of behavioral patterns in children with developmental disabilities. For example, after several consecutive days of the semantic label "attention deficit," the "running fast" behavior in the current environment can be accurately identified as "anxious hyperactivity" rather than "playful movement," avoiding semantic misjudgments caused by isolated analysis of single behaviors. This mechanism significantly improves the temporal coherence of behavioral assessments, making adjustments to rehabilitation training plans more aligned with the child's actual developmental trajectory.

[0101] This application further proposes a semantic category association module, including constructing a semantic category association network, where nodes in the semantic category association network are semantic categories, edges represent the development order relationship between semantic categories, and the weight of the edges is the transition probability between semantic categories obtained from historical data statistics; extracting the semantic category sequence output within a preset time period before the current time step, searching in the semantic category association network for all paths with the semantic category sequence as the starting point and ending point, and with the same length as the semantic category sequence; calculating the product of all edge weights in each path in the path set, selecting the path with the largest product as the development path, and extracting the next node pointed to by the development path as the reference category; calculating the semantic similarity between the current semantic category and the reference category, if it is lower than a preset consistency threshold, generating a development deviation warning signal, and fusing the prototype vector of the reference category into the current environment-behavior fusion vector in an attention-weighted manner to form a development context-enhanced fusion vector, and re-projecting semantics.

[0102] As a preferred embodiment, the specific implementation of this application is as follows: A network of associations containing semantic category nodes is constructed. Nodes are connected by edges to represent the development order relationship, and the edge weights are determined by the semantic category transition probabilities in historical data. When a semantic category sequence ["language imitation", "item identification"] from two weeks prior to the current time step is extracted, all paths with a length of 2 starting from this sequence are searched in the network, and the product of the edge weights of each path is calculated. The path with the largest product ["language imitation", "item identification", "active naming"] is selected, and "active naming" is extracted as the reference category. The similarity between the current semantic category "item identification" and the reference category is calculated. If it is below a threshold of 0.7, the prototype vector of the reference category is fused to the current environment-behavior fusion vector with an attention weight of 0.3, forming an enhanced vector containing development expectations, which is then reprojected into the semantic space.

[0103] Through the above technical solution, this application can dynamically correct the current semantic classification decision based on historical development patterns. When the semantic category output by the system deviates significantly from the typical development path, attention-weighted correction is performed by introducing the reference category predicted by the association network. This effectively avoids misclassification caused by isolated judgment of current behavioral characteristics, making the semantic annotation results more consistent with the continuous pattern of children's behavioral development and improving the consistency between automated recording and clinical assessment conclusions.

[0104] Example 2:

[0105] like Figure 3 As shown, the method for dynamically recording and tracking the growth profiles of children with developmental disabilities includes the following steps:

[0106] Acquire behavioral text records of the target object, synchronously acquired audio and video streams, and environmental context metadata; the environmental context metadata includes course type, environmental noise spectrum, and interactive objects;

[0107] Based on the timestamps of the behavioral text records, behavioral feature vectors and environmental feature vectors are extracted from the audio and video streams and environmental context metadata.

[0108] By using a cross-attention fusion mechanism, the environmental feature vector is used as the query vector, and the behavioral feature vector is used as the key vector and value vector for fusion processing to obtain the environmental-behavior fusion vector;

[0109] The environment-behavior fusion vector is projected onto a predefined semantic space, and its semantic similarity with the prototype vectors of each semantic category in the semantic space is calculated.

[0110] Determine whether the difference between the highest and second-highest semantic similarity scores among all semantic similarities exceeds a preset confidence threshold:

[0111] If yes, output the corresponding semantic category; otherwise, re-extract the environmental feature vector based on the preset extended time window and perform weighted fusion with the original environmental feature vector to generate an environmental feature enhancement vector.

[0112] Based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

[0113] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0114] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0115] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A dynamic recording and tracking system for the growth portfolio of children with developmental disabilities, characterized in that, include: The data acquisition module is used to acquire behavioral text records of the target object, synchronously collected audio and video streams, and environmental context metadata; The environmental context metadata includes course type, environmental noise spectrum, and interactive objects; The feature extraction module is used to extract behavioral feature vectors and environmental feature vectors from the audio and video streams and the environmental context metadata based on the timestamps of the behavioral text records; The feature fusion module is used to fuse the environment feature vector as a query vector and the behavior feature vector as a key vector and value vector through a cross-attention fusion mechanism to obtain an environment-behavior fusion vector. The semantic projection module is used to project the environment-behavior fusion vector onto a predefined semantic space and calculate its semantic similarity with the prototype vectors of each semantic category in the semantic space. The semantic output module is used to determine whether the difference between the highest and second-highest semantic similarity scores among all semantic similarities exceeds a preset confidence threshold. If yes, output the corresponding semantic category; otherwise, re-extract the environmental feature vector based on the preset extended time window and perform weighted fusion with the original environmental feature vector to generate an environmental feature enhancement vector. Based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

2. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: The feature extraction module further includes: The environmental feature subunit is used to extract noise feature vectors by performing principal component analysis on the environmental noise spectrum. The course type and interactive object are mapped into continuous vectors through the embedding layer. The noise feature vector, course type vector and interactive object vector are concatenated to generate the environmental feature vector. The behavioral feature subunit is used to extract Mel spectrograms from audio segments and encode them into audio feature vectors through a convolutional neural network, extract optical flow features from video segments and encode them into video feature vectors through a convolutional neural network, and concatenate the audio feature vectors and video feature vectors to generate behavioral feature vectors.

3. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: The formula for calculating the weighted fusion-generated environmental feature enhancement vector is as follows: in, This is the original environmental feature vector. For the re-extracted environmental feature vector, For environmental feature enhancement vectors, These are the weighting coefficients.

4. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 3, characterized in that: The weighting coefficient can be adjusted based on the similarity difference, and the calculation formula is as follows: in, Scaling factor The similarity difference. This is the sigmoid function.

5. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: It also includes an environment pattern memory module: Construct an environment pattern memory library containing historical environment feature vectors and timestamps; Calculate the cosine similarity between the current environment feature vector and the feature vectors of each historical environment in the memory bank; Filter out historical environmental feature vectors with cosine similarity exceeding a preset similarity threshold and the average frequency distribution of their corresponding semantic categories; Calculate the KL divergence between the current semantic distribution and the average probability distribution. When the KL divergence exceeds a preset difference threshold, semantic correction is initiated. The current semantic distribution is a vector composed of the semantic similarities of each semantic category calculated at the moment.

6. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 5, characterized in that: The environment pattern memory module also includes: The similarity ranking unit sorts the cosine similarities from high to low and selects the top m historical environment feature vectors with the highest cosine similarity as similar historical environments; the correction formula is: in, Let i be the semantic similarity after semantic correction for the i-th class. The semantic similarity currently calculated for the i-th semantic class is... Let λ represent the average semantic similarity of the i-th semantic class in the j-th similar historical environment, and let λ represent the adaptive weighting coefficient, which takes values ​​in the range [0,1].

7. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: It also includes an interaction pattern analysis module: Obtain the sequence of interactive objects within a preset time window, wherein the time window expands symmetrically with the timestamp of the behavior occurrence as the center; Perform pattern analysis on the sequence of interactive objects to calculate the object switching frequency and the duration of the dominant object; The object switching frequency is obtained by dividing the number of times the interactive object changes within the statistical time window by the length of the time window; The duration of the dominant object is obtained by calculating the duration of each interactive object within the time window and taking the maximum value. The object switching frequency and the duration of the dominant object are combined into a two-dimensional vector, and then normalized to generate an interaction mode feature vector. The interaction mode feature vector is concatenated with the original environment feature vector to form an enhanced environment feature vector for cross-attention fusion and semantic projection.

8. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: It also includes a time series modeling module: Extract the semantic category sequence output within a preset time period before the current time step, encode the semantic category sequence using a temporal convolutional network or a long short-term memory network, and output a behavior development trend vector; The behavior development trend vector is concatenated with the environment-behavior fusion vector at the current time step to form a temporal enhanced fusion vector; The temporal enhancement fusion vector is projected onto the semantic space to output a semantic category enhanced by long-term context.

9. The dynamic recording and tracking system for growth records of children with developmental disabilities according to claim 1, characterized in that: It also includes a semantic category association module: Construct a semantic category association network, where nodes are semantic categories, edges represent the development order relationship between semantic categories, and the weight of the edges is the transition probability between semantic categories obtained from historical data statistics; Extract the semantic category sequence output within a preset time period before the current time step, and search in the semantic category association network for all paths that have the semantic category sequence as the starting point and the ending point, and whose length is the same as the semantic category sequence. Calculate the product of all edge weights in each path in the path set, select the path with the largest product as the development path, and extract the next node pointed to by the development path as the reference category. Calculate the semantic similarity between the current semantic category and the reference category. If it is lower than a preset consistency threshold, generate a development deviation warning signal, and fuse the prototype vector of the reference category into the current environment-behavior fusion vector in an attention-weighted manner to form a development context-enhanced fusion vector, and then re-project semantics.

10. A method for dynamically recording and tracking the growth profiles of children with developmental disabilities, including the following steps: Acquire behavioral text records of the target object, synchronously acquired audio and video streams, and environmental context metadata; The environmental context metadata includes course type, environmental noise spectrum, and interactive objects; Based on the timestamps of the behavioral text records, behavioral feature vectors and environmental feature vectors are extracted from the audio and video streams and the environmental context metadata. By using a cross-attention fusion mechanism, the environmental feature vector is used as the query vector, and the behavioral feature vector is used as the key vector and value vector for fusion processing to obtain the environment-behavior fusion vector; The environment-behavior fusion vector is projected onto a predefined semantic space, and its semantic similarity with the prototype vectors of each semantic category in the semantic space is calculated. Determine whether the difference between the highest and second-highest semantic similarity scores among all semantic similarities exceeds a preset confidence threshold: If yes, output the corresponding semantic category; otherwise, re-extract the environmental feature vector based on the preset extended time window and perform weighted fusion with the original environmental feature vector to generate an environmental feature enhancement vector. Based on the environmental feature enhancement vector, cross-attention fusion and semantic projection are performed again, and the corresponding semantic category is output.

Images