Videosemantic-based fall prediction method and system

Abstract

This invention provides a video semantics-based fall prediction method, system, electronic device, storage medium, and program product that interacts structural semantic information from different time periods, learns the relationships between structural semantic information from different time periods, enhances the ability to identify predictive behaviors, and further provides more accurate fall risk predictions. [Solution] The fall prediction method based on video semantics extracts visual input vectors corresponding to the behavioral video segments of the individual being measured, extracts multi-channel structural semantic information using a Transformer deep neural network to obtain multi-channel structural semantic information of the individual being measured, processes the multi-channel structural semantic information using a time-series aggregation method based on efficient channel attention, and then performs fall prediction using a support vector machine.

Description

Technical Field

[0001] (Cross - reference to related applications) This invention claims the priority of a Chinese patent application filed with the China National Intellectual Property on December 11, 2024, with an application number of 202411821804.8 and an invention title of "Fall Detection Method and System Based on Video Semantics". All of its contents are incorporated herein by reference, constitute a part of this invention, and are used for all purposes.

[0002] This invention belongs to the technical field related to fall prediction, and particularly relates to a fall prediction method and system based on video semantics.

Background Art

[0003] The description of this part only provides background technical information related to this invention and does not necessarily constitute prior art.

[0004] The prediction and prevention of falls have become a key research theme in the field of public health. Although several fall prediction systems have already emerged in the market, these systems still have some practical limitations. For example, there are problems such as insufficient adaptability to specific people (such as the elderly) and insufficient accuracy in feature selection.

[0005] Problems with conventional fall prediction methods include the following: Firstly, they lack versatility; conventional fall predictions are usually difficult to adapt to people of different age groups and physical conditions. While older adults and younger adults have significant differences in physiological characteristics, conventional fall predictions do not adequately distinguish these important differences. Secondly, feature selection is insufficient; many prediction methods rely on conventional feature selection methods, which may not adequately identify and utilize features that significantly impact fall prediction. This can limit predictive performance, making it impossible to extract features from a broad perspective and evaluate their actual utility. Thirdly, conventional methods have significant limitations, including insufficient accuracy in conventional time-based mobility tests. Collecting data on falls using laboratory equipment (gold standards) has certain limitations; for example, it may be difficult to use, relatively expensive, and some indicators and other monitoring data may need improvement. Conventional IMU (Inertial Measurement Unit) equipment cannot measure the general semantic information of falling behavior, which is an important indicator for predicting falls. However, while structural semantic information of the human body is an important indicator for assessing fall risk, current fall prediction systems often ignore the application of this crucial information. Because structural semantic information of the human body can reflect the stability and coordination of the body's morphology and movement, insufficient use of this information can lead to inaccurate assessments of individual fall risk, potentially affecting the effective implementation of preventive measures. Therefore, developing a system that integrates structural semantic information to accurately assess fall risk is particularly important. This can not only improve the accuracy of fall predictions but also help in adaptively formulating preventive measures that are more suitable for high-risk individuals.

[0006] Therefore, the current challenge is to efficiently extract and utilize structural semantic information that reflects the morphology and movement of the human body to predict falls and achieve more accurate predictions. [Overview of the project] [Problems that the invention aims to solve]

[0007] To overcome the shortcomings of the above-mentioned prior art, the present invention provides a fall prediction method and system based on video semantics, which utilizes an initial structural semantic vector and a visual input vector to perform feature extraction, aggregation, and prediction based on a Transformer network, an Efficient Channel Attention (ECA) module, and a Support Vector Machine (SVM), thereby improving the accuracy of fall prediction. [Means for solving the problem]

[0008] To achieve the above objective, the present invention employs the following technical approach.

[0009] According to a first aspect, the present invention provides a fall prediction method based on video semantics, the fall prediction method is A Transformer deep neural network is trained using N d-dimensional embedding vectors and the visual input vectors of the training samples. After training, the updated N d-dimensional embedding vectors are used as the initial structure semantic vectors learned through training. Obtaining behavioral video segments of the individual being measured, The process involves extracting frame images from the behavioral video segments of the subject individual, performing grid region division on the extracted frame images, and performing a linear transformation to obtain the visual input vector of the subject individual. The initial structural semantic vectors learned through the aforementioned training and the visual input vectors of the subject of measurement are both input into the Transformer network to obtain multi-channel structural semantic information. The process involves inputting multi-channel structural semantic information output from a Transformer network into a trained machine learning model, and The aforementioned trained machine learning model includes two parts: an ECA-based time series aggregation part and a support vector machine classification part. In the ECA-based time-series aggregation section, the ECA module assigns appropriate weights to the multi-channel structural semantic information output from the Transformer network to perform time-series aggregation, thereby obtaining the structural semantic information of the time-series aggregation. In the support vector machine classification section, the structural semantic information of the time-series aggregation is input to the support vector machine to predict whether or not the subject of measurement has fallen over. Finally, the trained machine learning model includes outputting a prediction of whether or not the individual being measured has fallen over.

[0010] According to a second aspect, the present invention provides a fall prediction system based on video semantics, the fall prediction system is An acquisition unit configured to acquire behavioral video segments of the individual being measured, A first feature extraction unit is configured to divide frame images corresponding to the behavioral video segments of the individual being measured into grid regions and perform a linear transformation to obtain the visual input vector of the individual being measured. A second feature extraction unit is configured to obtain multi-channel structural semantic information by inputting both the initial structural semantic vector learned through training and the visual input vector of the individual being measured into a Transformer network, In the time-series aggregation portion of the machine learning model based on ECA, the ECA module is configured to perform time-series aggregation by assigning appropriate weights to the multi-channel structural semantic information output from the Transformer network, thereby obtaining the structural semantic information of the time-series aggregation. The support vector machine classification portion of the machine learning model includes a prediction unit configured to input the structural semantic information of the time series aggregation into a support vector machine to predict whether or not the subject individual has fallen over, and ultimately output a prediction result of whether or not the subject individual has fallen over.

[0011] According to a third aspect, the present invention provides an electronic device comprising a memory, a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when run by the processor, complete the method according to the first aspect.

[0012] According to a fourth aspect, the present invention provides a computer-readable storage medium used to store computer instructions, and when the computer instructions are executed by a processor, the method according to the first aspect is completed.

[0013] According to a fifth aspect, the present invention provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements and completes the method according to the first aspect. [Effects of the Invention]

[0014] The above one or more technical proposals have the following beneficial effects:

[0015] In the present invention, a visual input vector corresponding to the behavior video segment of the measurement target individual is extracted, multi-channel structural semantic information extraction is performed using a Transformer deep neural network to obtain the multi-channel structural semantic information of the measurement target individual, and after processing the multi-channel structural semantic information using a time series aggregation method based on ECA, a support vector machine is used to perform a fall prediction. Using a Transformer deep neural network, compared with the conventional method and the deep learning method based on a convolutional neural network (CNN), this method improves the efficiency, convenience, connectivity, and globality of information acquisition, effectively solves the problem of insufficient connection between the monitoring data of the left and right feet and the problem of insufficient unity of each measurement data of the conventional method. By ECA, the structural semantic information at different times is made to interact, the relevance between the structural semantic information at different times is learned, the discrimination ability of the predicted behavior is enhanced, and a more accurate fall risk prediction is further provided.

[0016] Advantages of additional aspects of the present invention are given in part in the following description, become apparent in part from the following description, or are understood through the practice of the present invention.

[0017] The accompanying drawings of the specification constituting a part of the present invention are for providing a further understanding of the present invention. The exemplary embodiments and their descriptions of the present invention are for explaining the present invention and do not constitute an undue limitation of the present invention.

Brief Description of the Drawings

[0018] [Figure 1] It is a schematic diagram of the Transformer deep neural network structure in the first embodiment of the present invention. [Figure 2] It is a flowchart of the fall prediction method based on video semantics in the first embodiment of the present invention.

Modes for Carrying Out the Invention

[0019] Note that the following detailed descriptions are all illustrative and intended to further explain the present invention. Unless otherwise specified, all technical terms and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art.

[0020] It should be noted that the terms used in this specification are for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention.

[0021] When there is no conflict, the features of the embodiments of the present invention can be combined with each other.

[0022] Example 1 This example discloses a fall prediction method based on video semantics. First, use N d-dimensional embedding vectors and the visual input vectors of training samples to train a Transformer deep neural network. After the training is completed, the updated N d-dimensional embedding vectors are used as the initial structural semantic vectors learned through training, and then used to process the action video information of the actual measurement target individual. During training, what is obtained through learning is a set of initial structural semantic vectors that are advantageous for the model prediction effect regardless of any specific image. Obtain the action video segment of the measurement target individual. Divide the frame images corresponding to the action video segment of the measurement target individual into grid regions, and perform linear transformation to obtain the visual input vector of the measurement target individual. Input the initially learned structural semantic vector and the corresponding visual input vector of the measurement target individual into the Transformer network, and output to obtain multi-channel structural semantic information. The process involves inputting multi-channel structural semantic information output from a Transformer network into a trained machine learning model, and The aforementioned trained machine learning model includes two parts: an ECA-based time series aggregation part and a support vector machine classification part. In the ECA-based time-series aggregation section, the ECA module assigns appropriate weights to the multi-channel structural semantic information output from the Transformer network to perform time-series aggregation, thereby obtaining the structural semantic information of the time-series aggregation. In the support vector machine classification section, the structural semantic information of the time-series aggregation is input to the support vector machine to predict whether or not the subject of measurement has fallen over. Finally, the trained machine learning model includes outputting a prediction of whether or not the individual being measured has fallen over.

[0023] This embodiment uses a Transformer deep neural network to extract structural semantic information from behavioral video segments of the target individual, obtains structural semantic information of the target individual, processes the structural semantic information using an ECA-based time-series aggregation method, and then performs fall prediction using a predictive model. The semantic information obtained by video analysis not only includes parameter information in conventional methods but can also extract hidden information that cannot be obtained by conventional methods. This embodiment uses a Transformer deep neural network and, compared to conventional methods and CNN-based deep learning methods, improves the efficiency, convenience, connectivity, and overall scope of information acquisition. It effectively solves the problems of insufficient connectivity between monitoring data of the left and right feet and insufficient consistency of each measurement data in conventional methods, and provides even more accurate fall risk prediction. Such an innovative method not only improves the reliability of fall prediction but also provides strong technical support for the safety protection of the elderly, further improving the accuracy, speed of fall prediction and the overall performance of the system.

[0024] The following describes in detail the video semantics-based fall prediction method proposed in this embodiment, linked to Figures 1 and 2, and specifically includes the following:

[0025] Step 1: Obtain the behavioral video segment of the individual being measured, divide the frame image corresponding to the behavioral video segment into a grid region, and perform a linear transformation to obtain the visual input vector of the individual being measured. Input the initial structural semantic vector learned during Transformer network training and the corresponding visual input vector of the individual being measured into the Transformer network to obtain multi-channel structural semantic information.

[0026] This embodiment further includes preprocessing of the behavioral videos of the individuals being measured, the preprocessing of which includes video clipping, normalization, noise reduction, contrast enhancement, etc., to improve the accuracy of semantic information prediction and facilitate its use in further analysis and application.

[0027] In this embodiment, visual input vectors are extracted from pre-processed behavioral videos of the subject to be measured. Specifically, a 2D image of the current frame is acquired at regular intervals (e.g., period T=0.5s), with dimensions [H,W]. The 2D image of the current frame is divided into multiple grid regions of the same size, and the original video data is converted into an embedding vector in [1,d] format using a linear transformation. This is then used as input to a Transformer deep neural network, and structural semantic information of the 2D image is extracted using a Transformer deep neural network based on self-attention.

[0028] Specifically, in the process of acquiring the visual input vector, the image is first [P h ,P w The image information for each mesh is divided into meshes of size P h *P w It is flattened into a one-dimensional vector, and the number of such vectors is

number

[0029] Before inputting the visual input vectors of the individual being measured into the Transformer network, it is necessary to train the Transformer deep neural network multiple times to learn the initial structural semantic vectors of the visual input vector processing process that can be used for the obtained individual.

[0030] The initial structure semantic vector is represented by N learnable d-dimensional embedding vectors, and the initial structure semantic vector is a set of initial data that is favorable to the model prediction effect, regardless of any particular image.

[0031] In this embodiment, the learnable d-dimensional embedding vector is a fixed-length vector representing data points (e.g., words, nodes, etc.) in deep learning, and its dimension d can be dynamically adjusted during the training process to capture data features.

[0032] Here, a d-dimensional embedding vector is a low-dimensional representation that maps high-dimensional or complex data (e.g., text, images) into a d-dimensional space, and is obtained through neural network learning. Its "learnability" is that the parameters of the embedding vector are optimized by backpropagation during training to minimize the loss function (e.g., cross-entropy, mean squared error, etc.) for a particular task.

[0033] At the start of the first training session, these N d-dimensional embedding vectors are randomly initialized using a Gaussian distribution. During training, these N d-dimensional embedding vectors are input to the Transformer deep neural network along with the visual input vectors of the training samples, and these N d-dimensional embedding vectors are updated during training. After training is complete, the updated N d-dimensional embedding vectors are the initial structural semantic vectors obtained through model learning.

[0034] The input to a Transformer deep neural network consists of two parts: a visual input vector and an initial structure semantic vector.

[0035] The visual input vector and the initial structural semantic vector are stitched together and input into a Transformer deep neural network. The Transformer deep neural network mentioned in this embodiment employs only the encoder portion of a standard Transformer deep neural network, learning the structural semantic representation by stacking M encoder modules. Each module contains two components: a multi-head self-attention component and a feedforward neural network component. The output portion of each component employs residual connection and layer normalization structures to improve the stability of the model and extend its generalization ability.

[0036] The output of a Transformer deep neural network is a structural semantic vector based on the original 2D image of the current frame, and represents the structural semantic information extracted from the original 2D image as a matrix composed of N structural semantic vectors.

[0037] For a single 2D image, its structural semantic matrix is ​​obtained by a deep neural network using the method described above. For a given video data, by processing the 2D image of the current frame at regular intervals (e.g., period T=0.5s) and then inputting the result into a Transformer deep neural network, K structural semantic matrices can be obtained. These structural semantic matrices can then be stored in tensor form to form multi-channel structural semantic information.

[0038] In this example, a training sample dataset is constructed, which includes video data of known falls or non-falling states, where the video data is collected by capturing video with a camera. A Transformer deep neural network is trained using the training sample dataset, and during training, the structural semantic information output from the Transformer deep neural network is mapped in a linear projection layer to obtain a 2D heatmap of shape [H,W]. The Transformer deep neural network is optimized by the difference between the predicted 2D heatmap and the true 2D heatmap, and the loss function used is the Mean-Square Error (MSE).

[0039] Collecting and comprehensively analyzing multi-channel structural semantic information is crucial for accurately assessing and preventing fall risk. Furthermore, multi-channel structural semantic information can more accurately capture an individual's biological structure by effectively reflecting the stability of their gait. By utilizing multi-channel structural semantic information, we can enhance sensitivity to and predictive capabilities regarding potential fall risk, thereby providing a scientific basis for fall prevention.

[0040] Step 2: Using the ECA module, time-series aggregation is performed by assigning appropriate weights to multi-channel structural semantic information to obtain time-series aggregated structural semantic information. Based on the time-series aggregated structural semantic information, a support vector model is used to make predictions and obtain a prediction result of whether or not the measured individual fell over.

[0041] The machine learning model in this embodiment includes two parts: a time-series aggregation part based on ECA and a support vector machine classification part.

[0042] In this embodiment, the first part is time-series aggregation based on ECA, employing an ECA module to extract frames from video data, integrate the extracted multi-channel structural semantic information, and assign weights corresponding to the structural semantic information of different channels.

[0043] Specifically, time-series aggregation based on ECA is performed on multi-channel structural semantic information. The ECA module is used to learn the relationships between channels in the input feature diagram, and different weights are assigned to each channel. The specific method is as follows:

[0044] Step 201: First, global average pooling is performed on multi-channel structure semantic information with K channels to obtain channel features with a scale of 1 × 1 × K.

[0045] Step 202: Then, feature extraction is performed using a convolutional kernel with 1 channel and adaptive size to realize the interaction between channels.

[0046] During the convolution process, appropriate padding is used to ensure that the magnitude of the input matches the magnitude of the output.

[0047] Step 203: The output of the convolutional layer is processed using a sigmoid activation function to convert the output of the convolutional layer to an output on (0,1), and different weights assigned to each channel, i.e., weight features, are obtained.

[0048] Here, the sigmoid activation function can compress any input in the range (-inf, inf) to a certain value in the interval (0, 1), and its formula is:

number

[0049] The ECA module is particularly well-suited for processing initially extracted multi-channel structural semantic information because it can effectively consider the relationships between channels in the input multi-channel structural semantic information. In multi-channel structural semantic information, the channels represent the time dimension, and the structural semantic matrices of different channels represent structural semantic information at different discrete times within a continuous time frame.

[0050] This embodiment uses an ECA module to perform time-series aggregation on multi-channel structural semantic data based on ECA, interacts structural semantic information at different time points, learns the relationships between structural semantic information at different time points, and significantly improves the generalization ability of the model to diverse data and overall fall prediction performance, particularly by extending the model's ability to discriminate against unpredictable behavior. In this way, the model can not only learn a wide range of features of the data but also be precisely tuned to handle various complex situations, ensuring the accuracy and reliability of predictions.

[0051] The second part employs a support vector machine classification component. The structural semantic information of the time series aggregate is first flattened into vectors, and then input into a trained support vector machine to obtain a prediction of whether or not a fall occurred.

[0052] The structured semantic information, after time-series aggregation, is flattened into vectors and fed into a support vector machine trained as input samples. The support vector machine then outputs the final toppling prediction result. The support vector machine maps the input samples from the input space to a new feature space using a Gaussian kernel function (Radial Basis Function, RBF), and completes the classification of the semantic information by searching for a hyperplane that separates the semantic information using a penalty parameter C and a Gaussian kernel function parameter g.

[0053] Mathematical model of the SVM model:

number

[0054] The formula for the Gaussian kernel function is:

number

[0055] This embodiment evaluates the predictive performance index for each combination by inputting various gait parameters into a machine learning model for predicting fall risk and training it, ultimately obtaining the trained machine learning model.

[0056] Specifically, the multi-channel structured semantic data of all training samples is input into a machine learning model to predict the risk of falling, and the model is trained. If the number of training iterations exceeds a set number, or if the total loss function value is less than a set threshold, training is stopped, and the trained machine learning model is obtained.

[0057] By analyzing video, the video data is preprocessed, and visual input vectors are extracted. These visual input vectors, along with initial structural semantic vectors, are input into a Transformer deep neural network to obtain rudimentary structural semantic information. This structural semantic information can provide important information about the individual's behavioral state. Time-series aggregation based on ECA is performed on this rudimentary structural semantic information to analyze the fluctuations and changes in the individual's behavior over continuous time. By reflecting the individual's motor characteristics, this can be useful in assessing the risk of falls.

[0058] This invention extracts structural semantic information from an individual and simultaneously performs time-series aggregation based on ECA on that structural semantic information. The results after time-series aggregation reflect the stability of the individual's gait. By comprehensively utilizing this information, the system can capture the individual's behavior more comprehensively and improve its sensitivity to the risk of falling. This method of comprehensively utilizing structural semantic information and time-series aggregation operations based on ECA gives the system a greater advantage in predicting falls.

[0059] To effectively learn and adapt to different individual differences and accurately predict the risk of falling, the present invention employs a deep learning neural network model for extracting structural semantic information based on Transformers and a machine learning model for predicting the risk of falling based on ECA modules and support vector machines. The deep learning neural network model can adaptively adjust the model's weights and biases to enhance its ability to learn from unpredictable individuals. By introducing such a deep learning model, the system of the present invention can better adapt to diversity and improve its overall fall prediction performance.

[0060] This embodiment provides a real-time, highly adaptive fall prediction method applicable to multiple fields such as exercise biomechanics and rehabilitation medicine. This method incorporates a deep learning model to significantly improve the intelligence and accuracy of fall prediction, providing a novel technological tool for research and practical applications in these fields.

[0061] The design of this embodiment can be applied to individuals of different age groups and physiological states, significantly improving the system's versatility. It has broad application prospects, particularly in fields such as elderly health management and medical assistance devices.

[0062] Example 2 The objective of this embodiment is to provide a fall prediction system based on video semantics. An acquisition unit configured to acquire behavioral video segments of the individual being measured, A first feature extraction unit is configured to divide frame images corresponding to the behavioral video segments of the individual being measured into grid regions and perform a linear transformation to obtain the visual input vector of the individual being measured. First, a Transformer network is trained, and through training learning, a set of initial structural semantic vectors favorable to the model prediction effect, regardless of any specific image, are obtained. Then, the learned initial structural semantic vectors and the corresponding visual input vectors of the measured individual are input to the Transformer network, and a second feature extraction unit is configured to obtain multi-channel structural semantic information. In the time-series aggregation portion of the machine learning model based on ECA, the ECA module is configured to perform time-series aggregation by assigning appropriate weights to the multi-channel structural semantic information output from the Transformer network, thereby obtaining the structural semantic information of the time-series aggregation. The support vector machine classification portion of the machine learning model includes a prediction unit configured to input the structural semantic information of the time series aggregation into a support vector machine to predict whether or not the subject individual has fallen over, and ultimately output a prediction result of whether or not the subject individual has fallen over.

[0063] In more embodiments, Further providing is an electronic device including memory, a processor, and computer instructions stored in memory and running on the processor, wherein the computer instructions are executed by the processor, completing the method of Embodiment 1. For brevity, no further explanation is provided here.

[0064] It should be understood that in this embodiment, the processor may be a Central Processing Unit (CPU), or it may be another general-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor may be a microprocessor, or it may be any general-purpose processor, etc.

[0065] The memory may include read-only memory and random-access memory, providing instructions and data to the processor, and a portion of the memory may further include non-volatile random memory. For example, the memory may further store device type information.

[0066] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method according to Embodiment 1 is completed.

[0067] The method in Example 1 may be directly embodied as being executed and completed by a hardware processor, or by a combination of hardware and software modules in a processor. The software modules may reside in a storage medium mature in the art, such as random memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, or registers. This storage medium is memory, and the processor reads information in memory and combines its hardware to complete the steps of the method described above. To avoid repetition, no further details are provided here.

[0068] The computer program product includes a computer program, and when the computer program is executed by a processor, the method described in Embodiment 1 is implemented and completed.

[0069] The present invention further provides at least one computer program product tangibly stored in a non-temporary computer-readable storage medium. This computer program product includes instructions executable by a computer, for example, instructions in a program module, which are executed in a device on a target real or virtual processor to perform the processes / methods described above. Generally, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or realize a specific abstract data type. In various embodiments, the functions of program modules can be combined or divided amongst program modules as needed. The device-executable instructions for a program module can be executed in a local or distributed device. In a distributed device, the program module may reside in local and remote storage media.

[0070] Computer program code for implementing the method of the present invention can be written in one or more programming languages. By providing this computer program code to the processor of a general-purpose computer, a dedicated computer, or another programmable data processing device, the program code can cause the functions / operations defined in the flowchart and / or block diagram to be performed when executed by the computer or other programmable data processing device. The program code can be executed entirely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0071] In the context of the present invention, computer program code or associated data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc. Examples of signals may include electrical, optical, radio, audio, or other forms of propagating signals, such as carriers, infrared signals, etc.

[0072] Those skilled in the art will understand that the various example units and algorithmic steps described in relation to this embodiment can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the invention. Those skilled in the art may implement the described functions using different methods for each specific application, but should not consider such implementation to be beyond the scope of this application.

[0073] The above describes specific embodiments of the present invention with reference to the attached drawings. However, this does not limit the scope of the present invention, and it will be clear to those skilled in the art that various modifications and variations that can be carried out by those skilled in the art without requiring creative effort, based on the technical idea of ​​the present invention, are within the scope of the present invention.

Claims

1. A fall prediction method based on video semantics performed by a computer, Obtaining behavioral video segments of the individual being measured, The process involves extracting frame images from the behavioral video segments of the subject to measurement, performing a linear transformation, and obtaining the visual input vector of the subject to measurement. The initial structural semantic vectors obtained through training and the visual input vectors of the subject being measured are both input into the Transformer network to obtain multi-channel structural semantic information. This involves inputting multi-channel structural semantic information output from a transformer network into a trained machine learning model, The trained machine learning model includes two parts: a time-series aggregation part based on Efficient Channel Attention (ECA) and a support vector machine classification part. In the time-series aggregation part based on ECA, the ECA module assigns appropriate weights to the multi-channel structural semantic information output from the Transformer network to perform time-series aggregation and obtain the structural semantic information of the time-series aggregation. The support vector machine classification portion includes inputting the structural semantic information of the time-series aggregation into the support vector machine to predict whether or not the individual being measured has fallen over. Here, the input to the Transformer network includes two parts, and the Transformer network employs only the encoder portion of a standard Transformer network, learning a structural semantic representation by stacking M encoder modules, each module containing two components: a multi-head self-attention component and a feedforward neural network component, and the output portion of each component employs a residual connection and layer normalization structure. Here, obtaining the initial structure semantic vector through training specifically means, This involves randomly initializing N d-dimensional embedding vectors using a Gaussian distribution, inputting the initialized N d-dimensional embedding vectors and the visual input vectors of the training samples into a Transformer network, training the Transformer network, and after training, using the updated N d-dimensional embedding vectors as the initial structure semantic vectors learned through training. Specifically, obtaining time-series aggregated structural semantic information by assigning appropriate weights to the multi-channel structural semantic information output from a Transformer network using the ECA module involves the following steps: The multi-channel structural semantic information output from the transformer is subjected to a global average pooling process to obtain channel features, The channel features are extracted using convolution, and interaction features between channels are obtained. The interaction features between the aforementioned channels are calculated using an activation function to obtain weight features, The weight features are multiplied by the multi-channel structural semantic information output from the Transformer network to obtain time-series aggregated structural semantic information. Specifically, inputting the structural semantic information of the time-series aggregation into a support vector machine to predict whether or not the subject of measurement has fallen over means: The structural semantic information of the aforementioned time series aggregate is mapped to a feature space using a Gaussian kernel function, A fall prediction method characterized by using penalty parameters and Gaussian kernel function parameters to construct an optimal hyperplane for separating data information, classifying the structural semantic information of the time series aggregate, and obtaining a prediction result of whether or not the measured individual has fallen over.

2. Specifically, dividing the frame images corresponding to the behavioral video segments of the subject being measured into grid regions and performing a linear transformation to obtain a visual input vector means: The frame images corresponding to the behavioral video segments of the subject to measurement are divided into grid regions, and the image information of each grid region is flattened into a one-dimensional vector. Mapping the one-dimensional vector corresponding to each grid region to the embedding vector using a linear projection function, The method for predicting falls based on video semantics according to claim 1, characterized in that it involves coding the aforementioned embedding vector to obtain a visual input vector.

3. Specifically, inputting the initial structural semantic vector and the corresponding visual input vector into a Transformer network to obtain multi-channel structural semantic information involves: The process involves using a Transformer encoder to extract features from the visual input vector corresponding to the frame image and the initial structural semantic vector for learning, thereby obtaining a structural semantic matrix. The fall prediction method based on video semantics according to claim 1, characterized in that the structural semantic matrix for each frame image is stored in tensor format to form multi-channel structural semantic information.

4. A fall prediction system based on video semantics, An acquisition unit configured to acquire behavioral video segments of the individual being measured, A first feature extraction unit is configured to divide frame images corresponding to the behavioral video segments of the individual being measured into grid regions and perform a linear transformation to obtain the visual input vector of the individual being measured. A second feature extraction unit is configured to obtain multi-channel structural semantic information of a target individual by inputting both the initial structural semantic vector learned through training and the visual input vector of the target individual into a Transformer network, wherein the input to the Transformer network includes two parts, the Transformer network employs only the encoder portion of a standard Transformer network, learns a structural semantic representation by stacking M encoder modules, each module includes two components: a multi-head self-attention component and a feedforward neural network component, and after the output of each component, the second feature extraction unit employs residual connection and layer normalization structure. Specifically, obtaining the initial structural semantic vector through training means: This involves randomly initializing N d-dimensional embedding vectors using a Gaussian distribution, inputting the initialized N d-dimensional embedding vectors and the visual input vectors of the training samples into a Transformer network, training the Transformer network, and after training, using the updated N d-dimensional embedding vectors as the initial structure semantic vectors learned through training. A time-series aggregation unit is configured to perform time-series aggregation by assigning appropriate weights to multi-channel structural semantic information output from a Transformer network using an ECA module, thereby obtaining time-series aggregated structural semantic information, and specifically, The multi-channel structural semantic information output from the Transformer network is subjected to a global average pooling process to obtain channel features, The channel features are extracted using convolution, and interaction features between channels are obtained. The interaction features between the aforementioned channels are calculated using an activation function to obtain weight features, A time-series aggregation unit obtains time-series aggregation structure semantic information by multiplying the weight features and the multi-channel structural semantic information output from the Transformer network, A prediction unit configured to input the structural semantic information of the time-series aggregation into a support vector machine and predict whether or not the subject of measurement has fallen over, specifically, The structural semantic information of the aforementioned time series aggregate is mapped to a feature space using a Gaussian kernel function, A fall prediction system characterized by including a prediction unit that constructs an optimal hyperplane for separating data information using penalty parameters and Gaussian kernel function parameters, classifies the structural semantic information of the time series aggregate, and obtains a prediction result of whether or not the measured individual has fallen over.

5. Electronic device comprising memory, a processor, and computer instructions stored in memory and running on the processor, wherein the computer instructions, when run by the processor, complete the method according to any one of claims 1 to 3.

6. A computer-readable storage medium used for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in any one of claims 1 to 3 is completed.

7. A computer program product comprising a computer program, wherein, when the computer program is executed by a processor, the method described in any one of claims 1 to 3 is implemented and completed.

Images