Video-based moro reflex identification method and device, terminal equipment and storage medium

Abstract

The application relates to the technical field of biomedical engineering. The application discloses a Moro reflex identification method and device based on a video, a terminal device and a storage medium, which can automatically identify Moro reflex and improve the accuracy of Moro reflex identification. The method comprises the following steps: acquiring a target video of a target object; performing human body part segmentation and identification processing on each frame of image of the target video by using an image segmentation module of a Moro reflex identification model to obtain a target segmentation result corresponding to each human body part in each frame of image; performing optical flow calculation processing on the target video and all target segmentation results by using a motion feature module of the Moro reflex identification model to obtain a plurality of optical flow time sequence data corresponding to each human body part; and performing infant action classification and identification processing on all optical flow time sequence data by using a classification and discrimination module of the Moro reflex identification model to obtain a Moro reflex identification result of the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomedical engineering. More particularly, the present application relates to a video-based Moro reflex identification method and device, a terminal device and a storage medium. BACKGROUND

[0002] Moro reflex is the earliest primitive reflex of newborns after birth, which is usually triggered when the baby is suddenly stimulated (such as sound, strong light or body position change), and is characterized by rapid abduction of both arms or opening of fingers, followed by rapid retraction. This reflex is generally acquired immediately after birth and gradually disappears within 3 to 6 months, and is one of the important indicators for measuring the development status of the nervous system of infants. The absence, delay or asymmetry of Moro reflex may indicate abnormal function of the central nervous system of the infant, such as cerebral palsy, perinatal brain injury or other neurodevelopmental disorders. Therefore, Moro reflex is widely used for neonatal neurological function assessment and early screening of high-risk children.

[0003] At present, the identification of Moro reflex mainly relies on manual observation, and medical staff judge whether the baby's action conforms to the Moro reflex response mode with naked eyes. Although this method is simple to operate, it is limited by the experience of medical staff and has significant subjectivity and dependence, resulting in inaccurate identification of Moro reflex. Therefore, the prior art still needs to be improved and improved. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a video-based Moro reflex identification method, device, terminal device and storage medium, which can automatically identify Moro reflex and improve the accuracy of Moro reflex identification. The embodiments of the present application mainly realize the following technical solutions:

[0005] In a first aspect, the embodiments of the present application provide a video-based Moro reflex identification method, comprising:

[0006] obtaining a target video of a target object;

[0007] performing human body part segmentation and identification processing on each frame of image of the target video by using an image segmentation module of a Moro reflex identification model, to obtain a target segmentation result corresponding to each human body part in each frame of image;

[0008] performing optical flow calculation processing on the target video and all target segmentation results by using a motion feature module of the Moro reflex identification model, to obtain a plurality of optical flow time series data corresponding to each human body part;

[0009] performing infant action classification and identification processing on all optical flow time series data by using a classification and discrimination module of the Moro reflex identification model, to obtain a Moro reflex identification result of the target object.

[0010] According to one embodiment of this application, the step of using an image segmentation module employing a Moro reflection recognition model to perform human body part segmentation and recognition processing on each frame of the target video to obtain the target segmentation result corresponding to each human body part in each frame includes:

[0011] The shared feature encoder of the image segmentation module is used to perform hierarchical feature extraction processing on the target image and the next frame image of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame image. The target image is any frame image in the target video.

[0012] The image segmentation module uses the segmentation branch module to perform human body part segmentation and recognition processing on the second feature map to obtain the first segmentation result corresponding to each human body part in the next frame image;

[0013] The propagation branch module of the image segmentation module is used to perform human body part segmentation and recognition processing on the first segmentation result corresponding to each human body part in the first feature map, the second feature map and the next frame image, to obtain the target segmentation result corresponding to each human body part in the target image.

[0014] According to an embodiment of this application, the step of using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain the first segmentation result corresponding to each human body part in the next frame image includes:

[0015] The first residual group of the segmentation branch module is used to perform image feature extraction processing on the second feature map to obtain the third feature map;

[0016] The receptive field of the third feature map is expanded using the hollow space pyramid pooling module of the segmentation branch module to obtain the fourth feature map.

[0017] The fourth feature map is transformed using the fully connected layer of the segmentation branch module to obtain the first segmentation result corresponding to each human body part in the next frame image.

[0018] According to one embodiment of this application, the step of using the propagation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the first segmentation result corresponding to each human body part in the first feature map, the second feature map, and the next frame image, and obtaining the target segmentation result corresponding to each human body part in the target image includes:

[0019] The second residual group of the propagation branch module is used to perform image feature extraction processing on both the first feature map and the second feature map to obtain the fifth feature map corresponding to the first feature map and the sixth feature map corresponding to the second feature map;

[0020] A cosine similarity matrix is ​​constructed based on the pixels of the fifth feature map and the sixth feature map;

[0021] A first preset algorithm is used to perform weighted calculation on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image; or, a second preset algorithm is used to perform weighted calculation on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image. The true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

[0022] According to one embodiment of this application, the step of using the motion feature module of the Moro reflection recognition model to perform optical flow calculation processing on the target video and all target segmentation results to obtain multiple optical flow time series data corresponding to each human body part includes:

[0023] Human motion feature extraction processing is performed on the target image, the next frame image, and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image;

[0024] Multiple target motion features corresponding to each human body part in all images of the target video are concatenated in time sequence to obtain multiple optical flow time series data corresponding to each human body part.

[0025] According to an embodiment of this application, the step of performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation result corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image includes:

[0026] The target segmentation results corresponding to each human body part in the target image are converted into a mask format to obtain the target mask corresponding to each human body part in the target image;

[0027] For each human body part, the region of interest (ROI) is extracted from the mask to obtain the target ROI for each human body part in the target image.

[0028] Each pixel in the target region of interest corresponding to each human body part in the target image is taken as a target pixel. The optical flow method is used to calculate the target features of each target pixel between the target image and the next frame image. All target features are combined to form an optical flow cluster corresponding to each human body part in the target image.

[0029] According to preset rules, the optical flow clusters corresponding to each human body part in the target image are calculated and processed to obtain multiple target motion features corresponding to each human body part in the target image.

[0030] According to one embodiment of this application, the step of using the classification and discrimination module of the Moro reflection recognition model to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object includes:

[0031] A long short-term memory network was used to classify and recognize infant movements in all optical flow time series data to obtain the Moro reflection recognition results of the target object.

[0032] According to one embodiment of this application, the step of using the classification and discrimination module of the Moro reflection recognition model to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object includes:

[0033] The mean and standard deviation of each optical flow time series data are calculated to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data.

[0034] Calculate the average frequency, centroid frequency, root mean square frequency, or frequency variance of each optical flow time series data to obtain the target frequency characteristics corresponding to each optical flow time series data.

[0035] Analysis of variance was used to screen all first motion intensity features, all first motion locality features, and all target frequency features to obtain multiple features to be identified.

[0036] The multiple features to be identified are input into a support vector machine for infant action classification and recognition processing to obtain the Moro reflection recognition result of the target object.

[0037] A second aspect of this application provides a video-based Moro reflection recognition device, comprising:

[0038] The target video acquisition module is used to acquire the target video of the target object;

[0039] The segmentation and recognition module is used to perform human body part segmentation and recognition processing on each frame of the target video using the Moro reflection recognition model, so as to obtain the target segmentation result corresponding to each human body part in each frame of the image.

[0040] The optical flow calculation module is used to perform optical flow calculation on the target video and all target segmentation results using the motion feature module of the Moro reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part;

[0041] The classification and recognition module is used to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflection recognition model, so as to obtain the Moro reflection recognition result of the target object.

[0042] A third aspect of this application provides a terminal device, including a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the steps of the video-based Moro reflection recognition method provided in the first aspect of this application.

[0043] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the video-based Moro reflection recognition method provided in the first aspect of this application.

[0044] The beneficial effects of the embodiments of this application include:

[0045] This application presents a Moro reflector recognition model to achieve automatic recognition of the Moro reflector. Compared with existing techniques that rely on manual observation, this application introduces an image segmentation module of the Moro reflector recognition model to segment multiple human body parts in each frame of the target video. Then, a motion feature module of the Moro reflector recognition model performs optical flow tracing on the segmented human body parts in each frame. Finally, a classification module of the Moro reflector recognition model classifies the infant's movements based on the optical flow tracing data, thereby obtaining the Moro reflector recognition result. Thus, this application can automatically recognize the Moro reflector. Furthermore, the method provided in this application does not require manual observation, effectively reducing the influence of medical personnel's subjectivity on the Moro reflector recognition result, thereby improving the accuracy of Moro reflector recognition. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 The flowcharts for some embodiments of the video-based Moro reflection recognition method of this application are shown below;

[0048] Figure 2 The flowcharts are shown in some other embodiments of the video-based Moro reflection recognition method of this application;

[0049] Figure 3 The flowcharts are shown in some further embodiments of the video-based Moro reflection recognition method of this application;

[0050] Figure 4 The flowcharts are shown in some further embodiments of the video-based Moro reflection recognition method of this application;

[0051] Figure 5 This is a flowchart illustrating the training steps of the original image segmentation module in this application;

[0052] Figure 6 The diagram below shows the principle block diagram of the video-based Moro reflection recognition device of this application in some embodiments.

[0053] Figure 7 This is a schematic block diagram of the terminal device of this application in some embodiments. Detailed Implementation

[0054] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0055] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0056] The terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0058] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0059] Currently, the identification of the Moro reflex mainly relies on manual observation, where medical staff visually judge whether the infant's movements conform to the Moro reflex response pattern. Although this method is simple to operate, it is limited by the experience of medical staff and has significant subjectivity and dependence, resulting in inaccurate identification of the Moro reflex.

[0060] To overcome the subjectivity of manual observation, recent research has attempted to achieve automatic recognition of infant movements and behaviors through technological means. These technologies can be broadly categorized into three types: methods based on wearable sensors, 3D recognition methods based on depth cameras or motion capture systems, and computer vision methods based on ordinary video images. Wearable sensor-based methods involve attaching devices such as accelerometers, gyroscopes, or electromyography (EMG) sensors to the infant's limbs to collect motion signals for behavior recognition. However, for infants, wearing sensors may cause discomfort or interfere with their natural movements. Furthermore, sensors are prone to detachment or misalignment, leading to signal distortion and making them unsuitable for automatic infant movement recognition. 3D recognition methods based on depth cameras or motion capture systems can achieve high-precision modeling and tracking of infant movements. However, these methods require expensive hardware, complex deployment, and are highly dependent on shooting angles and ambient lighting, making them difficult to widely apply in resource-constrained primary healthcare institutions or home environments. Computer vision methods based on ordinary video images have made preliminary explorations in recognizing infant behaviors such as sleeping, rolling over, and head lifting. However, these methods currently only focus on general behaviors (such as sleeping and crying) or pathological movement patterns (such as epileptic seizures). None of these three types of techniques can be used to identify the Moro reflex. Therefore, identifying the Moro reflex has not yet attracted much attention from researchers, related research is extremely limited, and no practical technical path has yet been developed. Based on this, this application proposes a video-based Moro reflex recognition method for identifying the Moro reflex.

[0061] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0062] refer to Figure 1 As shown, this is a first aspect of an embodiment of the present application, providing a video-based Moro reflection recognition method, including:

[0063] S1. Obtain the target video of the target object.

[0064] The target group is newborns.

[0065] Each frame of the target video contains multiple human body parts of the target object. These multiple human body parts include the head, arms, legs, and torso.

[0066] S2. The image segmentation module using the Moro reflection recognition model performs human body part segmentation and recognition processing on each frame of the target video to obtain the target segmentation result corresponding to each human body part in each frame.

[0067] Further, refer to Figure 3 As shown, step S2 includes:

[0068] S21. The shared feature encoder of the image segmentation module is used to perform hierarchical feature extraction processing on the target image and the next frame image of the target image to obtain the first feature map corresponding to the target image and the second feature map corresponding to the next frame image. The target image is any frame image in the target video.

[0069] More specifically, step S21 includes: performing hierarchical feature extraction processing on the target image using the shared feature encoder to obtain a first feature map corresponding to the target image; and performing hierarchical feature extraction processing on the next frame image using the shared feature encoder to obtain a second feature map corresponding to the next frame image.

[0070] The shared feature encoder consists of the first three modules of a ResNet-101 (Residual Network 101, 101 layers) network. The ResNet-101 network is a deep convolutional neural network belonging to the residual network family. The first three modules include the third residual group, the fourth residual group, and the fifth residual group.

[0071] The third residual group consists of three third residual blocks. Each third residual block includes a first convolutional layer, a second convolutional layer, and a third convolutional layer. The first convolutional layer has a 1×1 kernel and 64 output channels; the second convolutional layer has a 3×3 kernel and 64 output channels; and the third convolutional layer has a 1×1 kernel and 256 output channels.

[0072] The fourth residual group consists of three fourth residual blocks. Each fourth residual block includes a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer. The fourth convolutional layer has a 1×1 kernel and 128 output channels; the fifth convolutional layer has a 3×3 kernel and 128 output channels; and the sixth convolutional layer has a 1×1 kernel and 512 output channels.

[0073] The fifth residual group consists of three fifth residual blocks. Each fifth residual block includes a seventh convolutional layer, an eighth convolutional layer, and a ninth convolutional layer. The seventh convolutional layer has a 1×1 kernel and 256 output channels; the eighth convolutional layer has a 3×3 kernel and 256 output channels; and the ninth convolutional layer has a 1×1 kernel and 1024 output channels.

[0074] Further, the step of performing hierarchical feature extraction processing on the target image using the shared feature encoder to obtain the first feature map corresponding to the target image includes: performing low-level feature extraction processing on the target image using the third residual group to obtain the first low-level feature map; performing mid-level feature extraction processing on the first low-level feature map using the fourth residual group to obtain the first mid-level feature map; and performing high-level feature extraction processing on the first mid-level feature map using the fifth residual group to obtain the first feature map corresponding to the target image.

[0075] Low-level features refer to the edges, colors, textures, and corners of an image.

[0076] Intermediate features refer to the component outlines, local textures, shapes, and spatial relationships of an image.

[0077] High-level features refer to the object components and context-dependent features of an image.

[0078] Further, the step of using the shared feature encoder to perform hierarchical feature extraction processing on the next frame image to obtain the second feature map corresponding to the next frame image includes: using the third residual group to perform low-level feature extraction processing on the next frame image to obtain the second low-level feature map; using the fourth residual group to perform mid-level feature extraction processing on the second low-level feature map to obtain the second mid-level feature map; and using the fifth residual group to perform high-level feature extraction processing on the second mid-level feature map to obtain the second feature map corresponding to the next frame image.

[0079] S22. The second feature map is processed by the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing to obtain the first segmentation result corresponding to each human body part in the next frame image.

[0080] Further, step S22 includes: performing image feature extraction processing on the second feature map using the first residual group of the segmentation branch module to obtain a third feature map; performing receptive field expansion processing on the third feature map using the hollow spatial pyramid pooling module of the segmentation branch module to obtain a fourth feature map; and performing transformation processing on the fourth feature map using the fully connected layer of the segmentation branch module to obtain a first segmentation result corresponding to each human body part in the next frame image.

[0081] In the step of performing image feature extraction processing on the second feature map using the first residual group of the segmentation branch module to obtain the third feature map, the extracted features are the object category and scene semantics of the second feature map.

[0082] The first residual group is composed of the fourth module of the ResNet-101 network. The first residual group consists of three first residual blocks, each of which includes a tenth convolutional layer, an eleventh convolutional layer, and a twelfth convolutional layer. The tenth convolutional layer has a 1×1 kernel and 512 output channels; the eleventh convolutional layer has a 3×3 kernel and 512 output channels; and the twelfth convolutional layer has a 1×1 kernel and 2048 output channels.

[0083] The Atrous Spatial Pyramid Pooling (ASPP) module does not change the image size. It makes the convolution kernel sample the input feature map in a sparser manner by inserting "holes" between the elements of the standard convolution kernel. However, the inserted "holes" only make the pixel (i.e., element) distribution sparser and do not participate in the convolution calculation.

[0084] In the hollow space pyramid pooling module, the number of channels in the third feature map is adjusted to be the same as the number of categories in the target classification, so that each channel corresponds to one category. The target classification includes three categories: Moro reflex actions, general movement actions, and resting actions.

[0085] The receptive field expansion processing refers to multi-scale dilated convolution processing.

[0086] The step of using the hollow space pyramid pooling module of the segmentation branch module to expand the receptive field of the third feature map to obtain the fourth feature map can be implemented by existing technology, and will not be described in detail in this paper.

[0087] Further, the step of transforming the fourth feature map using the fully connected layer of the segmentation branch module to obtain the first segmentation result corresponding to each human body part in the next frame image includes: performing a linear transformation on the fourth feature map to obtain a ninth feature map; performing spatial dimension restoration on the ninth feature map to obtain a tenth feature map; and using the Softmax (i.e., normalization exponent) function to normalize the tenth feature map along the category dimension to obtain the first segmentation result corresponding to each human body part in the next frame image.

[0088] The first segmentation result for each body part is a segmentation probability map.

[0089] S23. The propagation branch module of the image segmentation module is used to perform human body part segmentation and recognition processing on the first segmentation result corresponding to each human body part in the first feature map, the second feature map and the next frame image, to obtain the target segmentation result corresponding to each human body part in the target image.

[0090] Further, refer to Figure 4 As shown, step S23 includes:

[0091] S231. The second residual group of the propagation branch module is used to perform image feature extraction processing on both the first feature map and the second feature map to obtain the fifth feature map corresponding to the first feature map and the sixth feature map corresponding to the second feature map.

[0092] More specifically, step S231 includes: performing image feature extraction processing on the first feature map using the second residual group to obtain the fifth feature map; and performing image feature extraction processing on the second feature map using the second residual group to obtain the sixth feature map. It should be understood that in the step of performing image feature extraction processing on the first feature map using the second residual group to obtain the fifth feature map, the object category and scene semantics of the first feature map are extracted; and in the step of performing image feature extraction processing on the second feature map using the second residual group to obtain the sixth feature map, the object category and scene semantics of the second feature map are extracted.

[0093] The second residual group consists of the fourth module of the ResNet-101 network. The second residual group comprises three second residual blocks, each including a thirteenth, fourteenth, and fifteenth convolutional layer. The thirteenth convolutional layer has a 1×1 kernel and 512 output channels; the fourteenth convolutional layer has a 3×3 kernel and 512 output channels; and the fifteenth convolutional layer has a 1×1 kernel and 2048 output channels.

[0094] S232. Construct a cosine similarity matrix based on the pixels of the fifth feature map and the sixth feature map.

[0095] Furthermore, the calculation formula for step S232 is as follows:

[0096] ;

[0097] in, It is the fifth feature map. The pixel and the sixth feature map The cosine similarity matrix of pixels; It is the fifth feature map The Middle 1 pixel; It is the sixth feature map The Middle 1 pixel; This represents the data processed by the propagation branch module; It is the norm symbol.

[0098] S233. A first preset algorithm is used to perform weighted calculation on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image; or, a second preset algorithm is used to perform weighted calculation on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image. The true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

[0099] The semantic segmentation algorithm can be HRNet (High-Resolution Network) or other deep learning networks.

[0100] The preset category can be a Moro reflex action, a general movement action, or a resting action.

[0101] In other embodiments, the true probability map may also be a manually annotated reference map used to represent the probability that each pixel in the next frame belongs to a preset category.

[0102] Furthermore, the first preset algorithm is used to perform weighted calculation on the true probability map corresponding to the next frame image and the cosine similarity matrix. The calculation formula for obtaining the target segmentation result corresponding to each human body part in the target image is as follows:

[0103] ;

[0104] in, It is the target segmentation result corresponding to each human body part. The 1 pixel; It is a normalized exponential function; It is the first in the cosine similarity matrix Highly similar pixels, Set to 20; It is the first of the cosine similarity matrices. Row data, i.e., the fifth feature map The similarity between each pixel and each pixel in the sixth feature map; It is the true probability map corresponding to the next frame image. In the The probability distribution of pixels.

[0105] Furthermore, the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix are weighted and calculated using the second preset algorithm. The calculation formula for obtaining the target segmentation result corresponding to each human body part in the target image is as follows:

[0106] ;

[0107] in, It is the first segmentation result corresponding to each human body part in the next frame image. In the The probability distribution of pixels.

[0108] The target segmentation result for each human body part is a segmentation probability map.

[0109] S3. The motion feature module of the Moro reflection recognition model is used to perform optical flow calculation on the target video and all target segmentation results to obtain multiple optical flow time series data corresponding to each human body part.

[0110] Further, step S3 includes: performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image; and concatenating the multiple target motion features corresponding to each human body part in all images in the target video in a temporal sequence to obtain multiple optical flow time series data corresponding to each human body part.

[0111] Further, the step of extracting human motion features from the target image, the next frame image, and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image includes: performing mask format conversion processing on the target segmentation results corresponding to each human body part in the target image to obtain a target mask corresponding to each human body part in the target image; performing region of interest extraction processing on each mask corresponding to each human body part to obtain a target region of interest corresponding to each human body part in the target image; taking each pixel point in the target region of interest corresponding to each human body part in the target image as a target pixel point, calculating the target features of each target pixel point between the target image and the next frame image using optical flow method, and combining all target features to form an optical flow cluster corresponding to each human body part in the target image; and performing calculation processing on the optical flow cluster corresponding to each human body part in the target image according to preset rules to obtain multiple target motion features corresponding to each human body part in the target image.

[0112] Further, the step of converting the target segmentation results corresponding to each human body part in the target image to obtain the target mask for each human body part in the target image involves performing mask format conversion on the target segmentation results corresponding to each human body part in the target image using an adaptive thresholding algorithm. The adaptive thresholding algorithm can be the Otsu algorithm (i.e., an automatic thresholding method based on maximizing inter-class variance) or the percentile method.

[0113] Furthermore, the step of calculating the target features of each target pixel between the target image and the next frame image using optical flow mainly involves calculating the target features of each target pixel in multiple predetermined directions between the target image and the next frame image using optical flow. These multiple predetermined directions are horizontal and vertical directions. In other embodiments, those skilled in the art can set other directions according to actual needs. The target feature can be a velocity component. In other embodiments, those skilled in the art can set other features according to actual needs, such as an acceleration component.

[0114] The optical flow method can be different optical flow estimation methods such as Farnback (also known as Farnback optical flow algorithm), TV-L (Total Variation-L1, total variation-L1 norm optical flow algorithm), or RAFT (Recurrent All-Pairs Field Transforms for Optical Flow).

[0115] Further, the step of calculating and processing the optical flow clusters corresponding to each human body part in the target image according to preset rules to obtain multiple target motion features corresponding to each human body part in the target image includes: calculating the average value of all target features in the optical flow clusters corresponding to each human body part in the target image to obtain a second motion intensity feature corresponding to each human body part in the target image; calculating the standard deviation of all target features in the optical flow clusters corresponding to each human body part in the target image to obtain a second motion locality feature corresponding to each human body part in the target image; the second motion intensity feature and the second motion locality feature corresponding to each human body part in the target image constitute multiple target motion features corresponding to each human body part in the target image.

[0116] The second motion intensity feature is used to characterize the intensity of motion.

[0117] The second motion locality feature is used to characterize the locality of motion.

[0118] Furthermore, the plurality of optical flow time series data includes first optical flow time series data, second optical flow time series data, third optical flow time series data, fourth optical flow time series data, fifth optical flow time series data, sixth optical flow time series data, seventh optical flow time series data, and eighth optical flow time series data.

[0119] Further, when the multiple human body parts include the head, arms, legs, and torso, the step of sequentially concatenating multiple target motion features corresponding to each human body part in all images of the target video to obtain multiple optical flow time-series data corresponding to each human body part includes: sequentially concatenating the second motion intensity features corresponding to the head in all images of the target video to obtain first optical flow time-series data corresponding to the head; sequentially concatenating the second motion locality features corresponding to the head in all images of the target video to obtain second optical flow time-series data corresponding to the head; sequentially concatenating the second motion intensity features corresponding to the arms in all images of the target video to obtain third optical flow time-series data corresponding to the arms; and sequentially concatenating the second motion intensity features corresponding to the arms in all images of the target video to obtain third optical flow time-series data corresponding to the arms. The second motion locality features corresponding to the arms in all images are concatenated in time to obtain the fourth optical flow time series data corresponding to the arms; the second motion intensity features corresponding to the legs in all images of the target video are concatenated in time to obtain the fifth optical flow time series data corresponding to the legs; the second motion locality features corresponding to the legs in all images of the target video are concatenated in time to obtain the sixth optical flow time series data corresponding to the legs; the second motion intensity features corresponding to the torso in all images of the target video are concatenated in time to obtain the seventh optical flow time series data corresponding to the torso; the second motion locality features corresponding to the torso in all images of the target video are concatenated in time to obtain the eighth optical flow time series data corresponding to the torso.

[0120] S4. The classification and discrimination module of the Moro reflection recognition model is used to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object.

[0121] In some implementations, step S4 includes using a long short-term memory network to perform infant movement classification and recognition processing on all optical flow time-series data to obtain the Moro reflection recognition result of the target object. That is, the classification and discrimination module is the long short-term memory network.

[0122] In the Long Short-Term Memory (LSTM) network, this embodiment classifies infant movements into three categories: Moro reflex movements, general motor movements, and resting movements. The LTM network is mapped to the final infant movement classification result through fully connected layers. The final infant movement classification result is either Moro reflex movements, general motor movements, or resting movements. Moro reflex movements are triggered when the infant is subjected to sudden stimuli (such as sound, bright light, or a change in body position), manifested as a characteristic movement of rapidly extending the arms or spreading the fingers, followed by rapid retraction. General motor movements refer to infant movements including head lifting, body twisting, rolling over, independent sitting with arm support for balance, crawling, or independent walking. Resting movements refer to limb movements where the infant's body is relatively still or with low activity, such as infant sleep movements.

[0123] Furthermore, using Optuna (an automated hyperparameter optimization software framework primarily used for tuning machine learning and deep learning models), this embodiment of the application determined that the long short-term memory network has two hidden layers, each with 128 neurons, on the collected Moro reflection dataset.

[0124] In some other embodiments, step S4 includes: calculating the mean and standard deviation for each optical flow time series data to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data; calculating the mean frequency, centroid frequency, root mean square frequency, or frequency variance of each optical flow time series data to obtain the target frequency feature corresponding to each optical flow time series data; using variance analysis to filter all the first motion intensity features, all the first motion locality features, and all the target frequency features to obtain multiple features to be identified; and inputting the multiple features to be identified into a support vector machine for infant action classification and recognition processing to obtain the Moro reflection recognition result of the target object.

[0125] It should be understood that the steps of calculating the mean and standard deviation for each optical flow time series data to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data are implemented in the time domain; while the steps of calculating the mean frequency, centroid frequency, root mean square frequency or frequency variance of each optical flow time series data to obtain the target frequency feature corresponding to each optical flow time series data are implemented in the frequency domain.

[0126] The number of features to be identified is 10. In this embodiment, variance analysis is used to filter all first motion intensity features, all first motion locality features, and all target frequency features to obtain the optimal 10 features as the plurality of features to be identified. In other embodiments, the number of features to be identified can be set by those skilled in the art according to actual needs.

[0127] In the analysis of variance method, multiple optical flow features (i.e. multiple target features) corresponding to Moro reflex actions, general movement actions, and resting actions show significant differences, especially the optical flow features of the arm region, which is consistent with the fact that Moro reflexes have movement features of the hand region.

[0128] In other implementations, other feature reduction methods can be used to replace the analysis of variance method, such as PCA (Principal Component Analysis).

[0129] Further, the step of calculating the mean and standard deviation for each optical flow time series data to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data includes: calculating the mean for the first optical flow time series data to obtain the first motion intensity feature corresponding to the first optical flow time series data; calculating the standard deviation for the first optical flow time series data to obtain the first motion locality feature corresponding to the first optical flow time series data; calculating the mean for the second optical flow time series data to obtain the first motion intensity feature corresponding to the second optical flow time series data; calculating the standard deviation for the second optical flow time series data to obtain the first motion locality feature corresponding to the second optical flow time series data; calculating the mean for the third optical flow time series data to obtain the first motion intensity feature corresponding to the third optical flow time series data; calculating the standard deviation for the third optical flow time series data to obtain the first motion locality feature corresponding to the third optical flow time series data; calculating the mean for the fourth optical flow time series data to obtain the first motion intensity feature corresponding to the fourth optical flow time series data; and calculating the mean for the fourth optical flow time series data to obtain the first motion intensity feature corresponding to the fourth optical flow time series data. The following methods are applied to obtain the first motion locality feature corresponding to the fourth optical flow time series data: Standard deviation calculation is performed on the fifth optical flow time series data to obtain the first motion intensity feature; standard deviation calculation is performed on the fifth optical flow time series data to obtain the first motion locality feature; average calculation is performed on the sixth optical flow time series data to obtain the first motion intensity feature; standard deviation calculation is performed on the sixth optical flow time series data to obtain the first motion locality feature; average calculation is performed on the seventh optical flow time series data to obtain the first motion intensity feature; standard deviation calculation is performed on the seventh optical flow time series data to obtain the first motion locality feature; average calculation is performed on the eighth optical flow time series data to obtain the first motion intensity feature; standard deviation calculation is performed on the eighth optical flow time series data to obtain the first motion locality feature.

[0130] In other embodiments, the classification and discrimination module may also be other time series modeling structures (such as GRU, Gated Recurrent Unit) or traditional machine learning methods (such as random forest).

[0131] In the above embodiments, this application presents a Morro reflection recognition model to achieve automatic Morro reflection recognition. Compared with existing technologies that rely on manual observation, this application introduces an image segmentation module of the Morro reflection recognition model to segment multiple human body parts in each frame of the target video. Then, the motion feature module of the Morro reflection recognition model performs optical flow tracing on the segmented human body parts in each frame. Finally, the classification module of the Morro reflection recognition model classifies the optical flow-traced data to obtain the Morro reflection recognition result. Therefore, this application can automatically recognize Morro reflections. Furthermore, the method provided by this application does not require manual observation, effectively reducing the influence of medical personnel's subjectivity on the Morro reflection recognition result, thereby improving the accuracy of Morro reflection recognition.

[0132] In addition, the image segmentation module of this application embodiment can achieve automatic image segmentation without the need for manual selection of regions for segmentation. This allows the application to not only eliminate the interference of background information in the image, but also to perform more refined and specialized feature extraction and analysis operations according to the requirements of the recognition target.

[0133] In some implementations, the video-based Moro reflection recognition method further includes a training step for the image segmentation module. This training step includes: acquiring a training dataset and a set of real labels. Each training data point in the training dataset is a training video, and each frame of each training data point corresponds to a subset of real labels in the set of real labels. More specifically, the head region, arm region, torso region, and leg region of each frame of each training data point have a one-to-one correspondence with a real label in the corresponding set of real labels. The original shared feature encoder of the original image segmentation module is used to perform hierarchical feature extraction processing on the target training image and the subsequent frame of the target training image to obtain a first training feature map corresponding to the target training image and a second training feature map corresponding to the subsequent frame. The target training image is any frame in the target training data. The training data is any training data in the training dataset; the original segmentation branch module of the original image segmentation module is used to perform human body part segmentation and recognition processing on the first training feature map to obtain a first training segmentation probability map corresponding to each human body part in the target training image; the original segmentation branch module is used to perform human body part segmentation and recognition processing on the second training feature map to obtain a second training segmentation probability map corresponding to each human body part in the next frame image; the second residual training group of the original propagation branch module of the original image segmentation module is used to perform image feature extraction processing on the first training feature map to obtain a seventh training feature map; the second residual training group of the original propagation branch module is used to perform image feature extraction processing on the second training feature map to obtain an eighth training feature map; a cosine similarity training matrix is ​​constructed based on the pixels of the seventh and eighth training feature maps (refer to...). Figure 5The process involves the following steps: (1) A first preset training algorithm is used to perform a weighted calculation on the true label subset corresponding to the next frame image and the cosine similarity training matrix to obtain a third training segmentation probability map corresponding to each human body part in the target training image; or, a second preset training algorithm is used to perform a weighted calculation on the second training segmentation probability map corresponding to each human body part in the next frame image and the cosine similarity training matrix to obtain a third training segmentation probability map corresponding to each human body part in the target training image; and a third preset training algorithm is used to perform a weighted calculation on the true label subset corresponding to the target training image and the cosine similarity training matrix to obtain a fourth training segmentation probability map corresponding to each human body part in the next frame image; or, a fourth preset training algorithm is used to perform a weighted calculation on the first training segmentation probability map corresponding to each human body part in the target training image and the cosine similarity training matrix. A fourth training segmentation probability map corresponding to each human body part in the next frame image is obtained; a first loss function is calculated based on the first training segmentation probability map corresponding to each human body part in the target training image and the subset of real labels corresponding to the target training image; a second loss function is calculated based on the second training segmentation probability map corresponding to each human body part in the next frame image and the subset of real labels corresponding to the next frame image; a third loss function is calculated based on the third training segmentation probability map corresponding to each human body part in the target training image and the subset of real labels corresponding to the target training image; a fourth loss function is calculated based on the fourth training segmentation probability map corresponding to each human body part in the next frame image and the subset of real labels corresponding to the next frame image; a total loss function is calculated based on the first, second, third, and fourth loss functions; and the parameters of the original image segmentation module are adjusted based on the total loss function to form the image segmentation module. The structure and function of the original shared feature encoder are the same as those of the shared feature encoder, the only difference being the parameters.

[0134] The training dataset includes 68 infant videos, containing 2149 labeled frames and 4790 unlabeled frames. All frames have been resized to 256×256 pixels, and the labels include five regions: background, head, arms, torso, and legs. In other embodiments, the number of videos in the training dataset can be set by those skilled in the art according to actual needs.

[0135] The structure of the second residual training group is the same as that of the second residual group; the only difference between them is the parameters.

[0136] The specific implementation process of using the original shared feature encoder of the original image segmentation module to perform hierarchical feature extraction processing on the target training image and the next frame image of the target training image to obtain the first training feature map corresponding to the target training image and the second training feature map corresponding to the next frame image can be referred to. This will not be elaborated here.

[0137] The original splitting branch module has the same structure as the splitting branch module, the only difference being the parameters.

[0138] Further, the step of using the original segmentation branch module of the original image segmentation module to perform human body part segmentation and recognition processing on the first training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image includes: using the first residual training group of the original segmentation branch module to perform image feature extraction processing on the first training feature map to obtain a third training feature map; using the original hollow spatial pyramid pooling module of the original segmentation branch module to perform receptive field expansion processing on the third training feature map to obtain a fifth training feature map; and using the original fully connected layer of the original segmentation branch module to transform the fifth training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image. The above steps can be referred to... Figure 5 The steps are "first training feature map", "first residual training group", "third training feature map", "image segmentation" and "first training segmentation probability map".

[0139] The two steps, "using the original hollow spatial pyramid pooling module of the original segmentation branch module to expand the receptive field of the third training feature map to obtain the fifth training feature map; and using the original fully connected layer of the original segmentation branch module to transform the fifth training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image," can correspond to... Figure 5 The steps are "Third Training Feature Map", "Image Segmentation", and "First Training Segmentation Probability Map".

[0140] The structure of the first residual training group is the same as that of the first residual group; the only difference between the two is the parameters.

[0141] The original void space pyramid pooling module and the void space pyramid pooling module have the same structure; the difference between them lies in their parameters.

[0142] Further, the step of using the original segmentation branch module to perform human body part segmentation and recognition processing on the second training feature map to obtain the second training segmentation probability map corresponding to each human body part in the subsequent frame image includes: using the first residual training group to perform image feature extraction processing on the second training feature map to obtain a fourth training feature map; using the original hollow spatial pyramid pooling module to perform receptive field expansion processing on the fourth training feature map to obtain a sixth training feature map; and using the original fully connected layer to transform the sixth training feature map to obtain the second training segmentation probability map corresponding to each human body part in the subsequent frame image. The above steps can be referenced. Figure 5 The steps are "Second Training Feature Map", "First Residual Training Group", "Fourth Training Feature Map", "Image Segmentation" and "Second Training Segmentation Probability Map".

[0143] "The receptive field of the fourth training feature map is expanded using the original hollow spatial pyramid pooling module to obtain the sixth training feature map; the sixth training feature map is transformed using the original fully connected layer to obtain the second training segmentation probability map corresponding to each human body part in the subsequent frame image." This can be interpreted as... Figure 5 The steps are "Fourth Training Feature Map", "Image Segmentation", and "Second Training Segmentation Probability Map".

[0144] Furthermore, the formula for calculating the cosine similarity training matrix based on the pixels of the seventh and eighth training feature maps is as follows: ;in, It is the seventh training feature map. The pixel and the eighth training feature map Cosine similarity training matrix for pixels; It is the seventh training feature map The Middle 1 pixel; It is the eighth training feature map The Middle 1 pixel; This represents the data processed by the original propagation branch module.

[0145] Furthermore, the calculation formula for the step of using the first preset training algorithm to perform weighted calculation processing on the true label subset corresponding to the next frame image and the cosine similarity training matrix to obtain the third training segmentation probability map corresponding to each human body part in the target training image is as follows: ;in, The target training image The third training segmentation probability map corresponding to each human body part The 1 pixel; It is the first in the cosine similarity training matrix Highly similar pixels, Set to 20; It is the first cosine similarity training matrix. Row data, i.e., the seventh feature map The similarity between each pixel and each pixel in the eighth feature map; The first tag source is formed by combining the subset of real tags corresponding to the next frame image. In the The probability distribution of pixels.

[0146] Furthermore, the calculation formula for obtaining the third training segmentation probability map corresponding to each human body part in the target training image by weighting the second training algorithm on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the cosine similarity training matrix is ​​as follows: ;in, It is the second training segmentation probability map corresponding to each human body part in the next frame image. In the The probability distribution of pixels.

[0147] Furthermore, the calculation formula for the step of using a third preset training algorithm to perform weighted calculation processing on the true label subset corresponding to the target training image and the cosine similarity training matrix to obtain the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image is as follows: ;in, The next frame image The fourth training segmentation probability map corresponding to each human body part The 1 pixel; The second label source is formed by combining the subset of real labels corresponding to the target training images. In the The probability distribution of pixels.

[0148] Furthermore, the calculation formula for obtaining the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image by weighting the first training segmentation probability map and the cosine similarity training matrix using the fourth preset training algorithm is as follows: ;in, It is the first training segmentation probability map corresponding to each human body part in the target training image. In the The probability distribution of pixels.

[0149] Furthermore, the formula for calculating the first loss function based on the first training segmentation probability map corresponding to each human body part in the target training image and the subset of real labels corresponding to the target training image is as follows: ;in, It is the first loss function; This refers to the total number of categories (which can also be understood as the total number of human body parts). In this application, the total number of categories is 4. The target training image The first in the corresponding real tag subset One real label; It is the first in the target training image First training segmentation probability map corresponding to individual body parts ; This represents the data processed by the original splitting and branching module.

[0150] Furthermore, the formula for calculating the second loss function based on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the subset of real labels corresponding to the subsequent frame image is as follows: ;in, It is the second loss function; The next frame image The first in the corresponding real tag subset One real label; It is the first in the next frame image Second training segmentation probability map corresponding to individual body parts .

[0151] Furthermore, the formula for calculating the third loss function based on the third training segmentation probability map corresponding to each human body part in the target training image and the true label subset corresponding to the target training image is as follows: ; It is the third loss function; The target training image The first in the corresponding real tag subset One real label; It is the first in the target training image First training segmentation probability map corresponding to individual body parts ; This represents the data processed by the original propagation branch module.

[0152] Furthermore, the formula for calculating the fourth loss function based on the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image and the subset of real labels corresponding to the subsequent frame image is as follows: ; It is the fourth loss function; The next frame image The first in the corresponding real tag subset One real label; It is the first in the next frame image Second training segmentation probability map corresponding to individual body parts .

[0153] Furthermore, the formula for calculating the total loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function is as follows:

[0154] ;

[0155] in, It is the total loss function; yes , Its purpose is to let and The scale is consistent with the first loss function, the second loss function, the third loss function, and the fourth loss function; It is the fifth loss function; It is the sixth loss function.

[0156] Furthermore, the formula for calculating the fifth loss function is as follows: Alternatively, the formula for calculating the fifth loss function is: .

[0157] Furthermore, the formula for calculating the sixth loss function is as follows: Alternatively, the formula for calculating the sixth loss function is: .

[0158] The image segmentation module uses an SPN (SiamParseNet, a semi-supervised body parsing model for baby motion videos, designed to jointly learn single-frame body parsing and inter-frame label propagation) neural network structure, which allows the embodiments of this application to eliminate background information interference without the need for manual region selection.

[0159] The SPN neural network structure can be trained under three different conditions. Specifically, the first condition is supervised learning where both the target training image and the subsequent frame image have real labels; the second condition is semi-supervised learning where only one of the target training image and the subsequent frame image has a real label; and the third condition is unsupervised learning where neither the target training image nor the subsequent frame image has a real label.

[0160] In actual training, most of the training data (i.e., baby videos) is unlabeled. Therefore, if all the training data is used directly for training, the following problems arise: In the early stages of training, when the original image segmentation module is unstable, the pseudo-labels generated by the original segmentation branch module are of poor quality; if the original image segmentation module prematurely enters semi-supervised learning, it will mislead the training. In the later stages of training, if fully supervised learning is relied upon continuously, the original image segmentation module cannot utilize the large amount of unlabeled data. To overcome these problems, this application adopts an AAT (Adaptive Alternative Training) training strategy to adaptively adjust the frequency of use of the three training modes during training. Assume the probability of selecting the fully supervised mode is... The selection probability of semi-supervised learning is The probability of unsupervised selection is ,So The dynamic update formula is: ,in, This is the current training epoch number; It is a pre-set lower probability limit, set to ; This is the maximum number of training rounds, set to 20; This is the annealing temperature parameter that controls the training process (annealing temperature refers to the annealing temperature used for learning rate optimization in deep learning), set to 0.4. and The remaining probability is divided equally.

[0161] In other embodiments, the image segmentation module may also be other structures with temporal modeling capabilities, such as a Transformer network based on an attention mechanism.

[0162] refer to Figure 6 The diagram shown is a schematic block diagram of a video-based Moro reflection recognition device provided in the second aspect of an embodiment of this application. Figure 6 The video-based Moro reflection recognition device 100 includes:

[0163] Target video acquisition module 101 is used to acquire the target video of the target object;

[0164] The segmentation and recognition module 102 is used to perform human body part segmentation and recognition processing on each frame of the target video using the image segmentation module with the Moro reflection recognition model, so as to obtain the target segmentation result corresponding to each human body part in each frame of the image.

[0165] The optical flow calculation module 103 is used to perform optical flow calculation processing on the target video and all target segmentation results using the motion feature module of the Moro reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part;

[0166] The classification and recognition module 104 is used to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflection recognition model, so as to obtain the Moro reflection recognition result of the target object.

[0167] A third aspect of this application provides a terminal device, the schematic diagram of which is as follows: Figure 7 As shown. The terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the terminal device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a video-based Moro reflection recognition method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0168] Those skilled in the art will understand that Figure 7 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0169] In some embodiments, this application provides a terminal device including a processor and a memory. The memory stores a computer program, and the processor calls and runs the computer program stored in the memory to perform the steps of the video-based Moro reflection recognition method provided in the first aspect of this application. In a fourth aspect, this application provides a computer-readable storage medium storing a computer program that causes a computer to perform the steps of the video-based Moro reflection recognition method provided in the first aspect of this application.

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0171] The technical features of the above embodiments can be combined without changing the basic principles of this application. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the patent protection scope of this application should be determined by the appended claims.

Claims

1. A video-based Moro reflection recognition method, characterized in that, include: Obtain the target video of the target object; The image segmentation module using the Moro reflection recognition model performs human body part segmentation and recognition processing on each frame of the target video to obtain the target segmentation result corresponding to each human body part in each frame. The motion feature module of the Moro reflection recognition model is used to perform optical flow calculation on the target video and all target segmentation results to obtain multiple optical flow time series data corresponding to each human body part; The classification and discrimination module of the Moro reflection recognition model is used to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object; The image segmentation module employing the Moro reflection recognition model performs human body part segmentation and recognition processing on each frame of the target video to obtain the target segmentation result corresponding to each human body part in each frame. This process includes: using the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on the target image and the next frame of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame, where the target image is any frame in the target video; using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain a first segmentation result corresponding to each human body part in the next frame; and using the propagation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the first feature map, the second feature map, and the first segmentation result corresponding to each human body part in the next frame to obtain the target segmentation result corresponding to each human body part in the target image. The step of using the motion feature module of the Moro reflection recognition model to perform optical flow calculation on the target video and all target segmentation results to obtain multiple optical flow time series data corresponding to each human body part includes: performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image; and concatenating the multiple target motion features corresponding to each human body part in all images of the target video in a time sequence to obtain multiple optical flow time series data corresponding to each human body part. The steps of extracting human motion features from the target image, the next frame image, and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image include: converting the target segmentation results corresponding to each human body part in the target image into a mask format to obtain a target mask corresponding to each human body part in the target image; extracting regions of interest (ROIs) from each mask corresponding to each human body part to obtain a target ROI corresponding to each human body part in the target image; treating each pixel within the ROI as a target pixel, calculating the target features of each target pixel between the target image and the next frame image using optical flow, and combining all target features to form an optical flow cluster corresponding to each human body part in the target image; and calculating the optical flow cluster corresponding to each human body part in the target image according to a preset rule to obtain multiple target motion features corresponding to each human body part in the target image.

2. The video-based Moro reflection recognition method according to claim 1, characterized in that, The steps of using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain the first segmentation result corresponding to each human body part in the next frame image include: The first residual group of the segmentation branch module is used to perform image feature extraction processing on the second feature map to obtain the third feature map; The receptive field of the third feature map is expanded using the hollow space pyramid pooling module of the segmentation branch module to obtain the fourth feature map. The fourth feature map is transformed using the fully connected layer of the segmentation branch module to obtain the first segmentation result corresponding to each human body part in the next frame image.

3. The video-based Moro reflection recognition method according to claim 1, characterized in that, The steps of using the propagation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the first segmentation results corresponding to each human body part in the first feature map, the second feature map, and the next frame image to obtain the target segmentation result corresponding to each human body part in the target image include: The second residual group of the propagation branch module is used to perform image feature extraction processing on both the first feature map and the second feature map to obtain the fifth feature map corresponding to the first feature map and the sixth feature map corresponding to the second feature map; a cosine similarity matrix is ​​constructed based on the pixels of the fifth feature map and the sixth feature map; A first preset algorithm is used to perform weighted calculation on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image; or, a second preset algorithm is used to perform weighted calculation on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain the target segmentation result corresponding to each human body part in the target image. The true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

4. The video-based Moro reflection recognition method according to claim 1, characterized in that, The step of using the classification and discrimination module of the Moro reflection recognition model to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object includes: using a long short-term memory network to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflection recognition result of the target object; or... The steps for classifying and recognizing infant movements using the classification and discrimination module of the Moro reflection recognition model to obtain the Moro reflection recognition result of the target object include: For each optical flow time series data, the mean and standard deviation are calculated to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data. Calculate the average frequency, centroid frequency, root mean square frequency, or frequency variance of each optical flow time series data to obtain the target frequency characteristics corresponding to each optical flow time series data. Analysis of variance was used to screen all first motion intensity features, all first motion locality features, and all target frequency features to obtain multiple features to be identified. The multiple features to be identified are input into a support vector machine for infant action classification and recognition processing to obtain the Moro reflection recognition result of the target object.

5. A video-based Moro reflection recognition device, characterized in that, include: The target video acquisition module is used to acquire the target video of the target object; The segmentation and recognition module is used to perform human body part segmentation and recognition processing on each frame of the target video using the Moro reflection recognition model, so as to obtain the target segmentation result corresponding to each human body part in each frame of the image. The optical flow calculation module is used to perform optical flow calculation on the target video and all target segmentation results using the motion feature module of the Moro reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part; The classification and recognition module is used to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflection recognition model, so as to obtain the Moro reflection recognition result of the target object; The segmentation and recognition module is further configured to use the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on the target image and the next frame image of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame image, wherein the target image is any frame image in the target video; The image segmentation module uses a segmentation branch module to perform human body part segmentation and recognition processing on the second feature map to obtain a first segmentation result corresponding to each human body part in the next frame image; the image segmentation module uses a propagation branch module to perform human body part segmentation and recognition processing on the first feature map, the second feature map, and the first segmentation result corresponding to each human body part in the next frame image to obtain a target segmentation result corresponding to each human body part in the target image. The optical flow calculation module is also used to perform human motion feature extraction processing on the target image, the next frame image and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image. Multiple target motion features corresponding to each human body part in all images of the target video are concatenated in time sequence to obtain multiple optical flow time series data corresponding to each human body part; The optical flow calculation module is also used to perform mask format conversion processing on the target segmentation results corresponding to each human body part in the target image to obtain the target mask corresponding to each human body part in the target image; Regions of interest (ROIs) are extracted from the mask corresponding to each human body part to obtain the target ROI for each human body part in the target image. Each pixel within the ROI is treated as a target pixel, and optical flow is used to calculate the target features of each target pixel between the target image and the next frame. All target features are combined to form an optical flow cluster for each human body part in the target image. The optical flow clusters for each human body part in the target image are then processed according to preset rules to obtain multiple target motion features for each human body part in the target image.

6. A terminal device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the steps of the video-based Moro reflection recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the steps of the video-based Moro reflection recognition method according to any one of claims 1 to 4.

Images