Video-based Moire reflection identification method and device, terminal equipment and storage medium

Abstract

The invention relates to the technical field of biomedical engineering. The invention discloses a video-based Moire reflection recognition method and device, terminal equipment and a storage medium, which can automatically recognize Moire reflection and improve the accuracy of Moire reflection recognition. The method comprises the following steps: acquiring a target video of a target object; performing human body part segmentation recognition processing on each frame of image of the target video by adopting an image segmentation module of a Moire reflection recognition model to obtain a target segmentation result corresponding to each human body part in each frame of image; performing optical flow calculation processing on the target video and all target segmentation results by adopting a motion feature module of a Moire reflection recognition model to obtain a plurality of pieces of optical flow time sequence data corresponding to each human body part; and adopting a classification and discrimination module of the Moire reflection recognition model to carry out infant motion classification and recognition processing on all the optical flow time sequence data to obtain a Moire reflection recognition result of the target object.

Description

Technical Field

[0001] The present application relates to the field of biomedical engineering technology. More specifically, the present application relates to a video-based Morrow reflex recognition method, apparatus, terminal device, and storage medium. Background Art

[0002] The Moro reflex is one of the earliest primitive reflexes to appear in newborns after birth. It is usually triggered by sudden stimulation (such as sound, bright light, or a change in body position), manifesting as a characteristic movement of rapid extension of the arms or spread of the fingers, followed by rapid retraction. This reflex is generally present at birth and gradually disappears within approximately 3 to 6 months. It is one of the important indicators of the infant's neurological development. The absence, bluntness, or asymmetry of the Moro reflex may indicate central nervous system dysfunction, such as cerebral palsy, perinatal brain injury, or other neurodevelopmental disorders. Therefore, the Moro reflex is widely used in neonatal neurological assessment and early screening of high-risk infants.

[0003] Currently, the identification of the Moro reflex relies primarily on manual observation, where medical staff visually determine whether the infant's movements conform to the Moro reflex response pattern. While this method is simple to use, it is limited by the medical staff's experience and is subject to significant subjectivity and dependency, resulting in inaccurate Moro reflex identification. Therefore, the existing technology needs to be improved and enhanced. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a video-based Moro reflex recognition method, apparatus, terminal device, and storage medium that can automatically recognize Moro reflexes and improve the accuracy of Moro reflex recognition. The embodiments of the present application are mainly achieved through the following technical solutions: A first aspect of an embodiment of the present application provides a video-based Morrow reflection recognition method, comprising: Get the target video of the target object; An image segmentation module of a Morrow reflection recognition model is used to perform human body part segmentation and recognition processing on each frame of the target video to obtain a target segmentation result corresponding to each human body part in each frame of the image; Using the motion feature module of the Morrow reflection recognition model to perform optical flow calculation processing on the target video and all target segmentation results, a plurality of optical flow time series data corresponding to each human body part are obtained; The classification and discrimination module of the Moro reflex recognition model is used to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflex recognition result of the target object.

[0005] According to one embodiment of the present application, the image segmentation module of the Morrow reflection recognition model is used to perform human body part segmentation and recognition processing on each frame of the target video, and the step of obtaining the target segmentation result corresponding to each human body part in each frame of the image includes: Using the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on a target image and a next frame image of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame image, wherein the target image is any frame image in the target video; Using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map, to obtain a first segmentation result corresponding to each human body part in the next frame of image; The propagation branch module of the image segmentation module is used to perform human body part segmentation and recognition processing on the first feature map, the second feature map and the first segmentation result corresponding to each human body part in the next frame image to obtain a target segmentation result corresponding to each human body part in the target image.

[0006] According to one embodiment of the present application, the step of using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain a first segmentation result corresponding to each human body part in the next frame image includes: Performing image feature extraction processing on the second feature map using the first residual group of the segmentation branch module to obtain a third feature map; Using the dilated spatial pyramid pooling module of the segmentation branch module to perform receptive field expansion processing on the third feature map to obtain a fourth feature map; The fourth feature map is transformed using the fully connected layer of the segmentation branch module to obtain a first segmentation result corresponding to each human body part in the next frame image.

[0007] According to one embodiment of the present application, the propagation branch module of the image segmentation module performs human body part segmentation and recognition processing on the first feature map, the second feature map, and the first segmentation result corresponding to each human body part in the next frame image, and the step of obtaining a target segmentation result corresponding to each human body part in the target image includes: Performing image feature extraction processing on both the first feature map and the second feature map using the second residual group of the propagation branch module to obtain a fifth feature map corresponding to the first feature map and a sixth feature map corresponding to the second feature map; Constructing a cosine similarity matrix based on the pixels of the fifth feature map and the sixth feature map; A first preset algorithm is used to perform weighted calculation processing on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image; or a second preset algorithm is used to perform weighted calculation processing on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image, wherein the true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

[0008] According to one embodiment of the present application, the steps of performing optical flow calculation processing on the target video and all target segmentation results using the motion feature module of the Morrow reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part include: Performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation result corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image; The multiple target motion features corresponding to each human body part in all images in the target video are connected in time sequence to obtain multiple optical flow time series data corresponding to each human body part.

[0009] According to one embodiment of the present application, performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation result corresponding to each human body part in the target image, and obtaining multiple target motion features corresponding to each human body part in the target image includes: Performing mask format conversion processing on the target segmentation result corresponding to each human body part in the target image to obtain a target mask corresponding to each human body part in the target image; Performing region of interest extraction processing on the mask corresponding to each human body part to obtain a target region of interest corresponding to each human body part in the target image; Taking each pixel point in the target region of interest corresponding to each human body part in the target image as a target pixel point, calculating the target feature of each target pixel point between the target image and the next frame image using the optical flow method, and combining all target features to form an optical flow cluster corresponding to each human body part in the target image; The optical flow cluster corresponding to each human body part in the target image is calculated and processed according to preset rules to obtain multiple target motion features corresponding to each human body part in the target image.

[0010] According to one embodiment of the present application, the steps of performing infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Morrow reflex recognition model to obtain the Morrow reflex recognition result of the target object include: A long short-term memory network is used to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflex recognition result of the target object.

[0011] According to one embodiment of the present application, the steps of performing infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Morrow reflex recognition model to obtain the Morrow reflex recognition result of the target object include: The average value and standard deviation of each optical flow time series data are calculated and processed to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data; Calculate the average frequency, centroid frequency, root mean square frequency or frequency variance of each optical flow time series data to obtain the target frequency feature corresponding to each optical flow time series data; A variance analysis method is used to screen all first motion intensity features, all first motion locality features, and all target frequency features to obtain multiple features to be identified; The multiple features to be identified are input into a support vector machine for classification and identification of infant movements to obtain a Moro reflex identification result of the target object.

[0012] A second aspect of the embodiments of the present application provides a video-based Morrow reflex recognition device, comprising: A target video acquisition module is used to acquire a target video of a target object; A segmentation and recognition module is used to perform human body part segmentation and recognition processing on each frame of the target video using the image segmentation module of the Morrow reflection recognition model to obtain a target segmentation result corresponding to each human body part in each frame of the image; An optical flow calculation module is used to perform optical flow calculation processing on the target video and all target segmentation results using a motion feature module of a Morrow reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part; The classification and recognition module is used to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflex recognition model to obtain the Moro reflex recognition result of the target object.

[0013] In a third aspect of an embodiment of the present application, a terminal device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the steps of the video-based Morrow reflex recognition method provided in the first aspect of the embodiment of the present application.

[0014] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the video-based Morrow reflection recognition method provided in the first aspect of the embodiment of the present application.

[0015] The beneficial effects of the embodiments of the present application include: The embodiment of the present application designs a Moro reflex recognition model to realize the automatic recognition of the Moro reflex. Compared with the existing technology using manual observation, the embodiment of the present application introduces the image segmentation module of the Moro reflex recognition model to segment multiple human body parts in each frame image of the target video, and then uses the motion feature module of the Moro reflex recognition model to perform optical flow tracking on the multiple human body parts in each frame image after segmentation. Finally, the classification and discrimination module of the Moro reflex recognition model is used to classify the data after optical flow tracking into infant movements, thereby obtaining the Moro reflex recognition result. Therefore, the embodiment of the present application can realize the automatic recognition of the Moro reflex. In addition, the method provided in the embodiment of the present application does not require manual observation, effectively reducing the influence of the subjectivity of medical staff on the Moro reflex recognition result, thereby improving the accuracy of Moro reflex recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a video-based Morrow reflection recognition method in some embodiments of the present application; Figure 2 Flowcharts of other embodiments of the video-based Morrow reflection recognition method of the present application; Figure 3 Flowcharts of the video-based Morrow reflection recognition method in some further embodiments of the present application; Figure 4 Flowcharts of the video-based Morrow reflection recognition method in some further embodiments of the present application; Figure 5This is a flowchart of the training steps of the original image segmentation module of this application; Figure 6 This is a principle block diagram of a video-based Morrow reflection recognition device in some embodiments of the present application; Figure 7 This is a principle block diagram of the terminal device of the present application in some embodiments. DETAILED DESCRIPTION

[0018] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0019] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0020] The terms "exemplary" or "for example" are used to indicate an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0021] The terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0022] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more of the relevant listed items.

[0023] Currently, the identification of the Moro reflex relies primarily on manual observation, where medical staff visually determine whether the infant's movements conform to the Moro reflex response pattern. While this method is simple to use, it is limited by the medical staff's experience and is subject to significant subjectivity and dependency, resulting in inaccurate Moro reflex identification.

[0024] To overcome the subjectivity of manual observation, recent research has attempted to automatically identify infant movements and behaviors through technical means. These approaches can be broadly categorized into three categories: wearable sensor-based methods, three-dimensional recognition methods using depth cameras or motion capture systems, and computer vision methods based on standard video images. Wearable sensor-based methods utilize devices such as accelerometers, gyroscopes, or electromyography sensors attached to infants' limbs to collect motion signals for behavioral recognition. However, wearing these sensors can cause discomfort or interfere with natural movements. Furthermore, sensors are prone to signal distortion due to problems such as detachment and misalignment, making them unsuitable for automatic recognition of infant movements and behaviors. Three-dimensional recognition methods based on depth cameras or motion capture systems can achieve high-precision modeling and tracking of infant movements. However, these methods are expensive, complex to deploy, and require strict camera angles and ambient lighting, making them difficult to implement in resource-limited primary healthcare settings or at home. Computer vision methods based on ordinary video images have been initially explored in identifying infant behaviors such as sleeping, rolling over, and raising the head. However, this method currently focuses only on general behaviors (such as sleeping, crying, etc.) or pathological movement patterns (such as epileptic seizures). None of the above three types of technical means can be used to identify the Moro reflex. Therefore, it is determined that the Moro reflex has not yet received the attention of researchers, related research work is extremely limited, and no practical technical path has yet been formed. Based on this, this application proposes a video-based Moro reflex recognition method for identifying the Moro reflex.

[0025] The specific implementation of this application is further described below with reference to the accompanying drawings.

[0026] refer to Figure 1 As shown in the first aspect of the embodiment of the present application, a video-based Morrow reflection recognition method is provided, comprising: S1. Obtain a target video of a target object.

[0027] The target subject is a newborn baby.

[0028] Each frame of the target video contains multiple body parts of the target object, including the head, arms, legs, and torso.

[0029] S2. Using the image segmentation module of the Morrow reflection recognition model, each frame of the target video is subjected to human body part segmentation and recognition processing to obtain a target segmentation result corresponding to each human body part in each frame of the image.

[0030] Further, refer to Figure 3 As shown, step S2 includes: S21. Use the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on the target image and the next frame image of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame image, where the target image is any frame image in the target video.

[0031] More specifically, step S21 includes: using the shared feature encoder to perform hierarchical feature extraction processing on the target image to obtain a first feature map corresponding to the target image; using the shared feature encoder to perform hierarchical feature extraction processing on the next frame image to obtain a second feature map corresponding to the next frame image.

[0032] The shared feature encoder consists of the first three modules of the ResNet-101 (Residual Network 101) network. The ResNet-101 network is a deep convolutional neural network that belongs to the residual network family. These first three modules include the third residual group, the fourth residual group, and the fifth residual group.

[0033] The third residual group is composed of three third residual blocks, each of which includes a first convolutional layer, a second convolutional layer and a third convolutional layer. The convolution kernel of the first convolutional layer is 1×1 and the number of output channels is 64; the convolution kernel of the second convolutional layer is 3×3 and the number of output channels is 64; the convolution kernel of the third convolutional layer is 1×1 and the number of output channels is 256.

[0034] The fourth residual group is composed of three fourth residual blocks, each of which includes a fourth convolutional layer, a fifth convolutional layer and a sixth convolutional layer. The convolution kernel of the fourth convolutional layer is 1×1 and the number of output channels is 128; the convolution kernel of the fifth convolutional layer is 3×3 and the number of output channels is 128; the convolution kernel of the sixth convolutional layer is 1×1 and the number of output channels is 512.

[0035] The fifth residual group is composed of three fifth residual blocks, each of the fifth residual blocks includes a seventh convolutional layer, an eighth convolutional layer and a ninth convolutional layer, the convolution kernel of the seventh convolutional layer is 1×1, and the number of output channels is 256; the convolution kernel of the eighth convolutional layer is 3×3, and the number of output channels is 256; the convolution kernel of the ninth convolutional layer is 1×1, and the number of output channels is 1024.

[0036] Furthermore, the shared feature encoder is used to perform hierarchical feature extraction processing on the target image to obtain a first feature map corresponding to the target image, including: using the third residual group to perform low-level feature extraction processing on the target image to obtain a first low-level feature map; using the fourth residual group to perform intermediate feature extraction processing on the first low-level feature map to obtain a first intermediate feature map; using the fifth residual group to perform high-level feature extraction processing on the first intermediate feature map to obtain a first feature map corresponding to the target image.

[0037] Low-level features refer to image edges, colors, textures, and corners.

[0038] Mid-level features refer to the image's component contours, local textures, shapes, and spatial relationships.

[0039] High-level features refer to object parts and context-dependent features of images.

[0040] Furthermore, the shared feature encoder is used to perform hierarchical feature extraction processing on the next frame image, and the step of obtaining a second feature map corresponding to the next frame image includes: using the third residual group to perform low-level feature extraction processing on the next frame image to obtain a second low-level feature map; using the fourth residual group to perform intermediate feature extraction processing on the second low-level feature map to obtain a second intermediate feature map; using the fifth residual group to perform high-level feature extraction processing on the second intermediate feature map to obtain a second feature map corresponding to the next frame image.

[0041] S22. Use the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain a first segmentation result corresponding to each human body part in the next frame image.

[0042] Furthermore, step S22 includes: using the first residual group of the segmentation branch module to perform image feature extraction processing on the second feature map to obtain a third feature map; using the void space pyramid pooling module of the segmentation branch module to perform receptive field expansion processing on the third feature map to obtain a fourth feature map; using the fully connected layer of the segmentation branch module to transform the fourth feature map to obtain a first segmentation result corresponding to each human body part in the next frame image.

[0043] In the step of performing image feature extraction processing on the second feature map using the first residual group of the segmentation branch module to obtain the third feature map, object categories and scene semantics of the second feature map are extracted.

[0044] The first residual group is composed of the fourth module of the ResNet-101 network. The first residual group is composed of three first residual blocks, each of which includes the tenth convolutional layer, the eleventh convolutional layer, and the twelfth convolutional layer. The tenth convolutional layer has a 1×1 convolution kernel and 512 output channels; the eleventh convolutional layer has a 3×3 convolution kernel and 512 output channels; and the twelfth convolutional layer has a 1×1 convolution kernel and 2048 output channels.

[0045] The Atrous Spatial Pyramid Pooling (ASPP) module does not resize the image. Instead, it inserts "holes" between the elements of the standard convolution kernel, allowing the kernel to sample the input feature map more sparsely. However, these inserted "holes" only make the pixel (element) distribution more sparse and do not participate in the convolution calculation.

[0046] In the dilated spatial pyramid pooling module, the number of channels of the third feature map is adjusted to be the same as the number of target classification categories, so that each channel corresponds to one category. The target classification includes three categories: Moro reflex action, general movement action, and resting action.

[0047] The receptive field expansion process refers to multi-scale dilated convolution processing.

[0048] The step of performing receptive field expansion processing on the third feature map using the dilated spatial pyramid pooling module of the segmentation branch module to obtain the fourth feature map can be achieved by existing technologies and will not be described in detail herein.

[0049] Furthermore, the fourth feature map is transformed using the fully connected layer of the segmentation branch module to obtain the first segmentation result corresponding to each human body part in the next frame image, including the following steps: performing linear transformation on the fourth feature map to obtain a ninth feature map; performing spatial dimension restoration on the ninth feature map to obtain a tenth feature map; and using a Softmax (also known as a normalization exponent) function to normalize the ten feature maps along the category dimension to obtain the first segmentation result corresponding to each human body part in the next frame image.

[0050] The first segmentation result corresponding to each human body part is a segmentation probability map.

[0051] S23. Use the propagation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the first feature map, the second feature map and the first segmentation result corresponding to each human body part in the next frame image to obtain a target segmentation result corresponding to each human body part in the target image.

[0052] Further, refer to Figure 4 As shown, step S23 includes: S231. Use the second residual group of the propagation branch module to perform image feature extraction processing on the first feature map and the second feature map to obtain a fifth feature map corresponding to the first feature map and a sixth feature map corresponding to the second feature map.

[0053] More specifically, step S231 includes: performing image feature extraction processing on the first feature map using the second residual group to obtain the fifth feature map; and performing image feature extraction processing on the second feature map using the second residual group to obtain the sixth feature map. It should be understood that in the step of performing image feature extraction processing on the first feature map using the second residual group to obtain the fifth feature map, the object category and scene semantics of the first feature map are extracted; and in the step of performing image feature extraction processing on the second feature map using the second residual group to obtain the sixth feature map, the object category and scene semantics of the second feature map are extracted.

[0054] The second residual group is composed of the fourth module of the ResNet-101 network. The second residual group is composed of three second residual blocks, each of which includes a 13th convolutional layer, a 14th convolutional layer, and a 15th convolutional layer. The 13th convolutional layer has a 1×1 convolution kernel and 512 output channels; the 14th convolutional layer has a 3×3 convolution kernel and 512 output channels; and the 15th convolutional layer has a 1×1 convolution kernel and 2048 output channels.

[0055] S232: Construct a cosine similarity matrix based on the pixels of the fifth feature map and the sixth feature map.

[0056] Furthermore, the calculation formula of step S232 is: ; in, It is the fifth feature map pixel points and the first pixel in the sixth feature map The cosine similarity matrix of pixels; is the fifth characteristic map Middle pixels; is the sixth characteristic map Middle pixels; Represents the data processed by the propagation branch module; is the norm symbol.

[0057] S233. Use a first preset algorithm to perform weighted calculation processing on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image; or use a second preset algorithm to perform weighted calculation processing on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image, wherein the true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

[0058] The semantic segmentation algorithm may be HRNet (High-Resolution Network) or other deep learning networks.

[0059] The preset category may be a Moro reflex action, a general movement action, or a resting action.

[0060] In other implementations, the true probability map may also be a manually labeled reference map used to represent the probability that each pixel in the next frame of image belongs to a preset category.

[0061] Furthermore, a first preset algorithm is used to perform weighted calculation processing on the true probability map corresponding to the next frame image and the cosine similarity matrix, and the calculation formula for obtaining the target segmentation result corresponding to each human body part in the target image is: ; in, is the target segmentation result corresponding to each human body part No. pixels; is the normalized exponential function; is the first in the cosine similarity matrix Highly similar pixels, Set to 20; is the first cosine similarity matrix Row data, that is, the first The similarity between each pixel and each pixel in the sixth feature map; is the true probability map corresponding to the next frame image In the The probability distribution of pixels.

[0062] Furthermore, a second preset algorithm is used to perform weighted calculation processing on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix, and a calculation formula for obtaining the target segmentation result corresponding to each human body part in the target image is: ; in, is the first segmentation result corresponding to each human body part in the next frame image In the The probability distribution of pixels.

[0063] The target segmentation result corresponding to each human body part is a segmentation probability map.

[0064] S3. Use the motion feature module of the Morrow reflection recognition model to perform optical flow calculation on the target video and all target segmentation results to obtain multiple optical flow time series data corresponding to each human body part.

[0065] Furthermore, step S3 includes: performing human motion feature extraction processing on the target image, the next frame image and the target segmentation results corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image; and connecting the multiple target motion features corresponding to each human body part in all images in the target video in time sequence to obtain multiple optical flow time series data corresponding to each human body part.

[0066] Furthermore, the target image, the next frame image and the target segmentation results corresponding to each human body part in the target image are subjected to human motion feature extraction processing, and the step of obtaining multiple target motion features corresponding to each human body part in the target image includes: performing mask format conversion processing on the target segmentation results corresponding to each human body part in the target image to obtain the target mask corresponding to each human body part in the target image; performing region of interest extraction processing on the mask corresponding to each human body part to obtain the target region of interest corresponding to each human body part in the target image; taking each pixel point in the target region of interest corresponding to each human body part in the target image as a target pixel point, using the optical flow method to calculate the target feature of each target pixel point between the target image and the next frame image, and combining all target features to form an optical flow cluster corresponding to each human body part in the target image; calculating and processing the optical flow cluster corresponding to each human body part in the target image according to preset rules to obtain multiple target motion features corresponding to each human body part in the target image.

[0067] Furthermore, the step of performing mask format conversion processing on the target segmentation results corresponding to each human body part in the target image to obtain a target mask corresponding to each human body part in the target image is performed by performing mask format conversion processing on the target segmentation results corresponding to each human body part in the target image using an adaptive threshold algorithm, thereby obtaining a target mask corresponding to each human body part in the target image. The adaptive threshold algorithm can be an Otsu (i.e., an automatic threshold selection method based on maximizing inter-class variance) algorithm or a percentile method.

[0068] Furthermore, the step of using the optical flow method to calculate the target features of each target pixel between the target image and the next frame image mainly uses the optical flow method to calculate the target features of each target pixel in multiple predetermined directions between the target image and the next frame image. The multiple predetermined directions are horizontal and vertical directions. In other embodiments, those skilled in the art can set other directions according to actual needs. The target feature can be a velocity component. In other embodiments, those skilled in the art can set it to other features, such as an acceleration component, according to actual needs.

[0069] The optical flow method can be different optical flow estimation methods such as Farnback (also known as Farnback optical flow algorithm), TV-L (Total Variation-L1, total variation-L1 norm optical flow algorithm) or RAFT (Recurrent All-Pairs Field Transforms for Optical Flow, recursive full-pairs field transform optical flow method).

[0070] Furthermore, the optical flow cluster corresponding to each human body part in the target image is calculated and processed according to preset rules, and the step of obtaining multiple target motion features corresponding to each human body part in the target image includes: calculating the average value of all target features in the optical flow cluster corresponding to each human body part in the target image to obtain the second motion intensity feature corresponding to each human body part in the target image; calculating the standard deviation of all target features in the optical flow cluster corresponding to each human body part in the target image to obtain the second motion locality feature corresponding to each human body part in the target image; the second motion intensity feature corresponding to each human body part in the target image and the second motion locality feature corresponding to each human body part in the target image constitute multiple target motion features corresponding to each human body part in the target image.

[0071] The second motion intensity feature is used to characterize the intensity of the motion.

[0072] The second motion locality feature is used to characterize the locality of motion.

[0073] Further, the multiple optical flow time series data include first optical flow time series data, second optical flow time series data, third optical flow time series data, fourth optical flow time series data, fifth optical flow time series data, sixth optical flow time series data, seventh optical flow time series data and eighth optical flow time series data.

[0074] Furthermore, in the case where the multiple human body parts include the head, arms, legs and torso, the multiple target motion features corresponding to each human body part in all images in the target video are connected in time sequence to obtain multiple optical flow time series data corresponding to each human body part, and the steps include: connecting the second motion intensity features corresponding to the head in all images in the target video in time sequence to obtain the first optical flow time series data corresponding to the head; connecting the second motion locality features corresponding to the head in all images in the target video in time sequence to obtain the second optical flow time series data corresponding to the head; connecting the second motion intensity features corresponding to the arms in all images in the target video in time sequence to obtain the third optical flow time series data corresponding to the arms; The second motion local features corresponding to the arms in all images are connected in time sequence to obtain the fourth optical flow time series data corresponding to the arms; the second motion intensity features corresponding to the legs in all images in the target video are connected in time sequence to obtain the fifth optical flow time series data corresponding to the legs; the second motion local features corresponding to the legs in all images in the target video are connected in time sequence to obtain the sixth optical flow time series data corresponding to the legs; the second motion intensity features corresponding to the torso in all images in the target video are connected in time sequence to obtain the seventh optical flow time series data corresponding to the torso; the second motion local features corresponding to the torso in all images in the target video are connected in time sequence to obtain the eighth optical flow time series data corresponding to the torso.

[0075] S4. Using the classification and discrimination module of the Moro reflex recognition model to perform infant action classification and recognition processing on all optical flow time series data, to obtain the Moro reflex recognition result of the target object.

[0076] In some embodiments, step S4 includes using a long short-term memory network to perform infant action classification and recognition processing on all optical flow time series data to obtain the target object's Moro reflex recognition result. In other words, the classification and recognition module is the long short-term memory network.

[0077] In the long short-term memory network, the embodiment of the present application classifies infant movements into three categories: Moro reflex, general motor movements, and resting movements. The long short-term memory network is mapped to the final infant movement classification result through a fully connected layer. The final infant movement classification result is also Moro reflex, general motor movement, or resting movement. The Moro reflex is stimulated when the infant is suddenly stimulated (such as sound, bright light, or change in body position), and is manifested as a characteristic movement of the arms rapidly extending or the fingers opening, followed by rapid retraction. The general motor movements refer to infant movements including the infant's head-up movements, body twisting movements, turning over movements, sitting alone with arm support to maintain balance, crawling movements, or independent walking movements. The resting movements refer to limb movements when the infant's body is relatively still or has low activity, such as the infant's sleeping movements.

[0078] Furthermore, through Optuna (Optuna is a software framework for automated hyperparameter optimization, mainly used for parameter adjustment of machine learning and deep learning models), the embodiment of the present application determined that the number of hidden layers of the long short-term memory network is 2 layers, and the size of each layer is 128 neurons on the collected Moro reflex data set.

[0079] In other embodiments, step S4 includes: calculating the mean and standard deviation of each optical flow time series data to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data; calculating the average frequency, center of gravity frequency, root mean square frequency or frequency variance of each optical flow time series data to obtain the target frequency feature corresponding to each optical flow time series data; using the variance analysis method to screen all the first motion intensity features, all the first motion locality features and all the target frequency features to obtain multiple features to be identified; inputting the multiple features to be identified into the support vector machine for infant movement classification and recognition processing to obtain the Moro reflex recognition result of the target object.

[0080] It should be understood that the step of calculating the mean value and standard deviation of each optical flow time series data and obtaining the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data is implemented in the time domain; the step of calculating the average frequency, center of gravity frequency, root mean square frequency or frequency variance of each optical flow time series data and obtaining the target frequency feature corresponding to each optical flow time series data is implemented in the frequency domain.

[0081] The number of the multiple features to be identified is 10. In the embodiment of the present application, a variance analysis method is used to screen all first motion intensity features, all first motion locality features, and all target frequency features to obtain the best 10 features as the multiple features to be identified. In other embodiments, the number of the multiple features to be identified can be set by those skilled in the art according to actual needs.

[0082] In the variance analysis method, multiple optical flow features (i.e., multiple target features) corresponding to Moro reflex movements, general movement movements, and resting movements have significant differences, especially the optical flow features of the arm area, which is consistent with the movement characteristics of the Moro reflex in the hand area.

[0083] In other implementations, other feature dimension reduction methods may be used to replace the variance analysis method, for example, a PCA (Principal Component Analysis) method.

[0084] Furthermore, the steps of calculating the average value and standard deviation of each optical flow time series data and obtaining the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data include: calculating the average value of the first optical flow time series data to obtain the first motion intensity feature corresponding to the first optical flow time series data; calculating the standard deviation of the first optical flow time series data to obtain the first motion locality feature corresponding to the first optical flow time series data; calculating the average value of the second optical flow time series data to obtain the first motion intensity feature corresponding to the second optical flow time series data; calculating the standard deviation of the second optical flow time series data to obtain the first motion locality feature corresponding to the second optical flow time series data; calculating the average value of the third optical flow time series data to obtain the first motion intensity feature corresponding to the third optical flow time series data; calculating the standard deviation of the third optical flow time series data to obtain the first motion locality feature corresponding to the third optical flow time series data; calculating the average value of the fourth optical flow time series data to obtain the first motion intensity feature corresponding to the fourth optical flow time series data; calculating the standard deviation of the fourth optical flow time series data to obtain the first motion locality feature corresponding to the fourth optical flow time series data. Perform standard deviation calculation processing on the fourth optical flow time series data to obtain the first motion local feature corresponding to the fourth optical flow time series data; perform average value calculation processing on the fifth optical flow time series data to obtain the first motion intensity feature corresponding to the fifth optical flow time series data; perform standard deviation calculation processing on the fifth optical flow time series data to obtain the first motion local feature corresponding to the fifth optical flow time series data; perform average value calculation processing on the sixth optical flow time series data to obtain the first motion intensity feature corresponding to the sixth optical flow time series data; perform standard deviation calculation processing on the sixth optical flow time series data to obtain the first motion local feature corresponding to the sixth optical flow time series data; perform average value calculation processing on the seventh optical flow time series data to obtain the first motion intensity feature corresponding to the seventh optical flow time series data; perform standard deviation calculation processing on the seventh optical flow time series data to obtain the first motion local feature corresponding to the seventh optical flow time series data; perform average value calculation processing on the eighth optical flow time series data to obtain the first motion intensity feature corresponding to the eighth optical flow time series data; perform standard deviation calculation processing on the eighth optical flow time series data to obtain the first motion local feature corresponding to the eighth optical flow time series data.

[0085] In other embodiments, the classification and discrimination module may also be other time series modeling structures (such as GRU, Gated Recurrent Unit, gated recurrent unit) or traditional machine learning methods (such as random forest).

[0086] In the above-mentioned embodiment, the embodiment of the present application designs a Morrow reflex recognition model to realize the automatic recognition of the Morrow reflex. Compared with the existing technology using manual observation, the embodiment of the present application introduces the image segmentation module of the Morrow reflex recognition model to segment the multiple human body parts in each frame image in the target video, and then uses the motion feature module of the Morrow reflex recognition model to perform optical flow tracking on the multiple human body parts in each frame image after segmentation. Finally, the classification and discrimination module of the Morrow reflex recognition model is used to classify the data after optical flow tracking, thereby obtaining the Morrow reflex recognition result. As a result, the embodiment of the present application can realize the automatic recognition of the Morrow reflex. In addition, the method provided in the embodiment of the present application does not require manual observation, effectively reducing the influence of the subjectivity of medical staff on the Morrow reflex recognition result, thereby improving the accuracy of Morrow reflex recognition.

[0087] In addition, the image segmentation module of the embodiment of the present application can realize automatic image segmentation without the need for manual selection of areas for segmentation, so that the present application can not only eliminate the interference of background information of the image, but also further perform more refined and specialized feature extraction and analysis operations according to the recognition target requirements.

[0088] In some embodiments, the video-based Morrow reflex recognition method further includes a training step of the image segmentation module, and the training step of the image segmentation module includes: obtaining a training data set and a real label set, wherein each training data in the training data set is a training video, and each frame image of each training data corresponds to one of the real label subsets of the real label set. More specifically, the head area, arm area, torso area and leg area of ​​each frame image of each training data have a one-to-one correspondence with one of the real labels of the corresponding real label set; using the original shared feature encoder of the original image segmentation module to perform hierarchical feature extraction processing on the target training image and the subsequent frame image of the target training image to obtain a first training feature map corresponding to the target training image and a second training feature map corresponding to the subsequent frame image, wherein the target training image is any frame image in the target training data, and the target The training data is any one of the training data sets; the original segmentation branch module of the original image segmentation module is used to perform human body part segmentation and recognition processing on the first training feature map to obtain a first training segmentation probability map corresponding to each human body part in the target training image; the original segmentation branch module is used to perform human body part segmentation and recognition processing on the second training feature map to obtain a second training segmentation probability map corresponding to each human body part in the subsequent frame image; the second residual training group of the original propagation branch module of the original image segmentation module is used to perform image feature extraction processing on the first training feature map to obtain a seventh training feature map; the second residual training group of the original propagation branch module is used to perform image feature extraction processing on the second training feature map to obtain an eighth training feature map; a cosine similarity training matrix is ​​constructed based on the pixel points of the seventh training feature map and the eighth training feature map (refer to Figure 5, performing weighted calculation on the true label subset corresponding to the subsequent frame image and the cosine similarity training matrix using a first preset training algorithm to obtain a third training segmentation probability map corresponding to each human body part in the target training image, or performing weighted calculation on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the cosine similarity training matrix using a second preset training algorithm to obtain a third training segmentation probability map corresponding to each human body part in the target training image; performing weighted calculation on the true label subset corresponding to the target training image and the cosine similarity training matrix using a third preset training algorithm to obtain a fourth training segmentation probability map corresponding to each human body part in the subsequent frame image, or performing weighted calculation on the first training segmentation probability map corresponding to each human body part in the target training image and the cosine similarity training matrix using a fourth preset training algorithm. Obtain a fourth training segmentation probability map corresponding to each human body part in the subsequent frame image; calculate a first loss function based on the first training segmentation probability map corresponding to each human body part in the target training image and the true label subset corresponding to the target training image; calculate a second loss function based on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the true label subset corresponding to the subsequent frame image; calculate a third loss function based on the third training segmentation probability map corresponding to each human body part in the target training image and the true label subset corresponding to the target training image; calculate a fourth loss function based on the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image and the true label subset corresponding to the subsequent frame image; calculate a total loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function; and adjust the parameters of the original image segmentation module based on the total loss function to form the image segmentation module. The structure and function of the original shared feature encoder are the same as those of the shared feature encoder, and the difference between the two is only in the parameters.

[0089] The training dataset includes 68 infant videos, comprising 2,149 labeled frames and 4,790 unlabeled frames. All frames are resized to 256×256, and the annotations include five regions: background, head, arms, torso, and legs. In other embodiments, the number of videos in the training dataset can be adjusted by those skilled in the art based on actual needs.

[0090] The structure of the second residual training group is the same as the structure of the second residual group, and the difference between the two lies only in the difference in parameters.

[0091] The specific implementation process of the step of using the original shared feature encoder of the original image segmentation module to perform hierarchical feature extraction processing on the target training image and the next frame image of the target training image to obtain the first training feature map corresponding to the target training image and the second training feature map corresponding to the next frame image can refer to the specific implementation process of using the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on the target image and the next frame image of the target image to obtain the first feature map corresponding to the target image and the second feature map corresponding to the next frame image, which will not be repeated in this article.

[0092] The structure of the original segmentation branch module is the same as that of the segmentation branch module, and the difference between the two is only in the parameters.

[0093] Furthermore, the steps of using the original segmentation branch module of the original image segmentation module to perform human body part segmentation and recognition processing on the first training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image include: using the first residual training group of the original segmentation branch module to perform image feature extraction processing on the first training feature map to obtain the third training feature map; using the original void space pyramid pooling module of the original segmentation branch module to perform receptive field expansion processing on the third training feature map to obtain the fifth training feature map; using the original fully connected layer of the original segmentation branch module to perform transformation processing on the fifth training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image. The above steps can refer to Figure 5 The "first training feature map", "first residual training group", "third training feature map", "image segmentation" and "first training segmentation probability map" steps in .

[0094] The two steps of "using the original hollow space pyramid pooling module of the original segmentation branch module to expand the receptive field of the third training feature map to obtain the fifth training feature map; using the original fully connected layer of the original segmentation branch module to transform the fifth training feature map to obtain the first training segmentation probability map corresponding to each human body part in the target training image" can correspond to Figure 5 The "third training feature map", "image segmentation" and "first training segmentation probability map" steps in .

[0095] The structure of the first residual training group is the same as the structure of the first residual group, and the difference between the two lies only in the difference in parameters.

[0096] The original atrous spatial pyramid pooling module and the atrous spatial pyramid pooling module have the same structure, and the difference between the two lies in the difference in parameters.

[0097] Furthermore, the step of using the original segmentation branch module to perform human body part segmentation and recognition processing on the second training feature map to obtain the second training segmentation probability map corresponding to each human body part in the next frame image includes: using the first residual training group to perform image feature extraction processing on the second training feature map to obtain a fourth training feature map; using the original void space pyramid pooling module to perform receptive field expansion processing on the fourth training feature map to obtain a sixth training feature map; using the original fully connected layer to transform the sixth training feature map to obtain the second training segmentation probability map corresponding to each human body part in the next frame image. The above steps can be referred to Figure 5 The "second training feature map", "first residual training group", "fourth training feature map", "image segmentation" and "second training segmentation probability map" steps in .

[0098] “Using the original void space pyramid pooling module to expand the receptive field of the fourth training feature map to obtain a sixth training feature map; using the original fully connected layer to transform the sixth training feature map to obtain a second training segmentation probability map corresponding to each human body part in the subsequent frame image” can correspond to Figure 5 The "fourth training feature map", "image segmentation" and "second training segmentation probability map" steps in .

[0099] Furthermore, the calculation formula for constructing the cosine similarity training matrix based on the pixels of the seventh training feature map and the eighth training feature map is: ;in, is the seventh training feature map pixel points and the eighth training feature map The cosine similarity training matrix of pixels; is the seventh training feature map Middle pixels; is the eighth training feature map Middle pixels; Represents the data processed by the original propagation branch module.

[0100] Furthermore, a first preset training algorithm is used to perform weighted calculation processing on the true label subset corresponding to the subsequent frame image and the cosine similarity training matrix to obtain a third training segmentation probability map corresponding to each human body part in the target training image. The calculation formula is: ;in, is the target training image The third training segmentation probability map corresponding to each human body part No. pixels; is the first cosine similarity in the training matrix Highly similar pixels, Set to 20; is the first cosine similarity training matrix Row data, that is, the seventh feature map The similarity between each pixel and each pixel in the eighth feature map; It is the first label source formed by merging the subset of true labels corresponding to the next frame of image In the The probability distribution of pixels.

[0101] Furthermore, a second preset training algorithm is used to perform weighted calculation processing on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the cosine similarity training matrix, and the calculation formula for the step of obtaining a third training segmentation probability map corresponding to each human body part in the target training image is: ;in, is the second training segmentation probability map corresponding to each human body part in the next frame image In the The probability distribution of pixels.

[0102] Furthermore, a third preset training algorithm is used to perform weighted calculation processing on the true label subset corresponding to the target training image and the cosine similarity training matrix, and the calculation formula for the step of obtaining the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image is: ;in, The next frame image The fourth training segmentation probability map corresponding to each human body part No. pixels; It is the second label source formed by merging the subset of true labels corresponding to the target training image In the The probability distribution of pixels.

[0103] Furthermore, a fourth preset training algorithm is used to perform weighted calculation processing on the first training segmentation probability map corresponding to each human body part in the target training image and the cosine similarity training matrix, and the calculation formula for the step of obtaining the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image is: ;in, is the first training segmentation probability map corresponding to each human body part in the target training image In the The probability distribution of pixels.

[0104] Furthermore, the calculation formula for the step of calculating the first loss function based on the first training segmentation probability map corresponding to each human body part in the target training image and the true label subset corresponding to the target training image is: ;in, is the first loss function; is the total number of categories (which can also be understood as the total number of multiple human body parts). In this application, the total number of categories is 4; is the target training image The corresponding true label subset True labels; is the first The first training segmentation probability map corresponding to each body part ; Represents the data processed by the original segmentation branch module.

[0105] Furthermore, the calculation formula for calculating the second loss function based on the second training segmentation probability map corresponding to each human body part in the subsequent frame image and the true label subset corresponding to the subsequent frame image is: ;in, is the second loss function; The next frame image The corresponding true label subset True labels; is the first Second training segmentation probability map corresponding to individual body parts .

[0106] Furthermore, the calculation formula for the step of calculating the third loss function based on the third training segmentation probability map corresponding to each human body part in the target training image and the true label subset corresponding to the target training image is: ; is the third loss function; is the target training image The corresponding true label subset True labels; is the first The first training segmentation probability map corresponding to each body part ; Represents the data processed by the original propagation branch module.

[0107] Furthermore, the calculation formula for the step of calculating the fourth loss function based on the fourth training segmentation probability map corresponding to each human body part in the subsequent frame image and the true label subset corresponding to the subsequent frame image is: ; is the fourth loss function; The next frame image The corresponding true label subset True labels; is the first Second training segmentation probability map corresponding to individual body parts .

[0108] Furthermore, a calculation formula for calculating the total loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function is: ; in, is the total loss function; yes , The role of and The scale of is consistent with the first loss function, the second loss function, the third loss function and the fourth loss function; is the fifth loss function; is the sixth loss function.

[0109] Furthermore, the calculation formula of the fifth loss function is: Alternatively, the calculation formula of the fifth loss function is .

[0110] Furthermore, the calculation formula of the sixth loss function is: Alternatively, the calculation formula of the sixth loss function is .

[0111] The image segmentation module adopts the SPN (SiamParseNet, SiamParseNet is a semi-supervised body parsing model for infant movement videos, which aims to jointly learn single-frame body parsing and inter-frame label propagation) neural network structure, so that the embodiment of the present application can eliminate the interference of background information without manually selecting areas.

[0112] The SPN neural network structure can be trained in three different situations. Specifically, the first situation is supervised learning in which both the target training image and the subsequent frame image have real labels; the second situation is semi-supervised learning in which only one of the target training image and the subsequent frame image has a real label; the third situation is unsupervised learning in which neither the target training image nor the subsequent frame image has a real label.

[0113] In actual training, most of the training data (i.e., baby videos) are not labeled. Therefore, if all the training data are used directly for training, the following problems will arise: in the early stages of training, when the original image segmentation module is unstable, the pseudo-labels generated by the original segmentation branch module are of poor quality, and if the original image segmentation module enters semi-supervised learning too early, it will mislead the training; in the later stages of training, if full supervision is always relied upon, the original image segmentation module will not be able to utilize a large amount of unlabeled data. To overcome the above problems, the embodiment of the present application adopts the AAT (Adaptive Alternative Training) training strategy to adaptively adjust the frequency of use of the above three training modes during the training process. Assume that the selection probability of the full supervision mode is , the probability of semi-supervised selection is , the unsupervised selection probability is ,So The dynamic update formula is ,in, is the number of rounds of current training; It is the lower limit of probability set in advance, set to ; is the maximum number of training rounds, set to 20; It is a parameter that controls the annealing temperature of the training process (annealing temperature refers to the annealing temperature of learning rate optimization in deep learning) and is set to 0.4. and Split the remaining probabilities equally.

[0114] In other embodiments, the image segmentation module may also be other structures with temporal modeling capabilities, such as a Transformer network based on an attention mechanism.

[0115] refer to Figure 6 The figure shows a principle block diagram of a video-based Morrow reflection recognition device provided in the second aspect of the embodiment of the present application. Figure 6 In the embodiment, the video-based Morrow reflection recognition device 100 includes: The target video acquisition module 101 is used to acquire a target video of a target object; The segmentation and recognition module 102 is configured to perform human body part segmentation and recognition processing on each frame of the target video using the image segmentation module of the Morrow reflection recognition model to obtain a target segmentation result corresponding to each human body part in each frame of the image; An optical flow calculation module 103 is configured to perform optical flow calculation processing on the target video and all target segmentation results using a motion feature module of a Morrow reflection recognition model to obtain a plurality of optical flow time series data corresponding to each human body part; The classification and recognition module 104 is configured to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflex recognition model to obtain a Moro reflex recognition result of the target object.

[0116] The third aspect of the embodiment of the present application provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 7 As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video-based Morrow reflection recognition method is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0117] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0118] In some embodiments, embodiments of the present application provide a terminal device comprising a processor and a memory, the memory being configured to store a computer program, the processor being configured to call and execute the computer program stored in the memory to perform the steps of the video-based Moro reflex recognition method provided in the first aspect of the embodiments of the present application. A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium being configured to store a computer program that causes a computer to perform the steps of the video-based Moro reflex recognition method provided in the first aspect of the embodiments of the present application.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0120] The technical features of the above embodiments can be combined without changing the basic principles of this application. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of patent protection for the present application shall be determined by the appended claims.

Claims

1. A video-based Morrow reflection recognition method, characterized in that: include: Get the target video of the target object; An image segmentation module of a Morrow reflection recognition model is used to perform human body part segmentation and recognition processing on each frame of the target video to obtain a target segmentation result corresponding to each human body part in each frame of the image; Using the motion feature module of the Morrow reflection recognition model to perform optical flow calculation processing on the target video and all target segmentation results, a plurality of optical flow time series data corresponding to each human body part are obtained; The classification and discrimination module of the Moro reflex recognition model is used to perform infant action classification and recognition processing on all optical flow time series data to obtain the Moro reflex recognition result of the target object.

2. The video-based Morrow reflection recognition method according to claim 1, characterized in that: The image segmentation module of the Morrow reflection recognition model is used to perform human body part segmentation and recognition processing on each frame of the target video, and the steps of obtaining the target segmentation result corresponding to each human body part in each frame of the image include: Using the shared feature encoder of the image segmentation module to perform hierarchical feature extraction processing on a target image and a next frame image of the target image to obtain a first feature map corresponding to the target image and a second feature map corresponding to the next frame image, wherein the target image is any frame image in the target video; Using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map, to obtain a first segmentation result corresponding to each human body part in the next frame of image; The propagation branch module of the image segmentation module is used to perform human body part segmentation and recognition processing on the first feature map, the second feature map and the first segmentation result corresponding to each human body part in the next frame image to obtain a target segmentation result corresponding to each human body part in the target image.

3. The video-based Morrow reflection recognition method according to claim 2, characterized in that: The step of using the segmentation branch module of the image segmentation module to perform human body part segmentation and recognition processing on the second feature map to obtain a first segmentation result corresponding to each human body part in the next frame image includes: Performing image feature extraction processing on the second feature map using the first residual group of the segmentation branch module to obtain a third feature map; Using the dilated spatial pyramid pooling module of the segmentation branch module to perform receptive field expansion processing on the third feature map to obtain a fourth feature map; The fourth feature map is transformed using the fully connected layer of the segmentation branch module to obtain a first segmentation result corresponding to each human body part in the next frame image.

4. The video-based Morrow reflection recognition method according to claim 2, characterized in that: The steps of performing human body part segmentation and recognition processing on the first feature map, the second feature map, and the first segmentation result corresponding to each human body part in the next frame image using the propagation branch module of the image segmentation module to obtain a target segmentation result corresponding to each human body part in the target image include: Performing image feature extraction processing on both the first feature map and the second feature map using the second residual group of the propagation branch module to obtain a fifth feature map corresponding to the first feature map and a sixth feature map corresponding to the second feature map; constructing a cosine similarity matrix based on pixels of the fifth feature map and the sixth feature map; A first preset algorithm is used to perform weighted calculation processing on the true probability map corresponding to the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image; or a second preset algorithm is used to perform weighted calculation processing on the first segmentation result corresponding to each human body part in the next frame image and the cosine similarity matrix to obtain a target segmentation result corresponding to each human body part in the target image, wherein the true probability map is a reference map generated by a semantic segmentation algorithm, representing the probability that each pixel in the next frame image belongs to a preset category.

5. The video-based Morrow reflection recognition method according to claim 2, characterized in that: The steps of performing optical flow calculation processing on the target video and all target segmentation results using the motion feature module of the Morrow reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part include: Performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation result corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image; The multiple target motion features corresponding to each human body part in all images in the target video are connected in time sequence to obtain multiple optical flow time series data corresponding to each human body part.

6. The video-based Morrow reflection recognition method according to claim 5, characterized in that: The step of performing human motion feature extraction processing on the target image, the next frame image, and the target segmentation result corresponding to each human body part in the target image to obtain multiple target motion features corresponding to each human body part in the target image includes: Performing mask format conversion processing on the target segmentation result corresponding to each human body part in the target image to obtain a target mask corresponding to each human body part in the target image; Performing region of interest extraction processing on the mask corresponding to each human body part to obtain a target region of interest corresponding to each human body part in the target image; Taking each pixel point in the target region of interest corresponding to each human body part in the target image as a target pixel point, calculating the target feature of each target pixel point between the target image and the next frame image using the optical flow method, and combining all target features to form an optical flow cluster corresponding to each human body part in the target image; The optical flow cluster corresponding to each human body part in the target image is calculated and processed according to preset rules to obtain multiple target motion features corresponding to each human body part in the target image.

7. The video-based Morrow reflection recognition method according to claim 1, characterized in that: The steps of performing infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflex recognition model to obtain the Moro reflex recognition result of the target object include: performing infant action classification and recognition processing on all optical flow time series data using a long short-term memory network to obtain the Moro reflex recognition result of the target object; or The steps of performing infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflex recognition model to obtain the Moro reflex recognition result of the target object include: The average value and standard deviation of each optical flow time series data are calculated and processed to obtain the first motion intensity feature and the first motion locality feature corresponding to each optical flow time series data; Calculate the average frequency, centroid frequency, root mean square frequency or frequency variance of each optical flow time series data to obtain the target frequency feature corresponding to each optical flow time series data; A variance analysis method is used to screen all first motion intensity features, all first motion locality features, and all target frequency features to obtain multiple features to be identified; The multiple features to be identified are input into a support vector machine for classification and identification of infant movements to obtain a Moro reflex identification result of the target object.

8. A video-based Morrow reflection recognition device, characterized in that: include: A target video acquisition module is used to acquire a target video of a target object; A segmentation and recognition module is used to perform human body part segmentation and recognition processing on each frame of the target video using the image segmentation module of the Morrow reflection recognition model to obtain a target segmentation result corresponding to each human body part in each frame of the image; An optical flow calculation module is used to perform optical flow calculation processing on the target video and all target segmentation results using a motion feature module of a Morrow reflection recognition model to obtain multiple optical flow time series data corresponding to each human body part; The classification and recognition module is used to perform infant action classification and recognition processing on all optical flow time series data using the classification and discrimination module of the Moro reflex recognition model to obtain the Moro reflex recognition result of the target object.

9. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the video-based Morrow reflection recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program enables a computer to execute the steps of the video-based Morrow reflection recognition method according to any one of claims 1 to 7.

Images