Classification system, classification method and equipment suitable for cerebral palsy scene and storage medium

Abstract

The invention discloses a classification system, a classification method and equipment suitable for a cerebral palsy scene and a storage medium. A classification system suitable for a cerebral palsy scene comprises a data processing module and a classification module, the data processing module is used for obtaining video data of a target object, and the video data comprises a plurality of video frames; extracting two-dimensional key point information of the target object in each video frame, and converting the two-dimensional key point information into three-dimensional key point information; and the classification module is used for carrying out classification processing on the three-dimensional key point information corresponding to each video frame based on a pre-trained classification model to obtain a classification result of the target object in the cerebral palsy scene. The processing efficiency of the video data is improved, and the classification accuracy of the video data in the cerebral palsy scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a classification system, classification method, device and storage medium suitable for cerebral palsy scenarios. Background Art

[0002] Research in the field of cerebral palsy is an important branch of medical research. The high-quality evidence in the "Guidelines for Rehabilitation of Cerebral Palsy in China (2022)" shows that abnormal general movements trust (GMs) assessment or Hammersmith infant neurological examination (HINE) score trajectory, combined with abnormal MRI examination, is more accurate than clinical assessment alone, and can be combined with assessments with strong predictive validity and clinical inference to make early diagnosis before 6 months of corrected age.

[0003] Although GMs or HINE assessment technology has extremely high application value in the early identification of cerebral palsy and the prediction of severe developmental delay, it has obvious limitations in large-scale promotion, especially at the grassroots level, because the assessment requires high technical skills from medical personnel and is time-consuming. Summary of the invention

[0004] The present invention provides a classification system, classification method, device and storage medium applicable to cerebral palsy scenarios to solve the problems of high technical requirements and long time consumption in the cerebral palsy assessment process.

[0005] According to one aspect of the present invention, a classification system applicable to cerebral palsy scenarios is provided, the classification system comprising a data processing module and a classification module, wherein:

[0006] The data processing module is used to obtain video data of the target object, wherein the video data includes a plurality of video frames; extract the two-dimensional key point information of the target object in each video frame, and convert the two-dimensional key point information into three-dimensional key point information;

[0007] The classification module is used to classify the 3D key point information corresponding to each video frame based on the pre-trained classification model to obtain the classification results of the target object in the cerebral palsy scene. This improves the accuracy of the classification of the target object in the cerebral palsy scene.

[0008] According to another aspect of the present invention, a classification method applicable to cerebral palsy scenarios is provided, the classification method comprising:

[0009] Acquire video data of a target object, the video data including a plurality of video frames; extract two-dimensional key point information of the target object in each video frame, and convert the two-dimensional key point information into three-dimensional key point information;

[0010] Based on the pre-trained classification model, the three-dimensional key point information corresponding to each video frame is classified and processed to obtain the classification result of the target object in the cerebral palsy scene.

[0011] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to at least one processor; wherein,

[0014] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the classification method applicable to cerebral palsy scenarios according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the classification method applicable to cerebral palsy scenarios according to any embodiment of the present invention when executed.

[0016] The technical solution of the embodiment of the present invention obtains video data of the target object, extracts two-dimensional key point information of the target object in each video frame, and converts the two-dimensional key point information into three-dimensional key point information, and classifies the three-dimensional key point information corresponding to each video frame based on a pre-trained classification model to obtain the classification result of the target object in the cerebral palsy scene, which solves the problem that the cerebral palsy assessment process has high technical requirements and long time consumption, shortens the time required for the target object in the cerebral palsy detection process, and improves the classification efficiency of the target object in the cerebral palsy scene.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a structural diagram of a classification system applicable to cerebral palsy scenarios provided by the first embodiment of the present invention;

[0020] Figure 2is a structural diagram of a classification model provided by an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a classification method applicable to cerebral palsy scenarios provided in Embodiment 2 of the present invention;

[0022] Figure 4 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1 1 is a schematic diagram of a classification system applicable to cerebral palsy scenarios provided in Embodiment 1 of the present invention. This embodiment can be applied to the case where video data of a target object is collected and the target object is classified in a cerebral palsy scenario through the video data. The classification system applicable to cerebral palsy scenarios can be integrated in an electronic device, and the electronic device can include an electronic device such as a mobile terminal, a server or a computer, and the mobile terminal can include a mobile phone and a tablet computer, etc.

[0027] like Figure 1As shown, the classification system includes a data processing module 110 and a classification module 120, wherein the data processing module 110 is used to obtain video data of the target object, the video data including multiple video frames; extract the two-dimensional key point information of the target object in each video frame, and convert the two-dimensional key point information into three-dimensional key point information; the classification module 120 is used to classify the three-dimensional key point information corresponding to each video frame based on a pre-trained classification model, and obtain the classification result of the target object in the cerebral palsy scene.

[0028] In this embodiment, the target object may be an object that needs to be classified in the cerebral palsy scenario, such as infants, young children, and high-risk infants. Among them, high-risk infants refer to special groups of people who have various risk factors that are unfavorable to the growth and development of the fetus and infant during the mother's pregnancy and delivery period, the neonatal period, and the infant period. High-risk infants often suffer brain damage due to various reasons, resulting in impaired development of one or more energy areas such as movement, and even long-term sequelae such as mental retardation and cerebral palsy.

[0029] In the process of classifying the target object in the cerebral palsy scene, the classification can be performed by analyzing the video data of the target object. The video data of the target object can be composed of multiple video frames, wherein the video frame can be a static image of the video constituting the target object. The video data can be information of the target object in a preset time period recorded in the form of a video, which can be acquired by a video acquisition device. The preset time period can be a pre-set time period for video data acquisition, for example, the preset time period can be 10 minutes. The video acquisition device can be a variety of terminal smart devices, etc., wherein the terminal smart devices include mobile phones, tablet computers and PCs, etc. It can be understood that the video acquisition device and the electronic device in this embodiment can be the same device. Taking the electronic device as a mobile phone as an example, the video data of the target object can be acquired through the camera function of the mobile phone, and the acquired video data is processed by the data processing module 110 configured in the mobile phone. The video acquisition device and the electronic device in this embodiment can be different devices, for example, the video acquisition device is a camera or a mobile terminal, and the electronic device is a computer; for example, the video acquisition device can be a user-end device, and the electronic device can be a doctor-end device, wherein the user-end device acquires video data and transmits it to the doctor-end device, and the doctor-end device receives the video data transmitted by the video acquisition device, wherein the user-end device can be a camera or a mobile terminal, etc., and the doctor-end device can be a mobile terminal, a server or a computer, etc.

[0030] The video data of the target object can also be retrieved from a database, which stores video data of multiple objects, and the target object identification can be used to match the video data of the target object in the database. The target object identification can be the identity information of the target object or other unique identification.

[0031] The key points may be the main joints or main parts of the target object required in the classification process of the cerebral palsy scene. Optionally, the key points include one or more of the nose, left and right eyes, left and right ears, left and right shoulder joints, left and right joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints. The two-dimensional key point information may be the two-dimensional coordinates of the key point, and the three-dimensional key point information may be the three-dimensional coordinates of the key point. Optionally, the two-dimensional key point information may be the two-dimensional coordinates of the key point in the same coordinate system. The classification result can be used to determine whether the target object has cerebral palsy, for example, the classification result may be that the target object is not cerebral palsy.

[0032] The two-dimensional key point information can be obtained through a machine learning model, such as a target detection model. Optionally, the machine learning model for obtaining the two-dimensional key point information can be configured in the data processing module.

[0033] Optionally, the data processing module 110 is further configured to perform key point recognition on a plurality of video frames respectively through a target detection model to obtain two-dimensional key point information of a target object in each video frame.

[0034] Specifically, the target detection model can be used to detect the position of the target object, for example, it can be a convolutional neural network model or a Transformer model, which can be selected according to the needs of the user and is not limited here. Optionally, the target detection model can be a Detectron2 model. The Detectron2 model is a deep learning framework that can be used for target detection. In the classification of cerebral palsy scenes, it can identify key points of multiple video frames and determine the two-dimensional key point information of the target object in each video frame, providing detailed data support for subsequent analysis.

[0035] In the process of obtaining the two-dimensional key point information, there may be abnormal two-dimensional key point information. For example, in the process of obtaining the two-dimensional key point information, the two-dimensional key point information of some key points cannot be obtained, resulting in missing values ​​in the corresponding three-dimensional key point information converted from the two-dimensional key point information of some key points, or an abnormality occurs in the process of providing the two-dimensional key point information, resulting in the extracted two-dimensional key point information exceeding the reasonable data range. In order to reduce the impact of the abnormal situation, the video frames corresponding to the two-dimensional key point information with abnormalities can be deleted.

[0036] Optionally, the data processing module 110 is also used to perform anomaly judgment on the two-dimensional key point information corresponding to each video frame, wherein the anomaly judgment includes one or more of missing value judgment and data range anomaly judgment; if there is an anomaly in the two-dimensional key point information corresponding to any video frame, the video frame is deleted.

[0037] Specifically, missing value determination can be used to determine whether two-dimensional key point information is missing, and data range anomaly determination can be used to determine whether two-dimensional key point information exceeds a preset range. For example, the preset range can be [-2000, 2000]. It can be understood that the preset range can be determined based on the resolution of the video data.

[0038] For any video frame, if the two-dimensional key point information of any key point in the video frame is missing or is not within the data range, the video frame is determined to be an abnormal video frame and deleted to avoid interference of the abnormal video frame on the classification process.

[0039] During the video capture process, the target object is in a free-moving state. For example, the two-dimensional key point information of a key point in the video frame cannot be read due to occlusion, resulting in the loss of the two-dimensional key point information of the key point. For example, during the rotation of the target object's head, the side face of the target object is captured in the video frame, and only the two-dimensional key point information of the left eye can be captured, while the two-dimensional key point information of the right eye cannot be captured, resulting in the loss of the two-dimensional key point information.

[0040] For example, the preset range of the two-dimensional key point information may be [-2000, 2000], and the two-dimensional key point information of a key point is (x, y). When there is at least one data less than -2000 or greater than 2000 in x or y, it can be determined that the data range of the two-dimensional key point information of the key point is abnormal.

[0041] The resolution of video data collected by different devices may be different, which may cause errors in coordinates to a certain extent. Therefore, before converting the two-dimensional key point information into three-dimensional key point information, the two-dimensional key point information can be standardized.

[0042] Optionally, the data processing module 110 is further configured to standardize the two-dimensional key point information based on the resolution information of the video frame before converting the two-dimensional key point information into three-dimensional key point information, so as to obtain the two-dimensional key point information in the standard resolution video frame.

[0043] Specifically, the resolution information can be used to indicate the clarity of the video data, which may include width and height. Different video data may have different resolution information, such as 1980*1080 and 1280*720. Standardization can be used to limit the two-dimensional key point information to a certain range, which is helpful for subsequent processing. Taking 17 key points such as nose, left and right eyes, left and right ears, left and right shoulder joints, left and right joints, left and right wrist joints, left and right hip joints, left and right knee joints and left and right ankle joints as examples, the width of the video frame resolution information is used to determine the key point information. and high The two-dimensional key point information is standardized and converted to the range of [-1,1]. The 17 key points in each video frame are represented as , the two-dimensional key point information of each key point is expressed as , the standardized formula is:

[0044] .

[0045] The two-dimensional key point information is standardized by resolution information. and When the two-dimensional key point information is divided by the same width w, it can be ensured that the obtained two-dimensional key point information has a consistent scale, rather than having different scaling in different directions, thereby ensuring the consistency of the input data. By subtracting [1, h / w], the y coordinate can be adjusted to maintain the correct aspect ratio. This standardized processing method not only eliminates the influence of resolution information on the two-dimensional key point information, but also ensures the generalization ability of the data processing module 110 for video data of different target objects.

[0046] Optionally, the two-dimensional key point information of each key point can also be standardized by setting standard resolution information, calculating the change ratio between the width and height of the current resolution information and the width and height of the standard resolution information. The standard resolution information can be resolution information set in advance, by calculating the first change ratio between the width of the current resolution information and the width of the standard resolution information, calculating the second change ratio between the height of the current resolution information and the height of the standard resolution information, converting the width of the current resolution information according to the first change ratio, and converting the height of the current resolution information based on the second change ratio, so as to achieve standardized processing of the two-dimensional key point information. For example, the width and height of the current resolution information are 1980 and 1080 respectively, the width and height of the standard resolution information are 1280 and 720 respectively, the change ratio of the width of the current resolution information is -35%, and the change ratio of the height of the current resolution information is -33%, where "-" indicates decrease and "+" indicates increase, and the two-dimensional key point information of each key point is and A decrease of 35% and 33% respectively.

[0047] Optionally, the data processing module 110 is further used to perform dimension conversion processing on the two-dimensional key point information based on a data conversion model to obtain three-dimensional key point information corresponding to the two-dimensional key point information, wherein the data conversion model includes multiple one-dimensional convolution blocks.

[0048] Specifically, the data conversion model can be used as a model for data conversion processing, which can be a model for converting the dimension of data. For example, a machine learning model can be used to convert the dimension of data to convert two-dimensional data into three-dimensional data. Optionally, the data conversion model can be a Videopose3D model. The Videopose3D model includes multiple one-dimensional convolution blocks, and generates three-dimensional coordinates through multiple layers of one-dimensional convolution: [N,T,C_2] -> [N,T,C_3], where N represents the batch size, T represents the time series length, C_2 represents two-dimensional key point information, and C_3 represents three-dimensional key point information. Among them, the one-dimensional convolution block can be a one-dimensional convolution block with a length of 3 and 1 respectively and a stride of 1.

[0049] Since the target object can be a human body, by converting the two-dimensional key point information of the target object into three-dimensional key point information, it is convenient to obtain the spatial characteristics of the key point data during the classification process, provide a more comprehensive feature basis for the classification process, and further improve the accuracy of classification.

[0050] Optionally, the classification model includes two feature extraction units and a classification unit, wherein any feature extraction unit includes a long short-term memory processing network block and a dropout layer, and the classification unit includes a fully connected layer and a pooling layer.

[0051] Specifically, the feature extraction unit can be used to extract features of input data. Taking 17 key points such as nose, left and right eyes, left and right ears, left and right shoulder joints, left and right joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints as an example, the input data can be the three-dimensional key point information of the 17 key points extracted from each video frame, and the three-dimensional key point information extracted from each video frame is stacked into an array, that is, [N, 17, 3], where N represents the number of video frames contained in the video, 17 is the number of key points, and 3 is the dimension of each key point.

[0052] The feature extraction unit includes a long short-term memory processing network block (Long Short-Term Memory, LSTM) and a dropout layer. Optionally, the long short-term memory processing network block includes a two-layer long short-term memory network. The two-layer long short-term memory network can extract features of different depths of input data, thereby improving the feature extraction capability of the long short-term memory processing network block. The dropout layer is a commonly used regularization technique that can be used to randomly discard a certain proportion of neurons after the first feature extraction unit extracts the features, so that the classification model does not rely on specific information, retains more features, and prevents overfitting of the long short-term memory processing network block. The feature extraction unit adopts the structure of the long short-term memory processing network block and the dropout layer, which can retain more features and prevent overfitting of the long short-term memory network.

[0053] The classification unit can be used to classify the features extracted by the feature extraction unit. The classification unit includes a fully connected layer and a pooling layer, wherein the fully connected layer can map the features extracted by the feature extraction unit into a binary classification vector, the binary classification vector can be used to determine whether the target object has cerebral palsy, and the pooling layer can output the result of the binary classification vector. Optionally, the pooling layer can be a maximum pooling layer.

[0054] For example, Figure 2 1 is a schematic diagram of a classification model provided by an embodiment of the present invention. The classification model includes two feature extraction units and a classification unit, each of which includes two LSTM blocks connected in series and a dropout layer. The classification unit includes a fully connected layer and a pooling layer.

[0055] In this embodiment, the multi-instance learning method in a three-dimensional coordinate system is applied to the field of posture recognition. The classification model uses multiple LSTM layers in series to transfer the output of the previous layer to the reinitialized LSTM module to ensure that each layer can independently learn different time series features. In terms of time series feature processing, compared with the simple superposition of multiple layers in traditional LSTM, the LSTM structure in series in the classification model is in a separated state, which helps to capture dynamic changes at different levels and improve the processing ability of long time series data. While increasing the depth of the network, the time series information is gradually extracted and refined, and the patterns in the data are gradually captured from coarse granularity to fine granularity. Make the network more suitable for complex time series tasks and capture more time series features.

[0056] Optionally, the classification system also includes a model training module, which is used to obtain clipped video data, where the clipped video data includes marked video frames in the sample video data, and the marked video frames are local video frames in the sample video data; pre-training a classification model to be trained based on the clipped video data and classification labels corresponding to the clipped video data to obtain a pre-trained classification model; and training the pre-trained classification model based on the sample video data and the classification labels corresponding to the sample video data to obtain a trained classification model.

[0057] Specifically, the model training module can be used to train the classification model to be trained. The sample video data can be collected by a video device, for example, the video data of the sample object in a preset time period can be collected by a camera; it can also be obtained by retrieving video data stored in a database, which stores video data of multiple sample objects.

[0058] The video data includes valid video frames that affect cerebral palsy decision-making, and also includes invalid video frames that have no effect on cerebral palsy decision-making. Invalid video frames interfere with the processing process of the classification model to a certain extent. In order to speed up the training efficiency of the classification model, the valid video frames in the video data are marked, and clipped video data are formed by marking the video frames. For example, a doctor or expert can mark the valid video frames in the video data. The marked video frames are local video frames in the sample video data, which are also marked video frames. The clipped video data can be data obtained by clipping the video data. For example, the marked video frames in the video data are identified, other video frames other than the marked video frames are deleted, the marked video frames are retained, and the retained marked video frames are clipped according to the timestamp to obtain clipped video data. The clipped video data can be used as new sample data for pre-training the classification model to be trained.

[0059] Optionally, the number of video frames in the clipped video data may be greater than N. By limiting the minimum number of video frames in the clipped video data, the video frames in the clipped video data can provide features for decision-making, thereby avoiding the situation where the classification result cannot be inferred due to insufficient features.

[0060] Optionally, the number of video frames in the clipped video data can be less than M, where N<M. By limiting the maximum number of video frames in the clipped video data, the data processing amount of the classification model can be reduced and the training efficiency of the classification model can be accelerated while ensuring that features are provided for the classification model to make decisions.

[0061] Optionally, a sample data set is obtained, which may include multiple sample video data. Local sample video data in the multiple sample video data may be marked to obtain clipped video data corresponding to the local sample video data. A classification model to be trained is pre-trained with a small amount of clipped video data to achieve rapid learning of the classification model in the clipped video data, thereby obtaining a pre-trained classification model. Furthermore, the pre-trained classification model is further trained with sample video data in the sample data set to achieve learning of the sample video data in the classification model, thereby obtaining a trained classification model.

[0062] The classification label can be used to characterize the actual classification results of the sample video data in the cerebral palsy scene. By pre-training the classification model to be trained with the clipped video data and the classification labels corresponding to the clipped video data, the characteristics of the video data that affect the cerebral palsy decision can be learned more quickly. Optionally, the classification model to be trained can be pre-trained by K-fold training to obtain multiple pre-trained classification models, and the above multiple pre-trained classification models are evaluated according to one or more of the indicators such as AUC, accuracy, specificity, sensitivity and precision to determine the final pre-trained classification model.

[0063] The pre-trained classification model is trained by using sample video data and the classification labels corresponding to the sample video data to obtain a trained classification model, which improves the generalization ability of the classification model, can more accurately predict the features of the key points of the target object, and improves the accuracy of the classification model prediction.

[0064] Optionally, the model training module is also used to obtain sample video data, train the classification model to be trained based on the sample video data and the classification labels corresponding to the sample video data, and adjust the model parameters of the classification model to be trained according to one or more indicators such as AUC, accuracy, specificity, sensitivity and precision to obtain a trained classification model.

[0065] The technical solution of this embodiment obtains video data of the target object through a data processing module, performs key point recognition on multiple video frames through a target detection model, obtains two-dimensional key point information of the target object in each video frame, performs abnormality judgment on the two-dimensional key point information corresponding to each video frame, deletes the video frame when there is an abnormality in the two-dimensional key point information corresponding to any video frame, standardizes the two-dimensional key point information based on the resolution information of the video frame to obtain the two-dimensional key point information located in the standard resolution video frame, performs dimension conversion processing on the two-dimensional key point information located in the standard resolution video frame based on the data conversion model to obtain the three-dimensional key point information corresponding to the two-dimensional key point information, pre-trains the classification model to be trained by clipping video data and the classification labels corresponding to the clipped video data to obtain a pre-trained classification model, and trains the pre-trained classification model by sample video data and the classification labels corresponding to the sample video data to obtain a trained classification model, classifies the three-dimensional key point information corresponding to each video frame by the trained classification model, obtains the classification result of the target object in the cerebral palsy scene, and improves the classification efficiency of the target object in the cerebral palsy scene.

[0066] Embodiment 2

[0067] Figure 3 is a flow chart of a classification method applicable to cerebral palsy scenes provided in the second embodiment of the present invention. This embodiment can be applied to the classification of target objects in cerebral palsy scenes. Figure 3 As shown, the classification method includes:

[0068] S210, obtaining video data of a target object, wherein the video data includes a plurality of video frames; extracting two-dimensional key point information of the target object in each video frame, and converting the two-dimensional key point information into three-dimensional key point information.

[0069] Optionally, the key points include one or more of a nose, left and right eyes, left and right ears, left and right shoulder joints, left and right joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints.

[0070] The two-dimensional key point information can be obtained through a related model, such as a target detection model.

[0071] Optionally, extracting the two-dimensional key point information of the target object in each video frame includes: performing key point recognition on multiple video frames respectively through a target detection model to obtain the two-dimensional key point information of the target object in each video frame.

[0072] Optionally, an abnormality determination is performed on the two-dimensional key point information corresponding to each video frame, wherein the abnormality determination includes one or more of missing value determination and data range abnormality determination; if an abnormality exists in the two-dimensional key point information corresponding to any video frame, the video frame is deleted.

[0073] Since the resolution of video data collected by different devices may be different, it may cause errors in coordinates to a certain extent. Therefore, before converting the two-dimensional key point information into three-dimensional key point information, the two-dimensional key point information can be standardized.

[0074] Optionally, before converting the two-dimensional key point information into three-dimensional key point information, the two-dimensional key point information is standardized based on the resolution information of the video frame to obtain the two-dimensional key point information in the standard resolution video frame.

[0075] Optionally, dimension conversion is performed on the two-dimensional key point information based on a data conversion model to obtain three-dimensional key point information corresponding to the two-dimensional key point information, wherein the data conversion model includes multiple one-dimensional convolution blocks.

[0076] In the process of obtaining two-dimensional key point information and converting the two-dimensional key point information into three-dimensional key point information, there may be abnormal three-dimensional key point information. For example, in the process of obtaining two-dimensional key point information, the two-dimensional key point information of some key points cannot be obtained, resulting in missing values ​​in the corresponding three-dimensional key point information converted from the two-dimensional key point information of some key points, or the three-dimensional key point information exceeds a certain range. In order to reduce the impact of abnormal situations, the video frames corresponding to the three-dimensional key point information with abnormalities can be deleted.

[0077] S220 , based on the pre-trained classification model, classify the three-dimensional key point information corresponding to each video frame to obtain a classification result of the target object in the cerebral palsy scene.

[0078] Optionally, the classification model includes two feature extraction units and a classification unit, wherein any feature extraction unit includes a long short-term memory processing network block and a dropout layer, and the classification unit includes a fully connected layer and a pooling layer.

[0079] Optionally, the classification method also includes a model training method, which is used to obtain clipped video data, where the clipped video data includes marked video frames in the sample video data, and the marked video frames are local video frames in the sample video data; pre-training a classification model to be trained based on the clipped video data and classification labels corresponding to the clipped video data to obtain a pre-trained classification model; and training the pre-trained classification model based on the sample video data and the classification labels corresponding to the sample video data to obtain a trained classification model.

[0080] The technical solution of this embodiment obtains video data of the target object, identifies key points of multiple video frames respectively through the target detection model, obtains two-dimensional key point information of the target object in each video frame, standardizes the two-dimensional key point information based on the resolution information of the video frame, performs dimension conversion processing on the standardized two-dimensional key point information based on the data conversion model to obtain the corresponding three-dimensional key point information, performs abnormality judgment on the three-dimensional key point information corresponding to each video frame, deletes the video frame when there is an abnormality in the three-dimensional key point information corresponding to any video frame, and classifies the three-dimensional key point information corresponding to each video frame based on the pre-trained classification model to obtain the classification result of the target object in the cerebral palsy scene, thereby shortening the time required for the target object in the cerebral palsy detection process and improving the classification efficiency of the target object in the cerebral palsy scene.

[0081] Embodiment 3

[0082] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0083] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the random access memory (RAM) 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory (ROM) 12, and the random access memory (RAM) 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0084] A number of components in the electronic device 10 are connected to an input / output (I / O) interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0085] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a classification method applicable to cerebral palsy scenarios.

[0086] In some embodiments, the classification method applicable to cerebral palsy scenarios may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via a read-only memory (ROM) 12 and / or a communication unit 19. When the computer program is loaded into a random access memory (RAM) 13 and executed by the processor 11, one or more steps of the classification method applicable to cerebral palsy scenarios described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the classification method applicable to cerebral palsy scenarios in any other appropriate manner (e.g., by means of firmware).

[0087] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] The computer programs for implementing the classification method applicable to cerebral palsy scenarios of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0089] Embodiment 4

[0090] Embodiment 4 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a classification method applicable to cerebral palsy scenarios, the method comprising:

[0091] The video data of the target object is obtained, wherein the video data includes a plurality of video frames; the two-dimensional key point information of the target object in each video frame is extracted, and the two-dimensional key point information is converted into three-dimensional key point information; based on a pre-trained classification model, the three-dimensional key point information corresponding to each video frame is classified and processed to obtain the classification result of the target object in the cerebral palsy scene.

[0092] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0094] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0095] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0097] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A classification system suitable for cerebral palsy scenarios, characterized in that: include: Data processing module and classification module, wherein, The data processing module is used to obtain video data of the target object, wherein the video data includes a plurality of video frames; extract the two-dimensional key point information of the target object in each of the video frames, and convert the two-dimensional key point information into three-dimensional key point information; The classification module is used to classify the three-dimensional key point information corresponding to each video frame based on a pre-trained classification model to obtain a classification result of the target object in the cerebral palsy scene.

2. The system according to claim 1, characterized in that The key points include one or more of nose, left and right eyes, left and right ears, left and right shoulder joints, left and right joints, left and right wrist joints, left and right hip joints, left and right knee joints and left and right ankle joints; The data processing module is also used to perform key point recognition on the multiple video frames respectively through a target detection model to obtain two-dimensional key point information of the target object in each of the video frames.

3. The system according to claim 1, characterized in that The data processing module is also used for: The two-dimensional key point information is dimensionally converted based on a data conversion model to obtain three-dimensional key point information corresponding to the two-dimensional key point information, wherein the data conversion model includes multiple one-dimensional convolution blocks.

4. The system according to claim 1 or 3, characterized in that: The data processing module is also used to, before converting the two-dimensional key point information into three-dimensional key point information, perform standardization processing on the two-dimensional key point information based on resolution information of the video frame to obtain the two-dimensional key point information in the standard resolution video frame.

5. The system according to claim 1, characterized in that The data processing module is also used for: Performing anomaly determination on the two-dimensional key point information corresponding to each of the video frames, wherein the anomaly determination includes one or more of missing value determination and data range anomaly determination; When there is an abnormality in the two-dimensional key point information corresponding to any video frame, the video frame is deleted.

6. The system according to claim 1, characterized in that The classification model includes two feature extraction units and a classification unit, wherein any of the feature extraction units includes a long short-term memory processing network block and a dropout layer, and the classification unit includes a fully connected layer and a pooling layer.

7. The system according to claim 1 or 6, characterized in that: The system further includes a model training module for acquiring clipped video data, wherein the clipped video data includes a marked video frame in the sample video data, and the marked video frame is a local video frame in the sample video data; Pre-training a classification model to be trained based on the clipped video data and classification labels corresponding to the clipped video data to obtain a pre-trained classification model; The pre-trained classification model is trained based on the sample video data and the classification labels corresponding to the sample video data to obtain a trained classification model.

8. A classification method suitable for cerebral palsy scenarios, characterized in that: include: Acquire video data of a target object, wherein the video data includes a plurality of video frames; extract two-dimensional key point information of the target object in each of the video frames, and convert the two-dimensional key point information into three-dimensional key point information; Based on the pre-trained classification model, the three-dimensional key point information corresponding to each of the video frames is classified and processed to obtain the classification result of the target object in the cerebral palsy scene.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the classification method applicable to cerebral palsy scenarios as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the classification method applicable to cerebral palsy scenarios described in claim 8 when executed.

Images