Scoliosis auxiliary screening method based on symmetric contrast learning and contour perception

Abstract

The invention discloses a scoliosis auxiliary screening method based on symmetric comparative learning and contour perception. The scoliosis auxiliary screening method mainly solves the problems that in the prior art, the scale and quality of a data set are too affected, and the accuracy rate is low. According to the implementation scheme, the method comprises the steps of collecting a forward flexion test video and constructing a scoliosis data set; performing edge mask labeling and category label labeling on the data set; constructing a contour edge prediction network, and training the network in combination with comparative learning and edge mask information to obtain a trained model weight omega; constructing a scoliosis screening network based on edge perception and symmetry contrast learning, and loading a trained model weight omega to the network; calculating a loss function of the network by using a category label, and training the loss function to obtain a trained screening network; and inputting the test set into the trained screening network to detect a spine identification result. According to the method, the scoliosis identification cost is reduced, the identification accuracy is improved, and the method can be used for remote large-scale scoliosis screening in hospitals.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and image processing technology, and in particular relates to a scoliosis-assisted screening method, which can be used for remote large-scale scoliosis screening in hospitals. Background Art

[0002] Scoliosis is a three-dimensional deformity involving abnormal curvature and rotation of the spine in multiple dimensions. Adolescent idiopathic scoliosis (AIS) is the most common type, usually affecting adolescents aged 10 to 18 years old. Statistics show that the global incidence of AIS is about 1% to 3%, and it is increasing year by year. For patients with scoliosis, the severity of the impact varies from person to person. Mild cases may only show low back pain or mild neurological symptoms, while severe cases may have restricted cardiopulmonary development due to chest deformity, leading to cardiopulmonary insufficiency and even life-threatening conditions. In addition, the appearance deformity caused by scoliosis often has an adverse effect on the patient's mental health. Many patients develop inferiority complex due to body changes, which leads to social disorders.

[0003] Since scoliosis has a high incidence and potential serious harm, early detection, early diagnosis and early treatment are crucial. Timely intervention can not only reduce the impact of the disease on physical health, but also help patients regain their self-confidence and avoid deepening psychological trauma. In view of this, promoting the normalization and systematization of scoliosis screening is of great significance to reducing the incidence of scoliosis and its adverse effects.

[0004] The patent document with publication number CN114287915A discloses a non-invasive scoliosis screening method and system based on back color images. Its implementation scheme is: first, the RGB and depth images of the back human body are collected by RBG-D camera, and they are saved as color images and depth maps respectively; then, the human body segmentation model is obtained by training the Mask-RCNN network, and the human body color image without background is segmented by the model; then, the back recognition model is obtained by training the YOLOv5 network, and the back area is intercepted to obtain the back color image; further, the back depth map is obtained, and the maximum ATR angle value is calculated; finally, the classification standard is formulated and the category is marked, and the back color image and category label are input into the EfficientNet network for training to obtain the spine classification model. However, since this method uses RBG-D cameras, it is expensive and costly. At the same time, since the classification standard and the category marking need to be based on the calculated maximum ATR angle value, the implementation process is complicated and inefficient, and the recognition accuracy is not high.

[0005] The patent document with publication number CN116258897A discloses a scoliosis screening method based on back images. The implementation steps of the method include: first, building a network model and preparing two scoliosis data sets; then, inputting the first data set into the network model for training, and optimizing the model by selecting the minimized loss function and the optimal evaluation index; then, using the second data set to fine-tune the model to obtain stable model parameters; finally, solidifying the model parameters. During screening, only the image needs to be input into the model to obtain the screening results. Although this method can achieve large-scale scoliosis screening with high accuracy and efficiency. However, since it depends on the size and quality of the data set, if the sample size in the data set is insufficient or the sample annotation is inaccurate, it will affect the accuracy and generalization ability of the model.

[0006] In addition, the above existing methods only rely on static RGB images of the back of a standing human body, without analyzing the dynamic forward bending video of the subject, and without obtaining edge features related to scoliosis judgment. Therefore, the information for judging human scoliosis is incomplete, which may lead to missed detection. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a scoliosis auxiliary screening method based on symmetric contrast learning and contour perception, so as to reduce the screening cost by using a low-cost ordinary RGB camera to collect the subject's flexion movement video, making it easier to popularize and promote; by analyzing the subject's spinal dynamic information in the ordinary RGB video of the flexion movement, and capturing the contour edge features of the subject's shoulders, waistline, pelvis and back, the accuracy of scoliosis identification is improved and the possibility of missed detection is reduced.

[0008] To achieve the above object, the implementation scheme of the present invention includes the following steps:

[0009] 1. A scoliosis auxiliary screening method based on symmetric contrast learning and contour perception, characterized by comprising the following:

[0010] (1) Collect videos and divide them into training set D t and the test set D v :

[0011] (1a) Use a common RGB camera to collect flexion test videos of different ages, genders, and scoliosis degrees to construct a scoliosis screening task dataset D;

[0012] (1b) Annotate the edge mask and category labels of the video data in the dataset D to obtain its edge mask image and category label;

[0013] (1c) performing flip preprocessing on the video data and the edge mask image to obtain a left image and a right flipped image thereof, as well as a corresponding left edge mask annotated image and a right flipped edge mask image;

[0014] (1d) Divide all left and right image data into training set D according to the ratio of 8:2 t and the test set D v ;

[0015] (2) Using the Swin Unet segmentation network including the Swin Transformer codec as the contour edge prediction network;

[0016] (3) Use contrastive learning to train the Swin Unet contour edge prediction network:

[0017] (3a) The training set D t Divide into multiple batches and input them into the contour edge prediction network to obtain its semantic features and edge prediction mask images;

[0018] (3b) Calculate the symmetry comparison learning loss value L between the semantic features of the left image and the right flipped image of each sample in the same batch ita and the Dice loss value L of the edge prediction mask image and edge mask label D ;

[0019] (3c) Calculate the loss function L1 = αL ita +βL D , by minimizing L1, the Swin Unet contour edge prediction network is back-propagated to adjust the model weight parameters of the network, where α+β=1, α and β are weight coefficients;

[0020] (3d) Repeat steps (3a) to (3d) until the number of training times reaches a set threshold or the value of the loss function converges, and obtain the weight of the contour edge prediction model;

[0021] (4) Constructing a scoliosis screening network based on symmetric contrast learning and contour perception, which uses two Swin Transformer encoders, feature decoders, and one contour feature encoder;

[0022] (5) Training of scoliosis screening network based on symmetric contrast learning and contour perception:

[0023] (5a) Load the contour edge prediction model weights obtained in (3d) into the Swin Transformer encoder and feature decoder, and freeze their weights;

[0024] (5b) The training set D tThe left image and the right flipped image in the image are divided into multiple batches and input into the scoliosis screening network based on symmetric contrast learning and contour perception, and the semantic features of the left and right images and the features of the left and right edge mask superimposed images are extracted, and they are spliced ​​to obtain the fusion features;

[0025] (5d) Input the fused features into the existing multi-layer perceptron module, and then perform a softmax operation to obtain the predicted category results of all samples in the batch;

[0026] (5e) The predicted category results and actual category labels of all samples in the batch are input into the Focal loss function and the cross entropy loss function to calculate the Focal loss loss L foc and the cross entropy loss L cls ;

[0027] (5f) Calculate the loss function L2 = γL foc +δL cls , by minimizing L2, the scoliosis screening network based on symmetric contrast learning and contour perception is back-propagated to adjust the model weight parameters of the network, where γ+δ=1, γ and δ are weight coefficients;

[0028] (5g) Repeat steps (5b) to (5f) until the number of training times reaches a set threshold or the value of the loss function converges, thereby obtaining a trained scoliosis screening model based on symmetric contrast learning and contour perception;

[0029] (6) Test set D v The data is input into the trained scoliosis screening model based on symmetric contrast learning and contour perception, and the visual physical examination strategy in the model is used to obtain the spinal status:

[0030] If the test set D v If all four test indicators of a sample are normal, it is confirmed that the spine of the sample is normal;

[0031] If the test set D v If one test indicator in a sample is abnormal, it is confirmed that the sample suffers from scoliosis.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. The present invention extracts image features and contour edge masks through the Swin Transformer encoder-decoder structure, and introduces the contour edge mask loss, which can prompt the model to enhance the contour edge features of the key area of ​​scoliosis; and extracts the superimposed contour edge features through the Swin encoder for symmetry judgment, which can constrain the model to recognize the morphological differences between the left and right sides of the spine.

[0034] 2. The present invention uses the focal loss function to constrain the model to handle the category imbalance problem, so that the model can focus on the relatively small categories such as shoulders and other heights and difficult-to-classify samples.

[0035] 3. Since the present invention constructs the intra-sample symmetry contrast loss, the contour edge prediction model can compare the symmetry differences between the left and right sides of the same sample, thereby effectively learning the symmetry characteristics within the sample. The symmetry characteristics can significantly improve the screening accuracy of the scoliosis screening model.

[0036] 4. Since the present invention uses a common RGB camera to collect scoliosis data video, compared with the existing method of using RGB-D camera for scoliosis screening, it greatly reduces the required hardware equipment cost and is suitable for large-scale screening of adolescents. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the implementation of the present invention;

[0038] Figure 2 It is a structural diagram of the scoliosis screening model in the present invention;

[0039] Figure 3 is the edge mask image of the shoulder area prediction in the present invention;

[0040] Figure 4 is an edge mask image for waistline region prediction in the present invention;

[0041] Figure 5 is an edge mask image of the pelvic region prediction in the present invention;

[0042] Figure 6 is an edge mask image predicted by the back region in the present invention;

[0043] Figure 7 is the superimposed image of the left and right masks of the shoulder area in the present invention;

[0044] Figure 8 is the superimposed image of the left and right masks of the waistline region in the present invention;

[0045] Fig. 9 It is the superimposed image of the left and right masks of the pelvic region in the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without creative work should all fall within the scope of protection of the present invention.

[0047] It should be noted that the step numbers in the specification and claims of the present invention and in the claims are only for a clear description of the implementation scheme of the present invention and for ease of understanding, and the sequence of the numbers is not limited.

[0048] Reference Figure 1 , the implementation steps of this example are as follows:

[0049] Step 1: Use a common RGB camera to collect flexion test videos of people of different ages, genders, and scoliosis degrees to construct a scoliosis screening task dataset D.

[0050] (1.1) Set the collection requirements for the forward flexion test:

[0051] The male subjects were naked, while the female subjects wore underwear.

[0052] The person being tested takes off his shoes, faces a clean wall, stands naturally, with his feet shoulder-width apart, his eyes looking straight ahead, his arms hanging naturally, and his palms facing inwards;

[0053] (1.2) Before the start of the data collection, inform the test subjects of the data collection requirements of the forward flexion test to ensure that the forward flexion test standards are available during the formal recording;

[0054] (1.3) After the data collection begins, the subject begins to bend forward, including straightening the knees, putting the feet together, standing at attention, stretching the arms and putting the palms together, lowering the head and slowly bending forward to about 90 degrees, putting the palms together and gradually placing them between the knees. The tester shoots the subject's overall forward bending action video and saves it in data set D.

[0055] Step 2: Preprocess the data and divide the training set D t and the test set D v .

[0056] (2.1) Perform edge mask annotation on the images of video samples in dataset D:

[0057] 2.1.1) Perform pixel-level annotation on the first frame of the video sample in dataset D, i.e., annotate the edges of the waistline, shoulders, and pelvic contour, and generate binary mask labels for the waistline area and shoulder area in the frame;

[0058] 2.1.2) For the bending action images of the video samples in the dataset D, pixel-level annotation is performed every ten frames to mark the edges of the back contour and generate binary mask labels for the back area in these images;

[0059] (2.2) Set up a visual physical examination strategy and label the video samples in the dataset D:

[0060] 2.2.1) Observe whether the shoulders of the person being measured are at the same height in the standing image. If they are, mark them as normal shoulders. If they are not, mark them as abnormal shoulders.

[0061] 2.2.2) Observe whether the waistlines on both sides of the person being measured in the standing image are symmetrical. If the waistlines on both sides are symmetrical, it is marked as normal waistline; if the waistlines on both sides are asymmetrical, it is marked as abnormal waistline;

[0062] 2.2.3) Observe whether the left and right pelvis of the person being tested in the standing image are symmetrical. If the left and right pelvis are symmetrical, mark it as a normal pelvis. If the left and right pelvis are asymmetrical, mark it as an abnormal pelvis.

[0063] 2.2.4) Find out the condition of the back of the person being tested by observing the overall forward bending action video, and mark the back image frames with left-right asymmetry during the bending process as razor back labels, and mark the left-right symmetrical back image frames as normal back labels.

[0064] (2.3) Perform flip preprocessing on the video data and edge mask image:

[0065] 2.3.1) The first frame standing RGB image and its annotated mask image in the video of dataset D are divided into left and right parts with the human body as the center, and the back RGB image and its annotated mask image of the video bending action in dataset D are also divided into left and right parts with the human body as the center;

[0066] 2.3.2) Horizontally flip the right RGB image and the right mask image of the standing and bending state to obtain a right flipped RGB image and a right flipped mask image;

[0067] 2.3.3) Resize the left RGB image, right flipped RGB image, left mask image, and right flipped mask image to 224×224 to match the input size of the model;

[0068] (2.4) All samples in data set D are randomly divided into training set D according to the ratio of 8:2 t and the test set D v .

[0069] Step 3: Build the Swin Unet contour edge prediction network.

[0070] (3.1) adopting the existing Swin Transformer-based encoder, which has a four-layer Swin Transformerblock encoding module, for efficiently capturing the multi-scale features of the input image using hierarchical feature extraction and sliding window self-attention mechanism;

[0071] (3.2) An existing Swin Transformer-based decoder is selected, which has a four-layer Swin Transformerblock decoding module, which gradually restores the spatial details of the image through layer-by-layer upsampling and multi-scale fusion;

[0072] (3.3) The output of each layer of the Swin Transformer encoder is jump-connected with the input of each layer of the corresponding decoder to construct the Swin Unet contour edge prediction network. This network fully combines the powerful feature extraction capability of the encoder and the layer-by-layer detail recovery capability of the decoder to achieve multi-scale encoding and decoding of the input image, ensuring the efficient combination of global information and local details, thereby achieving high-precision prediction of contour edges.

[0073] Step 4: Use contrastive learning to train the Swin Unet contour edge prediction network.

[0074] (4.1) The training set D t Divide into multiple batches and input them into the contour edge prediction network to obtain the semantic features of the image and the predicted mask image containing the waistline, shoulders and back edges:

[0075] 4.1.1) The training set D t The left image and the right flipped image are input into the Swin Transformer encoder of the Swin Unet contour edge prediction network. The spatial size of the feature map is gradually reduced through the hierarchical structure of the encoder, and four layers of multi-scale feature images of 56×56, 28×28, 14×14, and 7×7 are extracted. The 7×7 scale feature is pooled to obtain the semantic features of the image.

[0076] 4.1.2) Input the four-layer multi-scale feature map extracted in step (4.1.1) into the Swin Transformer decoder for upsampling, and then generate a segmented image with the same size as the input image by fusing these multi-scale features, and finally obtain the edge prediction mask images of the waistline area, shoulder area and back area;

[0077] (4.2) Calculate the symmetry comparison learning loss value L between the semantic features of the left image and the right flipped image of each sample in the same batch ita :

[0078] 4.2.1) Take the semantic features of the left image of each sample in the same batch as the anchor point. When the category label of the sample is normal, take the semantic features of the right flipped image as the first positive sample z of the anchor point. p1 ; When the category label of the sample is abnormal, the semantic features of the right-flipped image are used as the first negative sample z of the anchor point n1 ;

[0079] 4.2.2) Copy the anchor image features as the second positive sample z of the anchor p2 ;

[0080] 4.2.3) Rotate the anchor image feature by a small angle of 5° to 10° and use the rotated image feature as the second negative sample z of the anchor n2 ;

[0081] 4.2.4) Input the semantic features of the left image, the semantic features of the right flipped image, and the image features of the anchor image rotated at a small angle into the intra-sample contrastive learning loss function, and calculate the contrastive learning loss value L between the semantic features of the left image and the right flipped image of each sample in the same batch ita :

[0082]

[0083] Where K is the set of semantic features of the left image, the right flipped image, and the image features of the anchor image rotated at a small angle in the same batch; k represents the kth element of the set K; z k is the feature of the anchor image of the kth sample; P(k) is the set of positive sample image features of the kth sample; A(k) is the set of all image features of the kth sample except the anchor; p represents the pth element of the set P(k); a represents the ath element of the set A(k); z p =z p1 +z p2 , z a =z p1 +z p2 +z n1 +z n2 ; τ is the temperature coefficient, which is used to control the model's discrimination of negative samples;

[0084] (4.3) Calculate the Dice loss value L of the edge prediction mask images and edge mask labels of the same batch D :

[0085] 4.3.1) Calculate the number of intersection pixels |M∩G| between the edge prediction mask image M and the edge mask label G;

[0086] 4.3.2) Calculate the Dice loss value L for each regionD :

[0087]

[0088] Wherein, |M∩G| represents the number of pixels of the intersection of the edge prediction mask image M and the edge mask label G, |M| represents the total number of pixels of the edge prediction mask image M, and |G| represents the total number of pixels of the edge mask label G;

[0089] (4.4) Calculate the loss function L1 = αL ita +βL D , by minimizing L1 to back-propagate the Swin Unet contour edge prediction network, the model weight parameters of the network are adjusted, where α+β=1, α and β are weight coefficients used to balance the impact of the two parts of loss on model training;

[0090] (4.5) Repeat steps (4.1) to (4.4) until the number of training times reaches the set threshold or the value of the loss function converges, and obtain the weight of the contour edge prediction model.

[0091] Step 5: Construct a scoliosis screening network based on symmetric contrast learning and contour perception.

[0092] (5.1) Jump-connect each layer of the Swin Transformer block encoding module of the Swin Transformer image feature encoder and each layer of the Swin Transformer block encoding module of the Swin Transformer image feature decoder to form a codec structure;

[0093] (5.2) Copy one codec structure in (5.1) to obtain a second codec structure, and arrange the two codec structures in parallel through contrastive learning to form a twin network for contrastive learning, and the two codecs share weights;

[0094] (5.3) Connect one contour feature encoder to the back end of the twin network to form a scoliosis screening network based on symmetric contrast learning and contour perception, such as Figure 2 shown.

[0095] Step 6: Train the scoliosis screening network based on symmetric contrast learning and contour perception:

[0096] (6.1) Load the weights of the contour edge prediction model obtained in (4.5) into the Swin Transformer image feature encoder and feature decoder, and freeze their weights, which will no longer be updated during subsequent training.

[0097] (6.2) Input the left image and the right flipped image into the Swin Transformer image encoder to obtain four-layer multi-scale features of 56×56, 28×28, 14×14, and 7×7, and perform pooling operations on the left and right image features of 7×7 scale to obtain the semantic features of the left image F image-L and right flip image semantic feature F image-R ;

[0098] (6.3) Input the multi-scale features of the left and right images into the feature decoder for upsampling decoding to obtain the edge mask image F of the left image with the same scale of 224×224 as the original image. mask-L and the edge mask image F of the right flipped image mask-R The edge mask images of the shoulders, waistline, pelvis and back regions are as follows: Figure 3 , Figure 4 , Figure 5 , Figure 6 As shown;

[0099] (6.4) After adding the left and right edge mask images pixel by pixel, normalization is performed, that is, when the pixel value is greater than 1, it is set to 1, thereby obtaining the superimposed image of the left and right edge masks, where the superimposed images of the left and right edge masks of the shoulders, waistline and pelvis areas are respectively as follows: Figure 7 , Figure 8 , Fig. 9 As shown;

[0100] (6.5) The left and right edge mask superimposed images are passed through the contour encoder to extract the features F of the superimposed image edge ;

[0101] (6.6) Semantic features F of the left and right images image-K 、F image-R The feature F of the image superimposed with the left and right edge masks edge Perform splicing operation to obtain fusion feature F final ; Input the fused features into the existing multi-layer perceptron module, and then perform a softmax operation to obtain the predicted category results of all samples in the batch;

[0102] (6.7) The predicted category results and actual category labels of all samples in the batch are input into the Focal loss function to calculate the Focal loss loss L foc Value:

[0103]

[0104] Where B is the number of samples in a single batch in the training set, and C is the number of categories;

[0105] θ cis the weight of category c, which is used to balance the influence between different categories. In this example, it is set to but not limited to 0.2;

[0106] μ is a parameter that adjusts the impact of difficult and easy samples on the loss. In this example, its value is set to, but not limited to, 3;

[0107] y i,c is the actual category label of the i-th sample. If sample i belongs to category c, then y i,c =1, otherwise y i,c =0;

[0108] pred i,c is the predicted probability that the i-th sample belongs to category c after MLP and softmax operation;

[0109] (6.8) The predicted category results and actual category labels of all samples in the batch are input into the cross entropy loss function to calculate the cross entropy loss L cls :

[0110]

[0111] Where B is the number of samples in the batch;

[0112] C is the number of categories;

[0113] y i ∈{0,1} represents the actual label of the i-th sample;

[0114] Represents the predicted probability that the i-th sample is classified correctly;

[0115] (6.9) According to the results of steps (6.7) and (6.8), the loss function L2 of the scoliosis screening network based on symmetric contrast learning and contour perception is calculated:

[0116] L2=γL foc +δL cls ;

[0117] (6.10) Back-propagation of the scoliosis screening network based on symmetric contrast learning and contour perception is performed by minimizing L2 to adjust the model weight parameters of the network, where γ+δ=1, γ and δ are weight coefficients;

[0118] (6.11) Repeat steps (6.2) to (6.10) until the number of training times reaches the set threshold or the value of the loss function converges, and a trained scoliosis screening model based on symmetric contrast learning and contour perception is obtained.

[0119] Step 7: Test set D vThe data are input into the trained scoliosis screening model based on symmetry contrast learning and contour perception, and the spinal status of the test set samples is obtained using a visual physical examination strategy.

[0120] (7.1) The test set D v The data is input into the trained scoliosis screening model based on symmetry contrast learning and contour perception, and the classification results of whether the shoulders of the test set samples are equal in height, whether the waistline is symmetrical, whether the pelvis is symmetrical, and whether there is a razor back are output. That is, the output is 1 for normal and 0 for abnormal.

[0121] (7.2) According to the classification results, the visual physical examination strategy is used to obtain the spinal status of the test set samples:

[0122] If the test set D v If the results of the four classifications of shoulders, waistline, pelvis and razorback of a sample are all normal, then the spine of the sample is confirmed to be normal;

[0123] If the test set D v If one of the four classification results of shoulders, waistline, pelvis and razor back of a sample is abnormal, it is confirmed that the sample suffers from scoliosis.

[0124] The effect of the present invention can be further illustrated by the following test experiments:

[0125] 1. Test conditions:

[0126] The testing equipment consists of a computer and a camera, which are connected and set up 2 to 3 meters behind the person with scoliosis being tested.

[0127] The test data is forward flexion test videos of 204 test subjects collected in a hospital. All test subjects were labeled according to visual physical examination, and the labels were all marked by hospital doctors.

[0128] The simulation test platform uses Intel(R)Core(TM)CPU E5-2683 v4@2.10GHz, equipped with NVIDIATITAN Xp graphics card, and memory capacity of 12GB. The platform is based on Linux operating system and implemented in Python language.

[0129] Evaluation indicators: A "visual physical examination" test is performed to obtain a series of results, including whether the shoulders are at the same height, whether the waist lines on both sides are symmetrical, whether the left and right pelvises are at the same height, whether there is a razorback deformity, and whether there is scoliosis.

[0130] 2. Test content and results:

[0131] Under the above test conditions, the method of the present invention is used to process the forward flexion test video data of 204 tested persons, 164 samples are used as the training set to train the scoliosis screening model constructed by the present invention, and the trained scoliosis screening model is used to test the remaining 40 samples, that is, to test whether the shoulders are equal in height, whether the waist lines on both sides are symmetrical, whether the left and right pelvises are symmetrical, whether there is a razor back deformity, and whether there is scoliosis, and the respective accuracy, precision, recall rate and F1 score are calculated. The results are shown in Table 1.

[0132] Table 1 Scoliosis screening test index table

[0133]

[0134] As can be seen from Table 1, the accuracy and F1 scores of all test results reached 85% and above. In terms of judging whether scoliosis is present, the accuracy also reached 85%, and the FI score reached 91.5%. This shows that the present invention has excellent accuracy and reliability in screening scoliosis, and provides strong support for the early detection and screening of scoliosis.

Claims

1. A scoliosis auxiliary screening method based on symmetric contrast learning and contour perception, characterized in that: These include: (1) Collect videos and divide them into training set D t and the test set D v : (1a) Use a common RGB camera to collect flexion test videos of different ages, genders, and scoliosis degrees to construct a scoliosis screening task dataset D; (1b) Annotate the edge mask and category labels of the video data in dataset D to obtain its edge mask image and category label; (1c) performing flip preprocessing on the video data and the edge mask image to obtain a left image and a right flipped image thereof, as well as a corresponding left edge mask annotated image and a right flipped edge mask image; (1d) Divide all left and right image data into training set D according to the ratio of 8:2 t and the test set D v ; (2) Using the Swin Unet segmentation network including the Swin Transformer codec as the contour edge prediction network; (3) Use contrastive learning to train the Swin Unet contour edge prediction network: (3a) The training set D t Divide into multiple batches and input them into the contour edge prediction network to obtain its semantic features and edge prediction mask images; (3b) Calculate the symmetry comparison learning loss value L between the semantic features of the left image and the right flipped image of each sample in the same batch ita and the Dice loss value L of the edge prediction mask image and edge mask label D ; (3c) Calculate the loss function L1 = αL ita +βL D , by minimizing L1, the Swin Unet contour edge prediction network is back-propagated to adjust the model weight parameters of the network, where α+β=1, α and β are weight coefficients; (3d) Repeat steps (3a) to (3d) until the number of training times reaches a set threshold or the value of the loss function converges, and obtain the weight of the contour edge prediction model; (4) Constructing a scoliosis screening network based on symmetric contrast learning and contour perception, which uses two SwinTransformer encoders, feature decoders and one contour feature encoder; (5) Training of scoliosis screening network based on symmetric contrast learning and contour perception: (5a) Load the contour edge prediction model weights obtained in (3d) into the Swin Transformer encoder and feature decoder, and freeze their weights; (5b) The training set D t The left image and the right flipped image in the image are divided into multiple batches and input into the scoliosis screening network based on symmetric contrast learning and contour perception, and the semantic features of the left and right images and the features of the left and right edge mask superimposed images are extracted, and they are spliced ​​to obtain the fusion features; (5d) Input the fused features into the existing multi-layer perceptron module, and then perform a softmax operation to obtain the predicted category results of all samples in the batch; (5e) The predicted category results and actual category labels of all samples in the batch are input into the Focal loss function and the cross entropy loss function to calculate the Focal loss loss L foc and the cross entropy loss L cls ; (5f) Calculate the loss function L2 = γL foc +δL cls , by minimizing L2, the scoliosis screening network based on symmetric contrast learning and contour perception is back-propagated to adjust the model weight parameters of the network, where γ+δ=1, γ and δ are weight coefficients; (5g) Repeat steps (5b) to (5f) until the number of training times reaches a set threshold or the value of the loss function converges, thereby obtaining a trained scoliosis screening model based on symmetric contrast learning and contour perception; (6) Test set D v The data is input into the trained scoliosis screening model based on symmetric contrast learning and contour perception, and the visual physical examination strategy in the model is used to obtain the spinal status: If the test set D v If all four test indicators of a sample are normal, it is confirmed that the spine of the sample is normal; If the test set D v If one test indicator in a sample is abnormal, it is confirmed that the sample suffers from scoliosis.

2. The method according to claim 1, characterized in that In step (1a), the flexion test videos of people of different ages, genders, and scoliosis degrees are collected. Before the test begins, the test subjects are first informed of the key points of the flexion test, which include: keeping straight knees, feet together, and standing in a standing position, with arms straight and palms together, and lowering the head and slowly bending forward when bending down until the upper body is approximately 90 degrees to the ground, so as to ensure that the flexion test standard is available during the formal recording; then a common RGB camera is used to record the overall flexion action video of the test subject for storage.

3. The method according to claim 1, characterized in that In step (1b), the edge mask and category label annotations are performed on the video data in the dataset D, which is implemented as follows: (1b1) Annotate the first frame of the video sample in dataset D at the pixel level, i.e., annotate the edges of the waistline, shoulders, and pelvic contours, and generate binary mask labels for the waistline and shoulder regions in the frame. Annotate the bending action images of the video sample in dataset D at the pixel level every ten frames, annotate the edges of the back contours, and generate binary mask labels for the back regions in these images. (1b2) Set up a visual physical examination strategy to label the video samples in the dataset D: By observing the standing image of the overall forward bending action video of the tested person, three conditions are marked: whether the shoulders of the tested person are at the same height, whether the waist lines on both sides of the sample are symmetrical, and whether the left and right pelvises of the sample are at the same height; By observing the overall forward bending movement video of the person being tested, the condition of his / her back is found out, and whether the person being tested has a razor back in the forward bending test is marked. The back image frames with left-right asymmetry during the bending process are marked as razor back labels, and the left-right symmetrical back image frames are marked as normal back labels every other frame.

4. The method according to claim 1, characterized in that: Step (1c) performs flip preprocessing on the video data and edge mask image, which is implemented as follows: (1c1) The standing RGB image of the first frame of the video in the dataset D and its annotated mask image are divided into two parts, left and right, with the human body as the center, and the back RGB image of the video bending action in the dataset D and its annotated mask image are also divided into two parts, left and right, with the human body as the center; (1c2) Horizontally flip the right RGB image of standing and bending and the right mask image; (1c3) Resize the left RGB image, the right flipped RGB image, the left mask image, and the right flipped mask image to 224×224.

5. The method according to claim 1, characterized in that In step (3a), the training set D t Divide into multiple batches and input them into the contour edge prediction network to obtain its semantic features and edge prediction mask image, which is implemented as follows: (3a1) The left image and the right flipped image in the training set are input into the Swin Transformer encoder of the Swin Unet contour edge prediction network. The spatial size of the feature map is gradually reduced through a hierarchical structure, and multi-scale features of 56×56, 28×28, 14×14, and 7×7 images are extracted. The 7×7 scale features are pooled to obtain their semantic features. (3a2) The multi-scale features of the image obtained in (3a1) are input into the Swin Transformer decoder, and these multi-scale features are gradually upsampled into segmented images of the same size as the input image to obtain edge prediction mask images of the waistline area, shoulder area and back area.

6. The method according to claim 1, characterized in that In step (3b), the symmetry comparison learning loss value L between the semantic features of the left image and the right flipped image of each sample in the same batch is calculated. ita , the implementation steps include the following: (3b1) The semantic features of the left image of each sample in the same batch are used as anchor points. When the category label of the sample is normal, the semantic features of the right flipped image are used as the first positive sample z of the anchor point. p1 ; When the category label of the sample is abnormal, the semantic features of the right-flipped image are used as the first negative sample z of the anchor point n1 ; (3b2) Copy the anchor image features as the second positive sample z of the anchor p2 ; (3b3) Rotate the anchor image feature by a small angle of 5° to 10° and use the rotated image feature as the second negative sample z of the anchor n2 ; (3b4) Input the semantic features of the left image, the semantic features of the right-flipped image, and the image features of the anchor image rotated at a small angle into the symmetry contrast learning loss function, and calculate the contrast learning loss value L between the semantic features of the left image and the right-flipped image of each sample in the same batch ita : Among them, K is the set of semantic features of the left image, the semantic features of the right-flipped image, and the image features of the anchor image rotated at a small angle in the same batch; k represents the kth element of set K; z k is the feature of the anchor image of the kth sample; P(k) is the set of positive sample image features of the kth sample; A(k) is the set of all image features of the kth sample except the anchor point; p represents the pth element of the set P(k); a represents the a-th element of the set A(k); With p =from p1 +with p2 ,With a =from p1 +with p2 +with n1 +with n2 ; τ is the temperature coefficient, which is used to control the model's discrimination against negative samples.

7. The method according to claim 1, characterized in that In step (3b), the Dice loss value L of the edge prediction mask images and edge mask labels of the same batch is calculated. D , implemented as follows: (3b5) Calculate the number of intersection pixels |M∩G| between the edge prediction mask image M and the edge mask label G. (3b6) Calculate the Dice loss value L of each region based on the number of intersection pixels |M∩G| D : Among them, |M| represents the total number of pixels of the edge prediction mask image M, and |G| represents the total number of pixels of the edge mask label G.

8. The method according to claim 1, characterized in that In step (4), a scoliosis screening network based on symmetric contrast learning and contour perception is constructed. The network uses two Swin Transformer encoders, a feature decoder and a contour feature encoder, and is implemented as follows: (4a) Each layer structure of the Swin Transformer encoder is connected to each layer structure of the corresponding feature decoder through a skip connection to form a codec structure; (4b) duplicating one codec structure in (4a) to obtain a second codec structure, and placing the two codec structures in parallel through contrastive learning to form a twin network for contrastive learning, and the two codecs share weights; (4b) Connect a contour feature encoder to the back end of the twin network to form a scoliosis screening network based on symmetric contrast learning and contour perception.

9. The method according to claim 1, characterized in that: Step (5b) extracts the semantic features of the left and right images and the features of the left and right edge mask superimposed images, which is implemented as follows: Multiple batches of left-side images and right-side flipped images are input into the scoliosis screening network, and the multi-scale features and semantic features of the left and right images are first extracted through the SwinTransformer encoder; The multi-scale features and semantic features of the left and right images are extracted and then decoded by the input feature decoder to obtain the edge mask images of the left and right images, and the superimposed image of the left and right edge masks is obtained by pixel-by-pixel addition operation and normalization processing; The superimposed image of the left and right edge masks is finally passed through a contour encoder to output the features of the superimposed image.

10. The method according to claim 1, characterized in that In step (5e), ​​the focal loss L is calculated. foc and the cross entropy loss L cls , the formulas are as follows: Where B is the number of samples in a single batch in the training set, and C is the number of categories; θ c is the weight of category c, which is used to balance the influence between different categories; μ is a parameter that adjusts the impact of hard and easy samples on the loss; y i,c is the actual category label of the i-th sample. If sample i belongs to category c, then y i,c =1, otherwise y i,c =0; pred i,c is the predicted probability that the i-th sample belongs to category c after MLP and softmax operation; y i ∈{0,1} represents the actual label of the i-th sample; It represents the predicted probability that the i-th sample is classified correctly.

Images