Swallowing capacity screening method, system and equipment based on monocular color camera and medium

Abstract

The invention discloses a method, a system and equipment for screening swallowing ability based on a monocular color camera and a medium. The method comprises the following steps: acquiring a background image and a side swallowing image sequence of a subject; performing background removal and gray processing on the side swallowing image sequence of the subject to obtain a gray image sequence; performing correction rotation of each frame of image on the grayscale image sequence to obtain a binary image sequence; performing connected domain extraction and comparison between frames on the binary image sequence to obtain pixel coordinates of the connected domain; and evaluating the swallowing action of the subject according to the pixel coordinates of the connected domain to obtain a swallowing ability screening result. The system, the equipment and the medium are used for realizing the swallowing capability screening method based on the monocular color camera. The problems that in the prior art, a data set needs to be established for training, calculation is cumbersome, discomfort is brought to a subject due to the fact that a marker needs to be pasted to the throat of the subject, and pasting of the marker needs to be conducted by an experienced expert are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and particularly relates to a swallowing ability screening method, system, device and medium based on a monocular color camera. Background Art

[0002] Traditional screening methods for dysphagia generally include repeated saliva swallowing tests, water swallowing tests, modified water swallowing tests, dye tests, eating assessment questionnaires, etc. (Chinese Consensus Group on Rehabilitation Evaluation and Treatment of Dysphagia. Chinese Consensus on the Evaluation and Treatment of Dysphagia (2017 Edition), Part I: Evaluation [J]. Chinese Journal of Physical Medicine and Rehabilitation, 2017, 39(12): 881-892. DOI: 10.3760 / cma.j.issn.0254-1424.2017.12.001). Such screening methods rely on expert experience and have the deficiencies of strong subjectivity, difficult operation and low efficiency, and cannot conduct large-scale screening in the face of the needs of an aging society.

[0003] In recent years, with the development of computer science, the application of artificial intelligence methods for medical assessment has become more and more mature, providing inspiration and possibility for intelligent swallowing function screening. Although there are currently systems that use computer vision methods to screen swallowing function, such as the patent application document with publication number CN117274878A, which discloses a swallowing disorder screening method and system based on computer vision, the method includes: collecting video data of the side of several subjects when drinking water at multiple stages to obtain a first data set; marking key areas and categories of the pre-processed first data set to obtain a first training data set and a second training data set; training the target detection model with the first training data set; training the multi-branch feature extraction and fusion classification model with the second training data set; collecting the video data to be identified from the side of the subject to be tested when drinking water at multiple stages; inputting the video data to be identified into the trained target detection model; inputting the output of the target detection model into the trained multi-branch feature extraction and fusion classification model to obtain the classification result. However, it is necessary to establish a training data set, and the calculation is cumbersome. For example, the patent application document with the publication number CN114515395A discloses a swallowing detection method and device, equipment, and storage medium based on binocular vision. During the swallowing process of the patient, multiple consecutive image pairs of the patient are obtained, each image pair includes a first laryngeal image and a second laryngeal image, and the first laryngeal image and the second laryngeal image both include a mark pasted on the patient's throat; the three-dimensional laryngeal features of each image pair are obtained by identifying the features of the marks in the first laryngeal image and the second laryngeal image, and the three-dimensional laryngeal features include the three-dimensional position of the key points of the laryngeal ... Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a swallowing ability screening method, system, device and medium based on a monocular color camera, so as to solve the problems that the prior art needs to establish a data set for training, the calculation is cumbersome, and the markers need to be pasted on the subject's throat, causing discomfort to the subject, and the pasting of the markers also requires experienced experts.

[0005] The present invention is achieved through the following technical solutions:

[0006] A swallowing ability screening method based on a monocular color camera comprises the following steps:

[0007] Collecting background images and swallowing image sequences of the subject from the side;

[0008] Background removal and grayscale processing are performed on the sequence of lateral swallowing images of the subject to obtain a sequence of grayscale images;

[0009] The sequence of grayscale images is subjected to correction rotation for each frame image to obtain a sequence of binary images;

[0010] Connected component extraction and comparison are performed between each frame of the sequence of binary images to obtain the pixel coordinates of the connected components;

[0011] Based on the pixel coordinates of the connected components, the swallowing action of the subject is evaluated to obtain the screening result of swallowing ability.

[0012] Furthermore, the specific process of collecting the sequence of swallowing images of the subject is as follows: The center of the viewfinder of the collection device is at the pharyngeal position of the subject, and the entire head of the subject is within the field of view of the collection device, and the lateral side of the subject's throat is completely exposed within the field of view, and the collection duration is 2 - 4 seconds.

[0013] Furthermore, the background removal is specifically to subtract the background image from the sequence of lateral swallowing images of the subject.

[0014] Furthermore, the correction rotation of each frame image is specifically as follows: The first frame in the sequence of grayscale images is used as the reference image, SIFT features are adopted, the images in the sequence of grayscale images except the first frame are respectively registered with the reference image to obtain feature points, the RANSAC method is used to rotate and align all the images with the reference image, and taking the center point of the image as the reference point, the value obtained by rounding down 0.9 times the original resolution is used as the new cropping size to crop the aligned image sequence and perform frame difference operation to obtain a new sequence of grayscale images. After taking the absolute value of the pixels of the new sequence of grayscale images, binarization operation is performed to obtain a sequence of binary images.

[0015] Furthermore, the connected component extraction and comparison between each frame are specifically as follows: An OR operation is performed on the sequence of binary images to obtain a connected component result image. The two-step method or the seed filling method is used to mark each connected component, and the connected component region with the largest area is calculated as the active region of the pharyngeal swallowing action. The corner point range of this connected component is calibrated using a rectangular box, and the pixel coordinates of its upper left corner are recorded as a(x1, y1), and the lower right corner coordinates are recorded as b(x2, y2); At the same time, any frame in the sequence of grayscale images is selected, and the maximum inter-class variance method is used to perform binarization processing for alignment, and the connected component with the largest area is marked as the head contour of the subject. The corner point range of this connected component is calibrated using a rectangular box, and the pixel coordinates of its upper left corner are recorded as c(x3, y3), and the lower right corner coordinates are recorded as d(x4, y4).

[0016] Further, the evaluation of the subject's swallowing action specifically involves calculating the length pixel l of the pharyngeal activity area, where l = |y1 - y2|, and the width pixel of the head contour is w = |x3 - x4|. Then, the relative activity ratio r of the pharynx is r = k * l / w; where k is a proportionality constant. The side swallowing image sequence of the subject is collected N times continuously, and the average value of the relative activity ratio r of the pharynx is calculated. The activity and risk of the pharynx are screened by the magnitude of the R value. The larger the R value, the better the activity.

[0017] A swallowing ability screening system based on a monocular color camera includes:

[0018] An image acquisition module: acquiring a background image and a side swallowing image sequence of the subject;

[0019] An image initialization module: performing background removal and gray-scale processing on the side swallowing image sequence of the subject to obtain a gray-scale image sequence;

[0020] An image rotation correction module: performing correction rotation on each frame of the gray-scale image sequence to obtain a binary image sequence;

[0021] A connected component comparison module: extracting and comparing the connected components between each frame of the binary image sequence to obtain the pixel coordinates of the connected components;

[0022] A result output module: evaluating the subject's swallowing action according to the pixel coordinates of the connected components to obtain a swallowing ability screening result.

[0023] A swallowing ability screening device based on a monocular color camera includes:

[0024] A memory: storing a computer program for a swallowing ability screening method based on a monocular color camera, which is a computer-readable device;

[0025] A processor: used to implement a swallowing ability screening method based on a monocular color camera when executing the computer program.

[0026] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a swallowing ability screening method based on a monocular color camera.

[0027] Compared with the prior art, the advantages of the present invention are as follows:

[0028] The present invention adopts a non-contact screening technology to collect the swallowing videos of subjects through a standardized process. Subsequently, background removal, grayscale conversion, and rotation correction are performed on the video sequence to generate a binary image sequence. Through connected component analysis, based on the obtained pixel coordinates, the swallowing function can be evaluated, thereby improving the comfort of the screening process. In addition, the present invention supports the use of common electronic devices such as mobile phones to collect swallowing videos, and has the advantages of low cost and high efficiency. This system can operate without a training dataset, has a low computational cost, and is convenient for wide promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of the method of the present invention.

[0030] Figure 2 is a system block diagram of the present invention.

[0031] Figure 3 is a schematic diagram of the acquired image of the present invention.

[0032] Figure 4 is a result diagram of the connected components of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0033] The following further describes the present invention in detail with reference to specific embodiments, which are explanations of the present invention rather than limitations.

[0034] Refer to Figure 1 , a swallowing ability screening method based on a monocular color camera, comprising the following steps:

[0035] Collect a background image and a sequence of side swallowing images of the subject: First, collect a background image I0. Then, before collecting the swallowing image sequence, require the subject to rest as shown in Figure 3 . The shooting center of the shooter's shooting device should be as close as possible to the pharyngeal position of the subject, and the subject's head should be completely within the camera's field of view. The side of the subject's throat should be completely exposed within the field of view. The shooting requires a simple background. During the shooting process, it is required that the shooter's device and the subject's head remain as stationary as possible. The acquisition duration is 2 - 4 seconds. Determine the drinking volume according to the subject's height. Assume that the reference height is 160 cm and the reference drinking volume is 10 ml. When the height is greater than the reference height, the drinking volume is greater than the reference volume; when the height is less than the reference height, the drinking volume is less than the reference volume. The specific values of the above reference height and reference volume can be adjusted according to the actual situation. The image acquisition module 1 finally completes the acquisition of the side color video (i.e., the RGB image sequence {I t}, where t represents the number of frames) of the subject's complete swallowing action.

[0036] Perform background removal and grayscale processing on the sequence of side swallowing images of the subject to obtain a sequence of grayscale images: The background removal algorithm directly makes {It} Subtracting the background image I0 gives the image sequence {I′ after background removal t} = {I t} - I0. Continuing to perform grayscale processing on {I′ t} gives the grayscale image sequence {I″ t}.

[0037] Performing correction rotation on each frame of the grayscale image sequence gives a binary image sequence: Using the first frame in {I″ t} as the reference image, adopting SIFT features, registering the images in the {I″ t} sequence except the first frame with the reference image respectively. After obtaining the feature points, using the RANSAC method, rotating and aligning all the images with the reference image, and taking the integer value of 0.9 times the original resolution with the image center point as the reference point as the new cropping size to crop the aligned image sequence, obtaining the grayscale image sequence {G t}. Performing frame difference operation on the grayscale image {G t} gives a new grayscale image sequence {G′ t} = {G t} - {G t-1}. At this time, the grayscale image sequence {G′ t} only contains the pixel change part of the human swallowing action after image alignment. Further performing absolute value taking and binarization operations on the pixels of the grayscale image sequence {G′ t} gives the binary image sequence {B′ t}.

[0038] Performing connected component extraction and comparison between frames on the binary image sequence to obtain the pixel coordinates of the connected components: Using the OR operation on the binary image sequence {B′ t} gives the connected component result graph L = OR({B′ t}). Using the two-step method or the seed filling method to label each connected component, and calculating that the connected component region with the largest area is the active region of the pharyngeal swallowing action. Using a rectangular box to mark the corner point range of this connected component, recording its upper left pixel coordinate as a(x1, y1) and the lower right coordinate as b(x2, y2); at the same time, selecting any frame in the grayscale image sequence {I″ t}, performing binarization processing using the maximum inter-class variance method for alignment, marking the largest connected component among them, which is the head contour of the subject. Using a rectangular box to mark the corner point range of this connected component, recording its upper left pixel coordinate as c(x3, y3) and the lower right coordinate as d(x4, y4). The connected component result graph is as shown in Figure 4 Figure.

[0039] According to the pixel coordinates of the connected components, the swallowing actions of the subject are evaluated to obtain the screening results of swallowing ability: calculate the pixel length l of the pharyngeal activity area as l = |y1 - y2|, and the pixel width of the head contour is w = |x3 - x4|. Then the relative activity ratio r of the pharynx is calculated by the following formula: r = k * l / w; where k is a proportionality constant. Continuously collect the swallowing videos of the subject N times, and calculate the average value of the relative activity ratio r of the pharynx. Screen the pharyngeal activity and risks through the magnitude of the R value.

[0040] Such as Figure 2 As shown, a swallowing ability screening system based on a monocular color camera includes: an image acquisition module, an image initialization module, an image rotation correction module, a connected component comparison module, and a result output module;

[0041] The image acquisition module 1 collects the side swallowing video of the subject as input, and its output is connected to the input end of the image initialization module 2. The output of the image initialization module 2 is then connected to the input end of the image rotation correction module 3, and the output of the image rotation correction module 3 is connected to the input end of the connected component comparison module 4. One output port of the connected component comparison module 4 is connected to the input port of the result output module 5.

[0042] The image acquisition module 1 realizes the acquisition of the swallowing actions of the subject;

[0043] The image initialization module 2 realizes background removal and grayscale processing;

[0044] The image rotation correction module 3 is responsible for correcting each frame of the image to eliminate the deviation caused by the slight movement of the subject or the photographer during the acquisition process;

[0045] The connected component comparison module 4 realizes the extraction and comparison of the changing connected components between frames;

[0046] The result output module 5 realizes the quantitative evaluation and result display of the swallowing actions of the subject.

[0047] A swallowing ability screening device based on a monocular color camera includes:

[0048] A memory: stores a computer program for a swallowing ability screening method based on a monocular color camera, and is a computer-readable device;

[0049] A processor: is used to implement a swallowing ability screening method based on a monocular color camera when executing the computer program.

[0050] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a swallowing ability screening method based on a monocular color camera.

Claims

1. A swallowing ability screening method based on a monocular color camera, characterized in that, It includes the following steps: Collect the background image and the sequence of lateral swallowing images of the subject; Perform background removal and grayscale processing on the sequence of lateral swallowing images of the subject to obtain a sequence of grayscale images; Perform calibration rotation on each frame image of the sequence of grayscale images to obtain a sequence of binary images; Extract and compare the connected components between each frame of the sequence of binary images to obtain the pixel coordinates of the connected components; Evaluate the swallowing action of the subject according to the pixel coordinates of the connected components to obtain the screening result of swallowing ability.

2. The swallowing ability screening method based on a monocular color camera according to claim 1, characterized in that, The specific process of collecting the sequence of swallowing images of the subject is as follows: The center of the viewfinder of the collection device is at the pharyngeal position of the subject, and the head of the subject is completely within the field of view of the collection device. The lateral side of the subject's throat is completely exposed within the field of view, and the collection duration is 2 - 4 seconds.

3. The swallowing ability screening method based on a monocular color camera according to claim 2, characterized in that, The specific background removal method is to subtract the background image from the sequence of lateral swallowing images of the subject.

4. The swallowing ability screening method based on a monocular color camera according to claim 3, wherein, The specific calibration rotation of each frame image is as follows: Take the first frame in the sequence of grayscale images as the reference image, use SIFT features to register the images in the sequence of grayscale images except the first frame with the reference image respectively to obtain feature points, use the RANSAC method to rotate and align all images with the reference image, and take the center point of the image as the reference point, and use the rounded value of 0.9 times the original resolution as the new cropping size to crop the aligned image sequence and perform frame difference operation to obtain a new sequence of grayscale images. After taking the absolute value of the pixels of the new sequence of grayscale images, perform binarization operation to obtain a sequence of binary images.

5. A swallowing ability screening method based on a monocular color camera according to claim 4, characterized in that The specific extraction and comparison of connected components between each frame are as follows: Use the OR operation on the sequence of binary images to obtain the result image of connected components, use the two-step method or the seed filling method to mark each connected component, and calculate the connected component area with the largest area as the active area of the pharyngeal swallowing action. Use a rectangular frame to mark the corner range of this connected component, and record its upper left pixel coordinate as a(x1, y1) and the lower right coordinate as b(x2, y2); At the same time, select any frame in the sequence of grayscale images, use the maximum inter-class variance method to perform binarization processing for alignment, mark the connected component with the largest area as the head contour of the subject, use a rectangular frame to mark the corner range of this connected component, and record its upper left pixel coordinate as c(x3, y3) and the lower right coordinate as d(x4, y4).

6. The swallowing ability screening method based on a monocular color camera according to claim 5, characterized in that, The evaluation of the subject's swallowing action specifically includes calculating the length pixel l of the pharyngeal activity area, where l = |y1 - y2|, and the width pixel of the head contour is w = |x3 - x4|. Then the relative activity ratio r of the pharynx is r = k * l / w, where k is a proportionality constant. The lateral swallowing image sequence of the subject is collected continuously for N times, and the average value of the relative activity ratio r of the pharynx is calculated. The activity and risk of the pharynx are screened by the magnitude of the R value. The larger the R value, the better the activity.

7. A swallowing ability screening system based on a monocular color camera, characterized in that, It includes: Image acquisition module: Collect the background image and the sequence of lateral swallowing images of the subject; Image initialization module: Perform background removal and grayscale processing on the sequence of lateral swallowing images of the subject to obtain a sequence of grayscale images; Image rotation calibration module: Perform calibration rotation on each frame image of the sequence of grayscale images to obtain a sequence of binary images; Connected component comparison module: Extract and compare the connected components between each frame of the sequence of binary images to obtain the pixel coordinates of the connected components; Result output module: Evaluate the swallowing action of the subject according to the pixel coordinates of the connected components to obtain the screening result of swallowing ability.

8. A swallowing ability screening device based on a monocular color camera, characterized in that It includes: Memory: Store the computer program of a swallowing ability screening method based on a monocular color camera described in any one of claims 1 - 6, and it is a computer-readable device; Processor: When executing the computer program, it is used to implement a method for screening swallowing ability based on a monocular color camera according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a method for screening swallowing ability based on a monocular color camera according to any one of claims 1-6.

Images