Oral function evaluation device, oral function evaluation method, and program

Abstract

This invention provides an oral function evaluation device, oral function evaluation method, and program that can easily evaluate swallowing function. [Solution] The oral function evaluation device 4 comprises a throat measurement area setting unit 64 and a swallowing function evaluation unit 65. The throat measurement area setting unit 64 sets a throat measurement area, which is the region in the throat that includes the submandibular area and the laryngeal prominence, in a three-dimensional image of the target area 2a obtained when the depth camera 3 images the target area 2a including the throat of the subject 2. The swallowing function evaluation unit 65 evaluates at least one of the time required for the subject 2 to swallow, the number of swallows, the timing of swallowing, and the position change of the laryngeal prominence, based on the time-dependent changes in the vertical and depth coordinates of the swallowing movement measurement points set within the throat measurement area in a three-dimensional coordinate system that defines vertical, horizontal, and depth coordinates.

Description

Technical Field

[0001] The present invention relates to an oral function evaluation device, an oral function evaluation method, and a program.

Background Art

[0002] As a device for evaluating oral functions, Patent Document 1 describes an invention for evaluating swallowing functions using voice data obtained by non-contact sound collection of voices uttered by a subject with a predetermined syllable or a predetermined sentence, a first image obtained by imaging the face or neck of the subject, and a second image obtained by imaging the oral cavity of the subject. Further, Patent Document 2 describes a device that is wound around the neck of a subject and evaluates swallowing functions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention described in Patent Document 1, it is necessary to use voice data and images of the oral cavity when evaluating swallowing functions. Further, in the invention described in Patent Document 2, it is necessary to attach a dedicated instrument to the neck of the subject. Therefore, it is difficult to say that either invention has a configuration that can easily realize the evaluation of swallowing functions.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide an oral function evaluation device, an oral function evaluation method, and a program that can easily evaluate swallowing functions.

Means for Solving the Problems

[0006] To achieve the above objective, the oral function evaluation device according to the first aspect of the present invention is A throat measurement region setting unit sets a throat measurement region in a three-dimensional image of the target area, including the throat of a subject, obtained by imaging the target area including the throat of the subject with a depth camera, and the throat measurement region is the area including the submandibular region and the laryngeal prominence in the throat. The system includes a swallowing function evaluation unit that, in a three-dimensional coordinate system defining vertical, horizontal, and depth coordinates, evaluates at least one of the time required for the subject to swallow, the number of swallows, the timing of the swallows, and the positional change of the laryngeal prominence, based on the temporal changes in the vertical and depth coordinates of the swallowing movement measurement points set within the throat measurement area.

[0007] In the oral function evaluation device, The subject area includes the facial area of ​​the subject, A landmark detection unit that detects multiple landmarks in the facial area from the three-dimensional image, The system further includes a coordinate system setting unit that sets the three-dimensional coordinate system, which consists of a first axis for defining the horizontal coordinate, a second axis for defining the vertical coordinate, and a third axis for defining the depth coordinate, based on the plurality of landmarks detected by the landmark detection unit, The throat measurement area setting unit may set the throat measurement area directly below the position of the subject's chin, which is identified based on the plurality of landmarks detected by the landmark detection unit.

[0008] The throat measurement area setting unit may detect a specific point within the throat measurement area where the depth coordinate is closest to the depth camera, and set a plurality of points, including the detected specific point and arranged parallel to the second axis, as the swallowing movement measurement points.

[0009] In the oral function evaluation device, The aforementioned plurality of landmarks are located below the lower lip of the subject and include one or more mandibular movement measurement points, including the position of the chin. The system may further include a masticatory function evaluation unit that evaluates at least one of the following based on the temporal changes in the vertical, horizontal, and depth coordinates of the mandibular movement measurement point: the duration of chewing by the subject, the speed of chewing, the amount of mouth opening during chewing, the number of chews, and whether the subject is chewing on the left or right side.

[0010] The aforementioned multiple landmarks include the left corner of the eye, the right corner of the eye, the forehead, and the tip of the nose of the subject. The coordinate system setting unit may determine the first axis based on the left and right eye angles, the second axis based on the forehead and the tip of the nose, and the third axis based on the first and second axes.

[0011] To achieve the above objective, the oral function evaluation method according to the second aspect of the present invention is: The steps include setting a throat measurement region in a three-dimensional image of the target area, which is the region including the submandibular and laryngeal prominences in the throat, obtained by imaging the target area including the throat of a subject with a depth camera, The method includes a step of evaluating, based on the time-dependent changes in the vertical and depth coordinates of a swallowing movement measurement point set within the throat measurement area, the time required for the subject to swallow, the number of swallows, the timing of the swallows, and the change in the position of the laryngeal prominence in a three-dimensional coordinate system that defines vertical, horizontal, and depth coordinates.

[0012] To achieve the above objective, the program according to the third aspect of the present invention is: Computers, A throat measurement region setting means for setting a throat measurement region, which is the region of the throat that includes the submandibular and laryngeal prominences, in a three-dimensional image of the target area obtained by imaging the target area including the throat of a subject with a depth camera. In a three-dimensional coordinate system that defines a vertical coordinate, a horizontal coordinate, and a depth coordinate, based on the temporal changes of the vertical coordinate and the depth coordinate of the swallowing movement measurement points set within the throat measurement region, it functions as a swallowing function evaluation means for evaluating at least any one of the time required for the subject to swallow, the number of swallows, the timing of swallowing, and the positional change of the laryngeal elevation.

Effect of the Invention

[0013] According to the present invention, the swallowing function can be easily evaluated.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram showing the configuration of an oral function evaluation system according to an embodiment of the present invention. [Figure 2] It is a front view of a subject for explaining a plurality of landmarks and a throat measurement region according to the above embodiment. [Figure 3] It is a perspective view of a subject for explaining a plurality of landmarks and a throat measurement region according to the above embodiment. [Figure 4] (a) to (c) are graphs showing the experimental results of confirming whether the temporal data of the three-dimensional coordinates of the mandibular movement measurement points capture the mandibular movement of the subject. [Figure 5] (a) is a graph showing the temporal changes of five mandibular movement measurement points according to the above embodiment, and (b) is a partially enlarged view of the graph of FIG. 5(a). [Figure 6] It is a graph showing the temporal change in the left - right direction of the subject at an arbitrary one of the mandibular movement measurement points according to the above embodiment. [Figure 7] (a) is a graph showing the respective temporal changes of a plurality of swallowing movement measurement points according to the above embodiment, (b) is a graph showing the positions of a plurality of swallowing movement measurement points in the vertical and depth directions immediately after the start of measurement, and (c) is a graph showing the positions of a plurality of swallowing movement measurement points in the vertical and depth directions at a time point after a certain period has elapsed from the time point of FIG. 7(b). [Figure 8]It is a figure obtained by superimposing the graphs of FIGS. 7(b) and 7(c). [Figure 9] It is a graph showing the amount of movement in the depth direction of each of a plurality of swallowing motion measurement points from the time of FIG. 7(b) to the time of FIG. 7(c). [Figure 10] It is a figure showing an example of a three-dimensional image that can be displayed on the display unit according to the above-described embodiment.

Mode for Carrying Out the Invention

[0015] An embodiment of the present invention will be described with reference to the drawings.

[0016] The oral function evaluation system 1 shown in FIG. 1 is a system for evaluating a series of functions from chewing to swallowing as the oral functions of the subject 2, and includes a depth camera 3 and an oral function evaluation device 4.

[0017] The depth camera 3 is configured to include an RGB camera and a depth sensor for obtaining depth information, and acquires depth data of each pixel of the RGB image. The depth camera 3 generates three-dimensional visual information (hereinafter, three-dimensional image) by combining the depth data with the RGB image. As the depth camera 3, for example, a commercially available product such as an Intel RealSense (registered trademark) depth camera D435 can be used.

[0018] When evaluating the oral function with the oral function evaluation system 1, the depth camera 3 images the target part 2a of the subject 2 over a period from when the subject 2 eats until chewing to swallowing is completed. The target part 2a includes the facial part and the throat part of the subject 2. The depth camera 3 sends data of the three-dimensional image of the subject 2 (including data of the RGB image and depth data of each pixel of the RGB image) to a control unit 6 described later.

[0019] The oral function evaluation device 4 comprises a display unit 5 and a control unit 6. The display unit 5 is composed of a liquid crystal display, an organic EL display, etc., and displays images under the control of the control unit 6. For example, the display unit 5 can display a three-dimensional image of the subject 2 captured by the depth camera 3 in real time, or display the analysis process and evaluation results of oral function based on the three-dimensional image.

[0020] The control unit 6 controls the operation of the depth camera 3 and the display unit 5. The control unit 6 can be a microcontroller implemented in an information terminal such as a personal computer, smartphone, or tablet. At least one of the depth camera 3 and the display unit 5 may be included in the information terminal. In other words, part or all of the oral function evaluation device 4 may be composed of the information terminal.

[0021] The control unit 6 includes a landmark detection unit 61, a coordinate system setting unit 62, a mastication function evaluation unit 63, a throat measurement area setting unit 64, and a swallowing function evaluation unit 65, as functions for executing programs stored in the built-in memory.

[0022] (Landmark detection unit 61) The landmark detection unit 61 detects multiple pre-set landmarks (features) on the face of the subject 2 (target unit 2a) from a three-dimensional image acquired from the depth camera 3. The landmark detection unit 61 uses a machine learning model to detect multiple landmarks from the target unit 2a in the three-dimensional image. This machine learning model is a machine learning-treated neural network, and has been subjected to machine learning using training data so that it outputs multiple pre-set landmarks when an image of the target unit 2a is input. The machine learning model may be a configuration that the landmark detection unit 61 has, or it may be a configuration that the landmark detection unit 61 acquires or uses via a network. For example, as a machine learning model, a model that detects the position and features of a face in real time, provided by MediaPipe, an open-source machine learning library, can be used.

[0023] As shown in Figures 2 and 3, the pre-set landmarks on the face include a reference point P0, a maxillofacial movement observation point P1 (see Figure 3), and a mandibular movement measurement point P2. The landmark detection unit 61 can display these detected landmarks on the display unit 5 in real time.

[0024] Reference points P0 are used by the coordinate system setting unit 62 to set a three-dimensional coordinate system. In this embodiment, four reference points are set, representing the positions of the left eye corner, right eye corner, forehead (between the eyebrows), and nasal tip of the subject 2. The landmark detection unit 61 can also display the line segment connecting the left and right eye corners and the line segment connecting the forehead and nasal tip on the display unit 5 in real time, using them as reference lines. The number and position of reference points P0 are not limited to this example and can be changed arbitrarily as long as the coordinate system setting unit 62 can set a three-dimensional coordinate system. It is preferable that at least four reference points P0 are set in the facial region excluding the mandible.

[0025] The maxillofacial movement observation points P1 are used by evaluators, such as medical professionals, using the oral function evaluation system 1 to observe maxillofacial movements. In this embodiment, five points are set, indicating the positions of the subject 2's subnasal point, upper lip, lower lip, left corner of the mouth, and right corner of the mouth. The landmark detection unit 61 can also display the following observation lines used by the evaluator when observing maxillofacial movements on the display unit 5 in real time: a line segment connecting the forehead and the tip of the nose (same as the aforementioned reference line) defined by the reference point P0; a line segment connecting the tip of the nose and the subnasal point; a line segment connecting the subnasal point and the upper lip; a line segment connecting the upper lip and the lower lip; a line segment connecting the upper lip and the left corner of the mouth; a line segment connecting the left corner of the mouth and the lower lip; a line segment connecting the upper lip and the right corner of the mouth; and a line segment connecting the right corner of the mouth and the lower lip. The number and positions of the maxillofacial movement observation points P1 can be changed arbitrarily, not limited to this example, as long as the evaluator can appropriately evaluate the maxillofacial movements.

[0026] The mandibular movement measurement points P2 are used to measure mandibular movement. In this embodiment, a total of five points are set, located below the lower lip of the subject 2, including the position of the chin. Note that the number and position of the mandibular movement measurement points P2 can be arbitrarily changed as long as mandibular movement can be appropriately measured, as described later, and are not limited to this example.

[0027] (Coordinate system setting unit 62) The coordinate system setting unit 62 sets a three-dimensional coordinate system based on a plurality of landmarks detected by the landmark detection unit 61. This system consists of an X-axis (first axis defining the horizontal coordinate) that defines the horizontal direction of the subject 2, a Y-axis (second axis defining the vertical coordinate) that is perpendicular to the X-axis and defines the vertical direction of the subject 2, and a Z-axis (third axis defining the depth coordinate) that is perpendicular to the X-axis and defines the depth direction of the subject 2. For example, the coordinate system setting unit 62 sets the X-axis based on a reference point P0 indicating the positions of the left and right eye corners, sets the Y-axis based on a reference point P0 indicating the positions of the forehead (between the eyebrows) and the tip of the nose, and sets the Z-axis based on the X-axis and Y-axis thus set. The origin of the three-dimensional coordinate system can be set arbitrarily, but it is preferable that the Y-axis, when the X-coordinate is 0 (zero), is set to a position corresponding to the midline of the subject 2. For example, the coordinate system setting unit 62 can estimate the midline of the subject 2 as a perpendicular bisector passing through the midpoint of the line segment connecting the reference point P0 representing the left eye angle and the reference point P0 representing the right eye angle.

[0028] The three-dimensional coordinate system (three-dimensional Cartesian coordinate system) set as described above tracks the head of subject 2 in real time and is set to the area of ​​the face excluding the mandible, making it suitable as a reference coordinate system for measuring the mandibular movement of subject 2. Note that the setting of the three-dimensional coordinate system by the coordinate system setting unit 62 can be changed arbitrarily, not limited to this example, as long as the mandibular movement can be measured appropriately, as described later.

[0029] Here, the inventors of the present invention conducted an experiment to confirm whether the time-series data of the three-dimensional coordinates of the mandibular movement measurement point P2 captured the actual mandibular movement of subject 2. The graphs of the experimental results are shown in Figures 4(a) to (c). In the graphs in Figures 4(a) to (c), the dark waveforms show the coordinate changes (unit [mm]) of the mandibular movement measurement point P2 (any single point), and the light waveforms show the output changes (voltage signal changes: unit [V]) obtained from the K7 evaluation system (manufactured by Myotronics). In all of these graphs, the horizontal axis is the time axis. Specifically, Figure 4(a) corresponds to the coordinate change in the Y direction of the mandibular movement measurement point P2, Figure 4(b) corresponds to the coordinate change in the X direction of the mandibular movement measurement point P2, and Figure 4(c) corresponds to the coordinate change in the Z direction of the mandibular movement measurement point P2. These graphs show that the time-series data of the three-dimensional coordinates of mandibular movement measurement point P2 accurately captures the actual mandibular movement of subject 2. Based on these experimental results, the masticatory function evaluation unit 63 evaluates masticatory function as follows.

[0030] (Chewing function evaluation unit 63) The masticatory function evaluation unit 63 evaluates masticatory function based on the time-dependent changes in the three-dimensional coordinates of the mandibular movement measurement point P2. Here, Figure 5(a) is a graph showing the time-dependent changes in five mandibular movement measurement points P2 measured when subject 2 ingested one standardized gummy jelly. In this graph, the vertical axis corresponds to the Y axis and the horizontal axis is the time axis. Of the five graphs, the uppermost graph shows the time-dependent changes in the mandibular movement measurement point P2 closest to the lower lip, and the lowermost graph shows the time-dependent changes in the mandibular movement measurement point P2 representing the chin. Figure 5(b) is a magnified view of a part of the graph in Figure 5(a).

[0031] In Figure 5(a), T represents the chewing period from the start to the end of chewing the gummy jelly. Throughout the chewing period, the mandibular movement measurement point P2 oscillates up and down, and the mandibular movement measurement point P2 changes over time in a valley-like pattern corresponding to one chew. Utilizing this characteristic, the chewing function evaluation unit 63 evaluates (identifies) the duration of chewing, the speed of chewing, the amount of mouth opening during chewing, and the number of chews performed by the subject 2.

[0032] Specifically, the masticatory function evaluation unit 63 measures the period from the start to the end of the oscillation at any mandibular movement measurement point P2 (i.e., the period from the start to the end of chewing), and identifies the measured period as the masticatory period (unit: time, such as [s]). In addition, the masticatory function evaluation unit 63 measures the width t of the valley corresponding to one chew (i.e., the time required for one chew) within the masticatory period, as shown in Figure 5(b), and identifies this as the masticatory speed (unit: time, such as [s]). The masticatory speed may also be the average value over the masticatory period. Thus, the average value of the masticatory speed over the masticatory period can also be considered as the period of one chew (masticatory cycle) over the masticatory period. The masticatory function evaluation unit 63 may also calculate the reciprocal of the period to determine the frequency of mastication during the mastication period. Furthermore, as shown in Figure 5(b), the masticatory function evaluation unit 63 measures the depth A of the groove corresponding to one chew and identifies this as the opening amount during mastication (unit: length such as [mm]). The opening amount may also be the average value during the mastication period. Additionally, the masticatory function evaluation unit 63 calculates a value obtained by dividing the mastication period by the mastication speed and identifies this as the number of chews. In the above example, we focused on the valley-like movement (time-dependent change in a valley) of the mandibular movement measurement point P2, which oscillates up and down throughout the mastication period. However, the masticatory function evaluation unit 63 may also analyze the mountain-like movement (time-dependent change in a mountain-like movement) of the mandibular movement measurement point P2. This mountain-like movement represents the movement (chewing action) of the subject 2 during occlusion. For example, the masticatory function evaluation unit 63 can also evaluate (specify) the occlusal time by measuring the width of the peak of the mountain drawn by the mandibular movement measurement point P2 as it changes over time.

[0033] Furthermore, the masticatory function evaluation unit 63 evaluates (identifies) which side of the subject 2 chews on (hereinafter also referred to as the chewing side). Figure 6 is a graph showing the time-dependent change in the left-right direction of subject 2 at mandibular movement measurement point P2 (any one point) measured when subject 2 ingested one standardized gummy jelly. In this graph, the vertical axis corresponds to the X-axis, and the horizontal axis is the time axis. In this graph, the X-coordinate of the position corresponding to the midline of subject 2 is "0", negative values ​​in the vertical axis direction correspond to the right side of subject 2, and positive values ​​correspond to the left side of subject 2. For example, the masticatory function evaluation unit 63 identifies periods in which the X-coordinate of the mandibular movement measurement point P2 shows a negative value as periods in which subject 2 is performing right-side chewing, and periods in which the X-coordinate of the mandibular movement measurement point P2 shows a positive value as periods in which subject 2 is performing right-side chewing.

[0034] As described above, the masticatory function evaluation unit 63 evaluates the following masticatory evaluation indices based on the time-dependent changes in the three-dimensional coordinates of the mandibular movement measurement point P2: (i) the duration of mastication by the subject 2, (ii) the speed of mastication, (iii) the amount of mouth opening during mastication, (iv) the number of mastications, and (v) which side of the mouth the subject 2 uses for mastication (masticatory side), and displays the evaluation results on the display unit 5. The analysis method used by the masticatory function evaluation unit 63 to identify these masticatory evaluation indices is arbitrary, and well-known analysis methods such as threshold analysis and analysis to ignore outliers (trimming, winsorizing, etc.) can be used as appropriate.

[0035] The masticatory function evaluation unit 63 does not need to evaluate all of the masticatory evaluation indices (i) to (v), and may evaluate at least one of them. Furthermore, the masticatory function evaluation unit 63 may evaluate the risk of masticatory function decline based on at least one of the masticatory evaluation indices (i) to (v) and a machine learning model. This machine learning model is a machine learning-treated neural network, and only needs to be trained using training data to output a risk assessment result when at least one of the masticatory evaluation indices (i) to (v) is input. The training data consists of at least one of the masticatory evaluation indices (i) to (v) and the risk assessment result. This risk assessment result is not limited to a two-stage system of high and low risk, but may have three or more stages.

[0036] (Throat measurement area setting unit 64) The throat measurement area setting unit 64 sets a throat measurement area B (see Figures 2 and 3), which is the area in the throat of the subject 2 that includes the submandibular region and the laryngeal prominence, in the three-dimensional image acquired from the depth camera 3 in order to evaluate the swallowing function of the subject 2. In this embodiment, the throat measurement area setting unit 64 sets the throat measurement area B based on the reference point P0 and the mandibular movement measurement point P2 from among a plurality of landmarks detected by the landmark detection unit 61. Specifically, the throat measurement area setting unit 64 sets a rectangular throat measurement area B directly below the chin, which is identified by the mandibular movement measurement point P2. In this case, the throat measurement area setting unit 64 sets the center line of the rectangular throat measurement area B as a perpendicular bisector (a line parallel to the Y-axis) passing through the midpoint of the line segment connecting the reference point P0 indicating the left eye corner and the reference point P0 indicating the right eye corner.

[0037] Furthermore, the throat measurement area setting unit 64 detects a specific point Q1 from the point cloud set in a grid within the throat measurement area B, as shown in Figures 2 and 3, whose depth coordinate (Z coordinate) is closest to the depth camera 3. This specific point Q1 indicates the estimated position of the laryngeal prominence. The throat measurement area setting unit 64 then sets multiple points, including the detected specific point Q1 and arranged parallel to the Y axis, as swallowing movement measurement points Q2.

[0038] (Swallowing Function Evaluation Section 65) The swallowing function evaluation unit 65 evaluates swallowing function based on the time-dependent changes in the vertical coordinate (Y coordinate) and depth coordinate (Z coordinate) of the swallowing movement measurement point Q2 set within the throat measurement area B.

[0039] Here, Figure 7(a) is a graph showing the time course of the depth coordinates (Z coordinates) of multiple swallowing motion measurement points Q2 (60 swallowing motion measurement points Q2 were set in this example) measured when subject 2 swallowed 5 mL of water. In this graph, the vertical axis corresponds to the Z axis, and the horizontal axis is the time axis.

[0040] Furthermore, Figure 7(b) is a graph showing the vertical and depth positions of multiple swallowing motion measurement points Q2 immediately after the start of measurement (at 0.05 [sec]), and Figure 7(c) is a graph showing the vertical and depth positions of multiple swallowing motion measurement points Q2 after a certain period has elapsed since Figure 7(b) (at 2.02 [sec]). In other words, the vertical axis in Figures 7(b) and (c) corresponds to the Y axis, and the horizontal axis corresponds to the Z axis. Figure 8 is a superimposed view of the graphs in Figures 7(b) and (c). Figure 9 shows the Z coordinates of each swallowing motion measurement point Q2 in Figure 7(c) when the Z coordinate value of each swallowing motion measurement point in Figure 7(b) is set to 0 (zero). In other words, Figure 9 is a graph showing the amount of movement in the depth direction of each of the multiple swallowing motion measurement points from time point 7(b) (0.05 [sec]) to time point 7(c) (2.02 [sec]).

[0041] In Figure 7(a), D represents the time required for swallowing (swallowing period). It is known that during swallowing, the submandibular region temporarily bulges when the suprahyoid muscles contract and the hyoid bone elevates. It is also known that during swallowing, the larynx elevates and the bulge moves upward. Referring to Figure 7(a), it can be seen that the point representing a specific part of the submandibular region among the swallowing movement measurement points Q2 oscillates up and down throughout the swallowing period, and that the point representing the specific part changes over time in a mountain-like pattern corresponding to one swallow. Furthermore, focusing on the dashed arrows in Figures 8 and 9, it can be seen that a certain swallowing movement measurement point Q2 is significantly displaced in the depth direction (Z direction) in response to swallowing. In other words, it can be seen that the above phenomena are captured by the time-dependent changes in the swallowing movement measurement point Q2. Utilizing this characteristic, the swallowing function evaluation unit 65 evaluates (identifies) the time required for the subject 2 to swallow, the number of swallows, the timing of swallowing, and the change in the position of the laryngeal prominence.

[0042] Specifically, the swallowing function evaluation unit 65 measures the period from the start to the end of the oscillation at any mandibular movement measurement point P2 (i.e., the time required for swallowing), and identifies the measured period as the swallowing period (unit: time, such as [s]). Although only one swallow is shown in Figure 7(a), the swallowing function evaluation unit 65 can identify the number of swallows by counting the number of swallowing periods, and can also identify the timing of swallowing by extracting the start of the swallowing period. Furthermore, the swallowing function evaluation unit 65 can also identify changes in the position of the laryngeal prominence from the temporal changes of the aforementioned specific point Q1.

[0043] As described above, the swallowing function evaluation unit 65 evaluates the following swallowing evaluation indices based on the temporal changes in the vertical and depth coordinates of the swallowing movement measurement point Q2: (i) the time required for the subject 2 to swallow, (ii) the number of swallows, (iii) the timing of swallowing, and (iv) the change in the position of the laryngeal prominence. The evaluation results are then displayed on the display unit 5. The analysis method used by the swallowing function evaluation unit 65 to identify these swallowing evaluation indices is arbitrary, and well-known analysis methods such as threshold analysis and analysis to ignore outliers (trimming, winsorizing, etc.) can be used as appropriate. In addition to the above evaluation results, the swallowing function evaluation unit 65 may also display a three-dimensional image of the throat area showing the swallowing movement measurement point Q2, as shown in Figure 10, on the display unit 5 in real time.

[0044] The swallowing function evaluation unit 65 does not need to evaluate all of the swallowing evaluation indices (i) to (iv), and may evaluate at least one of them. Furthermore, the swallowing function evaluation unit 65 may evaluate the risk of impaired swallowing function (including the risk of aspiration and laryngoptosis) based on at least one of the swallowing evaluation indices (i) to (iv) and a machine learning model. This machine learning model is a machine learning-treated neural network, and only needs to be machine learning-treated using training data so that it outputs a risk assessment result when at least one of the swallowing evaluation indices (i) to (iv) is input. The training data consists of at least one of the swallowing evaluation indices (i) to (iv) and the risk assessment result. This risk assessment result is not limited to a two-stage system of high and low risk, but may have three or more stages.

[0045] The present invention is not limited by the embodiments and drawings described above. Modifications (including the deletion of components) can be made as appropriate without altering the essence of the invention.

[0046] The above example illustrates how the throat measurement area setting unit 64 mathematically sets the throat measurement area B based on the reference point P0 and the mandibular movement measurement point P2, but it is not limited to this. The throat measurement area setting unit 64 may also directly recognize the throat measurement area B by image recognition using a machine learning model from a three-dimensional image obtained when the depth camera 3 captures the target area 2a, including the throat of the subject 2. This machine learning model is a neural network that has undergone machine learning, and it is sufficient that it has been machine-learned using training data to output the throat measurement area B when an image of the target area 2a is input. Furthermore, the machine learning algorithm is not limited to the above example and can be changed at will.

[0047] The subjects of oral function evaluation using the oral function evaluation system 1 are not limited to humans (subjects 2), but may also include animals such as pets.

[0048] The program that performs each of the processes described above does not necessarily have to be pre-stored in the memory of the control unit 6, and may be distributed and provided on a removable recording medium. Furthermore, the program may be downloaded from other devices connected to the oral function evaluation device 4. The oral function evaluation device 4 may also perform each of the processes according to the program by exchanging various data with other devices via a telecommunications network or the like.

[0049] (1-1) The oral function evaluation device 4 described above is A throat measurement area setting unit 64 sets a throat measurement area B, which is the region of the throat that includes the submandibular area and the laryngeal prominence, in a three-dimensional image of the target area 2a obtained by the depth camera 3 imaging the target area 2a including the throat of the subject 2 (an example of a subject), and The system includes a swallowing function evaluation unit 65 that evaluates at least one of the following based on the time-dependent changes in the vertical and depth coordinates of a swallowing movement measurement point Q2 set within the throat measurement area B in a three-dimensional coordinate system that defines vertical, horizontal, and depth coordinates: the time required for swallowing by the subject 2, the number of swallows, the timing of swallowing, and the change in the position of the laryngeal prominence. (1-2) Furthermore, the oral function evaluation method that can be realized with the oral function evaluation device 4 is: The throat measurement area setting unit 64 sets the throat measurement area B, The swallowing function evaluation unit 65 includes the step of evaluating at least one of the time required for the subject 2 to swallow, the number of swallows, the timing of swallowing, and the positional change of the laryngeal prominence. (1-3) Furthermore, the program stored in the memory of the control unit 6 causes the computer to function as a throat measurement area setting unit 64 (throat measurement area setting means) and a swallowing function evaluation unit 65 (swallowing function evaluation means).

[0050] According to the features described in (1-1) to (1-3) above, swallowing function can be easily evaluated simply by photographing the subject (including subject 2) with the depth camera 3, the evaluation can be performed non-invasively on the subject, and the stress on the subject during the evaluation can be reduced.

[0051] (2) Furthermore, the target area 2a includes the face of the subject 2, The oral function evaluation device 4 includes a landmark detection unit 61 that detects multiple landmarks in the facial area from a three-dimensional image, The system further includes a coordinate system setting unit 62 that sets a three-dimensional coordinate system consisting of a first axis for defining the horizontal coordinate, a second axis for defining the vertical coordinate, and a third axis for defining the depth coordinate, based on a plurality of landmarks detected by the landmark detection unit 61. The throat measurement area setting unit 64 sets the throat measurement area B directly below the position of the chin, which is identified based on a plurality of landmarks detected by the landmark detection unit 61.

[0052] According to the features described in (2) above, the throat measurement area B can be set based on multiple landmarks used when evaluating masticatory function, making the sequence for evaluating swallowing function in conjunction with masticatory function more efficient.

[0053] (3) The throat measurement area setting unit 64 also detects a specific point Q1 within the throat measurement area B whose depth coordinate is closest to the depth camera 3, and sets the detected specific point Q1 and multiple points that are aligned parallel to the second axis as swallowing movement measurement points Q2.

[0054] According to the features described in (3) above, a specific point Q1 indicating the estimated position of the laryngeal prominence can be included in the swallowing movement measurement point Q2, thereby enabling efficient evaluation of swallowing function.

[0055] (4) In addition, multiple landmarks are located below the lower lip of subject 2 and include one or more mandibular movement measurement points P2, including the position of the chin. The oral function evaluation device 4 further includes a masticatory function evaluation unit 63 that evaluates at least one of the following based on the temporal changes in the vertical, horizontal, and depth coordinates of the mandibular movement measurement point P2: the duration of chewing by the subject 2, the speed of chewing, the amount of mouth opening during chewing, the number of chews, and whether the subject 2 chews on the left or right side.

[0056] According to the features described in (4) above, the depth camera 3 can be used to photograph the subject (including subject 2) and easily evaluate a series of functions from chewing to swallowing. The evaluation can be performed non-invasively on the subject and the stress on the subject during the evaluation can be reduced.

[0057] (5) As a specific example, the multiple landmarks include the left corner of the eye, the right corner of the eye, the forehead and the tip of the nose of subject 2. The coordinate system setting unit 62 may determine a first axis based on the left and right canthosopharyngeal angles, a second axis based on the forehead and the tip of the nose, and a third axis based on the first and second axes.

[0058] According to the features described in (5) above, a coordinate system suitable for measuring the mandibular movement of the subject (including subject 2) can be set.

[0059] In the above explanation, explanations of known technical matters have been omitted as appropriate in order to facilitate understanding of the present invention.

[0060] This invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of the invention. In other words, the scope of this invention is indicated not by the embodiments, but by the claims. Various modifications made within the scope of the claims and the equivalent scope of the meaning of the invention are considered to be within the scope of this invention. [Explanation of Symbols]

[0061] 1… Oral function evaluation system 2...Subject, 2a...Target area 3…Depth camera 4… Oral function evaluation device 5...Display section 6…Control Unit 61... Landmark detection unit 62... Coordinate system setting section 63…Chewing Function Evaluation Department 64... Throat measurement area setting unit 65…Swallowing Function Evaluation Department P0: Reference point, P1: Maxillofacial movement observation point, P2: Mandibular movement measurement point B... Throat measurement area, Q1... Specific point, Q2... Swallowing movement measurement point

Claims

1. A throat measurement region setting unit sets a throat measurement region in a three-dimensional image of the target area, including the throat of a subject, obtained by imaging the target area including the throat of the subject with a depth camera, and the throat measurement region is the region including the submandibular and laryngeal prominence in the throat. The system includes a swallowing function evaluation unit that, in a three-dimensional coordinate system defining vertical, horizontal, and depth coordinates, evaluates at least one of the following based on the temporal changes in the vertical and depth coordinates of a swallowing movement measurement point set within the throat measurement area: the time required for the subject to swallow, the number of swallows, the timing of the swallows, and the change in the position of the laryngeal prominence. Oral function evaluation device.

2. The subject area includes the facial area of ​​the subject, A landmark detection unit that detects multiple landmarks in the facial area from the three-dimensional image, The system further includes a coordinate system setting unit that sets the three-dimensional coordinate system, which consists of a first axis for defining the horizontal coordinate, a second axis for defining the vertical coordinate, and a third axis for defining the depth coordinate, based on the plurality of landmarks detected by the landmark detection unit, The throat measurement area setting unit sets the throat measurement area directly below the position of the subject's chin, which is identified based on the plurality of landmarks detected by the landmark detection unit. The oral function evaluation device according to claim 1.

3. The throat measurement area setting unit detects a specific point within the throat measurement area where the depth coordinate is closest to the depth camera, and sets a plurality of points, including the detected specific point and aligned parallel to the second axis, as the swallowing movement measurement points. The oral function evaluation device according to claim 2.

4. The aforementioned plurality of landmarks are located below the lower lip of the subject and include one or more mandibular movement measurement points, including the position of the chin. The system further includes a masticatory function evaluation unit that evaluates at least one of the following based on the temporal changes in the vertical, horizontal, and depth coordinates of the mandibular movement measurement point: the duration of chewing by the subject, the speed of chewing, the amount of mouth opening during chewing, the number of chews, and whether the subject chews on the left or right side. The oral function evaluation device according to claim 2 or 3.

5. The aforementioned multiple landmarks include the left corner of the eye, the right corner of the eye, the forehead, and the tip of the nose of the subject. The coordinate system setting unit determines the first axis based on the left eye angle and the right eye angle, determines the second axis based on the forehead and the tip of the nose, and determines the third axis based on the first axis and the second axis. The oral function evaluation device according to claim 2 or 3.

6. The steps include setting a throat measurement region in a three-dimensional image of the target area, which is the region including the submandibular and laryngeal prominences in the throat, obtained by imaging the target area including the throat of a subject with a depth camera, The method comprises the step of evaluating, based on the temporal changes in the vertical and depth coordinates of a swallowing movement measurement point set within the throat measurement area, the time required for the subject to swallow, the number of swallows, the timing of the swallows, and the positional changes of the laryngeal prominence, in a three-dimensional coordinate system that defines vertical, horizontal, and depth coordinates. Methods for evaluating oral function.

7. Computers, A throat measurement region setting means for setting a throat measurement region, which is the region of the throat that includes the submandibular and laryngeal prominences, in a three-dimensional image of the target area obtained by imaging the target area including the throat of a subject with a depth camera. In a three-dimensional coordinate system defining vertical, horizontal, and depth coordinates, the swallowing function evaluation means functions to evaluate at least one of the following based on the temporal changes in the vertical and depth coordinates of the swallowing movement measurement points set within the throat measurement area: the time required for the subject to swallow, the number of swallows, the timing of the swallows, and the change in the position of the laryngeal prominence. program.

Images