A 3D imaging and machine learning system for identifying bony landmarks used in the development of treatment protocols

Abstract

A method for determining a spinal treatment protocol for a patient in real time is provided, the method including selecting a three-dimensional imaging and processing system; the three-dimensional imaging and processing system identifying bony landmarks including the patient's head and shoulders; the three-dimensional imaging and processing system determining an edge of each shoulder; determining positions of the bony landmarks while tilting and rotating at least one of the patient's head and shoulders to provide an output; and the three-dimensional imaging and processing system processing the output to provide the spinal treatment protocol.

Description

[Technical Field]

[0001] The present technology is directed to methods that can be used to replace x-rays in locating joints and other bony features of a patient, and more specifically, methods that rely on three-dimensional imaging and subsequent processing of the digital data to accurately locate bony features of interest. [Background technology]

[0002] For many chiropractic adjustments, the practitioner needs knowledge of the patient's spinal alignment. Traditionally, x-rays are taken of the patient's spine to determine if any of the vertebrae are misaligned. These measurements are taken around the patient's x-, y-, and z-axes as defined by a Cartesian coordinate system. As disclosed in U.S. Patent No. 10,733,728, x-rays are also used to determine the mean axis of rotation. An alternative approach is disclosed in U.S. Patent Application Publication No. 20150305825, in which impulse placement and direction are derived from analysis of multiple precisely placed or acquired tomographic images, preferably MRI images. Yet another approach, disclosed in U.S. Patent No. 10,893,826, provides a system for displaying and collecting biomechanical measurements. The system includes an electronic caliper including a bar, two arms slidably attached to the bar and extending perpendicularly from the bar, a display module, and an electronic system housed within the display module and including a light-emitting diode string of lights, a nine-axis sensor, firmware, a radio, and a power connector for electronic communication with a power source; and a remote computing device, the radio communicating with the remote computing device. An electronic caliper and method for using the same are also provided. While highly useful, this system suffers from drawbacks such as the potential for human error, the need for patient contact, and the need to collect and process data before determining a treatment protocol.

[0003] U.S. Patent Application Publication No. 20210279967 discloses a device, system, and method for generating a three-dimensional (3D) model of an object based on images and the tracked position of the device during image acquisition. For example, an exemplary process can include acquiring sensor data while a device is moving in a physical environment containing the object, the sensor data including images of the physical environment acquired via a camera on the device; identifying the object in at least some of the images; tracking the position of the device during image acquisition based on identifying the object in at least some of the images, the position identifying and tracking the positioning of the device relative to a coordinate system defined based on the position and orientation of the object; and generating a 3D model of the object based on the images and the position of the device during image acquisition. This approach is not sufficiently accurate for identifying and tracking landmarks on the human body.

[0004] U.S. Patent Application Publication No. 20190291723 discloses that pixel image data of a scene is received, the pixel image data including a two-dimensional representation of an object in the scene. Point cloud data including three-dimensional point coordinates of physical objects in the scene corresponding to the two-dimensional representation of the object is received. The three-dimensional point coordinates include depth information of the physical objects. The point cloud data is mapped to an image plane of the pixel image data to form integrated pixel image data, in which one or more pixels of the pixel image data have depth information integrated with the pixel. A three-dimensional bounding box of the object is predicted using a convolutional neural network based on the integrated pixel image data. This approach is not sufficiently accurate for identifying and tracking landmarks on the human body.

[0005] U.S. Patent Application Publication No. 20190139297 discloses a technique for generating a three-dimensional (3D) skeleton of an object using images of the object taken from different viewpoints. Images of the object (e.g., a person) can be captured from different camera angles. Feature keypoints of the object can be identified in the captured images. Keypoints identifying the same feature in different images can be correlated using truncated epipolar lines. For example, depth information of the keypoints can be used to truncate the epipolar lines created using the keypoints. The correlated feature keypoints can be used to create 3D feature coordinates of related features of the object. The 3D feature coordinates can be used to generate a 3D skeleton. One or more 3D models can be mapped to the 3D skeleton and rendered. The rendered one or more 3D models can be displayed on one or more display devices. This approach is not accurate enough for identifying and tracking landmarks on the human body.

[0006] U.S. Patent Application Publication No. 20160073614 discloses a method for diagnosing lameness in quadrupeds, such as sport horses, using computer vision and a computerized depth perception system to scan the animal over time. This method allows for detailed analysis of the animal's movements and their changes over time without the need for sensors attached to the horse's body or the need for force plates or expensive high-speed cameras. A processing system receives this movement data and uses it to determine the severity of the animal's lameness signal. The system is inexpensive enough to be installed in a suitable location, such as a horse stable, allowing non-professionals, such as veterinarily trained quadruped owners, to identify lameness early to objectively analyze rehabilitation from injury and help correlate gait changes with changes in performance. While this method allows for the detection of gait abnormalities, it is not accurate enough to identify and track landmarks on the human body.

[0007] What is needed is a method for accurately identifying structural features of a patient's body without X-ray images. Preferably, landmarks can be identified by imaging one or more of the head, shoulders, and pelvis. Preferably, the method is based on 3D digital images. Preferably, it reduces or eliminates human error. Preferably, it is touchless. Preferably, data derived from the 3D images can be used to determine treatment and track treatment effectiveness in real time. Even more preferably, the data can be used to determine the patient's biomechanical relationships. Summary of the Invention

[0008] This technology is a method for accurately identifying structural features of a patient's body without X-ray images. Landmarks can be accurately located and identified by imaging one or more of the head, shoulders, and pelvis. The method is based on 3D digital images and 2D facial recognition software, which reduces or eliminates human error. It is touchless. Data derived from the 3D images can be used to determine treatment and track treatment effectiveness in real time. The data can be used to determine the patient's biomechanical relationships.

[0009] In one embodiment, a method for determining a spinal treatment protocol for a patient in real time is provided, the method including selecting a three-dimensional imaging system and a two-dimensional facial detection system; selecting a processing system; the three-dimensional imaging system and processing system identifying bony landmarks including the patient's head and shoulders; the processing system determining the edges of each shoulder; the three-dimensional imaging system and processing system rotating the patient's head and determining positions of the bony landmarks while rotating and tilting the patient's shoulders to provide an output; the processing system processing the output; the two-dimensional facial detection system detecting facial landmarks; the processing system determining positions of the facial landmarks while tilting the patient's head to provide an output; and the processing system processing the output, wherein the output provides the spinal treatment protocol.

[0010] The method can further include a three-dimensional imaging system and a processing system that identifies body landmarks of the patient's pelvis while rotating and tilting the patient's pelvis to provide an output, the output providing a spinal treatment protocol.

[0011] In one embodiment, a method is provided for determining a spinal treatment protocol for a patient in real time, the method including selecting a three-dimensional imaging and processing system; the three-dimensional imaging and processing system identifying bony landmarks including the patient's head and shoulders; the three-dimensional imaging and processing system determining an edge of each shoulder; determining positions of the bony landmarks while tilting and rotating at least one of the patient's head and shoulders to provide an output; and the three-dimensional imaging and processing system processing the output to provide the spinal treatment protocol.

[0012] The method further includes delivering the spinal treatment protocol in real time to an impulse therapy device, which may further include a stylus.

[0013] In the method, providing a spinal treatment protocol can include providing a treatment vector that defines an orientation of a stylus in three dimensions relative to the patient.

[0014] In the method, providing a spinal treatment protocol can include providing an impulse protocol.

[0015] In another embodiment, a system for determining a spinal treatment protocol for a patient in real time is provided, the system including a three-dimensional camera; a memory in electronic communication with the three-dimensional camera; a processor under control of the memory and configured to process depth, image, and inertial data from the three-dimensional camera to provide an input; a machine learning component configured to analyze the input to provide an output; and an analysis component configured to provide a spinal treatment protocol in real time from the output to an impulse therapy device.

[0016] In yet another embodiment, a method of treating a patient in need of treatment is provided, the method including selecting a three-dimensional imaging and processing system; the three-dimensional imaging and processing system identifying bony landmarks including the patient's head and shoulders; the three-dimensional imaging and processing system determining an edge of each shoulder; determining positions of the bony landmarks while tilting and rotating at least one of the patient's head and shoulders to provide an output; the three-dimensional imaging and processing system processing the output to provide a spinal treatment protocol; the three-dimensional imaging and processing system transmitting the spinal treatment protocol to an impulse therapy device including a stylus; and the impulse therapy device delivering the spinal treatment protocol to a patient in need of the spinal treatment protocol.

[0017] In the method, providing a spinal treatment protocol can include providing a treatment vector that defines an orientation of a stylus in three dimensions relative to the patient.

[0018] In the method, providing a spinal treatment protocol can include providing an impulse protocol. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of a three-dimensional imaging and processing system of the present technology. [Figure 2] FIG. 1 is a block diagram showing an interface between a three-dimensional imaging and processing system and an impulse processing device. DETAILED DESCRIPTION OF THE INVENTION

[0020] Unless otherwise expressly provided, the following rules of interpretation apply to this specification (the specification and claims): (a) all words used herein shall be construed in gender or number (singular or plural) as the context requires. (b) the singular terms "a," "a," and "the," as used in this specification and the appended claims, include plural references unless the context clearly dictates otherwise. (c) the antecedent word "about," applied to a recited range or value, indicates approximation within the deviation of the range or value known or expected in the art from the method of measurement. (d) the words "herein," "hereby," "hereof," "hereto," "hereinbefore," and "hereinafter," and words of similar import, refer to the specification as a whole and not to any particular paragraph, claim, or other portion, unless otherwise expressly stated. (e) Description headings are for convenience only and do not control or affect the meaning or construction of any portion of this specification. (f) "Or" and "any" are not exclusive, and "include" and "including" are not limiting. Further, the terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise indicated.

[0021] The recitation of ranges of values ​​herein, unless otherwise indicated herein, is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, and each separate value is incorporated herein as if it were individually listed herein. When a specific range of values ​​is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit between the upper and lower limits of that range, and any other stated or intervening value within that stated range, is included therein, unless the context clearly dictates otherwise. All smaller subranges are also included. The upper and lower limits of these smaller ranges are also included, subject to any specifically excluded limits in the stated range.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Although any methods and materials similar or equivalent to those described herein can also be used, the acceptable methods and materials are described herein.

[0023] As shown in FIG. 1 , a 3D imaging and processing system, generally designated 6, includes an imaging component 8, a machine learning component 10, and an analysis component 50. The imaging component 8 includes a camera 12, which in one embodiment is a 3D camera, a Microsoft Kinect Azure® Depth Camera (Kinect); a driver 14 for the camera's depth functionality, which in one embodiment is a Microsoft Kinect Azure SDK (Kinect SDK) driver for controlling the Kinect hardware and receiving and processing all depth sensor, image sensor, and inertial measurement unit (IMU) data; body tracking software 16, which in one embodiment is Microsoft Kinect Body Tracking SDK (KINECT_BT_SDK), for providing (x, y, z) coordinates of body joints in the Azure Kinect SDK depth sensor data; and facial landmark software 18, which in one embodiment is MediaPipe Face Mesh (Mediapipe), providing a red-green-blue (RGB) image providing the (x, y, z) coordinates of 468 3D facial landmarks. Details of the inputs and outputs to and from the imaging component 8 and the analysis using the machine learning component 10 are provided below.

[0024] head analysis JPEG2026504227000002.jpg138170

[0025] shoulder analysis JPEG2026504227000003.jpg221170

[0026] buttocks analysis JPEG2026504227000004.jpg114170

[0027] The system 6 has four modes of operation: 1. Cervical rotation 2. Lateral bending 3.Front 4.Backwards JPEG2026504227000005.jpg199170JPEG2026504227000006.jpg65170

[0028] As shown in FIG. 2, the output from the machine learning component 10 is transferred in real time to the analysis component 50, which processes the output to provide a spinal treatment protocol. The processing includes analyzing the output to define an alignment of the spine, or a particular region of the spine (one or more of the cervical, thoracic, and lumbar). Once the alignment is defined, the analysis component defines the treatment. The spinal treatment protocol is transferred in real time to the impulse therapy device 60. The treatment is defined by a list of treatment parameters. The treatment parameters are as follows:

[0029] a treatment vector defining the orientation of the stylus in the impulse treatment device 60 in three-dimensional space relative to the patient's position; and 1. An impulse protocol comprising: Amplitude (the amplitude of the stylus movement is controlled in terms of 5 up to 40 Hertz (Hz) and frequency from 41 Hz to 200 Hz); Frequency (the stylus is moved so that its tip position follows a sine wave of the selected frequency or chirp from the start frequency to the stop frequency with an amplitude of up to 3 mm); and Impulse protocols, including maximum number of periods / cycles at a specified frequency.

[0030] The output from the machine learning component and the treatment from the analysis component 50 are stored in a database 70. The treatment can be obtained from the database during subsequent patient visits.

[0031] While the exemplary embodiments have been described in connection with what are presently considered to be examples of the most practical and / or suitable embodiments possible, it should be understood that the description is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the exemplary embodiments. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific exemplary embodiments specifically described herein. Such equivalents are intended to be encompassed by the claims appended hereto or as subsequently filed.

Claims

1. 1. A method for determining a spinal treatment protocol for a patient in real time, the method comprising: selecting a three-dimensional imaging system and a two-dimensional facial detection system; selecting a processing system; the three-dimensional imaging system and the processing system identifying bony landmarks including the patient's head and shoulders; the processing system determining the edges of each shoulder; the three-dimensional imaging system and the processing system rotating the patient's head and determining positions of the bony landmarks while rotating and tilting the patient's shoulders to provide an output; the processing system processing the output; the two-dimensional facial detection system detecting facial landmarks; the processing system determining positions of the facial landmarks while tilting the patient's head to provide an output; and the processing system processing the output, wherein the output provides a spinal treatment protocol.

2. 10. The method of claim 1, further comprising identifying body landmarks of the patient's pelvis with the three-dimensional imaging system and the processing system while rotating and tilting the patient's pelvis to provide an output, the output providing a spinal treatment protocol.

3. 3. The method of claim 1 or 2, further comprising delivering the spinal treatment protocol in real time to an impulse therapy device, the impulse therapy device including a stylus.

4. The method of claim 3 , wherein providing a spinal treatment protocol includes providing a treatment vector that defines an orientation of the stylus in three dimensions relative to the patient.

5. 5. The method of claim 1, wherein providing a spinal treatment protocol comprises providing an impulse protocol.

6. 1. A system for determining a spinal treatment protocol for a patient in real time, the system comprising: a three-dimensional camera; a memory in electronic communication with the three-dimensional camera; a processor under control of the memory and configured to process depth, image, and inertial data from the three-dimensional camera to provide an input; a machine learning component configured to analyze the input to provide an output; and an analysis component configured to provide the spinal treatment protocol in real time from the output to an impulse therapy device.

7. The system of claim 6 , further comprising a two-dimensional face detection system in electronic communication with the memory.

Images