Method, system and equipment for measuring cervical flexion and extension range of motion
Through binocular cameras and key point detection models, the cervical flexion and extension angle is calculated, which solves the cumbersome and error problems of the measurement method of the forward and post-extension movement of the neck in the prior art, and realizes convenient and accurate detection of the neck mobility.
Patent Information
- Application Number
- CN202411161262.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing measurement methods for flexion and extension of the neck have problems such as high equipment costs, cumbersome measurement, relying on manual operations, unintuitive numerical reading, and large measurement errors, and lack convenient and accurate measurement methods.
The side face image of the subject is taken by a binocular camera device, and the normalized coordinate values of the nose tip and ear contour are obtained through the key point detection model, the spatial vector from ear to nose tip is calculated, and the cervical flexion and extension angle is calculated in combination with the triangulation method to realize the neck Activity detection.
It realizes that the human body does not need to touch equipment, is convenient to measure, is automated, has high accuracy, reduces detection steps, reduces measurement errors, and improves the accuracy of calculation results.
Smart Images

Figure CN118680553B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, system and device for measuring the range of motion of neck flexion and extension. Background Art
[0002] The neck mobility test is a test of the neck's mobility, which is used to diagnose and prevent diseases of the neck and its surrounding areas.
[0003] Among them, there are three common methods for checking the flexion / extension range of motion of the neck: the first is to use orthopedic angle measuring instruments, such as arm-type protractors and inclinometers, which require professionals to hold the instruments in hand to perform real-time measurements during the measurement process and read the data quickly and accurately. The characteristics of this type of measurement method are low equipment cost, but the measurement process is cumbersome, relying on manual operation, and the numerical reading is not intuitive and inconvenient to record. The second is a helmet-type cervical spine range of motion measuring device, which requires the measurement person to wear the device and perform neck movements according to the instructions. Although the numerical values of this measurement method are more intuitive and accurate, the test is cumbersome, time-consuming, and requires a high degree of cooperation from the measurement person. The third is machine vision measurement, which uses RGB or ToF (Time of Flight) depth cameras to capture the subject's neck movement screen, and uses machine vision technology to capture the dynamic activity trajectory of the neck in real time to detect and record the maximum flexion range of the neck. This method is convenient and fast in the measurement process, and realizes automatic data recording, but the forward and backward movement of the head will cause the neck to be blocked, and the activity trajectory is difficult to capture, which is easy to cause measurement errors.
[0004] Based on the above problems, there is a need for a convenient and accurate method for measuring the flexion and extension range of motion of the neck. Summary of the invention
[0005] In order to solve the above problems in the prior art, the present invention proposes a method, system and device for measuring the range of motion of neck flexion and extension, which is not only convenient and fast, but also effectively improves the accuracy of measurement.
[0006] In a first aspect of the present invention, a method for measuring the range of motion of neck flexion and extension is provided, the method comprising:
[0007] Using a binocular camera device to capture two images of the subject performing neck flexion / extension exercise; during the capture, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device;
[0008] Obtaining the position of the face rectangular frame in each image according to the two images, and then obtaining two side face images of the face;
[0009] Based on the two sets of side-face images of the human face, a pair of side-face images of the human face taken at the same time is input into the trained key point detection model, and an output result is obtained; the output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the side-face image pair of the human face, and the category of each key point;
[0010] Calculate the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame;
[0011] Repeat the steps of inputting the pair of side-face images of a person, obtaining the output result, and calculating the space vector until all the pair of side-face images of a person in the two paths of side-face images of a person are input;
[0012] The corresponding cervical flexion and extension angles are calculated according to the spatial vector obtained in each calculation, thereby obtaining the neck range of motion detection result.
[0013] Preferably, the step of "calculating the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame" includes:
[0014] According to the position of the face rectangular frame, respectively mapping the normalized coordinate values of the nose tip key point and the N ear contour key points back to the original scale when shooting, and obtaining the mapped coordinate values of the nose tip key point and the N ear contour key points;
[0015] Based on the internal parameters and external parameters of the binocular camera device, and the mapped coordinate values of the nose tip key point and the N ear contour key points, the 3D coordinates of the nose tip key point and the N ear contour key points are calculated according to a triangulation method;
[0016] Calculate the 3D coordinates of the center of gravity of the ear according to the 3D coordinates of the N ear contour key points;
[0017] Calculate the spatial vector of the line from the ear to the nose tip according to the 3D coordinates of the center of gravity of the ear and the 3D coordinates of the key point of the nose tip;
[0018] The step of “determining the corresponding cervical flexion and extension angle according to the spatial vector obtained by each calculation, and then obtaining the neck range of motion detection result” includes:
[0019] The angle between the spatial vector calculated each time and the starting vector is calculated to obtain a plurality of cervical flexion and extension angles; the starting vector is the spatial vector calculated before the subject's neck flexion / extension movement begins and the neck is in a neutral position;
[0020] Based on the multiple cervical flexion and extension angles, the maximum angles of cervical flexion and extension are respectively selected as the cervical range of motion detection results.
[0021] Preferably, the key point detection model comprises n layers of attention modules connected in series;
[0022] The first layer is the fusion attention module and is calculated as follows:
[0023] , where FusionAttention represents the fusion attention feature function; the face images X1 and X2 constitute the face profile image pair; Softmax represents the Softmax function, Q1 and Q2 represent the features of the face images X1 and X2 after the first mapping, respectively, Q1=X1W Q1, Q2=X2W Q2 ; K1 and K2 represent the features of face images X1 and X2 after the second mapping, K1=X1W K1, K2=X2W K2 ; V1=X1W V1 ; V2=X2W V2; W Q1 , W Q2 , W K1 , W K2 , W V1 and W V2 Both represent randomly initialized trainable parameters, and are the transpose of K1 and K2 respectively, and d represents the dimensions of Q1 and Q2;
[0024] The 2nd to n-1th layers are all fusion attention modules, and are calculated according to the following formula:
[0025] ; Among them, the output value of the t-1 layer is Y t , 2≤t≤n-1, Y t Divide equally into Y channels t1 and Y t2 , and Y t1 and Y t2 Same dimensions; Q t1 =Y t1 W Qt1 ;Q t2 =Y t2 W Qt2 , K t1 =V t1 =Y t1 W Kt1 ; K t2 =V t2 =Y t2 W Kt2;W Qt1、 W Qt2、 W Kt1 and W Kt2 All represent randomly initialized trainable parameters; and K t1 and K t2 The transpose of d t Indicates Q t1 and Q t2 Dimensions;
[0026] The nth layer is the self-attention module and is calculated as follows: , , where Y n Represents the output value of the n-1th layer; Q n =Y n W Qn , K n =V n =Y n W Kn , W Qn and W Kn Both represent randomly initialized trainable parameters; d n Indicates Q n dimension, Y represents the output value of the nth layer; MLP represents the linear mapping layer; Y is an M ×1 vector, M=2+(N+1)×4; Y is split sequentially according to the first dimension, and the first two digits are the classification results , the remaining M-2 bits are the coordinates of the two sets of 2D key points detected on the face images X1 and X2; It is a 2×1 vector, representing the classification result of whether it is a profile face or not; the coordinates of a set of 2D key points detected on each image are all P×1 vectors after splicing, P=(N+1) ×2.
[0027] Preferably, the training method of the key point detection model includes:
[0028] The binocular camera device is used to shoot frontal face, left side face and right side face images of persons of different ages, genders and heights to obtain an original image set; when shooting, the persons stand at a designated standing point with the frontal face, left side face or right side face facing the binocular camera device;
[0029] Cropping and labeling the face portion of each image in the original image set to generate a key point detection model training set;
[0030] Based on the key point detection model training set, a pair of side face images of a human face is input into the key point detection model, and a detection result is obtained; the detection result includes: the normalized coordinate prediction values of the nose tip key point and N ear contour key points corresponding to each image in the pair of side face images, the category of each key point, and the classification result of whether it is a side face;
[0031] Calculating a loss function based on the labeled data corresponding to the pair of side-face images and the detection result;
[0032] Repeat the steps of inputting the face profile image pair, obtaining the detection result, and calculating the loss function until the value of the loss function no longer decreases or the training is stopped when a preset number of training times is reached, thereby obtaining the trained key point detection model;
[0033] in,
[0034] The key point detection model training set includes: a preset number of face profile image pairs and corresponding annotation data.
[0035] Preferably, the step of “obtaining the position of the face rectangular frame in each image according to the two images, and then obtaining two side face images of the face” includes:
[0036] The two images are uniformly scaled to a preset first size to obtain two pre-processed images;
[0037] Input the two pre-processed images into the trained face detection model respectively, and output the position of the face rectangle frame in each image;
[0038] According to the position of the face rectangular frame, the two pre-processed images are respectively cropped and scaled to a preset second size to obtain two side face images of the face;
[0039] Preferably, the annotation data includes: an identification of whether each image is a profile, normalized pixel coordinates of the nose tip key point and N ear contour key points, and the category of each key point;
[0040] The step of “cropping and labeling the face portion of each image in the original image set to generate a key point detection model training set” includes:
[0041] Scaling each image in the original image set to a preset first size to generate a first image set;
[0042] Marking each image in the first image set with a face rectangle frame;
[0043] For each image in the first image set, cropping is performed according to the marked face rectangular frame, and scaling the image to a preset second size to generate a second image set;
[0044] marking whether each image in the second image set is a profile face;
[0045] For each side face image in the second image set, marking the nose tip key point and the N ear contour key points;
[0046] Recording the pixel coordinates of the nose tip key point and the N ear contour key points, and normalizing the recorded pixel coordinates according to the preset second size;
[0047] Marking the categories of the nose tip key point and the N ear contour key points;
[0048] Based on the human face profile image pairs in the second image set and the corresponding annotation data, generating the key point detection model training set;
[0049] in,
[0050] N is a preset value, and the N ear contour key points are evenly distributed on the outer edge contour of the ear;
[0051] The categories are nose tip or ear outline.
[0052] Preferably, the loss function is: in,
[0053] ; , L c is the classification loss of whether it is a side face, L o is the spatial loss, y is the true value of whether it is a side face, is the predicted value of whether it is a side face, N is the number of key points of the ear contour, (x i ,y i ) is the true value coordinate of the i-th key point, is the remapped coordinate of the i-th key point.
[0054] The remapping coordinate generation method is:
[0055] According to the coordinate prediction values output by the key point detection model, the 3D key point coordinates corresponding to each pair of 2D key points are obtained by using a triangulation method;
[0056] The 3D key point coordinates are mapped back to each image in the face profile image pair according to the following formula to obtain the remapped coordinates of each 2D key point: ,in, is the 3D coordinate of the i-th key point mapped back to the 2D homogeneous coordinate of the image, K is the camera intrinsic parameter, is the 3D homogeneous coordinate of the i-th key point in the camera coordinate system.
[0057] Preferably, the face detection model is based on the YOLOv5s model;
[0058] The training method of the face detection model includes:
[0059] For each image in the first image set, the pixel coordinates of the upper left corner and the lower right corner of the face rectangular frame are recorded according to the marked face rectangular frame, and the recorded pixel coordinates are normalized according to the preset first size to generate a face detection model training set;
[0060] Based on the face detection model training set, the face detection model is trained to obtain the trained face detection model.
[0061] A second aspect of the present invention provides a system for measuring cervical flexion and extension range of motion, the system comprising:
[0062] An image acquisition module is used to use a binocular camera device to capture two images of the subject performing neck flexion / extension exercise; during the shooting, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device;
[0063] A side face image acquisition module, used to acquire the position of the face rectangular frame in each image according to the two images, and then acquire two side face images of the face;
[0064] A key point detection module is used to input a pair of side face images of a person taken at the same time into the trained key point detection model based on the two side face images, and obtain an output result; the output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the side face image pair and the category of each key point;
[0065] A vector calculation module, used for calculating the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame;
[0066] A control module, used for repeatedly calling the key point detection module and the vector calculation module until all the pairs of side face images in the two channels of side face images are input;
[0067] The activity calculation module calculates the corresponding cervical flexion and extension angle according to the spatial vector obtained in each calculation, and then obtains the neck activity detection result.
[0068] According to a third aspect of the present invention, a computer-readable storage device is provided, storing a computer program that can be loaded by a processor and execute the method described above.
[0069] The present invention has the following beneficial effects:
[0070] The method for measuring the range of motion of neck flexion and extension proposed by the present invention does not require a human body to contact the device during the measurement process, thereby simplifying the maintenance of the device; during the neck flexion / extension activity of the tested person, the ears and nose tip are not easily blocked when the face is in profile, making the shooting process convenient and easier to cooperate, and no professional guidance is required, which effectively reduces the measurement error and improves the accuracy of the final calculation result, and the whole process from shooting to obtaining the calculation result can be automated; the key point detection model is used to simultaneously detect the key points of the nose tip and the auricle, thereby reducing the number of detection steps.
[0071] The present invention proposes for the first time a method for calculating space vectors by using key points of nose tip and auricle through side face shooting based on a triangulation algorithm.
[0072] The present invention also proposes a new key point detection model, which uses the synergistic effect of classification loss and spatial loss to train the model, and its effect is better than the existing technology that only uses key point pixel spatial loss for training. The fusion attention module therein bidirectionally fuses the features of two images from different angles, so that the model can pay attention to the key point information of the face in the two images at the same time, further improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 is a schematic diagram of a binocular camera device used in the present invention;
[0074] Figure 2 is a schematic diagram of the world coordinate system of the calibration plate in the present invention;
[0075] Figure 3 It is a schematic diagram of the main steps of an embodiment of a method for measuring the range of motion of neck flexion and extension in the present invention;
[0076] Figure 4 Schematic diagram of the main components of the key point detection model in an embodiment of the present invention;
[0077] Figure 5 Schematic diagram of the main steps of the key point detection model training method in an embodiment of the present invention;
[0078] Figure 6 Schematic diagram of the main steps of the face detection model training method in an embodiment of the present invention;
[0079] Figure 7 It is a schematic diagram of the main components of an embodiment of the neck flexion and extension range of motion measurement system in the present invention. DETAILED DESCRIPTION
[0080] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.
[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] It should be noted that, in the description of the present invention, the terms "first" and "second" are only for the convenience of description, and do not indicate or imply the relative importance of the devices, elements or parameters, and therefore cannot be understood as limiting the present invention. In addition, the term "and / or" in the present invention is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.
[0083] Figure 1 Schematic diagram of the binocular camera device used in the present invention. Figure 2 It is a schematic diagram of the world coordinate system of the calibration plate in the present invention. In the present invention, the following preliminary preparations (1)-(4) need to be performed before taking the side face image of the test subject:
[0084] (1) Use a common RGB camera to build Figure 1 The binocular camera device shown includes a first camera 1, a second camera 2 and a fixing bracket 3; the binocular camera device is used to shoot a calibration plate to obtain the calibration plate image of the first camera 1 and the calibration plate image of the second camera 2.
[0085] (2) Using the two-way calibration plate images, use Zhang Zhengyou’s calibration method to obtain the intrinsic parameter matrix of the first camera 1 and the second camera 2, and calculate the extrinsic parameter matrix between the first camera 1 and the second camera 2 based on the intrinsic parameter matrix. For the definition of the camera’s intrinsic and extrinsic parameter matrix, see “Fourteen Lectures on Visual SLAM: From Theory to Practice”.
[0086] (3) Use a binocular device to capture multiple pairs of images of a calibration plate at different angles, use the method in step (2) to obtain multiple sets of external parameters, and use the Karcher averaging method to fuse the multiple sets of external parameters to obtain accurate external parameters.
[0087] (4) Place the calibration plate on the ground and use the calibration plate corner point detection and triangulation method to obtain the 3D coordinates of all corner points of the calibration plate. For details on the calibration plate corner point detection and triangulation method, please refer to "Fourteen Lectures on Visual SLAM: From Theory to Practice". Figure 2 The world coordinate system shown in FIG. 1 marks the origin of the coordinate system on the ground, which is the designated standing point of the test subject.
[0088] Since the present invention calculates the neck mobility through the key points of the ear and the nose tip, which are relatively easy to identify and extract, the binocular camera device in the present invention does not require special equipment and can be built using an ordinary RGB camera.
[0089] Figure 3 Schematic diagram of the main steps of the embodiment of the method for measuring the range of motion of neck flexion and extension in the present invention. Figure 3 As shown, the measurement method of this embodiment includes steps A10-A60:
[0090] Step A10, using a binocular camera device to capture two images of the subject performing neck flexion / extension exercise. During the capture, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device.
[0091] In order to shorten the test time, the present invention tests the subject in a state where the position remains unchanged. The prescribed action and angle are set to facilitate the binocular camera device to collect standard images. Step A20, based on the two-way images, obtain the position of the face rectangular frame in each image, and then obtain two-way face profile images. This step can specifically include steps A21-A23:
[0092] Step A21, scaling the two channels of images uniformly to a preset first size to obtain two channels of pre-processed images.
[0093] Step A22, input the two pre-processed images into the trained face detection model respectively, and output the position of the face rectangle frame in each image.
[0094] Step A23, according to the position of the face rectangular frame, the two pre-processed images are cropped and scaled to a preset second size to obtain two side face images of the face.
[0095] The two images are preprocessed and then input into the trained face detection model to output the position of the face rectangular frame in each image. The images are then cropped according to the position of the face rectangular frame, the image outside the face rectangular frame is removed, and then scaled to obtain two side face images for subsequent automated processing using the key point detection model.
[0096] Step A30: Based on the two profile face images, a pair of profile face images taken at the same time is input into the trained key point detection model, and an output result is obtained. The output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the profile face image pair, and the category of each key point.
[0097] This embodiment selects a pair of side face images taken at the same time. The side face image pair introduces facial features of a person at different angles, provides data features for the key point detection model, and the key point detection model processes the side face image pair. It can detect the key points of the nose tip and the ear contour in one side face image at the same time, without the need to extract them separately, thus simplifying the steps. Step A40, calculates the spatial vector of the line from the ear to the nose tip based on the normalized coordinate values of the nose tip key point and N ear contour key points and the position of the face rectangular frame. This step can specifically include steps A41-A44:
[0098] Step A41, according to the position of the face rectangle, respectively map the normalized coordinate values of the nose tip key point and the N ear contour key points back to the original scale when shooting, and obtain the mapped coordinate values of the nose tip key point and the N ear contour key points. For details on the mapping method, see Chapter 3 of Part 0 (Basic Knowledge) of "Multiple View Geometry in Computer Vision".
[0099] Step A42, based on the internal and external parameters of the binocular camera device, and the mapped coordinate values of the nose tip key point and the N ear contour key points, the 3D coordinates of the nose tip key point and the N ear contour key points are calculated using a triangulation method. For details of the triangulation method, see Chapter 9, Part 2, 9.1 Triangulation Method of Multiple View Geometry in Computer Vision.
[0100] Step A43, calculate the 3D coordinates of the center of gravity of the ear according to the 3D coordinates of the N ear contour key points. The center of gravity calculation method is detailed in "Two Properties of the Center of Gravity of Polyhedrons" by Zhou Yongguo. Two Properties of the Center of Gravity of Polyhedrons. Mathematics Communications: Teacher Reading, 2003(17):1. DOI:CNKI:SUN:SXTX.0.2003-17-013.
[0101] Step A44, calculating the spatial vector of the line from the ear to the nose tip based on the 3D coordinates of the ear's center of gravity and the 3D coordinates of the nose tip key point.
[0102] This embodiment combines the acquired ear contour and nose tip key points, and uses the ear centroid and the nose tip line to calculate the space vector, which can avoid possible occlusion in single-direction measurement.
[0103] Step A50, go to step A30, until all the side face image pairs in the two channels of side face images are input.
[0104] Step A60, according to the spatial vector obtained in each calculation, the corresponding cervical flexion and extension angle is calculated, and then the neck mobility detection result is obtained. This step may specifically include steps A61-A62:
[0105] Step A61, finding the angle between the space vector obtained by each calculation and the starting vector, and obtaining a plurality of cervical flexion and extension angles.
[0106] The starting vector is a spatial vector calculated before the subject's neck flexion / extension movement begins and the neck is in a neutral position.
[0107] Step A62, based on multiple cervical flexion and extension angles, select the maximum angles of cervical flexion and extension respectively as the neck range of motion detection result.
[0108] Figure 4 Schematic diagram of the main components of the key point detection model in the embodiment of the present invention. Figure 4 As shown, the key point detection model in step A30 may include n layers of attention modules connected in series. The value of n determines the scale of the model and can be preset.
[0109] In the key point detection model, the first layer is the fusion attention module, and is calculated according to the following formula (1):
[0110] (1)
[0111] Wherein, FusionAttention represents the fusion attention feature function; face images X1 and X2 form a face profile image pair; Softmax represents the Softmax function, Q1 and Q2 represent the features of face images X1 and X2 after the first mapping, respectively, and the first mapping is Q1=X1W Q1, Q2=X2W Q2 ; K1 and K2 represent the features of the face images X1 and X2 after the second mapping, respectively. The second mapping is: K1=X1W K1, K2=X2W K2 ; V1=X1W V1 ; V2=X2W V2; W Q1 , W Q2 , W K1 , W K2 , W V1 and W V2 Both represent randomly initialized trainable parameters, and are the transpose of K1 and K2 respectively; d represents the dimension of Q1 and Q2.
[0112] The n-2 layers in the middle (from the 2nd layer to the n-1th layer) are all fusion attention modules, and are calculated according to the following formula (2):
[0113] (2)
[0114] Among them, the output value of the t-1 layer is Y t , 2≤t≤n-1, Y t Divide equally into Y channels t1 and Y t2 , and Y t1 and Y t2 Same dimensions; Q t1 =Y t1 W Qt1 ;Q t2 =Y t2 W Qt2 , K t1 =V t1 =Y t1 W Kt1 ; K t2 =V t2 =Y t2 W Kt2 ;W Qt1、 W Qt2、 W Kt1 and W Kt2 All represent randomly initialized trainable parameters; and K t1 and K t2 The transpose of d t Indicates Q t1 and Q t2 Dimensions;
[0115] The nth layer is the self-attention module and is calculated according to the following formulas (3) and (4):
[0116] (3)
[0117] (4)
[0118] Among them, Y n represents the output value of the n-1th layer (i.e. the input value of the nth layer); Q n =Y n W Qn , K n =V n =Y n W Kn , W Qn and W Kn Both represent randomly initialized trainable parameters; d n Indicates Qn dimension, Y represents the output value of the nth layer; MLP represents the linear mapping layer; Y is an M ×1 vector, M=2+(N+1)×4; N is the number of ear contour key points (50 in this embodiment), N+1 represents the sum of the number of ear contour key points and nose tip key points; Y is split sequentially according to the first dimension, and the first two digits are the classification results , the remaining M-2 bits are the coordinates of the two sets of 2D key points detected on the face images X1 and X2; It is a 2×1 vector, representing the classification result of whether it is a profile face or not; the coordinates of a set of 2D key points detected on each image are all P×1 vectors after splicing, P=(N+1) ×2.
[0119] In this embodiment, the output result Y of the nth layer is a vector with a shape of 206x1. Y is split sequentially according to the first dimension (206), and the first two digits are the classification results. , is a 2x1 vector, representing the classification result of whether it is a side face; in step A30, a set of key points are extracted from face image 1 and image 2 respectively, and each set of key point coordinates (51) are concatenated in sequence to form a vector of shape 102x1, representing the image pixel coordinates of the key points in the image. The network outputs the normalized coordinates of the key points of the two images and the classification result of whether it is a side face. Among them, face images 1 and 2 are images taken by two cameras at the same time, together forming a pair of side face images.
[0120] In this embodiment, a new fusion attention module FusionAttention is proposed, which bidirectionally fuses the features of two images from different angles, so that the model can focus on the key point information of the face in the two images at the same time, thereby improving the detection accuracy.
[0121] Figure 5 Schematic diagram of the main steps of the key point detection model training method in an embodiment of the present invention. Figure 5 As shown, the training method of the key point detection model composed of n layers of attention modules connected in series may include steps B10-B50:
[0122] Step B10, using a binocular camera device to capture frontal, left-side and right-side face images of persons of different ages, genders and heights to obtain an original image set; when shooting, the persons stand at a designated standing point with their frontal, left-side or right-side faces facing the binocular camera device.
[0123] Step B20, cropping and labeling the face part of each image in the original image set to generate a key point detection model training set.
[0124] The key point detection model training set includes: a preset number of face profile image pairs and corresponding annotation data. The annotation data includes: whether each image is a profile image, the normalized pixel coordinates of the nose tip key point and N ear contour key points, and the category of each key point.
[0125] The specific labeling method in this embodiment is: labeling whether the face image is a profile face (profile face is 1, non-profile face is 0); labeling a single key point of the nose tip of the profile face image, recorded as (x0, y0), and labeling 50 dense key points of the ear contour, and the labeling truth value is recorded as { (x1, y1) ,…(x 50 ,y 50 )} , dense key points are evenly distributed on the outer edge contour of the ear, and the pixel coordinates of the nose tip and 50 ear outer edge contours in all side face images are recorded (normalized by the length and width of the face image), and the category of the annotation points is recorded (nose tip 0, ear contour 1).
[0126] Step B30, based on the key point detection model training set, input a pair of side-face images of a human face into the key point detection model, and obtain the detection result.
[0127] The detection results include: the normalized coordinate prediction values of the nose tip key point and N ear contour key points corresponding to each image in the face profile image pair, the category of each key point, and the classification result of whether it is a profile face.
[0128] Step B40, calculating the loss function according to the labeled data corresponding to the face profile image pair and the detection result.
[0129] Specifically, the loss function is calculated according to formulas (5)-(7):
[0130] (5)
[0131] in,
[0132] (6)
[0133] (7)
[0134] L c is the classification loss of whether it is a side face, L o is the spatial loss, y is the true value of whether it is a side face, is the predicted value of whether it is a side face, N is the number of key points of the ear contour, (x i ,y i ) is the true value coordinate of the i-th key point, is the remapped coordinate of the i-th key point.
[0135] The remap coordinate generation method is:
[0136] (a) According to the coordinate prediction values output by the key point detection model, the 3D key point coordinates corresponding to each pair of 2D key points are obtained by triangulation. (b) According to the following formula (8), each 3D key point coordinate is mapped back to each image in the face profile image pair to obtain the remapped coordinates of each 2D key point:
[0137] (8)
[0138] in, is the 3D coordinate of the i-th key point mapped back to the 2D homogeneous coordinate of the image, K is the camera intrinsic parameter, is the 3D homogeneous coordinate of the i-th key point in the camera coordinate system.
[0139] Step B50, go to step B30, and stop training until the value of the loss function no longer decreases or reaches a preset number of training times, thereby obtaining a trained key point detection model.
[0140] In this embodiment, the classification loss and the spatial loss are used to train the model in synergy, and the effect is better than the prior art in which only the key point pixel spatial loss is used for training.
[0141] In an optional embodiment, step B20 may specifically include steps B21-B28:
[0142] Step B21, scaling each image in the original image set to a preset first size to generate a first image set.
[0143] Step B22: annotate each image in the first image set with a face rectangle.
[0144] Step B23, cropping each image in the first image set according to the marked face rectangle frame and scaling it to a preset second size to generate a second image set.
[0145] Step B24, marking whether each image in the second image set is a profile face.
[0146] Step B25: for each side face image in the second image set, mark the nose tip key point and N ear contour key points.
[0147] Step B26, recording the pixel coordinates of the nose tip key point and N ear contour key points, and normalizing the recorded pixel coordinates according to a preset second size.
[0148] Step B27, labeling the categories of the nose tip key point and N ear contour key points.
[0149] Step B28, generating a key point detection model training set based on the face profile image pairs and corresponding annotation data in the second image set.
[0150] Among them, N is a preset value, and N ear contour key points are evenly distributed on the outer edge contour of the ear; the category is nose tip or ear contour.
[0151] In another optional embodiment, the face detection model in step A22 is based on the YOLOv5s model. Figure 6 FIG. 1 is a schematic diagram of the main steps of the face detection model training method in an embodiment of the present invention. Figure 6 As shown, the training method of the face detection model of this embodiment includes steps C10-C20:
[0152] Step C10, for each image in the first image set, the pixel coordinates of the upper left corner and the lower right corner of the face rectangular frame are recorded according to the marked face rectangular frame, and the recorded pixel coordinates are normalized according to a preset first size to generate a face detection model training set;
[0153] Step C20, training the face detection model based on the face detection model training set to obtain a trained face detection model.
[0154] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.
[0155] Based on the same technical concept as the method embodiment, the present invention also provides an embodiment of a measurement system. Figure 7 FIG. 1 is a schematic diagram of the main components of an embodiment of a system for measuring the range of motion of neck flexion and extension in the present invention. Figure 7 As shown, the measurement system of this embodiment includes: an image acquisition module 10, a side face image acquisition module 20, a key point detection module 30, a vector calculation module 40, a control module 50 and an activity calculation module 60.
[0156] The image acquisition module 10 is used to use a binocular camera device to capture two images of the subject when the subject performs neck flexion / extension exercise; when shooting, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device;
[0157] The side face image acquisition module 20 is used to acquire the position of the face rectangular frame in each image according to the two images taken, and then acquire two side face images of the face. The module may specifically include a preprocessing unit, a face detection unit and a cropping unit.
[0158] Among them, the preprocessing unit is used to uniformly scale the two images to a preset first size to obtain two preprocessed images; the face detection unit is used to input the two preprocessed images into the trained face detection model respectively, and output the position of the face rectangular frame in each image; the cropping unit is used to crop the two preprocessed images respectively according to the position of the face rectangular frame and scale them to a preset second size to obtain two side face images.
[0159] The key point detection module 30 is used to input a pair of side face images of a person taken at the same time into the trained key point detection model based on the two side face images, and obtain an output result; the output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the side face image pair and the category of each key point;
[0160] The vector calculation module 40 is used to calculate the space vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame;
[0161] The control module 50 is used to repeatedly call the key point detection module and the vector calculation module until all the side face image pairs in the two paths of side face images are input;
[0162] The activity calculation module 60 calculates the corresponding cervical flexion and extension angle according to the spatial vector obtained in each calculation, and then obtains the neck activity detection result. The module may specifically include: an angle calculation unit and a detection result output unit.
[0163] Among them, the angle calculation unit is used to find the angle between the spatial vector calculated each time and the starting vector, so as to obtain multiple cervical flexion and extension angles; the starting vector is the spatial vector calculated when the neck of the subject is in a neutral position when the neck flexion / extension movement begins; the detection result output unit uses multiple cervical flexion and extension angles to select the maximum angles of neck flexion and extension as the neck mobility detection result and output them.
[0164] The neck flexion and extension range of motion measurement system provided by the present invention does not require a human body to contact the device during the measurement process, thereby simplifying the maintenance of the device; during the neck flexion / extension activity of the tested person, the ears and nose tip are not easily blocked when the face is in profile, making the shooting process convenient and easier to cooperate, and no professional guidance is required, which effectively reduces the measurement error and improves the accuracy of the final calculation result, and the whole process from shooting to obtaining the calculation result can be automated; the key point detection model is used to simultaneously detect the key points of the nose tip and the auricle, and the detection steps are also reduced.
[0165] The key point detection module 30 of the neck flexion and extension range of motion measurement system has a built-in key point detection model, which uses the synergistic effect of classification loss and spatial loss to train the model, and its effect is better than the existing technology of only using key point pixel space loss for training. The fusion attention module therein performs bidirectional fusion of the two-way image features at different angles, so that the model can simultaneously focus on the key point information of the face in the two-way image, further improving the detection accuracy.
[0166] The present invention also provides an embodiment of a computer-readable storage device. The storage device of this embodiment stores a computer program that can be loaded by a processor and execute the above method.
[0167] The computer-readable storage device may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0168] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0169] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A method for measuring the range of motion of neck flexion and extension, characterized in that: The method comprises: Using a binocular camera device to capture two images of the subject performing neck flexion / extension exercise; during the capture, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device; Obtaining the position of the face rectangular frame in each image according to the two images, and then obtaining two side face images of the face; Based on the two sets of side-face images of the human face, a pair of side-face images of the human face taken at the same time is input into the trained key point detection model, and an output result is obtained; the output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the side-face image pair of the human face, and the category of each key point; Calculate the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame; Repeat the steps of inputting the pair of side-face images of a person, obtaining the output result, and calculating the space vector until all the pair of side-face images of a person in the two paths of side-face images of a person are input; The corresponding cervical flexion and extension angles are calculated according to the spatial vector obtained in each calculation, thereby obtaining the neck range of motion detection result.
2. The method for measuring the range of motion of neck flexion and extension according to claim 1, characterized in that: The step of "calculating the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame" includes: According to the position of the face rectangular frame, respectively mapping the normalized coordinate values of the nose tip key point and the N ear contour key points back to the original scale when shooting, and obtaining the mapped coordinate values of the nose tip key point and the N ear contour key points; Based on the internal parameters and external parameters of the binocular camera device, and the mapped coordinate values of the nose tip key point and the N ear contour key points, the 3D coordinates of the nose tip key point and the N ear contour key points are calculated according to a triangulation method; Calculate the 3D coordinates of the center of gravity of the ear according to the 3D coordinates of the N ear contour key points; Calculate the spatial vector of the line from the ear to the nose tip according to the 3D coordinates of the center of gravity of the ear and the 3D coordinates of the key point of the nose tip; The step of "determining the corresponding cervical flexion and extension angle according to the spatial vector obtained by each calculation, and then obtaining the neck range of motion detection result" includes: The angle between the spatial vector calculated each time and the starting vector is calculated to obtain a plurality of cervical flexion and extension angles; the starting vector is the spatial vector calculated before the subject's neck flexion / extension movement begins and the neck is in a neutral position; Based on the multiple cervical flexion and extension angles, the maximum angles of cervical flexion and extension are respectively selected as the cervical range of motion detection results.
3. The method for measuring the range of motion of neck flexion and extension according to any one of claims 1 to 2, characterized in that: The key point detection model includes n layers of attention modules connected in series; The first layer is the fusion attention module and is calculated as follows: Wherein, FusionAttention represents the fusion attention feature function; the face images X1 and X2 constitute the face side face image pair; Softmax represents the Softmax function, Q1 and Q2 represent the features of the face images X1 and X2 after the first mapping, respectively, Q1=X1W Q1 , Q2=X2W Q2 ; K1 and K2 represent the features of face images X1 and X2 after the second mapping, K1=X1W K1 , K2=X2W K2 ; V1=X1W V1 ; V2=X2W V2 ; W Q1 , W Q2 , W K1 , W K2 , W V1 and W V2 Both represent randomly initialized trainable parameters, and are the transpose of K1 and K2 respectively, and d represents the dimensions of Q1 and Q2; The 2nd to n-1th layers are all fusion attention modules, and are calculated according to the following formula: Among them, the output value of the t-1 layer is Y t , 2≤t≤n-1, Y t Divide equally into Y channels t1 and Y t2 , and Y t1 and Y t2 Same dimensions; Q t1 =Y t1 W Qt1 ;Q t2 =Y t2 W Qt2 , K t1 =V t1 =Y t1 W Kt1 ; K t2 =V t2 =Y t2 W Kt2 ; W Qt1 , W Qt2 , W Kt1 and W K t2 represents randomly initialized trainable parameters; and K t1 and K t2 The transpose of t Indicates Q t1 and Q t2 Dimensions; The nth layer is the self-attention module and is calculated as follows: Y=MLP(Attention(Y)), where Y n Represents the output value of the n-1th layer; Q n =Y n W Qn , K n =V n =Y n W Kn , W Qn and W Kn Both represent randomly initialized trainable parameters; d n Indicates Q n dimension, Y represents the output value of the nth layer; MLP represents the linear mapping layer; Y is an M×1 vector, M=2+(N+1)×4; Y is split sequentially according to the first dimension, and the first two digits are the classification results The remaining M-2 bits are the coordinates of the two sets of 2D key points detected on the face images X1 and X2; It is a 2×1 vector, representing the classification result of whether it is a profile face or not; the coordinates of a set of 2D key points detected on each image are all P×1 vectors after splicing, P=(N+1)×2.
4. The method for measuring the range of motion of neck flexion and extension according to claim 3, characterized in that: The training method of the key point detection model includes: The binocular camera device is used to shoot frontal face, left side face and right side face images of persons of different ages, genders and heights to obtain an original image set; when shooting, the persons stand at a designated standing point with the frontal face, left side face or right side face facing the binocular camera device; Cropping and labeling the face portion of each image in the original image set to generate a key point detection model training set; Based on the key point detection model training set, a pair of side face images of a human face is input into the key point detection model, and a detection result is obtained; the detection result includes: the normalized coordinate prediction values of the nose tip key point and N ear contour key points corresponding to each image in the pair of side face images, the category of each key point, and the classification result of whether it is a side face; Calculating a loss function based on the labeled data corresponding to the pair of side-face images and the detection result; Repeat the steps of inputting the face profile image pair, obtaining the detection result, and calculating the loss function until the value of the loss function no longer decreases or the training is stopped when a preset number of training times is reached, thereby obtaining the trained key point detection model; in, The key point detection model training set includes: a preset number of face profile image pairs and corresponding annotation data.
5. The method for measuring the range of motion of neck flexion and extension according to claim 4, characterized in that: The step of "obtaining the position of the face rectangular frame in each image according to the two images, and then obtaining two side face images of the face" includes: The two images are uniformly scaled to a preset first size to obtain two pre-processed images; Input the two pre-processed images into the trained face detection model respectively, and output the position of the face rectangle frame in each image; According to the position of the face rectangular frame, the two pre-processed images are respectively cropped and scaled to a preset second size to obtain two side face images of the face.
6. The method for measuring the range of motion of neck flexion and extension according to claim 5, characterized in that: The annotation data includes: an identification of whether each image is a profile, normalized pixel coordinates of the nose tip key point and N ear contour key points, and the category of each key point; The step of "cropping and labeling the face portion of each image in the original image set to generate a key point detection model training set" includes: Scaling each image in the original image set to a preset first size to generate a first image set; Marking each image in the first image set with a face rectangle frame; For each image in the first image set, cropping is performed according to the marked face rectangular frame, and scaling the image to a preset second size to generate a second image set; marking whether each image in the second image set is a profile face; For each side face image in the second image set, marking the nose tip key point and the N ear contour key points; Recording the pixel coordinates of the nose tip key point and the N ear contour key points, and normalizing the recorded pixel coordinates according to the preset second size; Marking the categories of the nose tip key point and the N ear contour key points; Based on the human face profile image pairs in the second image set and the corresponding annotation data, generating the key point detection model training set; in, N is a preset value, and the N ear contour key points are evenly distributed on the outer edge contour of the ear; The categories are nose tip or ear outline.
7. The method for measuring the range of motion of neck flexion and extension according to claim 4, characterized in that: The loss function is: L = L c +yL o in, L c is the classification loss of whether it is a side face, L o is the spatial loss, y is the true value of whether it is a side face, is the predicted value of whether it is a side face, N is the number of key points of the ear contour, (x i ,y i ) is the true value coordinate of the i-th key point, is the remapped coordinate of the i-th key point.
8. The method for measuring the range of motion of neck flexion and extension according to claim 7, characterized in that: The remapping coordinate generation method is: According to the coordinate prediction values output by the key point detection model, the 3D key point coordinates corresponding to each pair of 2D key points are obtained by using a triangulation method; The 3D key point coordinates are mapped back to each image in the face profile image pair according to the following formula to obtain the remapped coordinates of each 2D key point: in, The 3D coordinates of the i-th key point are mapped back to the 2D homogeneous coordinates of the image, K is the camera internal parameter, [X i , Y i , Z i , 1] T is the 3D homogeneous coordinate of the i-th key point in the camera coordinate system.
9. The method for measuring the range of motion of neck flexion and extension according to claim 6, characterized in that: The face detection model is based on the YOLOv5s model; The training method of the face detection model includes: For each image in the first image set, the pixel coordinates of the upper left corner and the lower right corner of the face rectangular frame are recorded according to the marked face rectangular frame, and the recorded pixel coordinates are normalized according to the preset first size to generate a face detection model training set; Based on the face detection model training set, the face detection model is trained to obtain the trained face detection model.
10. A system for measuring the range of motion of neck flexion and extension, characterized in that: The system comprises: An image acquisition module is used to use a binocular camera device to capture two images of the subject performing neck flexion / extension exercise; during the shooting, the subject stands at a designated standing point with the left side or right side of the face facing the binocular camera device; A side face image acquisition module, used to acquire the position of the face rectangular frame in each image according to the two images, and then acquire two side face images of the face; A key point detection module is used to input a pair of side face images of a person taken at the same time into the trained key point detection model based on the two side face images, and obtain an output result; the output result includes: the normalized coordinate values of the nose tip key point and N ear contour key points corresponding to each image in the side face image pair and the category of each key point; A vector calculation module, used for calculating the spatial vector of the line from the ear to the nose tip according to the normalized coordinate values of the nose tip key point and the N ear contour key points and the position of the face rectangular frame; A control module is used to repeatedly call the key point detection module and the vector calculation module until all the side face images in the two sides of the side face images are input; an activity calculation module is used to calculate the corresponding cervical flexion and extension angle according to the spatial vector obtained by each calculation, so as to obtain the neck activity detection result.
11. A computer-readable storage device, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 9.
