A method and system for detecting the spinous process line of the human back

Abstract

The present invention relates to a method and system for detecting the spinous process line of the human back, belonging to the technical field of image processing. The method for detecting the spinous process line of the human back provided by the present invention, after obtaining a color image and a depth image, optimizes the depth image based on the obtained color image to obtain an optimized depth image, and then, the detection result of the spinous process line of the human back can be accurately obtained according to the color image and the optimized depth image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting spinous process lines on the back of a human body. Background Art

[0002] With the prevalence and harmfulness of adolescent scoliosis, spinous process line detection has become a topic of concern in the medical field. Computer algorithm-assisted screening of scoliosis can effectively prevent this problem. At present, the degree of scoliosis needs to be detected by taking X-rays in the hospital. This method is transparent and accurate, but it is time-consuming and harmful to patients. Therefore, whether the spinous process line can be detected by taking pictures has become a topic of concern. In addition, depth cameras have realized the application of many new real-time technologies in computer graphics, computer vision and other fields, and have made great progress in improving image quality and resolution. Although the current depth camera is still affected by severe sensor noise, resulting in only rough geometric figures per frame, the depth can be optimized by fusing color images and depth images to reduce the impact of noise and information loss. Therefore, the information that can be used includes not only color images, but also depth images. At present, due to the labor cost of collecting experimental data, the samples are not sufficient and the distribution is not wide enough. If the neural network is used, it is easy to cause overfitting. Therefore, how to establish better spinous process line constraints and obtain more accurate spinous process point results through traditional optimization algorithms is a technical problem that needs to be urgently solved in the field of human back spinous process line detection. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a method and system for detecting spinous process lines on the back of the human body.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for detecting spinous process lines on the back of a human body, comprising:

[0006] Acquire a color image and a depth image of the back of a human body to be detected;

[0007] Optimizing the depth image based on the color image to obtain an optimized depth image;

[0008] A human back spinous process line detection result is obtained according to the color image and the optimized depth image.

[0009] Preferably, the optimizing the depth image based on the color image to obtain the optimized depth image specifically includes:

[0010] Optimize the depth image using the shading information of the color image to obtain an optimized depth image.

[0011] Preferably, the process of optimizing the depth image using the shading information of the color image to obtain an optimized depth image specifically includes:

[0012] Construct a first energy function based on constraint terms; the constraint terms include: shading gradient constraint term, smooth constraint term, and depth constraint term;

[0013] Minimize the first energy function to obtain the optimized depth image.

[0014] Preferably, the construction process of the shading gradient constraint term includes:

[0015] Align the color image and the depth image to obtain an aligned image;

[0016] Based on the depth value, perform foreground segmentation on the aligned image to obtain a mark image of the human body;

[0017] Determine the spherical harmonic light coefficients and albedo of the mark image;

[0018] Determine the three-dimensional coordinate points of the depth image in the camera coordinate system;

[0019] Determine the surface normal vector according to the three-dimensional coordinate points;

[0020] Generate a rendered image according to the spherical harmonic light coefficients, the albedo, and the surface normal vector;

[0021] Determine the shading gradient constraint term according to the rendered image and the color image;

[0022] Preferably, the construction process of the shading gradient constraint term includes:

[0023] Determine the smooth constraint term according to the three-dimensional coordinate points.

[0024] Preferably, the depth constraint term is the loss value between the optimized depth image and the depth image.

[0025] Preferably, the first energy function is:

[0026]

[0027] Among them, E(d) is the first energy function, (i, j) is the pixel point in the foreground area of the mark image, w g is the weighting coefficient of the shading gradient constraint term, E g(i,j) is the shading gradient constraint term, w s is the weighting coefficient of the smoothing constraint term, E s (i,j) is the smoothing constraint term, w p is the weighting coefficient of the depth constraint term, E p (i,j) is the depth constraint term.

[0028] Preferably, obtaining the detection result of the spinal process line on the human back according to the color image and the optimized depth image specifically includes:

[0029] Constructing a second energy function;

[0030] Using the color image and the optimized depth image as the input of the second energy function to obtain the detection result of the spinal process line on the human back.

[0031] Preferably, the second energy function is:

[0032] E total = w d E d + w c E c + w q E q + w f E f ;

[0033] wherein, E total is the second energy function, w d is the weighting coefficient of the constraint term of the spinal process line in terms of depth value, E d is the constraint term of the spinal process line in terms of depth value, w c is the weighting coefficient of the constraint term of the spinal process line in terms of the contour information of the human back, E c is the constraint term of the spinal process line in terms of the contour information of the human back, w q is the weighting coefficient of the constraint term of the spinal process line in terms of continuity, E q is the constraint term of the spinal process line in terms of continuity, w f is the weighting coefficient of the constraint term of the starting point and the ending point of the spinal process line, E f is the constraint term of the starting point and the ending point of the spinal process line.

[0034] According to the specific embodiments provided by the present invention, the following technical effects are disclosed:

[0035] The method for detecting the spinal process line on the human back provided by the present invention optimizes the depth image based on the acquired color image to obtain the optimized depth image, and then, according to the color image and the optimized depth image, the detection result of the spinal process line on the human back can be accurately obtained.

[0036] Corresponding to the above-provided method for detecting the spinous process line of the human back, the present invention provides a system for detecting the spinous process line of the human back, which system includes: a processor and a memory;

[0037] The processor is connected to the memory; the memory is used for storing computer software programs; the computer software programs are used for implementing the above-provided method for detecting the spinous process line of the human back; the processor is used for executing the computer software programs.

[0038] Since the technical effects achieved by the system for detecting the spinous process line of the human back provided by the present invention are the same as those achieved by the above-provided method for detecting the spinous process line of the human back, no further elaboration will be provided herein. Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0040] Figure 1 It is a flowchart of the method for detecting the spinous process line of the human back provided by the present invention;

[0041] Figure 2 It is a flowchart of the depth image optimization provided by the embodiments of the present invention;

[0042] Figure 3 It is a flowchart of the detection and processing of the spinous process line of the human back provided by the embodiments of the present invention;

[0043] Figure 4 It is a schematic structural diagram of the system for detecting the spinous process line of the human back provided by the present invention. Detailed Embodiments

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, rather than all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0045] The objective of the present invention is to provide a method and a system for detecting the spinous process line of the human back, so as to obtain more accurate spinous process point results.

[0046] To make the above objectives, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0047] As Figure 1 shown, a method for detecting the spinous process line of the human back provided by the present invention includes:

[0048] Step 100: Obtain a color image and a depth image of the back of the human body to be detected. For example, use a Kinect device to take a picture, control the target object to be within the optimal shooting range of the device, and then collect the color image and the depth image of the back of the human body to be detected.

[0049] Step 101: Optimize the depth image based on the color image to obtain an optimized depth image.

[0050] In this step, the aligned color image and depth image are used as inputs, and the optimized depth image is output. Specifically:

[0051] Step 1010: Preprocessing stage: Align the color image and the depth image, and perform foreground segmentation based on the depth value to extract the mask image of the human body.

[0052] Step 1011: Calculate the illumination of the scene and the albedo of the target object in the mask image: Since calculating the two simultaneously is an ill-posed problem, the present invention simplifies the scene, makes assumptions of a Lambertian model and a distant light source, only considers diffuse reflection, ignores specular highlights, and models the illumination using the first three-order spherical harmonics. The normal vector can be calculated from the depth image. Thus, the rendered image can be represented as:

[0053]

[0054] where (i, j) is the pixel point in the foreground area of the mask image, B(i, j) is the rendered image, k(i, j) is the albedo, l t is the spherical harmonic coefficient, H t (n(i, j)) is the t-th third-order spherical harmonic basis function, which can be represented as H(n) = (1, n y , n z , n x , n x n y , n y n z , 3n z 2 -1, n x n z , n x 2 -n y 2 ), n is the normal vector, and n x 、n y 、n z represent the components of the normal vector on the x, y, and z axes, respectively.

[0055] At this stage, the present invention first initializes the albedo of the scene, assigns a unified value to it using prior knowledge, which is set to 1 here, and then calculates the albedo image by dividing the pixel values of the color image by the illumination part. Thus, the spherical harmonic coefficient l t can be calculated by minimizing the error sum of the pixel values of the rendered image B(i, j) and the input grayscale image I(i, j). The loss can be characterized as: E(i, j) = ∑(B(i, j) - I(i, j)) 2 , that is, solve the following matrix form equation:

[0056]

[0057] where, H i (*) is the i-th order spherical harmonic basis function, n i is the normal vector of the i-th pixel, with a total of m pixels, l is a 9-dimensional spherical harmonic coefficient array, I is an m-dimensional color image pixel value array, and the spherical harmonic coefficient l can be obtained by the least squares method: l = (A T A) - 1 A T I, Then through the formula: calculate the albedo, where m represents the m-th dimension.

[0058] Step 1012: Optimization of the depth image. At this stage, the present invention directly relates the depth value and the normal vector. The three-dimensional coordinate point p(i, j) of a point D(i, j) in the depth image in the camera coordinate system can be obtained according to the perspective projection formula as: where (u x , u y ) is the principal point of the camera, f x , f y are the focal lengths in the x and y directions respectively. The unnormalized surface normal vector at the point (i, j) can be calculated from the adjacent three-dimensional point coordinates: n(i, j) = (p(i, j - 1) - p(i, j)) × (p(i - 1, j) - p(i, j)). Considering that the normal vector obtained only from the surrounding two points is not accurate and sometimes even brings large errors. Therefore, in order to improve the accuracy, the method of taking the cross product of the direction vectors of the surrounding four-neighborhood coordinates and the current coordinate point pairwise and then taking the average is used to calculate the normal vector. This not only improves the accuracy of the obtained normal vector but also avoids some problems caused by missing or discontinuous depth points.

[0059] Since the illumination and albedo information of the scene have been obtained in the previous stage, the original rough depth image can be optimized through the shading information of the color image. Specifically:

[0060] The energy function designed for optimizing the depth value (i.e., the first energy function) is as follows:

[0061]

[0062] where, [w g , w s , w p[ represents the corresponding weighting coefficient.

[0063] The meanings of each constraint are introduced separately below:

[0064] E g is the shading gradient constraint term, which is represented as:

[0065]

[0066] The shading gradient constraint term calculates the loss between the gradient of the rendered image B and the color image I, and can add information to the depth value at the details through the color image.

[0067] E s is the smoothness constraint term, which is represented as:

[0068] E s (i, j) = [p(i, j) - w s (p(i - 1, j) + p(i, j - 1) + p(i + 1, j) + p(i, j + 1))] 2

[0069] where, w s is the weighting coefficient, and preferably 0.25 is adopted in the present invention. The smoothness constraint term calculates the loss between the three-dimensional coordinate p corresponding to the pixel and the surrounding points. Since it is continuous geometry, it prevents the coordinate deviation between adjacent points from being too large.

[0070] E p is the depth constraint term, which is represented as:

[0071] E p (i, j) = [D(i, j) - D′(i, j)] 2

[0072] The depth constraint term calculates the loss between the optimized depth image D′(i, j) and the depth image D(i, j) to prevent the depth value from being too smooth and having a large deviation from the initial value.

[0073] In summary, by minimizing the energy function E(d), an optimized depth image can be obtained, and the simplified process is as Figure 2 shown.

[0074] Step 102: Obtain the detection result of the spinous process line on the human back based on the color image and the optimized depth image.

[0075] In this step, the depth image optimized in the previous step (i.e., Step 101) and the color image obtained in Step 100 are used as inputs, and the output is the spinous process line on the human back. According to the prior knowledge and statistical results of the spinous process line, the following energy function (i.e., the second energy function) is designed:

[0076] E total =w d E d +w c E c +w q E q +w f E f

[0077] Among them, [w d , w c , w q , w f are the corresponding weighting coefficients.

[0078] The meanings of each constraint are introduced separately below. E d represents the constraint term designed for the spinous process line in terms of depth value, as shown below:

[0079]

[0080] Among them, p i represents the coordinates of the i-th spinous process point extracted, is the coordinates of the i-th spinous process point that is the best choice in terms of depth value, and is a constraint set based on the information that the spinous process line is at a local depression on the human back.

[0081] E c represents the constraint term designed for the spinous process line in terms of the human back contour information, as shown below:

[0082]

[0083] Among them, is the coordinates of the i-th spinous process point that is the best choice to meet the human contour information. Based on the symmetry of the human contour, the spinous process line has a certain correlation with the central axis of the human contour, and this constraint term is the loss designed for this contour information.

[0084] E q represents the constraint term designed for the spinous process line in terms of continuity, as shown below:

[0085]

[0086] Among them, p i,y represents the ordinate of the i-th spine point extracted, p i+1,y represents the ordinate of the (i + 1)-th spine point extracted, p i and p i+1 are adjacent coordinates. Based on the continuity of the spine, which is manifested on the spine line as the absolute value of the difference in the ordinates of adjacent spine points should be less than or equal to 1, so a penalty is imposed when the ordinates of adjacent spine points are greater than 1.

[0087] E f represents the constraint terms for the starting point and the ending point of the spine line, as follows:

[0088]

[0089] Among them, p beg represents the coordinates of the starting point of the spine line extracted, represents the reasonable coordinates of the starting point of the spine line, p end represents the coordinates of the ending point of the spine line extracted, represents the reasonable coordinates of the ending point of the spine line. Based on the statistical results of the samples, the starting point and the ending point of the spine line have certain rules in the distribution on the human back, so this constraint term is designed to prevent the detected starting point and ending point from deviating from the reasonable range.

[0090] Finally, by minimizing the energy function E total , the detection result of the spine line on the human back can be obtained, and the simplified process is as Figure 3 shown.

[0091] Corresponding to the above-provided method for detecting the spine line on the human back, the present invention provides a system for detecting the spine line on the human back, as Figure 4 shown, the system includes: a processor 400 and a memory 401.

[0092] The processor 400 is connected to the memory 401. The memory 401 is used to store computer software programs. The computer software programs are used to implement the above-provided method for detecting the spine line on the human back. The processor 400 is used to execute the computer software programs.

[0093] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0094] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. To sum up, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A method for detecting the spinous process line of the human back, characterized in that, it includes: Obtain the color image and depth image of the back of the human body to be detected; Optimize the depth image using the shading information of the color image to obtain an optimized depth image; Obtain the detection result of the spinous process line of the human back according to the color image and the optimized depth image; Among them, the process of optimizing the depth image using the shading information of the color image to obtain an optimized depth image specifically includes: Construct a first energy function based on the constraint term and the weighting coefficient; the constraint term includes: shading gradient constraint term, smooth constraint term and depth constraint term; Minimize the first energy function to obtain the optimized depth image; The process of obtaining the detection result of the spinous process line of the human back according to the color image and the optimized depth image specifically includes: Construct a second energy function based on the constraint term of the spinous process line on the depth value, the constraint term of the spinous process line on the contour information of the human back, the constraint term of the spinous process line on the continuity, the constraint term of the starting point and the ending point of the spinous process line, and the weighting coefficient; Use the color image and the optimized depth image as the input of the second energy function to obtain the detection result of the spinous process line of the human back.

2. The method for detecting the spinous process line of the human back according to claim 1, characterized in that, The construction process of the shading gradient constraint term includes: Align the color image and the depth image to obtain an aligned image; Perform foreground segmentation on the aligned image based on the depth value to obtain the mark image of the human body; Determine the spherical harmonic light coefficient and albedo of the mark image; Determine the three-dimensional coordinate points of the depth image in the camera coordinate system; Determine the surface normal vector according to the three-dimensional coordinate points; Generate a rendered image according to the spherical harmonic light coefficient, the albedo and the surface normal vector; Determine the shading gradient constraint term according to the rendered image and the color image.

3. The method for detecting the spinous process line of the human back according to claim 2, characterized in that, The construction process of the shading gradient constraint term includes: Determine the smooth constraint term according to the three-dimensional coordinate points.

4. The method for detecting the spinous process line of the human back according to claim 2, characterized in that, The depth constraint term is the loss value between the optimized depth image and the depth image.

5. The method for detecting the spinous process line of the human back according to claim 2, characterized in that, The first energy function is: Among them, E(d) is the first energy function, (i, j) is the pixel point in the foreground area of the mark image, and w g is the weighting coefficient of the shading gradient constraint term, E g (i, j) is the shading gradient constraint term, w s is the weighting coefficient of the smoothing constraint term, E s (i, j) is the smoothing constraint term, w p is the weighting coefficient of the depth constraint term, E p (i, j) is the depth constraint term.

6. The method for detecting the spinous process line of the human back according to claim 1, characterized in that, The second energy function is: E total = w d E d + w c E c + w q E q + w f E f ; Among them, E total is the second energy function, w d is the weighting coefficient of the constraint term of the spinous process line in the depth value, E d is the constraint term of the spinous process line in the depth value, w c is the weighting coefficient of the constraint term of the spinous process line in the human back contour information, E c is the constraint term of the spinous process line in the human back contour information, w q is the weighting coefficient of the constraint term of the spinous process line in the continuity, E q is the constraint term of the spinous process line in the continuity, w f is the weighting coefficient of the constraint term of the starting point and the ending point of the spinous process line, E f is the constraint term of the starting point and the ending point of the spinous process line.

7. A system for detecting the spinous process line of the human back, characterized in that, it includes: A processor and a memory; The processor is connected to the memory; the memory is used to store computer software programs; the computer software programs are used to implement the method for detecting the spinous process line of the human back according to any one of claims 1-6; the processor is used to execute the computer software programs.

Images