A fully automatic handheld measuring device for chronic wounds based on an RGB-D camera

Abstract

The application discloses a kind of chronic wound full-automatic handheld measuring equipment based on RGB-D camera, belong to chronic wound measurement technical field;Among them, image acquisition pre-processing module is used to control RGB-D camera to gather a group of color images and depth images of the wound to be measured, and record camera internal parameter, and bilateral filtering processing is carried out to depth image;Wound area segmentation module is used to adjust the size of color image after normalization, input into pre-trained wound segmentation model, obtain the mask of wound area, multiply the corresponding pixel points in depth image and mask, obtain the depth image of wound area;Wound area calculation module uses wound area depth image and camera internal parameter to calculate the area of wound area using the method based on wound area edge or based on wound area surface;Compared with other measuring equipment and method, the degree of automation is high, without human providing label, without placing marker, the measurement accuracy is high, miniaturization and low cost.

Description

Technical Field

[0001] This invention belongs to the field of chronic wound measurement technology, and more specifically, relates to a fully automatic handheld measurement device for chronic wounds based on an RGB-D camera. Background Technology

[0002] Chronic wounds are wounds that cannot recover their full anatomical and physiological function through normal repair processes. They mainly include diabetic foot ulcers, leg venous ulcers, and pressure sores. With the aging population, the number of patients with chronic diseases and chronic wounds is increasing annually. Because chronic wounds have a long healing period, patients often require long-term hospitalization, which consumes significant medical resources and places a heavy burden on both the healthcare system and patients' families. Furthermore, the healing of chronic wounds is difficult, requiring doctors to adjust treatment methods promptly based on the wound's condition to shorten the healing time. In wound assessment, changes in physical parameters (area) directly reflect the wound healing process, making accurate measurement of these parameters crucial for wound care. However, current clinical practice primarily uses rulers or transparent markers for contact-based measurements, which are subject to subjective influence and can cause pain and increase the risk of infection. Non-contact measurement devices are often expensive and difficult to use. Therefore, an efficient and accurate automated chronic wound measurement device can effectively solve clinical wound assessment problems and improve the efficiency of chronic wound care and treatment.

[0003] Currently, some commercially available non-contact wound measurement systems rely on laser 3D scanning for high accuracy, but the equipment is very expensive. To address this, some systems acquire 2D color images of the wound area and place markers within them to treat the wound area as a plane for analysis, thus determining its area. However, this method is only suitable for relatively smooth wounds; its accuracy is low when the wound area is uneven. Furthermore, this system requires markers to determine the actual size of the wound, making the operation complex. Therefore, to improve the accuracy of the above methods, when the wound area is uneven, researchers often need to acquire wound videos and perform 3D reconstruction on the 2D color image sequence from the acquired videos to obtain a 3D model of the wound area. Then, based on the markers, the surface area of ​​the 3D model is calculated to obtain the wound area. However, this method requires stable acquisition of 2D color image sequences, placing high demands on the user's acquisition stability and the anti-shake performance of the measurement equipment. It also requires markers to determine the actual size of the wound, making the operation complex, inflexible, and less user-friendly. Summary of the Invention

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a chronic wound full-automatic handheld measurement device based on an RGB-D camera, to solve the technical problem that the prior art cannot achieve high-precision and easy-to-use chronic wound measurement at a low cost.

[0005] In order to achieve the above-mentioned purpose, the present application provides a chronic wound full-automatic handheld measurement device based on an RGB-D camera, comprising:

[0006] An image acquisition pre-processing module is configured to control the RGB-D camera to acquire a set of color images and depth images of a wound to be measured, and record camera intrinsic parameters; perform bilateral filtering processing on the depth images, and output the acquired color images and the processed depth images to a wound area segmentation module, and output the camera intrinsic parameters to a wound area calculation module;

[0007] The wound area segmentation module is configured to input the color images after size normalization adjustment to a pre-trained wound segmentation model, to obtain a mask of the wound area, multiply corresponding pixel points in the depth images and the mask, to obtain a wound area depth image, and output the wound area depth image to the wound area calculation module;

[0008] The wound area calculation module is configured to calculate the area of the wound area based on the wound area depth image and the camera intrinsic parameters in a first mode or a second mode.

[0009] The first mode includes: calculating 3D point clouds of the wound area edge based on the wound area depth image and the camera intrinsic parameters, fitting the 3D point clouds of the wound area edge to the same plane to obtain a spatial polygon representation of the wound area, calculating the area of the spatial polygon, and obtaining the area of the wound area.

[0010] The second mode includes: calculating 3D point clouds of the wound area and the wound area edge based on the wound area depth image and the camera intrinsic parameters; reconstructing the 3D point clouds of the wound area into a surface, removing 3D points in the surface that are outside the wound area edge, and obtaining a complete wound 3D model after smoothing and denoising; calculating the surface area of the wound 3D model, and obtaining the area of the wound area.

[0011] Further preferably, the wound area calculation module first calculates the area of the wound area in the first mode, judges whether the area of the wound area obtained in the first mode is greater than a preset area, if yes, calculates the area of the wound area in the second mode as the final area measurement result, otherwise, takes the area of the wound area obtained in the first mode as the final area measurement result.

[0012] Further preferably, the wound area calculation module maps the pixel points in the wound area edge or the wound area in the wound area depth image through the following mapping formula to obtain corresponding 3D point clouds:

[0013]

[0014]

[0015]

[0016] wherein (x, y, z) is the 3D coordinate of a pixel point p on the wound area edge or in the wound area in the 3D point cloud of the wound area depth image; d is the pixel value of the pixel point p in the wound area depth image; depth scale is a fixed parameter in the RGB-D camera for converting the depth pixel value into the actual distance; (u, v) is the coordinate of the pixel point p in the wound area depth image; cx, cy, fx and fy are all camera intrinsic parameters.

[0017] Further preferably, in the above-mentioned mode one, the expression of the plane f is obtained by minimizing the distance between each 3D point in the 3D point cloud of the wound area edge and the plane f; and the spatial polygon representation of the wound area is obtained by orthogonally projecting all the 3D points in the 3D point cloud of the wound area edge onto the plane f.

[0018] Further preferably, the area of the wound area calculated by the above-mentioned mode one is:

[0019]

[0020] wherein n is the unit normal vector of the projection plane; cos(n, z) is the cosine of the angle between n and the coordinate axis z; N is the number of 3D points in the 3D point cloud of the wound area edge; (x i ,y i ,z i ) is the coordinate of the i-th 3D point in the 3D point cloud of the wound area edge; cos(n, x) is the cosine of the angle between n and the coordinate axis x; and cos(n, y) is the cosine of the angle between n and the coordinate axis y.

[0021] Further preferably, the above-mentioned mode two uses the Poisson surface reconstruction algorithm to reconstruct the 3D point cloud of the wound area into a surface.

[0022] Further preferably, in the above-mentioned mode two, the 3D points on the surface and the 3D point cloud of the wound area edge are projected onto the xy plane under the 3D point cloud coordinate system to obtain the 2D points corresponding to the 3D points on the surface in the xy plane and the polygon area determined by the wound area edge in the xy plane, and the 3D points on the surface corresponding to the 2D points outside the polygon area are removed, so as to remove the 3D points on the surface outside the wound area edge.

[0023] Further preferably, the wound segmentation model comprises:

[0024] an encoding network configured to extract image features of the color image at different resolutions and output to a feature enhancement network;

[0025] the feature enhancement network is configured to enhance the image features at different resolutions on different receptive fields;

[0026] a feature fusion network configured to fuse the enhanced image features at different resolutions to obtain fused features;

[0027] an output decoding network configured to up-sample the fused features to the original resolution of the color image to obtain a mask of the wound area.

[0028] Further preferably, the encoding network is a HarDNet model, which includes a plurality of cascaded HarDNet units and a pooling layer arranged between each two adjacent HarDNet units; the HarDNet unit includes a cascaded HarDBlock and a convolution layer.

[0029] After the color image is processed by the HarDNet model, three image features at different resolutions are output from the last three HarDNet units.

[0030] The feature enhancement network includes three RFB modules, which are connected one by one corresponding to the output ends of the last three HarDNet units of the HarDNet model, and respectively enhance the image features at different resolutions.

[0031] The output decoding network includes three cascaded decoding units and a convolution layer with a convolution kernel size of 1*1; the decoding unit includes a cascaded up-sampling layer and a convolution layer.

[0032] Further preferably, the handheld device further includes an embedded development board, an RGB-D camera, a display module and a power supply module; wherein the image acquisition and preprocessing module, the wound area segmentation module and the wound area calculation module are integrated on the embedded development board.

[0033] The RGB-D camera is configured to acquire color images and depth images of the wound to be measured and send them to the display module and the image acquisition and preprocessing module.

[0034] The display module is configured to display the preview image of the RGB-D camera, the actual distance from the center of the image to the RGB-D camera and the area of the wound area output by the wound area calculation module in real time.

[0035] The power supply module is configured to supply power to the embedded development board, the RGB-D camera and the display module.

[0036] Further preferably, the power supply in the power supply module is a lithium battery.

[0037] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0038] 1. The application provides a chronic wound full-automatic handheld measurement device based on an RGB-D camera, which adopts an RGB-D camera to collect a set of color images and depth images, segments the wound area in the color images through a wound segmentation model, determines the wound area in the depth images, and obtains a wound area depth image; then the application calculates the area of the wound area by introducing 3D point clouds based on the obtained wound area depth image, and provides two ways of calculating the area of the wound area based on the edge of the wound area and based on the surface of the wound area; compared with the method of using only 2D images, the application effectively improves the measurement accuracy by introducing 3D point clouds, and the application only needs to process a single image to realize 3D reconstruction, which meets the requirements of speed and accuracy; at the same time, the application does not need to collect image videos, but only needs to process color images and depth images, which reduces the requirements for the collection stability of the user and the anti-shake characteristics of the collection equipment, has a low cost, and the application also does not need to set a marker to determine the actual size of the wound, which is easy to operate, can realize flexible measurement, and has good usability; in addition, the application provides two ways of calculating the wound area, the way based on the edge of the wound area is suitable for approximately planar surfaces, has a fast speed and stability, and has a high accuracy in a very small area; the way based on the surface of the wound area is suitable for all surfaces and can reflect the curvature characteristics of the surface, as a more general measurement method, it can realize high-precision, easy-to-use chronic wound measurement at a low cost.

[0039] 2. The chronic wound full-automatic handheld measurement device based on an RGB-D camera provided by the application determines the wound area based on a wound segmentation model, extracts features level by level using a HarDNet model, uses only the last three levels of features finally because the low-resolution features also have enough original information, then respectively uses RFB modules to strengthen the information of the last three levels of features on different receptive fields, after fusion, finally uses three groups of up-sampling and convolution layers, and then realizes high-precision wound image segmentation through 1*1 convolution, and compared with other commonly used feature extraction networks, the application has a lower DRAM read-write frequency, greatly reduces the running power consumption while ensuring accuracy, has a faster running speed, and can be well integrated into a small mobile device.

[0040] 3. The chronic wound measurement process provided by the application has a small algorithm running overhead, a small hardware size and a low cost, and can be well integrated into a mobile device, so that the device has a wide application scenario. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A schematic diagram of a chronic wound full-automatic handheld measurement device based on an RGB-D camera provided by the present application is shown in

[0042] Figure 2 A schematic diagram of a hardware part of a chronic wound full-automatic handheld measurement device provided by an embodiment of the present application is shown in

[0043] Figure 3 A whole work flow of a chronic wound full-automatic handheld measurement device provided by an embodiment of the present application is shown in

[0044] Figure 4 A structure schematic diagram of a wound image segmentation model provided by an embodiment of the present application is shown in DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] In order to achieve the above-mentioned purpose, the present application provides a chronic wound full-automatic handheld measurement device based on an RGB-D camera, as shown in Figure 1 , comprising:

[0047] An image acquisition and preprocessing module is configured to control the RGB-D camera to acquire a set of color images and depth images of a wound to be measured and record camera intrinsic parameters; perform bilateral filtering processing on the depth images; output the acquired color images and the processed depth images to a wound area segmentation module; and output the camera intrinsic parameters to a wound area calculation module. It should be noted that the bilateral filtering processing on the depth images can smooth the integer pixel value with a precision of millimeter to floating point type to eliminate the step stratification phenomenon caused by the depth acquisition precision on the point cloud, and at the same time, the main edges can be retained.

[0048] The wound area segmentation module is configured to normalize the size of the color image, input the normalized color image into a pre-trained wound segmentation model, obtain a mask of a wound area, multiply corresponding pixel points in the depth image and the mask, obtain a wound area depth image, and output the wound area depth image to the wound area calculation module. Specifically, the wound segmentation model is a segmentation network, which specifically includes a cascaded encoding network, a feature enhancement network, a feature fusion network, and an output decoding network. The encoding network is configured to extract image features of the color image at different resolutions and output the image features to the feature enhancement network. The feature enhancement network is configured to enhance the image features at different resolutions on different receptive fields. The feature fusion network is configured to fuse the enhanced image features at different resolutions to obtain fused features. The output decoding network is configured to upsample the fused features to the original resolution of the color image to obtain the mask of the wound area. In an optional embodiment, the encoding network is a HarDNet model, which includes a plurality of cascaded HarDNet units and a pooling layer arranged between each two adjacent HarDNet units. The HarDNet unit includes a cascaded HarDBlock and a convolution layer. After the color image is processed by the HarDNet model, three image features at different resolutions are output from the last three HarDNet units. The feature enhancement network includes three RFB modules, which are connected to the output ends of the last three HarDNet units of the HarDNet model in a one-to-one correspondence, respectively, to enhance the image features at different resolutions. The output decoding network includes three cascaded decoding units and a convolution layer with a convolution kernel size of 1*1. The decoding unit includes a cascaded upsampling layer and a convolution layer.

[0049] The wound area calculation module is configured to calculate the area of the wound area based on the wound area depth image and the camera intrinsic parameter by using mode one or mode two.

[0050] Mode one includes the following steps: calculating 3D point clouds of a wound area edge based on the wound area depth image and the camera intrinsic parameter, fitting the 3D point clouds of the wound area edge onto a same plane to obtain a spatial polygon representation of the wound area, calculating the area of the spatial polygon to obtain the area of the wound area. Specifically, an expression of the plane f(x, y, z) is obtained by minimizing the distance between each 3D point in the 3D point clouds of the wound area edge and the plane f(x, y, z). All 3D points in the 3D point clouds of the wound area edge are projected onto the plane f(x, y, z) by vertical projection to obtain the spatial polygon representation of the wound area. The area of the wound area calculated by using mode one is as follows:

[0051]

[0052] wherein, n is the unit normal vector of the projection plane; cos(n,z) is the cosine of the angle between n and the coordinate axis z; N is the number of 3D points in the 3D point cloud of the wound area edge; (x i ,y i ,z i ) is the coordinate of the i-th 3D point in the 3D point cloud of the wound area edge; cos(n,x) is the cosine of the angle between n and the coordinate axis x; cos(n,y) is the cosine of the angle between n and the coordinate axis y.

[0053] The above method two includes: calculating the 3D point cloud of the wound area and the 3D point cloud of the wound area edge based on the wound area depth image and the camera intrinsic parameter respectively; reconstructing the 3D point cloud of the wound area into a surface by using a surface reconstruction algorithm, removing the 3D points in the surface which are outside the wound area edge, and obtaining a complete wound 3D model after smoothing and denoising; calculating the surface area of the wound 3D model to obtain the area of the wound area. Specifically, the surface reconstruction algorithm can be Poisson surface reconstruction algorithm, rolling ball method, Delaunay triangulation, etc.; preferably, the Poisson surface reconstruction algorithm is used to reconstruct the 3D point cloud of the wound area into a surface. Further, the method of removing the 3D points in the surface which are outside the wound area edge is: projecting the 3D points on the surface and the 3D point cloud of the wound area edge onto the xy plane under the 3D point cloud coordinate system to obtain the 2D points corresponding to the 3D points on the surface in the xy plane, and the polygonal region determined by the wound area edge in the xy plane, and removing the 3D points on the surface corresponding to the 2D points outside the polygonal region, so as to remove the 3D points in the surface which are outside the wound area edge. Preferably, the PNPOLY algorithm can be used to determine whether the above 2D points are outside the polygonal region.

[0054] Specifically, the wound area calculation module maps the pixel points in the wound area edge or the wound area in the wound area depth image by the following mapping formula to obtain the corresponding 3D point cloud:

[0055]

[0056]

[0057]

[0058] Wherein, (x, y, z) is the 3D coordinate of the edge of the wound area or a pixel point p in the wound area in the 3D point cloud in the wound area depth image; d is the pixel value of the pixel point p in the wound area depth image; depth scale is a fixed parameter in the RGB-D camera for converting the depth pixel value into the actual distance; (u, v) is the coordinate of the pixel point p in the wound area depth image; cx, cy, fx and fy are all camera parameters; wherein, fx and fy are the transformation scales when converting the coordinates of the physical imaging plane to the camera plane, indicating that on the imaging plane with a focal length of f, how many pixels correspond to each f length; cx and cy are the translation scales when converting the coordinates of the physical imaging plane to the camera plane, specifically, the origin of the physical imaging plane is located at the center, while the origin of the camera plane is located at the top left corner of the image, and cx and cy are used to translate the origin of the physical imaging plane to the origin of the camera plane.

[0059] It should be noted that, in an optional embodiment, the present application provides two modes for calculating the area of the wound area, corresponding to the above-mentioned mode one and mode two respectively, mode one is faster and less affected by noise, and is suitable for measuring smaller and approximately planar wounds; mode two can measure wound surfaces with a certain curvature; the above-mentioned chronic wound full-automatic handheld measuring device provides a measuring mode selection button, and the user can select the corresponding mode according to the specific wound condition.

[0060] Further, more intelligently, in an optional embodiment, the wound area calculation module first calculates the area of the wound area by mode one, judges whether the area of the wound area obtained by mode one is greater than a preset area, if yes, calculates the area of the wound area by mode two as the final area measurement result; otherwise, takes the area of the wound area obtained by mode one as the final area measurement result; specifically, in the present embodiment, the preset area is 5cm 2 .

[0061] Further, in an optional embodiment, the handheld device further comprises: an embedded development board, an RGB-D camera, a display module and a power supply module; wherein, the image acquisition and preprocessing module, the wound area segmentation module and the wound area calculation module are integrated on the embedded development board; the RGB-D camera is used to acquire the color image and the depth image of the wound to be measured, and send them to the display module and the image acquisition and preprocessing module; the display module is used to display the preview picture of the RGB-D camera, the actual distance from the center of the picture to the RGB-D camera, and the area of the wound area output by the wound area calculation module after processing; the power supply module is used to supply power to the embedded development board, the RGB-D camera and the display module. Preferably, the power supply in the above-mentioned power supply module is a lithium battery.

[0062] To further illustrate the fully automated handheld measurement device for chronic wounds based on an RGB-D camera provided by the present invention, a specific embodiment is described in detail below:

[0063] like Figure 2 The diagram shows the hardware components of a fully automated handheld measurement device for chronic wounds, including an RGB-D camera, an embedded development board, a display screen, and a power supply module.

[0064] RGB-D cameras can simultaneously acquire relatively accurate color and depth images, obtaining complete 3D information, and are small in size and low in cost. Therefore, this invention integrates an RGB-D camera into a fully automated handheld chronic wound measurement device to achieve more efficient wound measurement. The RGB-D camera continuously outputs color and depth video streams at a certain resolution and frame rate. After the user sends a acquisition command, it acquires a set of current color and depth images while recording camera intrinsic parameters. The image acquisition preprocessing module, wound region segmentation module, and wound area calculation module are integrated on an embedded development board. The embedded development board controls the RGB-D camera through a USB3 interface to complete image acquisition and subsequent processing, outputting real-time preview images and results to a display screen. The user controls the acquisition and results acquisition through touch control on the display screen. The power supply module continuously supplies power to the entire system. Specifically, in this embodiment, the aforementioned embedded development board is a small development board running a Linux system with a USB3 interface and display output. The display screen is a 7-inch touch screen; the power supply module includes a lithium battery and a power control circuit; the lithium battery supplies power to both the development board and the display screen simultaneously through the control circuit. The system displays a real-time preview of the current RGB-D camera on the screen, along with the actual distance from the center of the image to the camera. After the user presses the acquisition button, the system begins acquisition and processing, and displays the segmentation and area results on the color image upon completion. The user can then choose to start the next acquisition or exit.

[0065] The overall workflow of the fully automated handheld measuring device for chronic wounds provided by this invention is as follows: Figure 3 As shown.

[0066] First, before acquisition, the RGB-D camera continuously outputs a preview image at 640*480 resolution and 30FPS. Upon receiving the user's acquisition command, it acquires a set of color and depth images for the current frame and saves the camera's intrinsic parameters. Then, after performing bilateral filtering on the depth image, the acquired color image and the processed depth image are output to the wound region segmentation module.

[0067] Specifically, further, such as Figure 4As shown, the wound image segmentation model comprises an encoding network, a feature enhancement network, a feature fusion network and an output decoding network. Among them, the encoding network uses HarDNet68, and after the input image is processed by several groups of HarDBlock, 1*1 convolution layer and max-pooling with a step of 2, three encoded features are obtained from the output of the last three 1*1 convolution layers; the three features are processed through the feature enhancement network, i.e. the RFB module; then the feature fusion network is used for fusion; finally, through the output decoding network, three groups of up-sampling and convolution layers, and 1*1 convolution layer are used; finally, the output of the original input size is obtained, representing the segmentation result. Specifically, the size of the input wound image is adjusted from 640*480 to 512*512, and the features are extracted by the encoding network step by step. Since the low-resolution features also have enough original information, only the last three features are used finally, and their resolution and dimension are 64*64*320, 32*32*640 and 16*16*1024 respectively; the three features are input into the RFB module, which uses 1*3+3*1 convolution, 1*5+5*1 convolution, 1*7+7*1 convolution, 1*1 convolution on the input features respectively, and then the first three groups are further processed by 3*3 sparse convolution. Then, the four groups of features are stacked and 3*3 convolution is performed. After the whole process, the number of layers and dimensions of the input are uniformly adjusted to 32, and the output is obtained. This process enhances the information of the input features in different receptive fields. At this time, the three features with different resolutions and the same number of layers are input into the feature fusion network, which is up-sampled and multiplied by the corresponding elements to fuse into a feature with a resolution of 64*64. Finally, after the output decoding network, three groups of up-sampling and convolution layers are used, and then 1*1 convolution is performed to obtain the final output of 512*512*1.

[0068] It should be noted that the effect of traditional segmentation algorithms (such as threshold segmentation) is easily affected by the shooting environment, and in most methods, a color marker placed beside the wound is needed for standardization. The semi-automatic and manual segmentation method introduces the subjective influence of the operator, and the result lacks stability and the operation is complicated; while the existing high-precision segmentation method based on deep learning network has problems such as long running time and insufficient memory in mobile terminal hardware without GPU acceleration, which cannot be well integrated into the mobile device of the hand-held measuring device provided by the present application, and the network with reduced parameter quantity and model size will lose too much precision. Therefore, the above-mentioned wound image segmentation model is designed, and HarDNet with low memory read and write is used as the feature extraction network in the encoder, which optimizes the feature extraction step with the largest operation amount, greatly reduces the running time and power consumption.

[0069] In this embodiment, the loss function of the wound image segmentation model is the DICE coefficient; during the training process, the Adam optimizer is used for parameter updating. The training data of the segmentation model comes from the MICCAI2021 foot ulcer segmentation challenge, 810 images are used as the training set, 200 images are used as the verification set, and all images have fine-labeled labels. The DICE coefficient of the model on the verification set reaches 86.01%, and the DICE coefficient on the test set is 87.57%. After the model is converted into the ONNX standard format, it is deployed on the development board to run inference.

[0070] In the wound area calculation module, an edge-based method (the above-mentioned method one) or a surface-based method (the above-mentioned method two) is selected for calculation. First, edges are extracted on the wound area depth image, and 3D point clouds of the area and the edges are calculated respectively; the edge-based method fits all edge points to a plane, and calculates the area of the polygon; the surface-based method applies Poisson surface reconstruction on the point cloud of the area, and then removes redundant surfaces using the edge point cloud to obtain the complete 3D model of the wound area, and then outputs the surface area thereof.

[0071] In order to further verify the measurement accuracy of the chronic wound full-automatic handheld measurement device provided by the present application, three cases of foot ulcer wound cases were collected. In the experiment, three wounds of different sizes were measured at different distances, and the results are shown in Table 1. It can be seen that in actual application, the chronic wound full-automatic handheld measurement device provided by the present application still has good stability and accuracy, and the maximum error with the actual area is within 5%. According to the above experiment, the present application has high precision in the overall process of wound image segmentation, area calculation and combination of the two, and can provide a feasible measurement means for clinical chronic wound care.

[0072] Table 1

[0073]

[0074]

[0075] In summary, compared with other measurement devices and methods, the chronic wound full-automatic handheld measurement device provided by the present application has high automation degree, does not need to provide labels manually, does not need to place markers, has high measurement accuracy, is small in size and low in cost.

[0076] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fully automated handheld measuring device for chronic wounds based on an RGB-D camera, characterized in that, include: The image acquisition and preprocessing module is used to control the RGB-D camera to acquire a set of color and depth images of the wound to be measured, and to record the camera's intrinsic parameters; The depth image is subjected to bilateral filtering, and the acquired color image and the processed depth image are output to the wound region segmentation module. The camera intrinsic parameters are output to the wound area calculation module. The wound region segmentation module is used to normalize and adjust the size of the color image before inputting it into the pre-trained wound segmentation model to obtain a mask of the wound region. The depth image is then multiplied with the corresponding pixels in the mask to obtain the depth image of the wound region, which is then output to the wound area calculation module. The wound area calculation module is used to calculate the area of ​​the wound region based on the depth image of the wound region and camera intrinsic parameters, using either method one or method two. The first method includes: calculating the 3D point cloud of the wound region edge based on the wound region depth image and camera intrinsic parameters, fitting the 3D point cloud of the wound region edge onto the same plane to obtain a spatial polygon representation of the wound region, calculating the area of ​​the spatial polygon, and obtaining the area of ​​the wound region. The second method includes: calculating 3D point clouds of the wound region and the edge of the wound region based on the depth image of the wound region and camera intrinsic parameters respectively; reconstructing the 3D point cloud of the wound region into a surface, removing 3D points outside the edge of the wound region from the surface, and obtaining a complete 3D model of the wound after smoothing and denoising; calculating the surface area of ​​the 3D model of the wound to obtain the area of ​​the wound region. The wound segmentation model includes: An encoding network is used to extract image features of a color image at different resolutions and output them to a feature enhancement network. The encoding network is a HarDNet model, which includes multiple cascaded HarDNet units and pooling layers between every two adjacent HarDNet units. The HarDNet unit includes cascaded HarDBlock and convolutional layers. After the color image is processed by the HarDNet model, the last three HarDNet units output image features at three different resolutions. The feature enhancement network is used to enhance image features at different resolutions across different receptive fields. The feature enhancement network includes three RFB modules, which are connected one-to-one with the outputs of the last three HarDNet units of the HarDNet model, and enhance image features at different resolutions respectively. Feature fusion network is used to fuse image features at different resolutions after enhancement processing to obtain fused features; An output decoding network is used to upsample the fused features to the original resolution of the color image to obtain a mask for the wound region. The output decoding network includes three cascaded decoding units and a convolutional layer with a kernel size of 1*1. The decoding unit includes cascaded upsampling layers and convolutional layers.

2. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, The wound area calculation module maps the edges of the wound region or the pixels within the wound region in the wound region depth image using the following mapping formula to obtain the corresponding 3D point cloud: in, d represents the 3D coordinates of a pixel p in the 3D point cloud of the wound region edge or wound region in the wound region depth image; d represents the pixel value of pixel p in the wound region depth image. This is a fixed parameter used in RGB-D cameras to convert depth pixel values ​​into actual distance; Let p be the coordinates of pixel p in the depth image of the wound region; , , and All of these are camera internal parameters.

3. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, In the aforementioned method, by minimizing the interaction between each 3D point in the 3D point cloud at the edge of the wound region and the plane... The distance between them is used to obtain the plane. The expression; projecting all 3D points in the 3D point cloud of the wound region edge onto the plane by vertical projection. The spatial polygon representation of the wound region is obtained from the above.

4. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, The area of ​​the wound region calculated using the first method is: in, Let be the unit normal vector of the projection plane; for with coordinate axes The cosine of the included angle; N is the number of 3D points in the 3D point cloud at the edge of the wound area; The first point in the 3D point cloud of the wound area edge i 3D point coordinates; for with coordinate axes x The cosine of the included angle; for with coordinate axes y The cosine of the included angle.

5. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, Method 2 uses the Poisson surface reconstruction algorithm to reconstruct the 3D point cloud of the wound area into a surface.

6. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, In the second method, the 3D points on the surface and the 3D point cloud of the wound area edge are projected onto the 3D point cloud coordinate system. xy On a plane, obtain 3D points on the surface. xy The corresponding 2D points on the plane, and the edges of the wound area. xy The polygonal region defined on the plane is removed, and the 3D points on the surface corresponding to the 2D points located outside the polygonal region are removed, thereby removing the 3D points on the surface that are outside the edge of the wound region.

7. The fully automatic handheld measuring device for chronic wounds according to any one of claims 1-6, characterized in that, The wound area calculation module first calculates the area of ​​the wound region using the first method, and then determines whether the area of ​​the wound region obtained by the first method is greater than a preset area. If so, the area of ​​the wound region is calculated using the second method and used as the final area measurement result; otherwise, the area of ​​the wound region obtained by the first method is used as the final area measurement result.

8. The fully automatic handheld measuring device for chronic wounds according to claim 1, characterized in that, The handheld measuring device also includes: an embedded development board, an RGB-D camera, a display module, and a power supply module; wherein, the image acquisition and preprocessing module, the wound region segmentation module, and the wound area calculation module are integrated on the embedded development board; The RGB-D camera is used to acquire color and depth images of the wound to be measured and send them to the display module and the image acquisition preprocessing module. The display module is used to display in real time the preview image of the RGB-D camera, the actual distance from the center of the image to the RGB-D camera, and the area of ​​the wound region output by the wound area calculation module; The power supply module is used to supply power to the embedded development board and the display module.

Images