Nail fold video image registration method and device, electronic equipment and storage medium
The blue chromaticity component of nail fold video was calculated by variance projection method, which solved the problem of registration difficulties caused by micro-jitter in clinical detection of nail fold video images, and improved the image registration accuracy and diagnostic efficiency.
Patent Information
- Application Number
- CN202510331945.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-20
- Publication Date
- 2025-06-27
AI Technical Summary
In nail fold capillary video examination, slight shaking of the hand causes difficulty in registering the nail fold video image, affecting the accuracy of the diagnosis results.
The variance projection method is used to calculate the nail fold video of the blue chroma component, and obtain the vertical and horizontal offsets of the subsequent frame image and the first frame image, and then the image registration is carried out.
The variance projection method effectively captures the details in the image, improves the image registration accuracy of nail fold videos, and improves the diagnostic efficiency of nail fold videos.
Smart Images
Figure CN120219455A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of image processing. Specifically, it relates to a method, device, electronic device, and storage medium for registering nailfold video images. Background Art
[0002] Nailfold videocapillaroscopy (NVC), as a non-invasive, simple-to-operate, and intuitive detection method, is widely used in the diagnosis and monitoring of pathological conditions such as cardiovascular diseases, diabetes, and rheumatic immune diseases.
[0003] However, during the process of NVC collecting nailfold microcirculation videos, due to factors such as the difficulty in controlling cardiac pulsation and arm muscles, the hand will produce minute tremors, which will be amplified in the collected nailfold videos, thus interfering with doctors' observation of nailfolds and parameter measurement, and further affecting the accuracy of doctors' diagnosis.
[0004] Therefore, in order to solve the technical problem that the registration of nailfold video images is difficult due to micro-tremors in clinical detection and affects the diagnostic results, there is an urgent need for a method, device, electronic device, and storage medium for registering nailfold video images. Summary of the Invention
[0005] The purpose of this application is to provide a method, device, electronic device, and storage medium for registering nailfold video images. By calculating the vertical offset and horizontal offset between each subsequent frame image and the first frame image of the nailfold video obtained by the variance projection method for the blue chrominance component, image registration of the nailfold video is performed, solving the problem that the registration of nailfold video images is difficult due to micro-tremors in clinical detection and affecting the diagnostic results. The variance projection method can effectively capture the detailed changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnostic efficiency of the nailfold video.
[0006] In a first aspect, this application provides a method for registering nailfold video images, including: Obtain the nailfold video to be registered; Perform color space conversion on the nailfold video to be registered to obtain a nailfold video of the blue chrominance component; Using the variance projection method, with the first frame image as the reference frame image, move the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset between each subsequent frame image and the reference frame image; According to the vertical offset and the horizontal offset, perform image registration on the nailfold video to be registered to obtain the image registration result of the nailfold video to be registered.
[0007] The nailfold video image registration method provided by this application can achieve image registration for nailfold videos. By calculating the vertical and horizontal offsets of each subsequent frame image and the first frame image obtained by calculating the nailfold video of the blue chrominance component through the variance projection method, image registration of the nailfold video is performed, solving the problem that nailfold video images are difficult to register due to micro-vibrations in clinical examinations, which affects the diagnostic results. Through the variance projection method, the detailed changes in the image can be effectively captured, the image registration accuracy of the nailfold video is improved, and the diagnostic efficiency of the nailfold video is enhanced.
[0008] Optionally, performing color space conversion on the nailfold video to be registered to obtain a nailfold video of the blue chrominance component includes: According to a preset YCrCb color space conversion formula, calculating the luminance component and chrominance component when the nailfold video to be registered is converted from the RGB color space to the YCrCb color space; Based on the luminance component and the chrominance component, converting the nailfold video to be registered to the YCrCb color space; Extracting the blue chrominance component from the nailfold video to be registered in the YCrCb color space to obtain a nailfold video of the blue chrominance component.
[0009] The nailfold video image registration method provided by this application can achieve image registration for nailfold videos. By calculating the luminance component and chrominance component when the nailfold video to be registered is converted from the RGB color space to the YCrCb color space, the nailfold video to be registered is converted to the YCrCb color space to extract a nailfold video of the blue chrominance component. Due to the characteristic that the blue chrominance component is relatively insensitive to illumination changes, the influence of non-uniform illumination on image registration can be effectively reduced, the true structure of the image can be better reflected, and it is beneficial to improve the image registration efficiency of the nailfold video.
[0010] Optionally, through the variance projection method, using the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical and horizontal offsets of each subsequent frame image and the reference frame image includes: Through the variance projection method, calculating the projection variance of each frame image in the nailfold video of the blue chrominance component to obtain the projection vector of each frame image in the nailfold video of the blue chrominance component; Using the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to make the Pearson correlation coefficient between the projection vector of each subsequent frame image and the projection vector of the reference frame image reach the highest value, so as to calculate the vertical and horizontal offsets of each subsequent frame image and the reference frame image.
[0011] The nailfold video image registration method provided by this application can achieve image registration for nailfold videos. Through the variance projection method, while moving the subsequent frame images in the nailfold video of the blue chrominance component, calculate the Pearson correlation coefficient between the projection vectors of the subsequent frame images and the first frame image after each movement. Based on the projection vector corresponding to the highest value of the Pearson correlation coefficient for each subsequent frame image, calculate the vertical offset and horizontal offset when performing image registration for each subsequent frame image. The vertical offset and horizontal offset of each subsequent frame image calculated through the variance projection method are used to perform image registration on the nailfold video, which is beneficial to improving the registration efficiency of the nailfold video.
[0012] Optionally, the projection vector includes a vertical projection vector and a horizontal projection vector; through the variance projection method, calculating the projection variance of each frame image in the nailfold video of the blue chrominance component to obtain the projection vector of each frame image in the nailfold video of the blue chrominance component includes: Perform noise reduction processing on each frame image in the nailfold video of the blue chrominance component to obtain the blue chrominance component after noise reduction; Calculate the projection variances of the blue chrominance component after noise reduction in the horizontal and vertical directions to obtain a horizontal projection vector and a vertical projection vector.
[0013] Optionally, calculating the projection variances of the blue chrominance component after noise reduction in the horizontal and vertical directions to obtain a horizontal projection vector and a vertical projection vector includes: Calculate the variance of the pixel values in each row of the blue chrominance component after noise reduction row by row, and combine the variance values of each row to form a horizontal projection vector; Calculate the variance of the pixel values in each column of the blue chrominance component after noise reduction column by column, and combine the variance values of each column to form a vertical projection vector.
[0014] Optionally, taking the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to make the Pearson correlation coefficient between the projection vectors of each subsequent frame image and the projection vector of the reference frame image reach the highest value, so as to calculate the vertical offset and horizontal offset between each subsequent frame image and the reference frame image, includes: Taking the first frame image as the reference frame image, after moving the projection vectors of each subsequent frame image in the nailfold video of the blue chrominance component multiple times, calculate the Pearson correlation coefficient between the projection vector of each subsequent frame image after each movement and the projection vector of the reference frame image; Extract the highest value from the Pearson correlation coefficients belonging to the same subsequent frame image; According to the displacement amount of the projection vector corresponding to the highest value, the vertical offset and the horizontal offset between each subsequent frame image and the reference frame image are calculated.
[0015] Optionally, after performing image registration on the to-be-registered nailfold video according to the vertical offset and the horizontal offset to obtain the image registration result of the to-be-registered nailfold video, it further includes: Calculating the similarity between each subsequent frame image after registration and the reference frame image to determine the image registration effect of the to-be-registered nailfold video.
[0016] In a second aspect, the present application provides a nailfold video image registration device, including: An acquisition module, configured to acquire a to-be-registered nailfold video; A conversion module, configured to perform color space conversion on the to-be-registered nailfold video to obtain a nailfold video of the blue chrominance component; A calculation module, configured to use the variance projection method, with the first frame image as the reference frame image, move the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and the horizontal offset between each frame of the subsequent frame image and the reference frame image; A registration module, configured to perform image registration on the to-be-registered nailfold video according to the vertical offset and the horizontal offset to obtain the image registration result of the to-be-registered nailfold video.
[0017] The nailfold video image registration device calculates the vertical offset and the horizontal offset between each subsequent frame image and the first frame image obtained by performing variance projection method on the nailfold video of the blue chrominance component, and performs image registration on the nailfold video, solving the problem that the nailfold video image is difficult to register due to micro-vibration in clinical detection, which affects the diagnosis result. The variance projection method can effectively capture the detailed changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnosis efficiency of the nailfold video.
[0018] In a third aspect, the present application provides an electronic device, including a processor and a memory, where the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the nailfold video image registration method described above.
[0019] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it runs the steps in the nailfold video image registration method described above.
[0020] Beneficial effects: The nailfold video image registration method, device, electronic device, and storage medium provided by this application perform image registration on the nailfold video by calculating the vertical and horizontal offsets of each subsequent frame image and the first frame image of the nailfold video obtained by the variance projection method on the blue chrominance component. This solves the problem that the registration of nailfold video images is difficult due to micro-vibrations in clinical examinations, which affects the diagnostic results. The variance projection method can effectively capture the detailed changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnostic efficiency of the nailfold video. Description of the Drawings
[0021] Figure 1 It is a flowchart of the nailfold video image registration method provided by an embodiment of this application.
[0022] Figure 2 It is a schematic structural diagram of the nailfold video image registration device provided by an embodiment of this application.
[0023] Figure 3 It is a schematic structural diagram of the electronic device provided by an embodiment of this application.
[0024] Reference numeral description: 1. Acquisition module; 2. Conversion module; 3. Calculation module; 4. Registration module; 301. Processor; 302. Memory; 303. Communication bus. Detailed Embodiments
[0025] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Usually, the components of the embodiments of this application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of this application that is required to be protected, but only represents the selected embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of this application.
[0026] It should be noted that: Similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, terms such as "first" and "second" are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.
[0027] Please refer to Figure 1 , Figure 1 which is a nailfold video image registration method in some embodiments of this application for performing image registration on a nailfold video, including the steps: Step S101, obtain the to-be-registered nailfold video; Step S102, perform color space conversion on the to-be-registered nailfold video to obtain the nailfold video of the blue chrominance component; Step S103, through the variance projection method, using the first frame image as the reference frame image, move the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image; Step S104, according to the vertical offset and horizontal offset, perform image registration on the to-be-registered nailfold video to obtain the image registration result of the to-be-registered nailfold video.
[0028] In this nailfold video image registration method, the vertical offset and horizontal offset of each subsequent frame image calculated from the nailfold video of the blue chrominance component through the variance projection method are used to perform image registration on the nailfold video, solving the problem that the nailfold video image is difficult to register due to micro-vibrations in clinical detection, which affects the diagnosis result. The variance projection method can effectively capture the detailed changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnosis efficiency of the nailfold video.
[0029] Specifically, in step S101, the to-be-registered nailfold video can be obtained in various ways. For example, it can be obtained by real-time shooting and collecting with a nailfold capillary microscope device, or the nailfold video stored in a memory in advance can be obtained as the to-be-registered nailfold video.
[0030] Specifically, in step S102, performing color space conversion on the to-be-registered nailfold video to obtain the nailfold video of the blue chrominance component includes: According to the preset YCrCb color space conversion formula, calculate the luminance component and chrominance component when the to-be-registered nailfold video is converted from the RGB color space to the YCrCb color space; Based on the luminance component and chrominance component, convert the to-be-registered nailfold video to the YCrCb color space; Extract the blue chrominance component from the to-be-registered nailfold video in the YCrCb color space to obtain the nailfold video of the blue chrominance component.
[0031] In step S102, the fingernail video to be registered initially exists in the RGB color space format. To extract image features more suitable for registration, a conversion from the RGB color space to the YCrCb color space is performed. The YCrCb color space decomposes the image information into a luminance component (Y) and two chrominance components (the chrominance components include the blue chrominance component Cb and the red chrominance component Cr). First, according to a preset YCrCb color space conversion formula (the YCrCb color space conversion formula is a conversion formula based on the ITU-R BT.601 or BT.709 standard, which is prior art and will not be elaborated here), the luminance component Y, the blue chrominance Cb, and the red chrominance Cr when converted to the YCrCb color space are calculated. Based on the calculated luminance and chrominance components, the original RGB video is converted into a video in the YCrCb color space. In this converted video, the blue chrominance component video (i.e., the fingernail video of the blue chrominance component) is extracted for subsequent image registration processing. The blue chrominance component is selected considering its relatively insensitive characteristic to illumination changes, which can effectively reduce the impact of non-uniform illumination on image registration, better reflect the true structure of the image, and thus improve the accuracy and robustness of registration. Through color space conversion, video data suitable for subsequent registration steps can be effectively prepared.
[0032] Specifically, in step S103, by using the variance projection method, with the first frame image as the reference frame image, the subsequent frame images in the fingernail video of the blue chrominance component are moved to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image, including: By using the variance projection method, the projection variance of each frame image in the fingernail video of the blue chrominance component is calculated to obtain the projection vector of each frame image in the fingernail video of the blue chrominance component; With the first frame image as the reference frame image, the subsequent frame images in the fingernail video of the blue chrominance component are moved to make the Pearson correlation coefficient between the projection vector of each subsequent frame image and the projection vector of the reference frame image reach the highest value, so as to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image.
[0033] Specifically, the projection vector includes a vertical projection vector and a horizontal projection vector; in step S103, by using the variance projection method, the projection variance of each frame image in the fingernail video of the blue chrominance component is calculated to obtain the projection vector of each frame image in the fingernail video of the blue chrominance component, including: Noise reduction processing is performed on each frame image in the fingernail video of the blue chrominance component to obtain the blue chrominance component after noise reduction; The projection variances of the blue chrominance component after noise reduction in the horizontal and vertical directions are calculated to obtain the horizontal projection vector and the vertical projection vector.
[0034] In step S103, each frame image in the nailfold video of the blue chrominance component is denoised to obtain the denoised blue chrominance component. Image filtering algorithms such as Gaussian filtering or median filtering can be used for the denoising process to reduce the interference of image noise on the subsequent projection variance calculation and improve the accuracy of the projection vector.
[0035] In step S103, the projection variances of the denoised blue chrominance component in the horizontal and vertical directions are calculated to obtain the horizontal projection vector and the vertical projection vector, including: Calculate the variance of the pixel values in each row of the denoised blue chrominance component row by row, and combine the variance values of each row to form the horizontal projection vector; Calculate the variance of the pixel values in each column of the denoised blue chrominance component column by column, and combine the variance values of each column to form the vertical projection vector.
[0036] In step S103, when calculating the horizontal projection vector, scan the denoised Cb component image (each frame image in the denoised blue chrominance component) row by row. For each row of pixel values, calculate the variance of these pixel values. For example, for a certain row of pixel values, first calculate the average value of the pixel values in this row, then calculate the square of the difference between each pixel value and the average value, and then add all the squared values and divide by the number of pixels to obtain the variance of the pixel values in this row. Combine the variance values calculated for each row in the order of rows to form the horizontal projection vector. The calculation method of the vertical projection vector is similar to that of the horizontal projection vector, except that the vertical projection vector calculates the variance of the pixel values in each column of the denoised Cb component image column by column and combines the variance values of each column in the order of columns. Thus, through the horizontal projection vector and the vertical projection vector, the projection characteristics of the image in the horizontal and vertical directions can be fully described.
[0037] Specifically, in step S103, taking the first frame image as the reference frame image, move the subsequent frame images in the nailfold video of the blue chrominance component (the subsequent frame images are each frame image after the first frame image in the nailfold video) so that the Pearson correlation coefficient between the projection vector of each subsequent frame image and the projection vector of the reference frame image reaches the highest value, and calculate the vertical offset and horizontal offset between each subsequent frame image and the reference frame image, including: Taking the first frame image as the reference frame image, after moving the projection vectors of each subsequent frame image in the nailfold video of the blue chrominance component multiple times, calculate the Pearson correlation coefficient between the projection vector of each subsequent frame image after each movement and the projection vector of the reference frame image; Extract the highest value from the Pearson correlation coefficients belonging to the same subsequent frame image; According to the displacement of the projection vector corresponding to the highest value, the vertical offset and horizontal offset between each subsequent frame image and the reference frame image are calculated.
[0038] In step S103, the projection vector of the subsequent frame image will be moved horizontally and vertically multiple times within a preset range (the preset range can be set according to actual needs). For example, it is moved by one pixel as the step size. After each movement, the Pearson correlation coefficient between the moved projection vector and the projection vector of the reference frame image will be calculated. Thus, a series of Pearson correlation coefficient values can be obtained, and each value corresponds to a specific movement. After all the preset number of movements (the preset number can be set according to actual needs) are completed, the highest value will be found from these Pearson correlation coefficient values. The position of the projection vector corresponding to this highest value is considered the best projection vector matching position between the corresponding subsequent frame image and the reference frame image. The displacement corresponding to the projection vector corresponding to this highest value is the best displacement of the current subsequent frame image relative to the reference frame image. According to this best displacement, the vertical offset and horizontal offset of the subsequent frame image relative to the reference frame image can be determined. In summary, each subsequent frame image is moved in sequence to obtain the vertical offset and horizontal offset of each subsequent frame image during registration. Among them, the calculation method of the Pearson correlation coefficient is the prior art and will not be elaborated here.
[0039] For example, taking the first frame image as the reference frame image, the registration is performed for the second subsequent frame image. First, the displacement search ranges in both the horizontal and vertical directions are set to [-5, 5] pixels, and the displacement step size is 1 pixel. Then, the projection vector of the second frame image is moved horizontally and vertically within the range of [-5, 5] pixels at a step size of 1 pixel respectively. After each movement, the Pearson correlation coefficient between the moved projection vector of the second frame image and the projection vector of the first frame image is calculated. Thus, for the second frame image, 11 * 11 = 121 Pearson correlation coefficient values can be obtained. The highest value is extracted from these 121 correlation coefficient values. Suppose the displacement corresponding to the highest value is 3 pixels of horizontal movement and -2 pixels of vertical movement. Then, the horizontal offset of the second frame image relative to the first frame image is 3 pixels, and the vertical offset is -2 pixels. This offset will be used for subsequent image registration operations to correct the position of the second frame image so that it is aligned with the first frame image. By this method of multiple movements and highest value extraction, the best matching position between images can be found more accurately, improving the accuracy of image registration.
[0040] Specifically, in step S104, according to the vertical offset and horizontal offset corresponding to each subsequent frame image, the subsequent frame images in the to-be-registered nailfold video are offset to perform image registration on the to-be-registered nailfold video, and an image registration result of the to-be-registered nailfold video is obtained. This image registration result provides more accurate and reliable image information for the doctor, which helps him make a correct diagnosis.
[0041] Specifically, in step S104, after performing image registration on the to-be-registered nailfold video according to the vertical offset and horizontal offset to obtain an image registration result of the to-be-registered nailfold video, it further includes: Calculating the similarity between each subsequent frame image after registration and the reference frame image to determine the image registration effect of the to-be-registered nailfold video.
[0042] In the same nailfold video, there are only differences in the details between each frame of images (doctors can diagnose the nailfold video based on the differences in the details), that is, there is a certain similarity between each frame of images. Therefore, after obtaining the image registration result, calculation methods such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM) are used to calculate the similarity value between each subsequent frame image after registration and the reference frame image. And this similarity value is compared with a set threshold (the value of the threshold can be set according to actual needs). When the similarity value is higher than or equal to the threshold, it is considered that the image registration effect is good, and the quality of the registered video image meets the requirements of subsequent analysis or diagnosis. On the contrary, if the similarity value is lower than the threshold, it may be necessary to re-perform image registration or manual intervention. In summary, through the similarity calculation step, the image registration effect of the to-be-registered nailfold video can be determined. And the similarity calculation adds a quality control link to the entire registration process, which can more objectively understand the quality of the registration result, so as to judge whether the registered video can be used for subsequent observation analysis or disease diagnosis, ensuring the reliability and practicality of the registration result.
[0043] Specifically, the similarity calculation can be achieved through various image similarity measurement methods. For example, metrics such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM) can be adopted. The mean squared error calculates the average squared difference between the pixel values of two images. The smaller the value, the higher the image similarity. The peak signal-to-noise ratio is calculated based on the mean squared error. The larger the value, the higher the image quality and the similarity. The structural similarity index measures from three aspects of image brightness, contrast, and structure, with a value range from 0 to 1. The closer to 1, the higher the structural similarity of the image. In the specific implementation process, a suitable similarity measurement method is selected and a reasonable similarity threshold is set. The similarity value between each subsequent frame image and the reference frame image after registration is calculated and compared with the set threshold. If the similarity value is higher than or equal to the threshold, it is considered that the image registration effect is good, and the quality of the registered video image meets the requirements of subsequent analysis or diagnosis. Conversely, if the similarity value is lower than the threshold, it may be necessary to re-perform image registration or manual intervention.
[0044] As can be seen from the above, for this nailfold video image registration method, by obtaining the nailfold video to be registered, performing color space conversion on the nailfold video to be registered to obtain the nailfold video of the blue chrominance component, and through the variance projection method, using the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset of each subsequent frame image and the reference frame image, and according to the vertical offset and horizontal offset, performing image registration on the nailfold video to be registered to obtain the image registration result of the nailfold video to be registered; thus, through the vertical offset and horizontal offset of each subsequent frame image and the first frame image calculated by the variance projection method for the nailfold video of the blue chrominance component, performing image registration on the nailfold video, solving the problem that the nailfold video image is difficult to register due to micro-vibrations in clinical detection and affecting the diagnosis result. The variance projection method can effectively capture the detail changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnosis efficiency of the nailfold video.
[0045] Reference Figure 2 , this application provides a nailfold video image registration device for performing image registration on a nailfold video, including: An acquisition module 1 for acquiring the nailfold video to be registered; A conversion module 2 for performing color space conversion on the nailfold video to be registered to obtain the nailfold video of the blue chrominance component; A calculation module 3 for, through the variance projection method, using the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset of each subsequent frame image and the reference frame image; The registration module 4 is used to perform image registration on the to-be-registered nail fold video according to the vertical offset and the horizontal offset, and obtain the image registration result of the to-be-registered nail fold video.
[0046] For this nail fold video image registration device, the vertical offset and the horizontal offset between each subsequent frame image and the first frame image of the nail fold video obtained by calculating the nail fold video of the blue chrominance component through the variance projection method are used to perform image registration on the nail fold video, solving the problem that the registration of nail fold video images is difficult due to micro-vibrations in clinical detection and affecting the diagnostic results. Through the variance projection method, the detailed changes in the image can be effectively captured, the image registration accuracy of the nail fold video is improved, and the diagnostic efficiency of the nail fold video is improved.
[0047] Specifically, when the acquisition module 1 is executed, the to-be-registered nail fold video can be acquired in various ways. For example, the to-be-registered nail fold video can be acquired by real-time shooting and collection using a nail fold capillaroscope device, or the nail fold video stored in the memory in advance can be acquired as the to-be-registered nail fold video.
[0048] Specifically, when the conversion module 2 performs color space conversion on the to-be-registered nail fold video to obtain the nail fold video of the blue chrominance component, it executes: According to the preset YCrCb color space conversion formula, calculate the luminance component and the chrominance component when the to-be-registered nail fold video is converted from the RGB color space to the YCrCb color space; Based on the luminance component and the chrominance component, convert the to-be-registered nail fold video to the YCrCb color space; Extract the blue chrominance component from the to-be-registered nail fold video in the YCrCb color space to obtain the nail fold video of the blue chrominance component.
[0049] When the conversion module 2 is executing, the nailfold video to be registered initially exists in the RGB color space format. In order to extract image features more suitable for registration, a conversion from the RGB color space to the YCrCb color space is performed. The YCrCb color space decomposes the image information into a luminance component (Y) and two chrominance components (the chrominance components include the blue chrominance component Cb and the red chrominance component Cr). First, according to the preset YCrCb color space conversion formula (the YCrCb color space conversion formula is a conversion formula based on the ITU-R BT.601 or BT.709 standard, which is prior art and will not be elaborated here), the luminance component Y, the blue chrominance Cb, and the red chrominance Cr when converted to the YCrCb color space are calculated. Based on the calculated luminance and chrominance components, the original RGB video is converted into a video in the YCrCb color space. In this converted video, the blue chrominance component video (i.e., the nailfold video of the blue chrominance component) is extracted for subsequent image registration processing. The blue chrominance component is selected considering its relatively insensitive characteristic to illumination changes, which can effectively reduce the impact of non-uniform illumination on image registration, better reflect the true structure of the image, and thus improve the accuracy and robustness of registration. Through color space conversion, video data suitable for subsequent registration steps can be effectively prepared.
[0050] Specifically, when the calculation module 3 calculates the vertical offset and horizontal offset of each subsequent frame image in the nailfold video of the blue chrominance component with respect to the reference frame image (the first frame image) by the variance projection method, it performs the following: By the variance projection method, calculate the projection variance of each frame image in the nailfold video of the blue chrominance component to obtain the projection vector of each frame image in the nailfold video of the blue chrominance component; Taking the first frame image as the reference frame image, move the subsequent frame images in the nailfold video of the blue chrominance component to make the Pearson correlation coefficient between the projection vector of each subsequent frame image and the projection vector of the reference frame image reach the highest value, so as to calculate the vertical offset and horizontal offset of each subsequent frame image with respect to the reference frame image.
[0051] Specifically, the projection vector includes a vertical projection vector and a horizontal projection vector; when the calculation module 3 calculates the projection variance of each frame image in the nailfold video of the blue chrominance component by the variance projection method to obtain the projection vector of each frame image in the nailfold video of the blue chrominance component, it performs the following: Perform noise reduction processing on each frame image in the nailfold video of the blue chrominance component to obtain the noise-reduced blue chrominance component; Calculate the projection variances of the noise-reduced blue chrominance component in the horizontal and vertical directions to obtain the horizontal projection vector and the vertical projection vector.
[0052] When the calculation module 3 is executing, noise reduction processing is performed on each frame image in the nailfold video of the blue chrominance component to obtain the noise-reduced blue chrominance component. The noise reduction processing can adopt image filtering algorithms such as Gaussian filtering or median filtering, etc., to reduce the interference of image noise on the subsequent projection variance calculation and improve the accuracy of the projection vector.
[0053] Specifically, when the calculation module 3 calculates the projection variances of the noise-reduced blue chrominance component in the horizontal and vertical directions to obtain the horizontal projection vector and the vertical projection vector, it executes: Calculate the variance of the pixel values in each row of the noise-reduced blue chrominance component row by row, so as to combine the variance values of each row to form the horizontal projection vector; Calculate the variance of the pixel values in each column of the noise-reduced blue chrominance component column by column, so as to combine the variance values of each column to form the vertical projection vector.
[0054] When the calculation module 3 is executing, when calculating the horizontal projection vector, it scans the noise-reduced Cb component image (each frame image in the noise-reduced blue chrominance component) row by row, and calculates the variance of these pixel values for each row of pixel values. For example, for a certain row of pixel values, first calculate the average value of the pixel values in this row, then calculate the square of the difference between each pixel value and the average value, and then add up all the squared values and divide by the number of pixels to obtain the variance of the pixel values in this row. Combine the variance values calculated for each row in the order of rows, thereby forming the horizontal projection vector. The calculation method of the vertical projection vector is similar to that of the horizontal projection vector, the difference being that the vertical projection vector calculates the variance of the pixel values in each column of the noise-reduced Cb component image column by column, and combines the variance values of each column in the order of columns to form. Thus, through the horizontal projection vector and the vertical projection vector, the projection characteristics of the image in the horizontal and vertical directions can be fully described.
[0055] Specifically, when the calculation module 3 uses the first frame image as the reference frame image and moves the subsequent frame images (the subsequent frame images are each frame image after the first frame image in the nailfold video of the blue chrominance component) in the nailfold video of the blue chrominance component, so that the Pearson correlation coefficient between the projection vector of each subsequent frame image and the projection vector of the reference frame image reaches the highest value, to calculate the vertical offset and horizontal offset between each subsequent frame image and the reference frame image, it executes: Using the first frame image as the reference frame image, after moving the projection vectors of each subsequent frame image in the nailfold video of the blue chrominance component multiple times, calculate the Pearson correlation coefficient between the projection vector of each subsequent frame image after each movement and the projection vector of the reference frame image; Extract the highest value from the Pearson correlation coefficients belonging to the same subsequent frame image; According to the displacement of the projection vector corresponding to the highest value, the vertical offset and horizontal offset between each subsequent frame image and the reference frame image are calculated.
[0056] When the calculation module 3 executes, it will move the projection vector of the subsequent frame image horizontally and vertically multiple times within a preset range (the preset range can be set according to actual needs). For example, it moves with a step size of one pixel. After each move, the Pearson correlation coefficient between the moved projection vector and the projection vector of the reference frame image will be calculated. Thus, a series of Pearson correlation coefficient values can be obtained, and each value corresponds to a specific move. After completing all the preset number of moves (the preset number can be set according to actual needs), the highest value will be found from these Pearson correlation coefficient values. The position of the projection vector corresponding to this highest value is considered the best projection vector matching position between the corresponding subsequent frame image and the reference frame image. The displacement corresponding to the projection vector corresponding to this highest value is the best displacement of the current subsequent frame image relative to the reference frame image. According to this best displacement, the vertical offset and horizontal offset of the subsequent frame image relative to the reference frame image can be determined. In summary, each subsequent frame image is moved in sequence to obtain the vertical offset and horizontal offset of each subsequent frame image during registration. Among them, the calculation method of the Pearson correlation coefficient is a prior art and will not be elaborated here.
[0057] For example, taking the first frame image as the reference frame image, the registration is performed for the second subsequent frame image. First, the displacement search ranges in both the horizontal and vertical directions are set to [-5, 5] pixels, and the displacement step size is 1 pixel. Then, the projection vector of the second frame image moves horizontally and vertically within the range of [-5, 5] pixels with a step size of 1 pixel respectively. After each move, the Pearson correlation coefficient between the moved projection vector of the second frame image and the projection vector of the first frame image is calculated. Thus, for the second frame image, 11 * 11 = 121 Pearson correlation coefficient values can be obtained. The highest value is extracted from these 121 correlation coefficient values. Suppose the displacement corresponding to the highest value is 3 pixels in the horizontal direction and -2 pixels in the vertical direction. Then, the horizontal offset of the second frame image relative to the first frame image is 3 pixels, and the vertical offset is -2 pixels. This offset will be used for subsequent image registration operations to correct the position of the second frame image so that it is aligned with the first frame image. By this method of multiple moves and highest value extraction, the best matching position between images can be found more accurately, improving the accuracy of image registration.
[0058] Specifically, when the registration module 4 is executed, according to the vertical offset and horizontal offset corresponding to each subsequent frame image, the subsequent frame images in the to-be-registered nailfold video are offset to perform image registration on the to-be-registered nailfold video, and an image registration result of the to-be-registered nailfold video is obtained. This image registration result provides more accurate and reliable image information for the doctor, which helps him make a correct diagnosis.
[0059] Specifically, the nailfold video image registration device further includes: A determination module, configured to calculate the similarity between each subsequent frame image after registration and the reference frame image, so as to determine the image registration effect of the to-be-registered nailfold video.
[0060] In the same nailfold video, there are only differences in the detailed parts between each frame of images (doctors can diagnose the nailfold video based on the differences in the detailed parts), that is, there is a certain similarity between each frame of images. Therefore, after the determination module obtains the image registration result, calculation methods such as mean square error MSE, peak signal-to-noise ratio PSNR, or structural similarity index SSIM are used to calculate the similarity value between each subsequent frame image after registration and the reference frame image. And this similarity value is compared with a set threshold (the value of the threshold can be set according to actual needs). When the similarity value is higher than or equal to the threshold, it is considered that the image registration effect is good, and the quality of the registered video image meets the requirements of subsequent analysis or diagnosis. On the contrary, if the similarity value is lower than the threshold, it may be necessary to re-perform image registration or manual intervention. In summary, through the similarity calculation step, the image registration effect of the to-be-registered nailfold video can be determined. And the similarity calculation adds a quality control link to the entire registration process, which can more objectively understand the quality of the registration result, so as to judge whether the registered video can be used for subsequent observation analysis or disease diagnosis, ensuring the reliability and practicality of the registration result.
[0061] Specifically, the similarity calculation can be achieved through various image similarity measurement methods. For example, indicators such as the mean squared error (MSE), peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM) can be adopted. The mean squared error calculates the average squared difference between the pixel values of two images. The smaller the value, the higher the image similarity. The peak signal-to-noise ratio is calculated based on the mean squared error. The larger the value, the higher the image quality and the similarity. The structural similarity index measures from three aspects of image brightness, contrast, and structure, with a value range from 0 to 1. The closer to 1, the higher the structural similarity of the image. In the specific implementation process, a suitable similarity measurement method is selected and a reasonable similarity threshold is set. The similarity value of each subsequent frame image after registration and the reference frame image is calculated, and this similarity value is compared with the set threshold. If the similarity value is higher than or equal to the threshold, it is considered that the image registration effect is good, and the quality of the registered video image meets the requirements of subsequent analysis or diagnosis. On the contrary, if the similarity value is lower than the threshold, it may be necessary to re-perform image registration or perform manual intervention.
[0062] As can be seen from the above, the nailfold video image registration device obtains the nailfold video to be registered, performs color space conversion on the nailfold video to be registered to obtain the nailfold video of the blue chrominance component, and uses the variance projection method. With the first frame image as the reference frame image, the subsequent frame images in the nailfold video of the blue chrominance component are moved to calculate the vertical offset and horizontal offset of each subsequent frame image and the reference frame image. According to the vertical offset and horizontal offset, image registration is performed on the nailfold video to be registered to obtain the image registration result of the nailfold video to be registered; thus, through the vertical offset and horizontal offset of each subsequent frame image and the first frame image calculated by the variance projection method for the nailfold video of the blue chrominance component, image registration is performed on the nailfold video, solving the problem that the nailfold video image is difficult to register due to micro-vibrations in clinical detection, which affects the diagnosis result. The variance projection method can effectively capture the detailed changes in the image, improve the image registration accuracy of the nailfold video, and improve the diagnosis efficiency of the nailfold video.
[0063] Please refer to Figure 3 , Figure 3A schematic structural diagram of an electronic device provided by an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores a computer program executable by the processor 301. When the electronic device runs, the processor 301 executes the computer program to perform the nailfold video image registration method in any optional implementation manner of the above embodiment to achieve the following functions: obtaining a nailfold video to be registered, performing color space conversion on the nailfold video to be registered to obtain a nailfold video of the blue chrominance component, through the variance projection method, using the first frame image as a reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image, and performing image registration on the nailfold video to be registered according to the vertical offset and horizontal offset to obtain the image registration result of the nailfold video to be registered.
[0064] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it performs the nailfold video image registration method in any optional implementation manner of the above embodiment to achieve the following functions: obtaining a nailfold video to be registered, performing color space conversion on the nailfold video to be registered to obtain a nailfold video of the blue chrominance component, through the variance projection method, using the first frame image as a reference frame image, moving the subsequent frame images in the nailfold video of the blue chrominance component to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image, and performing image registration on the nailfold video to be registered according to the vertical offset and horizontal offset to obtain the image registration result of the nailfold video to be registered. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disc.
[0065] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical or other forms.
[0066] In addition, the units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0067] Furthermore, in each embodiment of this application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.
[0068] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.
[0069] The above are only the embodiments of this application and are not used to limit the protection scope of this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.
Claims
1. A nail fold video image registration method, used for performing image registration on a nail fold video, characterized in that: Includes steps: Obtain the nailfold video to be registered; Performing color space conversion on the nailfold video to be registered to obtain a nailfold video of a blue chromaticity component; By using a variance projection method, taking the first frame image as a reference frame image, the subsequent frame images in the nailfold video of the blue chromaticity component are moved to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image; Image registration is performed on the nailfold video to be registered according to the vertical offset and the horizontal offset to obtain an image registration result of the nailfold video to be registered.
2. The nailfold video image registration method according to claim 1, characterized in that: Performing color space conversion on the nailfold video to be registered to obtain a nailfold video of a blue chromaticity component, including: According to a preset YCrCb color space conversion formula, the brightness component and the chromaticity component of the nailfold video to be registered are calculated when the RGB color space is converted to the YCrCb color space; Based on the brightness component and the chrominance component, converting the nailfold video to be registered into a YCrCb color space; A blue chromaticity component is extracted from the nailfold video to be registered in the YCrCb color space to obtain a nailfold video of the blue chromaticity component.
3. The nailfold video image registration method according to claim 1, characterized in that: By using a variance projection method, taking the first frame image as a reference frame image, moving the subsequent frame images in the nailfold video of the blue chromaticity component, so as to calculate the vertical offset and horizontal offset of each subsequent frame image from the reference frame image, including: By using a variance projection method, the projection variance of each frame of the image in the nailfold video of the blue chromaticity component is calculated to obtain a projection vector of each frame of the image in the nailfold video of the blue chromaticity component; Taking the first frame image as the reference frame image, the subsequent frame images in the nailfold video of the blue chromaticity component are moved so that the Pearson correlation coefficient between the projection vector of each frame of the subsequent frame image and the projection vector of the reference frame image reaches a maximum value, so as to calculate the vertical offset and horizontal offset of each frame of the subsequent frame image and the reference frame image.
4. The nailfold video image registration method according to claim 3, characterized in that: The projection vector includes a vertical projection vector and a horizontal projection vector; the projection variance of each frame image in the nailfold video of the blue chromaticity component is calculated by a variance projection method to obtain the projection vector of each frame image in the nailfold video of the blue chromaticity component, including: Performing noise reduction processing on each frame of the nailfold video of the blue chromaticity component to obtain a noise-reduced blue chromaticity component; The projection variance of the blue chromaticity component after noise reduction in the horizontal direction and the vertical direction is calculated to obtain a horizontal projection vector and a vertical projection vector.
5. The nailfold video image registration method according to claim 4, characterized in that: Calculating the projection variance of the blue chromaticity component after noise reduction in the horizontal direction and the vertical direction to obtain a horizontal projection vector and a vertical projection vector, including: Calculating the variance of each row of pixel values in the blue chrominance component after noise reduction row by row, so as to combine the variance values of each row to form a horizontal projection vector; The variance of the pixel values in each column of the blue chrominance component after noise reduction is calculated column by column, so as to combine the variance values of each column to form a vertical projection vector.
6. The nailfold video image registration method according to claim 3, characterized in that: Taking the first frame image as the reference frame image, moving the subsequent frame images in the nailfold video of the blue chromaticity component so that the Pearson correlation coefficient between the projection vector of each frame of the subsequent frame image and the projection vector of the reference frame image reaches a maximum value, so as to calculate the vertical offset and horizontal offset of each frame of the subsequent frame image and the reference frame image, including: Taking the first frame image as the reference frame image, after moving the projection vector of each subsequent frame image in the nailfold video of the blue chromaticity component multiple times, calculate the Pearson correlation coefficient between the projection vector of each subsequent frame image after each movement and the projection vector of the reference frame image; The highest value is extracted from the Pearson correlation coefficients of subsequent frame images belonging to the same frame; According to the projection vector displacement corresponding to the highest value, the vertical offset and the horizontal offset between each subsequent frame image and the reference frame image are calculated.
7. The nailfold video image registration method according to claim 1, characterized in that: After performing image registration on the to-be-registered nailfold video according to the vertical offset and the horizontal offset to obtain the image registration result of the to-be-registered nailfold video, the method further includes: The similarity between each subsequent frame image and the reference frame image after registration is calculated to determine the image registration effect of the nailfold video to be registered.
8. A nail fold video image registration device, used for performing image registration on a nail fold video, characterized in that: include: An acquisition module, used for acquiring the nailfold video to be registered; A conversion module, used for performing color space conversion on the nailfold video to be registered to obtain a nailfold video of a blue chromaticity component; A calculation module, used to move the subsequent frame images in the nailfold video of the blue chromaticity component by using a variance projection method and taking the first frame image as a reference frame image, so as to calculate a vertical offset and a horizontal offset between each frame of the subsequent frame image and the reference frame image; A registration module is used to perform image registration on the nailfold video to be registered according to the vertical offset and the horizontal offset to obtain the image registration result of the nailfold video to be registered.
9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the nailfold video image registration method as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the nailfold video image registration method according to any one of claims 1 to 7 are performed.