A lean expression method for body fluid microscopy video based on image fusion

Abstract

The present invention discloses a lean expression method for body fluid microscopic video based on image fusion. The present invention is based on image fusion technology and combines artificial intelligence formed component recognition technology to extract, fuse, sort, suppress non-maximum and other technical processing on key information of formed components in each frame of image, and expresses effective information content in all videos of a field of view with pictures containing effective foreground and background information. This not only makes the detection result closer to the real state of the sample, further improves the intelligence level of the instrument, but also greatly improves the review speed of the detection result, and solves the problems of difficulty in focusing the microscope caused by the thick body fluid sample that cannot be completely precipitated when the body fluid formed component detection adopts a fully automatic microscope to take pictures or videos for analysis, difficulty in fully reflecting the foreground and background information of the same position of the sample at different levels, time-consuming and labor-intensive video viewing for manual review, and inability to directly and effectively express the sample state.

Description

Technical Field

[0001] The invention relates to a lean expression method of a body fluid microscopic video based on image fusion, and in particular to a lean expression method of a body fluid microscopic detection video. Background Art

[0002] At present, all microscopic examinations of formed components in body fluids mostly use fully automatic microscopes to take pictures or videos, and then perform image analysis. Image analysis is used because when the body fluid sample is thick, the formed components in the sample cannot be completely precipitated, or the precipitation time is long, resulting in low sample detection efficiency, or when the optical magnification is large, the image depth of field is small, the automatic focus control of the microscope is difficult to focus, and it is difficult to achieve complete detection of different formed components at all levels of the sample; although video shooting can be fully detected, it is time-consuming and laborious to view the video during manual review, and it cannot directly express the status of the formed components in the sample.

[0003] Therefore, based on the image fusion technology combined with artificial intelligence formed component recognition technology, the key information of formed components at each level of each field of view video is extracted, fused, sorted, non-maximum suppression and other technical processing is carried out, and an intuitive, concise and comprehensive expression technology for microscopic detection videos of body fluid samples is studied, and finally the video information of a field of view is expressed with pictures containing information at all levels. Not only does it make the test results closer to the real state of the sample, but it also further improves the intelligence of the instrument and greatly improves the review speed of the test results.

[0004] In summary, a lean expression method for body fluid microscopy videos based on image fusion is an effective method to improve the intelligence level of microscopic examination of formed elements in body fluids and increase the detection efficiency. Summary of the invention

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] The present invention proposes a lean expression method for body fluid microscopic video based on image fusion, which specifically comprises the following steps:

[0007] S1: Capture a video, split the video into several frames, sort them in sequence and summarize them into a test image set, and set the initial image to be the test image corresponding to the sequence number 1 in the test image set;

[0008] S2: sequentially extracting an image to be tested from the image set to be tested for recognition analysis, obtaining data of the image to be tested, and cropping the area where the data corresponds to the image to be tested as a fusion candidate image, and sequentially summarizing it into the fusion candidate image set;

[0009] S3: Perform clarity calculation on the image to be tested and the initial image, replace the initial image with the image corresponding to the calculation result, and output it as the fusion base;

[0010] S4: performing a non-maximum suppression operation on the fusion candidate image set, and outputting the fusion candidate image corresponding to the operation result as a fusion candidate region, and sequentially summarizing them into the fusion candidate region set;

[0011] S5: Determine the number of remaining images to be tested in the image set to be tested. If the number of remaining images to be tested is 0, perform gradient contrast fusion on the fusion base and the fusion candidate areas in the fusion candidate area set in sequence, and output the fused image, which is summarized into the fusion result set in order. If the number of remaining images to be tested is not 0, return to execute S2.

[0012] Furthermore, in S1, the video is split according to the video acquisition timeline.

[0013] Furthermore, in S2, a deep learning algorithm for identifying formed components in body fluids is combined to analyze and obtain parameters such as the formed component area, credibility, and type in the body fluid on the image to be tested.

[0014] Further, including:

[0015] S2.1: dividing the image to be tested into a plurality of small detection images, wherein the plurality of small detection images are in a grid shape;

[0016] S2.2: The target identified from the detection thumbnail is projected onto the image to be tested, a non-maximum suppression operation is performed, and the operation result is output as the recognition result.

[0017] Furthermore, the clarity calculation in S3 includes:

[0018] S3.1: Perform convolution operation on each pixel of the image to be tested using the Laplace gradient operator. , where I is the pixel value and L is the convolution result;

[0019] S3.2: Calculate the average Laplace gradient value of the image to be tested, denoted as Mi, and calculate the average Laplace gradient value of the initial image, denoted as Mmax;

[0020] S3.3: Compare the sizes of Mi and Mmax. If Mmax is greater than Mi, output the image corresponding to Mmax as the initial image. Otherwise, output the image corresponding to Mi as the initial image, and output the output image as the fusion basis.

[0021] Furthermore, the non-maximum suppression operation in S4 includes:

[0022] S4.1: Arrange the confidence of each fusion candidate image in the fusion candidate image set in descending order, name the fusion candidate image with the highest confidence as image A, and add it to the output list;

[0023] S4.2: Calculate the area ratio of the overlapping area between image A and all fusion candidate images, and delete the fusion candidate images whose formed element type is the same as image A and whose intersection-over-union ratio is greater than a set threshold from the fusion candidate image set;

[0024] S4.3: again selecting the fusion candidate image with the highest confidence among the remaining fusion candidate images and adding it to the output list;

[0025] S4.4: Repeat the above process until the number of remaining fusion candidate images in the fusion candidate image set is 0, and output the output list as the fusion candidate area.

[0026] Furthermore, S5 includes:

[0027] S5.1: performing Laplace gradient operation on the fusion candidate area and the corresponding area of ​​the fusion base;

[0028] S5.2: Compare the average Laplace gradient values ​​of the candidate fusion area and the corresponding area of ​​the fusion base;

[0029] S5.3: If the average Laplace gradient value of the fused candidate region is greater than the average Laplace gradient value of the region corresponding to the fused base, the fused candidate region is replaced with the region corresponding to the fused base;

[0030] S5.4: Perform gradient smoothing on the fused area.

[0031] The present invention proposes a lean expression method for body fluid microscopic video based on image fusion, which has the following beneficial effects:

[0032] The lean expression picture obtained by the present invention not only contains all the tangible component information in the entire video image data, but also has high credibility of each component and clear pictures. It can leanly describe and reflect the effective information in the entire video image data, so as to achieve the effect of seeing the whole picture at a glance. It not only makes the report result closer to the real state of the sample, but also further improves the intelligence level of the instrument, and greatly improves the review speed of the test results.

[0033] The method of the present invention effectively solves the problem of difficulty in reviewing the results of microscopic examination of body fluid samples in clinical medical equipment, greatly relieves the labor intensity of operators, and improves the accuracy of the review results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the data collection and processing algorithm involved in the present invention;

[0035] Figure 2 It is a schematic diagram of intersection-and-union ratio calculation in the present invention;

[0036] Figure 3 It is a schematic diagram of a preferred embodiment of the present invention;

[0037] Figure 4 It is a comparative schematic diagram before and after fusion of the present invention. DETAILED DESCRIPTION

[0038] The technical scheme of the present invention will be further described in detail below in conjunction with specific embodiments. The following embodiments are only exemplary to illustrate and explain the present invention, and should not be construed as limiting the scope of protection of the present invention. All technologies implemented based on the above content of the present invention are included in the scope that the present invention is intended to protect.

[0039] Unless otherwise specified, the raw materials and reagents used in the following examples are commercially available or can be prepared by known methods.

[0040] Example 1

[0041] The present invention proposes a lean expression method for body fluid microscopic video based on image fusion, which specifically comprises the following steps:

[0042] S1: Capture a video, split the video into several frames according to the video acquisition time axis, sort them in sequence and summarize them into a test image set, which is a collection of several frame images after the video is split, and set the initial image to be the test image corresponding to the sequence number 1 in the test image set;

[0043] S2: Sequentially extract one image to be tested from the set of images to be tested for recognition and analysis, obtain the data of the image to be tested, and combine it with the deep learning body fluid formed component recognition algorithm to analyze the formed component area, credibility, type and other parameters on the image to be tested. Use deep learning networks such as yolo, faster-CNN, EfficientDet, etc. First, manually collect the clearest and most perfect pictures of the actual samples for annotation, training, deployment and other steps:

[0044] In order to improve the recognition stability, random scaling, random cropping, occlusion, random arrangement, blurring, random rotation and other methods are used to enhance the image of the training target during training to make up for the defects of sparse and insufficient samples and unclear target micro-structure features;

[0045] S2.1: In order to adapt to the detection of tiny targets and improve the detection rate of micro cells, the image to be tested is divided into several detection small pictures. The detection small pictures are in a grid shape. In order to avoid the target being segmented and truncated, a certain segmentation overlap area is set, so that the segmented small pictures are used for recognition;

[0046] S2.2: The target identified from the detection image is projected onto the image to be tested, and a non-maximum suppression operation is performed. The operation result is output as the recognition result to realize the automatic recognition of the formed components of the body fluid microscopic video. The maximum recognition speed is greater than 30fps, the recognition rate is greater than 90%, and the recognition accuracy rate is greater than 95%.

[0047] The area where the data corresponds to the image to be tested is cropped as a fusion candidate image, and is summarized into a fusion candidate image set in order; there are many types of formed components in the sample, and the number is also large, that is, the area containing formed components on the sample is cropped as a fusion candidate image, and in the recognition process, different formed component types will have multiple recognition areas, and different formed component types will be calculated separately in the subsequent non-maximum suppression operation, that is, the fusion candidate image set contains multiple different formed component type subsets on a test image.

[0048] S3: Performing definition operation on the image to be tested and the initial image, replacing the initial image with the image corresponding to the definition operation result, and outputting it as the fusion basis. After each round of definition operation is completed, the initial image of the next round of definition operation is the fusion basis of this round.

[0049] S3.1: Perform convolution operation on each pixel of the image to be tested using the Laplace gradient operator. , where I is the pixel value and L is the convolution result;

[0050] S3.2: Calculate the average Laplace gradient value of the image to be tested, denoted as Mi, and calculate the average Laplace gradient value of the initial image, denoted as Mmax;

[0051] S3.3: Compare the size of Mi and Mmax. If Mmax is greater than Mi, then output the image corresponding to Mmax as the initial image, that is, the initial image after this round of operation is the initial image. The previous initial image can be regarded as a memory, and the next initial image is a frame image split from the video. Otherwise, output the image corresponding to Mi as the initial image, that is, the initial image after this round of operation is the image to be tested, and output the output image as the fusion basis; the fusion basis is actually to compare the two frames of images in the video in sequence, and output a clearer frame of image, that is, to compare all the images to be tested split from the video one by one, and output the clearest frame of image to be tested. Except for the first clarity operation, the initial image is the image to be tested corresponding to the sequence number 1 in the image set to be tested. The subsequent clarity operations are all the fusion basis of the previous round, that is, the clearer frame of the two frames in the previous round of clarity operation.

[0052] S4: Perform non-maximum suppression operation on the fusion candidate image set, and output the fusion candidate image corresponding to the operation result as the fusion candidate area, and summarize it into the fusion candidate area set in order, exclude the parts of the area with multiple repeated identification or low credibility, and retain the area with the highest confidence, that is, the area with the most obvious and credible features;

[0053] S4.1: Arrange the confidence of each fusion candidate image in the fusion candidate image set in descending order, name the fusion candidate image with the highest confidence as image A, and add it to the output list;

[0054] S4.2: Calculate the overlapping area ratio between image A and all fusion candidate images. The overlapping area ratio is the overlap rate, which is recorded as the IOU value. The fusion candidate images whose formed element type is the same as that of image A and whose intersection over union ratio is greater than the set threshold are deleted from the fusion candidate image set. At this time, only an area on image A, that is, the image to be selected (the fusion candidate image) is marked and saved.

[0055] S4.3: Select the fusion candidate image with the highest confidence from the remaining fusion candidate images again and add it to the output list; that is, find another area on the image to be selected again.

[0056] S4.4: Repeat the above process until the number of remaining fusion candidate images in the fusion candidate image set is 0, and output the output list as the fusion candidate area; after repeating the above process, the area with the highest confidence in the fusion candidate image set can be found. It can remove low-confidence areas and repeated identification areas at the same position, and retain the fusion candidate area with the highest confidence, which can also be called the optimal area to be fused.

[0057] That is, suppose there are 20 regions, sort them according to the classification category, suppose there are 6 types of ABM, and they are sorted in descending order of confidence as abm1, abm2, abm3, abm4, abm5, and abm6. Starting from the highest probability abm1, determine whether the overlap rate between abm2~abm6 and abm1 is greater than the preset threshold; if the overlap rate between abm2, abm4 and abm1 exceeds the threshold, then discard abm2 and abm4, and mark abm1 as what we have retained, add it to the output list, select abm3 with the highest confidence from the remaining abm3, abm5, and abm6, then determine the overlap rate between abm5, abm6 and abm3, and discard the region with an overlap rate greater than the preset threshold, and mark abm3 as the second retained region, and add it to the output list, repeat this process to find all the retained regions. Then perform the operation on the remaining categories in the same way.

[0058] S5: Determine the number of remaining images to be tested in the image set to be tested. If the number of remaining images to be tested is 0, perform gradient contrast fusion on the fusion base and the fusion candidate areas in the fusion candidate area set in sequence, and output the fused image, which is summarized into the fusion result set in order. If the number of remaining images to be tested is not 0, return to execute S2.

[0059] When the number of remaining images to be tested is 0, the fusion base is the clearest frame among the several frames of images to be tested divided from the entire video.

[0060] S5.1: performing Laplace gradient operation on the fusion candidate area and the corresponding area of ​​the fusion base;

[0061] S5.2: Compare the average Laplace gradient values ​​of the candidate fusion area and the corresponding area of ​​the fusion base;

[0062] S5.3: If the average Laplace gradient value of the fused candidate area is greater than the average Laplace gradient value of the area corresponding to the fused base, the pixels of the fused candidate area are replaced with the area corresponding to the fused base;

[0063] S5.4: Perform gradient smoothing on the fused area to make the fused edge transition naturally and obtain a lean expression image of the body fluid microscopy video.

[0064] The sample is observed through a microscope, and the image of the microscope is collected by an acquisition device. The microscope slowly moves upward, and the focus slowly moves upward, so that the sample is identified from bottom to top, thus forming a video. The video is split according to the acquisition timeline, split into several frames of images, and sorted in sequence and summarized into the image set to be tested.

[0065] An image to be tested is sequentially extracted from the image set to be tested for recognition and analysis, and data such as the type of formed elements on the image to be tested is obtained. The corresponding area on the image to be tested is cropped as a fusion candidate image, and is sequentially summarized into the fusion candidate image set. The fusion candidate image set can be numbered sequentially, such as 1 (the image to be tested corresponding to sequence number one in the image set to be tested) -1 (the first fusion candidate image), 2-4 (the fourth fusion candidate image of the image to be tested corresponding to sequence number two in the image set to be tested). In the first round, the image to be tested corresponding to sequence number one in the image set to be tested is extracted, and in the second round, the image to be tested corresponding to sequence number two in the image set to be tested is extracted, and so on.

[0066] Perform clarity calculation on the extracted image to be tested (the first round is the image to be tested corresponding to the sequence number 1 in the image set to be tested). At this time, the initial image is the image to be tested corresponding to the sequence number 1 in the image set to be tested. Perform convolution operation on the Laplace gradient operator of each pixel of the image to be tested. The convolution formula is: , where I is the pixel value, L is the convolution result, and then the average Laplace gradient value of the image to be tested is calculated, recorded as Mi, and the average Laplace gradient value of the initial image is calculated, recorded as Mmax, and the size of Mi and Mmax is compared, and the image corresponding to the larger number between the two is output, that is, when Mmax is greater than Mi, the initial image is output as the fusion basis, and the initial image is replaced by the initial image, when Mmax is less than Mi, the image to be tested is output as the fusion basis, and the image to be tested replaces the initial image, that is, the fusion basis output in this round is used as the initial image of the next round. The initial image of the first round is the image to be tested corresponding to the sequence number one in the image set to be tested, so the fusion basis output in the first round is the image to be tested corresponding to the sequence number one. The second round is to select a clearer frame of image between the sequence number one image to be tested and the sequence number two image to be tested.

[0067] All fusion candidate images in the fusion candidate image set are the cropped areas on the image to be tested extracted in this round. They are sorted in descending order according to the confidence, and the fusion candidate image with the highest confidence is selected and named as image A. Image A is added to the output list, and then image A is calculated with the remaining fusion candidate images in the fusion candidate image set for intersection and union ratio, that is, the overlap rate between image A and each remaining fusion candidate image is calculated, and the fusion candidate images with an overlap rate greater than the set threshold are deleted, and then the fusion candidate image with the highest confidence is extracted from it, named as image B, and the above operation is repeated until the number of remaining fusion candidate images in the fusion candidate image set is 0. The fusion candidate images in the output list of this round are used as the fusion candidate areas of the image to be tested extracted in this round, which can also be called the optimal candidate areas to be fused, and are summarized in the fusion candidate area set in order.

[0068] After the first round of testing image processing is completed, the number of remaining testing images in the testing image set is judged. If the number of remaining testing images is not 0, the next testing image is sequentially extracted from the testing image set for recognition analysis, clarity calculation and non-maximum suppression calculation until the number of remaining testing images in the testing image set is 0. When the number of remaining testing images in the testing image set is 0, the fusion base is the clearest frame of all testing images, and the fusion candidate area set includes multiple fusion candidate areas of each testing image. Each fusion candidate area in the fusion candidate area set is sequentially subjected to gradient contrast fusion with the fusion base, that is, multiple fusion candidate areas of each testing image are sequentially subjected to gradient contrast fusion with the fusion base. In the fusion process of multiple fusion candidate areas of a testing image and the fusion base, the fusion candidate areas are first subjected to gradient contrast fusion. A Laplace gradient operation is performed, and then a Laplace gradient operation is performed on the corresponding area of ​​the fusion candidate area on the fusion substrate, and then the average Laplace gradient value of the fusion candidate area and the corresponding area of ​​the fusion candidate area on the fusion substrate is calculated, and the average Laplace gradient value of the fusion candidate area is compared with the average Laplace gradient value of the area corresponding to the fusion candidate area on the fusion substrate. If the average Laplace gradient value of the fusion candidate area is greater than the average Laplace gradient value of the area corresponding to the fusion candidate area on the fusion substrate, the pixels of the fusion candidate area are replaced by the pixels of the area corresponding to the fusion candidate area on the fusion substrate, that is, the fusion candidate area is replaced by the area corresponding to the fusion candidate area on the fusion substrate, and the fusion area is gradient smoothed to make the fusion edge transition naturally, so as to obtain a frame of the image to be tested based on image fusion of the body fluid microscopy video lean expression picture. Repeat the above operation until all the candidate fusion areas of the tested images have completed gradient contrast fusion with the fusion base, and then summarize all the lean expression pictures into the fusion result set in order. Through the output of the peripheral processor or other devices, the video can be matched with the fusion result. When the player plays a certain frame of the video, it outputs the lean expression picture corresponding to the frame.

[0069] The present invention is an intuitive, concise and comprehensive expression technology for microscopic examination videos of formed components in body fluid samples. It uses pictures containing information at all levels to express the video information of a field of view (pictures containing effective foreground and background information express the effective information content in all videos of a field of view), so that the detection results are closer to the actual state of the sample, further improve the intelligence level of the instrument, greatly improve the review speed of the detection results, and effectively solve the problem of difficulty in reviewing the results of microscopic examination of body fluid samples in clinical medical devices. It solves the problems of difficulty in focusing the microscope caused by the thick body fluid sample that cannot be completely precipitated when the formed component detection of body fluid is carried out by using a fully automatic microscope to take pictures or videos for analysis, difficulty in fully reflecting the foreground and background information of different levels at the same position of the sample, time-consuming and labor-intensive video viewing for manual review, and inability to directly and effectively express the sample status. It greatly relieves the labor intensity of the operator, improves the accuracy of the review results, has a large leading position in technology, and has a very good market prospect.

[0070] The present invention is based on image fusion technology and combines artificial intelligence shaped component recognition technology to extract, fuse, sort, and perform non-maximum suppression on the key information of shaped components at each level in each frame of the field of view video.

[0071] The above are only preferred specific embodiments of the present invention. However, the present invention is not limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A lean expression method for body fluid microscopy video based on image fusion, characterized in that: The specific steps include: S1: Capture a video, split the video into several frames, sort them in sequence and summarize them into a test image set, and set the initial image to be the test image corresponding to the sequence number 1 in the test image set; S2: sequentially extracting an image to be tested from the image set to be tested for recognition analysis, obtaining data of the image to be tested, and cropping the area where the data corresponds to the image to be tested as a fusion candidate image, and sequentially summarizing it into the fusion candidate image set; S2.1: dividing the image to be tested into a plurality of small detection images, wherein the plurality of small detection images are in a grid shape; S2.2: Project the target identified from the detection thumbnail onto the image to be tested, perform non-maximum suppression operation, and output the operation result as the recognition result; S3: Perform clarity calculation on the image to be tested and the initial image, replace the initial image with the image corresponding to the calculation result, and output it as the fusion base; Clarity calculations include: S3.1: Perform convolution operation on each pixel of the image to be tested using the Laplace gradient operator. , where I is the pixel value and L is the convolution result; S3.2: Calculate the average Laplace gradient value of the image to be tested, denoted as Mi, and calculate the average Laplace gradient value of the initial image, denoted as Mmax; S3.3: Compare the sizes of Mi and Mmax. If Mmax is greater than Mi, output the image corresponding to Mmax as the initial image. Otherwise, output the image corresponding to Mi as the initial image, and output the output image as the fused basis. S4: performing a non-maximum suppression operation on the fusion candidate image set, and outputting the fusion candidate image corresponding to the operation result as a fusion candidate region, and sequentially summarizing them into the fusion candidate region set; Non-maximum suppression operations include: S4.1: Arrange the confidence of each fusion candidate image in the fusion candidate image set in descending order, name the fusion candidate image with the highest confidence as image A, and add it to the output list; S4.2: Calculate the area ratio of the overlapping area between image A and all fusion candidate images, and delete the fusion candidate images whose formed element type is the same as image A and whose intersection-over-union ratio is greater than a set threshold from the fusion candidate image set; S4.3: again selecting the fusion candidate image with the highest confidence among the remaining fusion candidate images and adding it to the output list; S4.4: repeat the above process until the number of remaining fusion candidate images in the fusion candidate image set is 0, and output the output list as the fusion candidate area; S5: Determine the number of remaining images to be tested in the image set to be tested. If the number of remaining images to be tested is 0, perform gradient contrast fusion on the fusion base and the fusion candidate areas in the fusion candidate area set in sequence, and output the fused image, which is summarized into the fusion result set in order. If the number of remaining images to be tested is not 0, return to execute S2; S5.1: performing Laplace gradient operation on the fusion candidate area and the corresponding area of ​​the fusion base; S5.2: Compare the average Laplace gradient values ​​of the candidate fusion area and the corresponding area of ​​the fusion base; S5.3: If the average Laplace gradient value of the fused candidate region is greater than the average Laplace gradient value of the region corresponding to the fused base, the fused candidate region is replaced with the region corresponding to the fused base; S5.4: Perform gradient smoothing on the fused area.

2. According to the method of lean expression of body fluid microscopic video based on image fusion according to claim 1, it is characterized in that: In S1, the video is split according to the video acquisition time axis.

3. The method for lean expression of body fluid microscopic video based on image fusion according to claim 1, characterized in that: In S2, the formed component area, credibility, and type parameters of the body fluid on the image to be tested are analyzed in combination with a deep learning body fluid formed component recognition algorithm.

Images