Training method and measurement device based on tight frame mark

Abstract

The present disclosure describes a training method and measurement device based on tight-frame labels, which is a training method for training a network module based on a tight-frame label of a target. The training method includes: constructing a training sample including input image data and label data, wherein the input image data includes multiple images to be trained, the multiple images to be trained include images containing targets belonging to at least one category, and the label data includes a gold standard of the category to which the target belongs and a gold standard of the tight-frame label of the target; inputting the training sample into the network module to obtain predicted data, wherein the predicted data includes predicted segmentation data output by a segmentation network and predicted offsets output by a regression network; and determining a training loss of the network module based on the label data, the predicted segmentation data, and the predicted offsets, and optimizing the network module based on the training loss.

Description

[0001] This application is a divisional application of the patent application with the application date of October 19, 2021, application number 2021112166277, and invention name as Measurement method and measurement device based on deep learning of tight frame mark. Technical Field

[0002] The present disclosure generally relates to the field of deep learning-based recognition technology, and more particularly to a tight-frame-based training method and measurement device. Background Art

[0003] Images often contain information about various objects. Image processing techniques can be used to automatically analyze these objects by identifying them. For example, in the medical field, tissue objects in medical images can be identified and their dimensions measured to monitor changes.

[0004] In recent years, artificial intelligence technologies, particularly deep learning, have experienced significant development, and their applications in areas such as object recognition and measurement have garnered increasing attention. Researchers are using deep learning techniques to identify and measure objects in images. Specifically, some deep learning-based research often uses labeled data to train deep learning-based neural networks to identify and segment objects in images, enabling them to be measured.

[0005] However, the aforementioned object recognition or measurement methods often require precise pixel-level annotated data for neural network training, and collecting pixel-level annotated data often consumes a significant amount of manpower and material resources. Furthermore, while some object recognition methods are not based on pixel-level annotated data, they simply identify objects within an image. However, their boundary recognition is not precise enough, or their accuracy is often low near the boundaries of the object, making them unsuitable for scenarios requiring precise measurement. In these cases, the accuracy of measuring objects in images needs to be improved. Summary of the Invention

[0006] The present disclosure is proposed in view of the above-mentioned situation, and its purpose is to provide a measurement method and a measurement device based on deep learning of tight-frame markers that can identify targets and accurately measure targets.

[0007] To this end, a first aspect of the present disclosure provides a measurement method based on tight bounding box deep learning. The method utilizes a network module trained based on a tight bounding box of a target to identify the target and thereby achieve measurement. The tight bounding box is the minimum bounding rectangle of the target. The measurement method comprises: obtaining an input image including at least one target, wherein the at least one target belongs to at least one category of interest; inputting the input image into the network module to obtain a first output and a second output, wherein the first output comprises the probability that each pixel in the input image belongs to each category, and the second output comprises the offset between the position of each pixel in the input image and the tight bounding box of the target of each category, and the offset in the second output is used as the target offset. The network module comprises a backbone network, a segmentation network for image segmentation based on weakly supervised learning, and a regression network based on bounding box regression. The backbone network is configured to extract a feature map of the input image. The segmentation network uses the feature map as input to obtain the first output, and the regression network uses the feature map as input to obtain the second output, wherein the feature map has the same resolution as the input image; and based on the first and second outputs, the target is identified to obtain the tight bounding box of the target of each category.

[0008] In this disclosure, a network module is constructed, including a backbone network, a segmentation network for image segmentation based on weakly supervised learning, and a regression network based on bounding box regression. The network module is trained based on the tight bounding box of the target. The backbone network receives an input image and extracts a feature map with the same resolution as the input image. The feature map is input into the segmentation network and the regression network, respectively, to obtain a first output and a second output. The tight bounding box of the target in the input image is then obtained based on the first and second outputs to achieve measurement. In this case, the network module trained based on the tight bounding box of the target can accurately predict the tight bounding box of the target in the input image, and thus can accurately measure the target based on the tight bounding box.

[0009] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the size of each target is measured based on the tight frame mark of the target, thereby enabling the target to be accurately measured based on the tight frame mark of the target.

[0010] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, the network module is trained by the following method: constructing a training sample, the input image data of the training sample includes multiple images to be trained, the multiple images to be trained include images containing targets belonging to at least one category, and the label data of the training sample includes a gold standard of the category to which the target belongs and a gold standard of the tight frame label of the target; obtaining the predicted segmentation data output by the segmentation network and the predicted offset output by the regression network corresponding to the training sample based on the input image data of the training sample by the network module; determining the training loss of the network module based on the label data corresponding to the training sample, the predicted segmentation data and the predicted offset; and training the network module based on the training loss to optimize the network module. In this way, an optimized network module can be obtained.

[0011] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, the determining the training loss of the network module based on the label data corresponding to the training sample, the predicted segmentation data and the predicted offset includes: obtaining the segmentation loss of the segmentation network based on the predicted segmentation data and label data corresponding to the training sample; obtaining the regression loss of the regression network based on the predicted offset corresponding to the training sample and the true offset corresponding to the label data, wherein the true offset is the offset between the position of the pixel point of the image to be trained and the gold standard of the tight frame mark of the target in the label data; and obtaining the training loss of the network module based on the segmentation loss and the regression loss. In this case, the predicted segmentation data of the segmentation network can be approximated to the label data through the segmentation loss, and the predicted offset of the regression network can be approximated to the true offset through the regression loss.

[0012] In addition, in the measurement method according to the first aspect of the present disclosure, the target offset is optionally a normalized offset based on the average width and average height of targets of each category, thereby improving the accuracy of identifying or measuring targets with small size variations.

[0013] In addition, in the measurement method of the first aspect of the present disclosure, optionally, the width and height of the tight-framed labels of the objects in the label data are averaged by category to obtain the average width and average height. In this way, the average width and average height of the objects can be obtained through training samples.

[0014] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, using multi-instance learning, multiple training packages are obtained based on the gold standard of the tight frame mark of the target in each image to be trained, and the segmentation loss is obtained based on the multiple training packages of each category, wherein the multiple training packages include multiple positive packages and multiple negative packages, and all pixel points on each straight line of the multiple straight lines connecting the two opposite sides of the gold standard of the tight frame mark of the target are divided into a positive package, and the multiple straight lines include at least one group of mutually parallel first parallel lines and mutually parallel second parallel lines respectively perpendicular to each group of first parallel lines, and the negative package is a single pixel point in the area outside the gold standard of the tight frame mark of all targets of a category. In this way, the segmentation loss can be obtained based on the positive and negative packages of multi-instance learning.

[0015] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the angle of the first parallel line is the angle between an extension of the first parallel line and an extension of any non-intersecting edge of a gold standard of the tight-frame marker of the target, and the angle of the first parallel line is greater than -90° and less than 90°. In this case, the segmentation network can be optimized by dividing the positive envelope at different angles. This can improve the accuracy of the segmentation data predicted by the segmentation network.

[0016] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, the segmentation loss includes a unary term and a paired term, wherein the unary term describes the degree to which each to-be-trained bag belongs to the gold standard of each category, and the paired term describes the degree to which the pixel of the to-be-trained image and the pixels adjacent to the pixel belong to the same category. In this case, the unary loss can be used to constrain the tight bounding box by both the positive and negative bags, and the paired loss can be used to smooth the predicted segmentation result.

[0017] In addition, in the measurement method of the first aspect of the present disclosure, optionally, pixels within a gold standard tight-framed label of at least one target are selected from the training image as positive samples for optimizing the regression network. In this case, optimizing the regression network based on pixels within the true tight-framed label of at least one target can improve the efficiency of regression network optimization.

[0018] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, pixels that fall within a gold standard of a tight frame marker of at least one target are selected from the training image by category as positive samples for each category, and matching tight frames corresponding to the positive samples are obtained to filter the positive samples for each category based on the matching tight frames. The regression network is then optimized using the filtered positive samples for each category, wherein the matching tight frames are the gold standards of the tight frames markers within which the positive samples fall, with the smallest true offset relative to the position of the positive samples. Thus, the regression network can be optimized using the positive samples for each category filtered based on the matching tight frames markers.

[0019] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, let the position of the pixel point be expressed as (x, y), the tight frame mark of a target corresponding to the pixel point is expressed as b = (xl, yt, xr, yb), and the offset of the tight frame mark b of the target relative to the position of the pixel point is expressed as t = (tl, tt, tr, tb), then tl, tt, tr, tb satisfy the formula: tl = (x-xl) / S c1 , tt=(y-yt) / S c2 , tr=(xr-x) / S c1 , tb=(yb-y) / S c2 , where xl, yt represents the position of the upper left corner of the target's tight frame, xr, yb represents the position of the lower right corner of the target's tight frame, S c1 represents the average width of the target of the cth category, S c2 represents the average height of the target of the cth category. Thus, the normalized offset can be obtained.

[0020] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the regression network is optimized by using the expected intersection-and-union (IoU) corresponding to the pixels of the image to be trained, and selecting pixels having an expected IoU greater than a preset IoU. Positive samples meeting the preset expected IoU can thereby be obtained.

[0021] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, multiple bounding boxes of different sizes are constructed with the pixel point of the training image as the center point, and the maximum value of the intersection-and-union ratios (IoU) between each of the multiple bounding boxes and the matching tight bounding box of the pixel point is obtained and used as the expected IoU, wherein the matching tight bounding box is the gold standard tight bounding box within which the pixel point of the training image falls, and the tight bounding box has the smallest actual offset relative to the position of the pixel point. In this way, the expected IoU can be obtained.

[0022] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the desired intersection over union satisfies the formula: where r1 and r2 are the relative positions of the pixel points of the image to be trained in the matching tight bounding box label, 0 < r1, r2 < 1, IoU1(r1, r2) = 4r1r2, IoU2(r1, r2) = 2r1 / (2r1(1 - 2r2) + 1), IoU3(r1, r2) = 2r2 / (2r2(1 - 2r1) + 1), IoU4(r1, r2) = 1 / (4(1 - r1)(1 - r2)). Thus, the desired intersection over union can be obtained.

[0023] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the minimum true offset is obtained by comparing the L1 norm of the true offset. In this case, the minimum true offset can be obtained based on the L1 norm, and then the matching tight bounding box label can be obtained.

[0024] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the regression loss satisfies the formula: where C represents the number of categories, M c represents the number of positive samples of the c-th category, t ic represents the true offset corresponding to the i-th positive sample of the c-th category, v ic represents the predicted offset corresponding to the i-th positive sample of the c-th category, and s(x) represents the sum of the smooth L1 losses of all elements in x. Thus, the regression loss can be obtained.

[0025] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the tight bounding box labels of the objects of each category obtained by identifying the object based on the first output and the second output are: the position of the pixel point with the largest local probability belonging to each category is obtained from the first output as the first position, and the tight bounding box labels of the objects of each category are obtained based on the position corresponding to the first position in the second output and the object offset of the corresponding category. In this case, one object or multiple objects of each category can be identified.

[0026] In addition, in the measurement method according to the first aspect of the present disclosure, optionally, the sizes of multiple objects of the same category differ from each other by less than 10 times. Thus, the accuracy of object recognition can be further improved.

[0027] In addition, in the measurement method involved in the first aspect of the present disclosure, optionally, the backbone network includes an encoding module and a decoding module, the encoding module is configured to extract image features at different scales, and the decoding module is configured to map the image features extracted at different scales back to the resolution of the input image to output the feature map. In this way, a feature map consistent with the resolution of the input image can be obtained.

[0028] According to a second aspect of the present disclosure, there is provided a measurement device based on deep learning of tight-frame labels, which is a measurement device that uses a network module trained based on a tight-frame label of a target to identify the target and thereby realize measurement. The tight-frame label is the minimum circumscribed rectangle of the target. The measurement device includes an acquisition module, a network module, and an identification module; the acquisition module is configured to acquire an input image including at least one target, and the at least one target belongs to at least one category of interest; the network module is configured to receive the input image and acquire a first output and a second output based on the input image, wherein the first output includes the probability that each pixel point in the input image belongs to each category, and the second output includes the probability that each pixel point in the input image belongs to each category. The offset between the position of each pixel point in the image and the tight frame mark of each category of the target is used, and the offset in the second output is used as the target offset, wherein the network module includes a backbone network, a segmentation network for image segmentation based on weakly supervised learning, and a regression network based on bounding box regression, the backbone network is used to extract a feature map of the input image, the segmentation network uses the feature map as input to obtain the first output, and the regression network uses the feature map as input to obtain the second output, wherein the feature map has the same resolution as the input image; and the recognition module is configured to recognize the target based on the first output and the second output to obtain the tight frame mark of each category of the target.

[0029] In this disclosure, a network module is constructed, including a backbone network, a segmentation network for image segmentation based on weakly supervised learning, and a regression network based on bounding box regression. The network module is trained based on the tight bounding box of the target. The backbone network receives an input image and extracts a feature map with the same resolution as the input image. The feature map is input into the segmentation network and the regression network, respectively, to obtain a first output and a second output. The tight bounding box of the target in the input image is then obtained based on the first and second outputs to achieve measurement. In this case, the network module trained based on the tight bounding box of the target can accurately predict the tight bounding box of the target in the input image, and thus can accurately measure the target based on the tight bounding box.

[0030] According to the present disclosure, a measurement method and a measurement device based on tight-frame deep learning are provided, which are capable of identifying a target and accurately measuring the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present disclosure will now be explained in further detail, by way of example only, with reference to the accompanying drawings, in which:

[0032] Figure 1 It is a schematic diagram showing an application scenario of the measurement method based on tight frame mark deep learning involved in the example of the present disclosure.

[0033] FIG. 2( a ) is a schematic diagram showing a fundus image according to an example of the present disclosure.

[0034] FIG2( b ) is a schematic diagram showing the recognition result of the fundus image according to the example of the present disclosure.

[0035] Figure 3 is a schematic diagram showing an example of a network module involved in the examples of the present disclosure.

[0036] Figure 4 is a schematic diagram illustrating another example of a network module according to an example of the present disclosure.

[0037] Figure 5 is a flowchart illustrating a training method of a network module according to an example of the present disclosure.

[0038] Figure 6 Schematic diagram showing a positive package involved in the examples of the present disclosure.

[0039] Figure 7 is a schematic diagram showing a border constructed with a pixel point as the center involved in the example of the present disclosure.

[0040] FIG8( a ) is a flowchart illustrating a measurement method based on tight-frame deep learning according to an example of the present disclosure.

[0041] FIG8( b ) is a flowchart illustrating another example of a measurement method based on tight-frame deep learning according to an example of the present disclosure.

[0042] FIG9( a ) is a block diagram illustrating a measurement device based on tight-frame deep learning according to an example of the present disclosure.

[0043] FIG9( b ) is a block diagram illustrating another example of a measurement device based on tight-frame deep learning according to an example of the present disclosure.

[0044] FIG9( c ) is a block diagram illustrating another example of a measurement device based on tight-frame deep learning according to an example of the present disclosure. DETAILED DESCRIPTION

[0045] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present disclosure will be described in detail. In the following description, identical symbols are assigned to identical components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components to each other or the shapes of the components, etc. may be different from the actual ones. It should be noted that the terms "including" and "having" in the present disclosure and any variations thereof, such as a process, method, system, product or device that includes or has a series of steps or units, are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. All methods described in the present disclosure may be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by the context.

[0046] The present disclosure relates to a measurement method and a measurement device based on deep learning using tight-frame markers, which can identify targets and improve the accuracy of target measurement. For example, the optic disc or its tight-frame marker in a fundus image can be identified, and the size of the optic disc or its tight-frame marker can be measured based on the tight-frame marker. The measurement method based on deep learning using tight-frame markers in the present disclosure can also be referred to as a recognition method, a tight-frame marker measurement method, a tight-frame marker recognition method, an automatic measurement method, an auxiliary measurement method, etc. The measurement method in the present disclosure can be applied to any application scenario in which the width and / or height of a target in an image are accurately measured.

[0047] The measurement method disclosed herein is a measurement method that uses a network module trained based on a tight frame mark of a target to identify the target and thereby realize measurement. The tight frame mark can be the minimum circumscribed rectangle of the target. In this case, the target is in contact with the four sides of the tight frame mark and does not overlap with the area outside the tight frame mark (that is, the target is tangent to the four sides of the tight frame mark). Thus, the tight frame mark can represent the width and height of the target. In addition, training the network module based on the tight frame mark of the target can reduce the time and labor cost of collecting pixel-level annotation data (also referred to as label data) and the network module can accurately identify the tight frame mark of the target.

[0048] The input images involved in the present disclosure may come from a camera, a CT scan, a PET-CT scan, a SPECT scan, an MRI, an ultrasound, an X-ray, an angiogram, a fluorescence image, an image captured by a capsule endoscope, or a combination thereof. In some examples, the input image may be an image of a tissue object (e.g., a fundus image). In some examples, the input image may be a natural image. A natural image may be an image observed or captured in a natural scene. Thus, it is possible to measure targets in a natural image. For example, the size of a face or the height of a pedestrian in a natural image may be measured. The following describes an example of the present disclosure using an example in which the input image is a fundus image captured by a fundus camera, and such a description does not limit the scope of the present disclosure.

[0049] Figure 1 2 is a schematic diagram illustrating an application scenario of the measurement method based on tight-frame deep learning involved in the examples of the present disclosure. FIG2(a) is a schematic diagram illustrating a fundus image involved in the examples of the present disclosure. FIG2(b) is a schematic diagram illustrating the recognition result of the fundus image involved in the examples of the present disclosure.

[0050] In some examples, the measurement methods disclosed herein can be applied to Figure 1 In the application scenario shown in FIG. 1 , an image of a target object 51 including a corresponding position of the target can be captured by a capture device 52 (such as a camera) as an input image (see FIG. Figure 1 ), the input image is input to the network module 20 to identify the target in the input image and obtain the target's tight frame mark B (see Figure 1 ), and the target can be measured based on the tight frame mark B. Taking a fundus image as an example, inputting the fundus image shown in Figure 2(a) into the network module 20 can obtain the recognition result shown in Figure 2(b). The recognition result can include tight frame marks for two types of targets: the optic cup and the optic disc. Tight frame mark B11 is the tight frame mark for the optic disc, and tight frame mark B12 is the tight frame mark for the optic cup. In this case, both the optic cup and the optic disc can be measured based on the tight frame marks.

[0051] The network module 20 involved in the present disclosure can be multi-task based. In some examples, the network module 20 can be a neural network based on deep learning. In some examples, the network module 20 can include two tasks, one task can be a segmentation network 22 for image segmentation based on weakly supervised learning (described later), and the other task can be a regression network 23 based on bounding box regression (described later).

[0052] In some examples, the segmentation network 22 can segment the input image to obtain targets (e.g., optic cup and / or optic disc). In some examples, the segmentation network 22 can be based on multiple-instance learning (MIL) and used to supervise tight box labeling. In some examples, the problem solved by the segmentation network 22 can be a multi-label classification problem. In some examples, the input image can contain targets of at least one category of interest (which can be simply referred to as a category). Thus, the segmentation network 22 can recognize input images that contain targets of at least one category of interest. In some examples, the input image may not contain any targets. In some examples, the number of targets of each category of interest can be at least greater than 1.

[0053] In some examples, the regression network 23 can be used to predict tight bounding boxes by category. In some examples, the regression network 23 can predict tight bounding boxes by predicting the offset of the tight bounding boxes relative to the positions of each pixel of the input image.

[0054] In some examples, the network module 20 may further include a backbone network 21. The backbone network 21 may be used to extract a feature map of an input image (i.e., the original image input to the network module 20). In some examples, the backbone network 21 may extract high-level features for object representation. In some examples, the resolution of the feature map may be consistent with that of the input image (i.e., the feature map may be of a single scale and consistent with the size of the input image). Thus, the accuracy of identifying or measuring targets whose size does not vary much can be improved. In some examples, a feature map consistent with the scale of the input image may be obtained by continuously fusing image features of different scales. In some examples, the feature map may serve as input to the segmentation network 22 and the regression network 23.

[0055] In some examples, backbone network 21 may include an encoding module and a decoding module. In some examples, the encoding module may be configured to extract image features at different scales. In some examples, the decoding module may be configured to map the image features extracted at different scales back to the resolution of the input image to output a feature map. In this way, a feature map consistent with the resolution of the input image can be obtained.

[0056] Figure 3 FIG. 1 is a schematic diagram showing an example of the network module 20 according to the present disclosure.

[0057] In some examples, such as Figure 3 As shown, the network module 20 may include a backbone network 21, a segmentation network 22, and a regression network 23. The backbone network 21 may receive an input image and output a feature map. The feature map may serve as input to the segmentation network 22 and the regression network 23 to obtain corresponding outputs. Specifically, the segmentation network 22 may take the feature map as input to obtain a first output, and the regression network 23 may take the feature map as input to obtain a second output. In this case, the input image can be input to the network module 20 to obtain the first output and the second output.

[0058] In some examples, the first output may be the result of image segmentation prediction. In some examples, the second output may be the result of bounding box regression prediction.

[0059] In some examples, the first output may include the probability that each pixel in the input image belongs to each category. In some examples, the probability that each pixel belongs to each category can be obtained through an activation function. In some examples, the first output may be a matrix. In some examples, the size of the matrix corresponding to the first output may be M×N×C, where M×N may represent the resolution of the input image, M and N may correspond to the rows and columns of the input image, respectively, and C may represent the number of categories. For example, for fundus images with two target categories, optic cup and optic disc, the size of the matrix corresponding to the first output may be M×N×2.

[0060] In some examples, the value corresponding to each pixel point at each position in the input image in the first output can be a vector, and the number of elements in the vector can be consistent with the number of categories. For example, for the pixel point at the kth position in the input image, the corresponding value in the first output can be the vector p k , vector p k It may include C elements, where C is the number of categories. In some examples, the vector p k The element value can be a number from 0 to 1.

[0061] In some examples, the second output may include the offset between the position of each pixel in the input image and the tight frame mark of each category of target. That is, the second output may include the offset of the tight frame mark of the target of a specific category. In other words, the regression network 23 may predict the offset of the tight frame mark of the target of a specific category. In this case, when the overlap between targets of different categories is high, the tight frame marks of the targets of the corresponding categories can be distinguished, and the tight frame marks of the targets of the corresponding categories can be obtained. This makes it possible to identify or measure targets with high overlap between different categories. In some examples, the offset in the second output can be used as the target offset.

[0062] In some examples, the target offset may be a normalized offset. In some examples, the target offset may be an offset normalized based on the average size of targets of each category. In some examples, the target offset may be an offset normalized based on the average width and average height of targets of each category. The target offset and the predicted offset (described later) may correspond to the true offset (described later). That is, if the true offset during training the network module 20 (which may be referred to as the training phase) is normalized, the target offset (corresponding to the measurement phase) and the predicted offset (corresponding to the training phase) predicted using the network module 20 (which may be referred to as the measurement phase) may also be normalized accordingly. In this way, the accuracy of identifying or measuring targets whose size changes little can be improved.

[0063] In some examples, the average size of the target can be obtained by averaging the average width and average height of the target. In some examples, the average size of the target can be an empirical value (that is, the average width and average height can be empirical values). In some examples, the average size of the target can be obtained by statistically analyzing the samples corresponding to the collected input image. In some examples, the width and height of the tight frame mark of the target in the label data of the sample can be averaged by category to obtain the average width and average height. In some examples, the average width and average height can be averaged to obtain the average size of the target of the category. In some examples, the sample can be a training sample (described later). That is, the average width and average width of the target and the average size of the target can be obtained by statistically analyzing the training samples. Thus, the average width and average width of the target, or the average size of the target, can be obtained through the training samples.

[0064] In some examples, the second output may be a matrix. In some examples, the size of the matrix corresponding to the second output may be M×N×A, where A may represent the size of all target offsets, M×N may represent the resolution of the input image, and M and N may correspond to the rows and columns of the input image, respectively. In some examples, if the size of a target offset is a 4×1 vector (i.e., it can be represented by 4 numbers), then A may be C×4, where C may represent the number of categories. For example, for fundus images with two target categories, optic cup and optic disc, the size of the matrix corresponding to the second output may be M×N×8.

[0065] In some examples, the value corresponding to each pixel point at each position in the input image in the second output can be a vector. For example, the value corresponding to the pixel point at the kth position in the input image in the second output can be expressed as: v k =[v k1 ,v k2 ,…,v kC ]. Where C is the number of categories, v k Each element in can be represented as the target displacement of each category of the target. Thus, it is convenient to represent the target displacement and the corresponding category. In some examples, v k The elements of can be 4-dimensional vectors.

[0066] In some examples, the backbone network 21 may be based on a U-net network. In this embodiment, the encoding module of the backbone network 21 may include unit layers and pooling layers. The decoding module of the backbone network 21 may include unit layers, up-sampling layers (up-sampling layers, Up-sampling) and skip-connection units (skip-connection units, Skip-connection).

[0067] In some examples, the unit layers can include convolutional layers, batch normalization layers, and rectified linear unit layers (ReLu). In some examples, the pooling layers (Pooling) can be max pooling layers (Max-poooling). In some examples, the skip connection units can be used to combine image features from deep layers with image features from shallow layers.

[0068] In addition, the segmentation network 22 can be a forward neural network. In some examples, the segmentation network 22 can include multiple unit layers. In some examples, the segmentation network 22 can include multiple unit layers and convolutional layers (Conv).

[0069] In addition, the regression network 23 may include dilated convolution layers (DilatedConv) and batch normalization layers (BN). In some examples, the regression network 23 may include dilated convolution layers, rectified linear unit layers, and convolution layers.

[0070] Figure 4 It is a schematic diagram showing another example of the network module 20 involved in the example of the present disclosure. It should be noted that in order to more clearly describe the network structure of the network module 20, Figure 4 , the network layers in the network module 20 are distinguished by the numbers in the arrows, where arrow 1 represents a network layer (i.e., a unit layer) composed of a convolutional layer, a batch normalization layer, and a rectified linear unit layer, arrow 2 represents a network layer composed of an expanded convolutional layer and a rectified linear unit, arrow 3 represents a convolutional layer, arrow 4 represents a maximum pooling layer, arrow 5 represents an upsampling layer, and arrow 6 represents a skip connection unit.

[0071] As an example of the network module 20. Figure 4 As shown, an input image with a resolution of 256×256 can be input into the network module 20, and image features are extracted through the unit layers (see arrow 1) and maximum pooling layers (see arrow 4) of different levels of the encoding module, and image features of different scales are continuously fused through the unit layers (see arrow 1), upsampling layers (see arrow 5) and jump connection units (see arrow 6) of different levels of the decoding module to obtain a feature map 221 consistent with the scale of the input image, and then the feature map 221 is respectively input into the segmentation network 22 and the regression network 23 to obtain the first output and the second output.

[0072] In addition, if Figure 4 As shown, the segmentation network 22 can be sequentially composed of a unit layer (see arrow 1) and a convolutional layer (see arrow 3), and the regression network 23 can be sequentially composed of a plurality of network layers consisting of dilated convolutional layers and rectified linear unit layers (see arrow 2), and a convolutional layer (see arrow 3). The unit layer can be composed of a convolutional layer, a batch normalization layer, and a rectified linear unit layer.

[0073] In some examples, the size of the convolution kernel of the convolution layer in the network module 20 can be set to 3×3. In some examples, the size of the convolution kernel of the maximum pooling layer in the network module 20 can be set to 2×2, and the convolution step can be set to 2. In some examples, the scale-factor of the upsampling layer in the network module 20 can be set to 2. In some examples, such as Figure 4 As shown, the dilation factors of the multiple dilated convolutional layers in the network module 20 can be set to 1, 1, 2, 4, 8 and 16 in sequence (see the numbers above arrow 2). Figure 4 As shown in FIG, the number of the maximum pooling layers may be 5. Thus, the size of the input image can be divided by 32 (32 may be 2 to the fifth power).

[0074] As described above, the measurement method involved in the present disclosure is a measurement method that uses a network module 20 trained based on a tight frame mark of the target to identify the target and thereby achieve measurement. The following describes in detail the training method of the network module 20 involved in the present disclosure (which may be referred to as the training method) in conjunction with the accompanying drawings. Figure 5 FIG. 1 is a flowchart illustrating a training method of the network module 20 according to an example of the present disclosure.

[0075] In some examples, the segmentation network 22 and the regression network 23 in the network module 20 can be trained simultaneously on an end-to-end basis.

[0076] In some examples, the segmentation network 22 and the regression network 23 in the network module 20 can be jointly trained to simultaneously optimize the segmentation network 22 and the regression network 23. In some examples, through joint training, the segmentation network 22 and the regression network 23 can adjust the network parameters of the backbone network 21 through back propagation, so that the feature map output by the backbone network 21 can better express the features of the input image and input the segmentation network 22 and the regression network 23. In this case, the segmentation network 22 and the regression network 23 are both processed based on the feature map output by the backbone network 21.

[0077] In some examples, multi-instance learning can be used to train the segmentation network 22. In some examples, the expected intersection-over-union ratio corresponding to the pixels of the image to be trained can be used to screen the pixels used to train the regression network 23 (described later).

[0078] In some examples, such as Figure 5 As shown, the training method may include constructing a training sample (step S120), inputting the training sample into the network module 20 to obtain prediction data (step S140), and determining the training loss of the network module 20 based on the training sample and the prediction data and optimizing the network module 20 based on the training loss (step S160). In this way, an optimized (also called trained) network module 20 can be obtained.

[0079] In some examples, in step S120, a training sample may be constructed. The training sample may include input image data and label data. In some examples, the input image data may include multiple images to be trained. For example, the images to be trained may be fundus images to be trained.

[0080] In some examples, the plurality of images to be trained may include images containing targets. In some examples, the plurality of images to be trained may include images containing targets and images not containing targets. In some examples, the targets may belong to at least one category. In some examples, the number of targets of each category in the images to be trained may be greater than or equal to 1. For example, taking fundus images as an example, if the optic cup and optic disc are to be identified or measured, the targets in the fundus image may be an optic disc and an optic cup. That is, there are two targets in the fundus image that need to be identified or measured, and the number of each target may be 1. If microaneurysms are to be identified or measured, the target in the fundus image may be at least one microaneurysm. The examples disclosed herein do not specifically limit the number of targets, the categories to which the targets belong, and the number of targets of each category.

[0081] In some examples, the label data may include a gold standard for the category to which the object belongs (the gold standard for the category may also be referred to as the true category) and a gold standard for the tight bounding box label of the object (the gold standard for the tight bounding box label may also be referred to as the true tight bounding box label). In other words, the label data may include the true category to which the object in the training image belongs and the true tight bounding box label of the object. It should be noted that, unless otherwise specified, the tight bounding box label or category of the object in the label data in the training method may be the gold standard by default.

[0082] In some examples, the training images can be annotated to obtain label data. In some examples, the training images can be annotated using an annotation tool, such as a line annotation system. Specifically, the annotation tool can be used to annotate the tight bounding box (i.e., the minimum bounding rectangle) of the object in the training image, and a corresponding category can be set for the tight bounding box to indicate the true category of the object.

[0083] In some examples, in order to suppress overfitting of the network module 20, data augmentation processing can be performed on the training samples. In some examples, the data augmentation processing can include but is not limited to flipping (e.g., flipping upside down or flipping left to right), amplification, rotation, adjusting contrast, adjusting brightness or color balance. In some examples, the same data augmentation processing can be performed on the input image data and label data in the training samples. In this way, the input image data and label data can be kept consistent.

[0084] In some examples, in step S140, the training sample can be input into the network module 20 to obtain predicted data. As described above, the network module 20 may include a segmentation network 22 and a regression network 23. In some examples, the network module 20 may obtain predicted data corresponding to the training sample based on the input image data of the training sample. The predicted data may include predicted segmentation data output by the segmentation network 22 and predicted offsets output by the regression network 23.

[0085] In addition, the predicted segmentation data may correspond to the first output, and the predicted offset may correspond to the second output (that is, may correspond to the target offset). That is, the predicted segmentation data may include the probability that each pixel in the image to be trained belongs to each category, and the predicted offset may include the offset between the position of each pixel in the image to be trained and the tight frame mark of the target of each category. In some examples, corresponding to the target offset, the predicted offset may be an offset normalized based on the average size of the targets of each category. In this way, the accuracy of identifying or measuring targets with little size variation can be improved. Preferably, the sizes of multiple targets of the same category may differ from each other by less than 10 times. For example, the sizes of multiple targets of the same category may differ from each other by 1 time, 2 times, 3 times, 5 times, 7 times, 8 times or 9 times, etc. In this way, the accuracy of target identification or measurement can be further improved.

[0086] To more clearly describe the offset between the pixel position and the target's tight frame marker, as well as the normalized offset, the following formula is used to describe it. It should be noted that the predicted offset, target offset, and true offset are all types of offsets and are also applicable to the following formula (1).

[0087] Specifically, the position of a pixel can be expressed as (x, y), the tight box mark of an object corresponding to the pixel can be expressed as b = (xl, yt, xr, yb), and the offset of the tight box mark b of the object relative to the position of the pixel (that is, the offset between the position of the pixel and the tight box mark of the object) can be expressed as t = (tl, tt, tr, tb). Then tl, tt, tr, tb can satisfy formula (1):

[0088] tl=(x-xl) / S c1 ,

[0089] tt=(y-yt) / S c2 ,

[0090] tr=(xr-x) / S c1 ,

[0091] tb=(yb-y) / S c2 ,

[0092] Among them, xl, yt can represent the position of the upper left corner of the target's tight frame mark, xr, yb can represent the position of the lower right corner of the target's tight frame mark, c can represent the index of the category to which the target belongs, S c1 It can represent the average width of the target of the cth category, S c2 It can represent the average height of the target of the cth category. Thus, the normalized offset can be obtained. In some examples, S c1 and S c2 can be the average size of the objects in the cth category.

[0093] However, the examples disclosed herein are not limited thereto. In other examples, the tight frame mark of an object may be represented by the position of the lower left corner and the position of the upper right corner, or by the position, length, and width of any corner. Furthermore, in other examples, normalization may be performed using other methods, for example, the offset may be normalized using the length and width of the tight frame mark of the object.

[0094] In addition, the pixels in formula (1) can be pixels of the training image or the input image. That is, formula (1) can be applied to the actual offset corresponding to the training image in the training phase and the target offset corresponding to the input image in the measurement phase.

[0095] Specifically, during the training phase, the pixel point can be a pixel point in the image to be trained, the target tight frame mark b can be a gold standard for the target tight frame mark in the image to be trained, and the offset t can be a true offset (also called a gold standard for the offset). Thus, the regression loss of the regression network 23 can be subsequently obtained based on the predicted offset and the true offset. Furthermore, if the pixel point is a pixel point in the image to be trained, and the offset t is a predicted offset, the predicted target tight frame mark can be inferred based on formula (1).

[0096] In addition, in the measurement phase, the pixel point can be a pixel point in the input image, and the offset t can be a target offset. Then, the tight frame mark of the target in the input image can be inferred based on formula (1) and the target offset (that is, the target offset and the position of the pixel point can be substituted into formula (1) to obtain the tight frame mark of the target). In this way, the tight frame mark of the target in the input image can be obtained.

[0097] In some examples, in step S160, a training loss of the network module 20 can be determined based on the training samples and the predicted data, and the network module 20 can be optimized based on the training loss. In some examples, the training loss of the network module 20 can be determined based on the label data corresponding to the training samples, the predicted segmentation data, and the predicted offset, and then the network module 20 is trained based on the training loss to optimize the network module 20.

[0098] As described above, the network module 20 may include a segmentation network 22 and a regression network 23. In some examples, the training loss may include the segmentation loss of the segmentation network 22 and the regression loss of the regression network 23. That is, the training loss of the network module 20 may be obtained based on the segmentation loss and the regression loss. Thus, the network module 20 may be optimized based on the training loss. In some examples, the training loss may be the sum of the segmentation loss and the regression loss. In some examples, the segmentation loss may represent the degree to which pixels in the training image in the predicted segmentation data belong to each true category, and the regression loss may represent the degree to which the predicted offset is close to the true offset.

[0099] Figure 6 Schematic diagram showing a positive package involved in the examples of the present disclosure.

[0100] In some examples, the segmentation loss of the segmentation network 22 can be obtained based on the predicted segmentation data and label data corresponding to the training samples. Thus, the predicted segmentation data of the segmentation network 22 can be approximated to the label data through the segmentation loss. In some examples, the segmentation loss can be obtained using multi-instance learning. In multi-instance learning, multiple packages to be trained can be obtained based on the true tight frame labels of the targets in each image to be trained according to category (that is, each category can correspond to multiple packages to be trained). The segmentation loss can be obtained based on the multiple packages to be trained of each category. In some examples, the multiple packages to be trained may include multiple positive packages and multiple negative packages. Thus, the segmentation loss can be obtained based on the positive packages and negative packages of multi-instance learning. It should be noted that, unless otherwise specified, the following positive packages and negative packages are for each category.

[0101] In some examples, multiple positive packets can be obtained based on the area within the target's true tight bounding box. Figure 6 As shown, the area A2 in the training image P1 is the area within the real tight frame B21 of the target T1.

[0102] In some examples, all pixels on each of the multiple straight lines connecting two opposite sides of the real tight frame mark of the target can be divided into a positive bag (that is, one straight line can correspond to one positive bag). Specifically, the two ends of each straight line can be at the upper end and the lower end, or the left end and the right end of the real tight frame mark. As an example, Figure 6 As shown, the pixels on lines D1, D2, D3, D4, D5, D6, D7, and D8 can each be divided into a positive packet. However, the examples disclosed herein are not limited thereto. In other examples, other methods of dividing positive packets can also be used. For example, the pixels at a specific position of the true tight-frame marker can be divided into a positive packet.

[0103] In some examples, the plurality of straight lines may include at least one group of mutually parallel first parallel lines. For example, the plurality of straight lines may include one group of first parallel lines, two groups of first parallel lines, three groups of first parallel lines, or four groups of first parallel lines. In some examples, the number of straight lines in the first parallel lines may be greater than or equal to 2.

[0104] In some examples, the plurality of straight lines may include at least one group of mutually parallel first parallel lines and mutually parallel second parallel lines that are respectively perpendicular to each group of first parallel lines. Specifically, if the plurality of straight lines include a group of first parallel lines, the plurality of straight lines may further include a group of second parallel lines that are respectively perpendicular to the group of first parallel lines. If the plurality of straight lines include multiple groups of first parallel lines, the plurality of straight lines may further include multiple groups of second parallel lines that are respectively perpendicular to each group of first parallel lines. Figure 6As shown, a group of first parallel lines may include parallel straight lines D1 and D2, and a group of second parallel lines corresponding to the group of first parallel lines may include parallel straight lines D3 and D4, where straight line D1 may be perpendicular to straight line D3; another group of first parallel lines may include parallel straight lines D5 and D6, and a group of second parallel lines corresponding to the group of first parallel lines may include parallel straight lines D7 and D8, where straight line D5 may be perpendicular to straight line D7. In some examples, the number of straight lines in the first and second parallel lines may be greater than or equal to 2.

[0105] As described above, in some examples, the plurality of straight lines may include multiple groups of first parallel lines (i.e., the plurality of straight lines may include parallel lines at different angles). In this case, the segmentation network 22 can be optimized by dividing the positive packets at different angles. This can improve the accuracy of the segmentation data predicted by the segmentation network 22.

[0106] In some examples, the angle of the first parallel line can be the angle between the extension line of the first parallel line and the extension line of any non-intersecting side of the real tight frame mark, and the angle of the first parallel line can be greater than -90° and less than 90°. For example, the angle can be -89°, -75°, -50°, -25°, -20°, 0°, 10°, 20°, 25°, 50°, 75° or 89°, etc. Specifically, if the extension line of the non-intersecting side is rotated clockwise by less than 90° to the extension line of the first parallel line, the angle can be greater than 0° and less than 90°; if the extension line of the non-intersecting side is rotated counterclockwise by less than 90° (that is, rotated clockwise by more than 270°) to the extension line of the first parallel line, the angle can be greater than -90° and less than 0°; if the non-intersecting side is parallel to the first parallel line, the angle can be 0°. Figure 6 As shown, the angles of lines D1, D2, D3, and D4 can be 0°, and the angles of lines D5, D6, D7, and D8 (i.e., angle C1) can be 25°. In some examples, the angles of the first parallel lines can be hyperparameters that can be optimized during the training process.

[0107] Alternatively, the angle of the first parallel line may be described in terms of a rotation of the image to be trained. The angle of the first parallel line may be a rotation angle. Specifically, the angle of the first parallel line may be the rotation angle required to rotate the image to be trained so that any side of the image to be trained that does not intersect with the first parallel line is parallel to the first parallel line. The angle of the first parallel line may be greater than -90° and less than 90°, and the rotation angle may be positive for a clockwise rotation and negative for a counterclockwise rotation.

[0108] However, the examples disclosed herein are not limited thereto. In other examples, depending on how the angle of the first parallel line is described, the angle of the first parallel line may also be within other ranges. For example, if the description is based on the sides of a real tight frame intersecting the first parallel line, the angle of the first parallel line may also be greater than 0° and less than 180°.

[0109] In some examples, multiple negative bags can be obtained based on the area outside the true tight bounding box of the target. Figure 6 As shown, region A1 in the training image P1 is outside the ground-truth tight bounding box B21 of target T1. In some examples, a negative bag can be a single pixel outside the ground-truth tight bounding box of all targets of a category (i.e., one pixel can correspond to one negative bag).

[0110] As described above, in some examples, segmentation loss can be obtained based on multiple training packages of each category. In some examples, the segmentation loss can include a unary term (also known as a unary loss) and a pairwise term (also known as a pairwise loss). In some examples, the unary term can describe the degree to which each training package belongs to each true category. In this case, the tight bounding box can be constrained by both the positive and negative packages through the unary loss. In some examples, the pairwise term can describe the degree to which a pixel of the image to be trained belongs to the same category as the pixel adjacent to the pixel. In this case, the pairwise loss smoothes the predicted segmentation result.

[0111] In some examples, the segmentation loss of each category can be obtained by category, and the segmentation loss (ie, the total segmentation loss) can be obtained based on the segmentation loss of each category. seg Can satisfy the formula:

[0112]

[0113] Among them, L c It can represent the segmentation loss of category c, and C can represent the number of categories. For example, if both the optic cup and the optic disc are identified in the fundus image, C can be 2, and if only the optic cup or only the optic disc is identified, C can be 1.

[0114] In some examples, the segmentation loss L for class c is c Can satisfy the formula:

[0115]

[0116] Among them, φ c can represent a unary term, It can represent a pair of terms, P can represent the degree to which each pixel predicted by the segmentation network 22 belongs to each category (also known as probability), Can represent a collection of multiple positive packages, Can represent a collection of multiple negative bags, and λ can represent a weight factor. The weight factor λ can be a hyperparameter that can be optimized during the training process. In some examples, the weight factor λ can be used to switch between two losses (i.e., unary term and pairwise term).

[0117] Generally speaking, in multi-instance learning, if each positive bag of a category contains at least one pixel belonging to the category, then the pixel with the highest probability of belonging to the category in each positive bag can be used as the positive sample of the category; if each negative bag of a category does not contain a pixel belonging to the category, then even the pixel with the highest probability in the negative bag is also a negative sample of the category. Based on this situation, in some examples, the unary term φ corresponding to category c c Can satisfy the formula:

[0118]

[0119] Among them, P c (b) can represent the probability that a packet to be trained belongs to category c (also known as the degree of belonging to category c or the probability of the packet to be trained), b can represent a packet to be trained, Can represent a collection of multiple positive packages, Can represent a collection of multiple negative packages, max can represent the maximum value function, can represent the cardinality of the set of multiple positive packets (that is, the number of elements in the set), β can represent the weight factor, and γ can represent the focusing parameter. In some examples, when the P corresponding to the positive packet is c (b) P is equal to 1 and corresponds to the negative bag c (b) The value of the unary term is minimum when it is equal to 0. In other words, the unary loss is minimum.

[0120] In some examples, the weighting factor β can be between 0 and 1. In some examples, the focus parameter γ can be greater than or equal to zero.

[0121] In some examples, P c (b) can be the maximum probability of a pixel in a training bag belonging to category c. In some examples, P c (b) can satisfy the formula: P c (b) = max k∈b (p kc ), where p kc It can represent the probability that the pixel at the kth position of the training bag b belongs to category c.

[0122] In some examples, the maximum probability of a pixel point in a training package belonging to a category can be obtained based on the maximum smooth approximation function (ie, obtaining P c (b)). In this way, a more stable maximum probability can be obtained.

[0123] In some examples, the maximum smoothing approximation function may be at least one of an α-softmax function and an α-quasimax function.

[0124] In some examples, for the maximum function f(x)=max 1≤i≤n x i , max can represent the maximum value function, n can represent the number of elements (which can correspond to the number of pixels in the training package), x i It can represent the value of the element (which can correspond to the probability that the pixel at the i-th position of the training package belongs to a category. In this case, the α-softmax function can satisfy the formula:

[0125]

[0126] Wherein, α can be a constant. In some examples, the larger α is, the closer it is to the maximum value of the maximum function.

[0127] In addition, the α-quasimax function can satisfy the formula:

[0128]

[0129] Wherein, α can be a constant. In some examples, the larger α is, the closer it is to the maximum value of the maximum function.

[0130] As mentioned above, in some examples, the paired term can describe the degree to which a pixel of the image to be trained belongs to the same category as the pixels adjacent to it. That is, the paired term can evaluate the closeness of the probability that adjacent pixels belong to the same category. In some examples, the paired term corresponding to category c is Can satisfy the formula:

[0131]

[0132] Among them, ε can represent the set of all adjacent pixel pairs, (k, k') can represent a pair of adjacent pixel points, k and k' can represent the positions of the two pixels of the adjacent pixel pair respectively, and p kc It can represent the probability that the pixel at the kth position belongs to category c, p k'c It can represent the probability that the pixel at the k'th position belongs to category c.

[0133] In some examples, the adjacent pixels may be pixels in an eight-neighborhood or a four-neighborhood. In some examples, the adjacent pixels of each pixel in the image to be trained may be obtained to obtain a set of adjacent pixel pairs.

[0134] As described above, the training loss may include a regression loss. In some examples, the regression loss of the regression network 23 may be obtained based on the predicted offset corresponding to the training sample and the true offset corresponding to the labeled data. In this case, the regression loss can be used to approximate the predicted offset of the regression network 23 to the true offset.

[0135] In some examples, the true offset can be the offset between the position of a pixel in the training image and the true tight bounding box of the object in the label data. In some examples, corresponding to the predicted offset, the true offset can be the offset normalized based on the average size of objects in each category. For details, see the description of the offset in formula (1) above.

[0136] In some examples, corresponding pixels in the image to be trained can be selected as positive samples to train the regression network 23. That is, the positive samples can be used to optimize the regression network 23. Specifically, a regression loss can be obtained based on the positive samples, and then the regression loss can be used to optimize the regression network 23.

[0137] In some examples, the regression loss can satisfy the formula:

[0138]

[0139] Among them, C can represent the number of categories, M c It can represent the number of positive samples of the cth category, t ic It can represent the true offset corresponding to the i-th positive sample of the c-th category, v ic It can represent the predicted offset corresponding to the i-th positive sample of the c-th category, and s(x) can represent the sum of the smooth L1 losses of all elements in x. In some examples, for x to be t ic -v ic ,s(t ic -v ic ) can represent the degree of consistency between the predicted offset corresponding to the i-th positive sample of the c-th category calculated using the smooth L1 loss and the true offset corresponding to the i-th positive sample. Here, the positive sample can be a pixel in the training image selected for training the regression network 23 (i.e., for calculating the regression loss). In this way, the regression loss can be obtained.

[0140] In some examples, the true offset corresponding to the positive sample can be the offset corresponding to the true tight bounding box. In some examples, the true offset corresponding to the positive sample can be the offset corresponding to the matching tight bounding box. This can be applied to situations where the positive sample falls within multiple true tight bounding boxes.

[0141] In some examples, the smooth L1 loss function can satisfy the formula:

[0142]

[0143] Among them, σ can represent a hyperparameter used to switch between the smooth L1 loss function and the smooth L2 loss function, and x can represent a variable of the smooth L1 loss function.

[0144] As described above, in some examples, corresponding pixel points may be selected from the pixel points in the image to be trained as positive samples to train the regression network 23 .

[0145] In some examples, positive samples can be pixels in the training image that fall within the true tight frame of at least one target (that is, pixels in the training image that fall within the true tight frame of at least one target can be selected as positive samples). In this case, optimizing the regression network 23 based on pixels that fall within the true tight frame of at least one target can improve the optimization efficiency of the regression network 23. In some examples, pixels in the training image that fall within the true tight frame of at least one target can be selected by category as positive samples for each category. In some examples, the regression loss for each category can be obtained based on the positive samples for each category.

[0146] As described above, pixels that fall within the true tight bounding box of at least one object can be selected from the training image by category as positive samples for each category. In some examples, the positive samples for each category can be filtered, and the regression network 23 can be optimized based on the filtered positive samples. In other words, the positive samples used to calculate the regression loss can be the filtered positive samples.

[0147] In some examples, after obtaining positive samples for each category (i.e., selecting pixels from the training image that fall within the true tight bounding box of at least one object as positive samples), the matching tight bounding box corresponding to the positive sample can be obtained, and then the positive samples for each category can be filtered based on the matching tight bounding box. Thus, the regression network 23 can be optimized using the positive samples for each category filtered based on the matching tight bounding box.

[0148] In some examples, the true tight frames that a pixel point (e.g., a positive sample) falls into can be screened to obtain a matching tight frame for the pixel point. In some examples, the matching tight frame can be the true tight frame that has the smallest true offset relative to the position of the pixel point among the true tight frames that the pixel point of the training image falls into. For a positive sample, the matching tight frame can be the true tight frame that has the smallest true offset relative to the position of the positive sample among the true tight frames that the positive sample falls into.

[0149] Specifically, within a category, if a pixel (e.g., a positive sample) falls within only one true tight frame of the object to be measured, then that true tight frame is used as the matching tight frame. If the pixel falls within multiple true tight frames of the objects to be measured, then the true tight frame with the smallest true offset relative to the pixel's position among the multiple true tight frames of the objects to be measured can be used as the matching tight frame. In this way, the matching tight frame corresponding to the pixel can be obtained.

[0150] In some examples, the minimum true offset (i.e., the true tight frame marker with the minimum true offset) can be obtained by comparing the L1 normal form of the true offsets. In this case, the minimum true offset can be obtained based on the L1 normal form, and thus the matching tight frame marker can be obtained. Specifically, the absolute value of each element of the true offset in multiple true offsets can be calculated and then summed to obtain multiple offset values. The multiple offset values ​​can then be compared to obtain the true offset with the minimum offset value as the minimum true offset.

[0151] In some examples, the expected intersection-over-union (IoU) of pixels (e.g., positive samples) can be used to screen positive samples for each category. In this case, pixels far from the center of the true tight frame mark or the matching tight frame mark can be screened out. This can reduce the adverse effects of pixels far from the center on the optimization of the regression network 23 and improve the efficiency of the optimization of the regression network 23.

[0152] In some examples, the expected IoU corresponding to the positive sample can be obtained based on the matching tight bounding box, and the positive samples of each category can be screened based on the expected IoU. Specifically, after obtaining the positive samples of each category, the matching tight bounding box corresponding to the positive sample can be obtained, and then the expected IoU corresponding to the positive sample can be obtained based on the matching tight bounding box, and the positive samples of each category can be screened based on the expected IoU. Finally, the regression network 23 can be optimized using the positive samples of each category after screening. However, the examples of the present disclosure are not limited to this. In some examples, the pixels of the training image can be screened by category using the expected IoU corresponding to the pixels of the training image (that is, the pixels of the training image can be screened using the expected IoU without first selecting pixels that fall within at least one of the true tight bounding boxes of the target from the training image as positive samples). In addition, pixels that do not fall within any true tight bounding box (that is, pixels that do not have a matching tight bounding box) can be identified. This facilitates subsequent screening of the pixel. For example, the expected IoU of the pixel can be set to 0 to identify the pixel. Specifically, the pixels of the training image may be screened by category based on the expected intersection-union (IoU) corresponding to the pixels of the training image, and the regression network 23 may be optimized based on the screened pixels.

[0153] In some examples, the pixels whose expected intersection-and-union ratio is greater than the preset expected intersection-and-union ratio can be screened out from the pixels of the image to be trained to optimize the regression network 23. In some examples, the positive samples whose expected intersection-and-union ratio is greater than the preset expected intersection-and-union ratio can be screened out from the positive samples of each category to optimize the regression network 23. In this way, pixels (such as positive samples) that meet the preset expected intersection-and-union ratio can be obtained. In some examples, the preset expected intersection-and-union ratio can be greater than 0 and less than or equal to 1. For example, the preset expected intersection-and-union ratio can be 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 or 1, etc. In some examples, the preset expected intersection-and-union ratio can be a hyperparameter. The preset expected intersection-and-union ratio can be adjusted during the training process of the regression network 23.

[0154] In some examples, the expected intersection-and-union (IoU) corresponding to a pixel point can be obtained based on the matching tight box mark of the pixel point (e.g., a positive sample). In some examples, if a pixel point does not correspond to a matching tight box mark, the pixel point can be ignored or the expected IoU corresponding to the pixel point can be set to 0. In this case, the pixel without a matching tight box mark can be not used for training the regression network 23 or reduce its contribution to the regression loss. It should be noted that, unless otherwise specified, the following description of the expected IoU corresponding to a pixel point also applies to the expected IoU corresponding to a positive sample.

[0155] In some examples, the desired intersection over union (IoU) can be the maximum value among the IoUs of the matching tight bounding boxes of pixel points with multiple bounding boxes constructed centered on the pixel points. Thus, the desired IoU can be obtained. However, the examples of the present disclosure are not limited thereto. In other examples, the desired IoU can be the maximum value among the IoUs of the true tight bounding boxes of pixel points with multiple bounding boxes constructed centered on the pixel points.

[0156] In some examples, multiple bounding boxes constructed centered on pixel points of the image to be trained can be used to obtain the maximum value among the IoUs of the multiple bounding boxes with the matching tight bounding boxes of the pixel points as the desired IoU. In some examples, the sizes of the multiple bounding boxes can be different. Specifically, the widths or heights of each of the multiple bounding boxes can be different from those of other bounding boxes.

[0157] Figure 7 FIG. is a schematic diagram of a bounding box constructed centered on a pixel point involved in an example of the present disclosure.

[0158] For a clearer description of the desired IoU, the following is described in conjunction with Figure 7 As shown in Figure 7 , pixel point M1 has a matching tight bounding box B31, and bounding box B32 is an exemplary bounding box constructed centered on pixel point M1.

[0159] In some examples, let W be the width of the matching tight bounding box, H be the height of the matching tight bounding box, (r1W, r2H) represent the position of the pixel point, r1 and r2 be the relative positions of the pixel point in the matching tight bounding box, and satisfy the condition: 0 < r1, r2 < 1. Multiple bounding boxes can be constructed based on the pixel point. As an example, as shown in Figure 7 , the position of pixel point M1 can be expressed as (r1W, r2H), and the width and height of the matching tight bounding box B31 can be W and H respectively.

[0160] In some examples, the two center lines of the matching tight bounding box can be used to divide the matching tight bounding box into four regions. The four regions can be the upper left region, the upper right region, the lower left region, and the lower right region. For example, as shown in Figure 7 , the center line D9 and center line D10 of the matching tight bounding box B31 can divide the matching tight bounding box B31 into the upper left region A3, the upper right region A4, the lower left region A5, and the lower right region A6.

[0161] The following takes the case where the pixel point is in the upper left region (that is, r1 and r2 satisfy the condition: 0 < r1, r2 ≤ 0.5) as an example to describe the desired IoU. For example, as shown in Figure 7 , pixel point M1 can be a point in the upper left region A3.

[0162] First, construct multiple bounding boxes centered on pixel points. Specifically, for r1 and r2 satisfying the condition: 0 < r1, r2 ≤ 0.5, the four boundary conditions corresponding to pixel point M1 can be respectively:

[0163] w1 = 2r1W, h1 = 2r2H;

[0164] w2 = 2r1W, h2 = 2(1 - r2)H;

[0165] w3 = 2(1 - r1)W, h,3 = 2r2H;

[0166] w4 = 2(1 - r1)W, h4 = 2(1 - r 2 )H;

[0167] Among them, w1 and h1 can represent the width and height of the first boundary condition, w2 and h2 can represent the width and height of the second boundary condition, w3 and h3 can represent the width and height of the third boundary condition, and w4 and h4 can represent the width and height of the fourth boundary condition.

[0168] Secondly, calculate the intersection over union (IoU) of the bounding boxes under each boundary condition with the matching tight bounding box label. Specifically, the IoU corresponding to the above four boundary conditions can satisfy formula (2):

[0169] IoU1(r1, r2) = 4r1r2,

[0170] IoU(r1, r2) = 2r1 / (2r1(1 - 2r2)+1),

[0171] IoU3(r1, r2) = 2r2 / (2r2(1 - 2r1)+1),

[0172] IoU4(r1, r2) = 1 / (4(1 - r1)(1 - r2)),

[0173] Among them, IoU1(r1, r2) can represent the IoU corresponding to the first boundary condition, IoU2(r1, r2) can represent the IoU corresponding to the second boundary condition, IoU3(r1, r2) can represent the IoU corresponding to the third boundary condition, and IoU4(r1, r2) can represent the IoU corresponding to the fourth boundary condition. In this case, the IoU corresponding to each boundary condition can be obtained. [[ID=3?]]

[0174] Finally, the maximum IoU among the IoUs of multiple boundary conditions is the desired IoU. In some examples, for r1 and r2 satisfying the condition: 0 < r1, r2 ≤ 0.5, the desired IoU can satisfy formula (3):

[0175] In addition, the expected intersection over union (IoU) for pixel points located in other regions (i.e., the upper right region, the lower left region, and the lower right region) can be obtained based on a similar method for the upper left region. In some examples, for r1 satisfying the condition: 0.5 ≤ r1 < 1, r1 in formula (3) can be replaced with 1 - r1, and for r2 satisfying the condition: 0.5 ≤ r2 < 1, r2 in formula (3) can be replaced with 1 - r2. Thus, the expected IoU for pixel points located in other regions can be obtained. That is, pixel points located in other regions can be mapped to the upper left region through coordinate transformation, and then the expected IoU can be obtained in a manner consistent with that of the upper left region. Therefore, for r1, r2 satisfying the condition: 0 < r1, r2 < 1, the expected IoU can satisfy formula (4):

[0176]

[0177] where IoU1(r1, r2), IoU2(r1, r), IoU2(r1, r), and IoU2(r1, r) can be obtained from formula (2). Thus, the expected IoU can be obtained.

[0178] As described above, in some examples, the expected IoU corresponding to a pixel point can be obtained based on the matching tight bounding box label of the pixel point (e.g., a positive sample). However, the examples of the present disclosure are not limited thereto. In other examples, during the process of screening positive samples of each category or pixel points of the image to be trained, the matching tight bounding box label may not be obtained. Specifically, the expected IoU corresponding to a pixel point can be obtained based on the true tight bounding box label corresponding to the pixel point (e.g., a positive sample), and each category of pixel points can be screened based on the expected IoU. In this case, the expected IoU can be the maximum value among the expected IoUs corresponding to each true tight bounding box label. The description related to obtaining the expected IoU corresponding to a pixel point based on the matching tight bounding box label of the pixel point can be referred to when obtaining the expected IoU corresponding to a pixel point based on the true tight bounding box label.

[0179] Hereinafter, the measurement method related to the present disclosure will be described in detail with reference to the accompanying drawings. The network module 20 involved in the measurement method can be trained by the above training method. FIG. 8(a) is a flowchart showing a deep learning-based measurement method for tight bounding boxes according to an example of the present disclosure.

[0180] In some examples, as shown in FIG. 8(a), the measurement method may include obtaining an input image (step S220), inputting the input image into the network module 20 to obtain a first output and a second output (step S240), and identifying a target based on the first output and the second output to obtain the tight bounding box of the target of each category (step S260).

[0181] In some examples, in step S220, an input image may be acquired. In some examples, the input image may include at least one target. In some examples, the at least one target may belong to at least one category of interest (the category of interest may be referred to as a category). Specifically, if the input image includes one target, the target may belong to one category of interest; if the input image includes multiple targets, the multiple targets may belong to at least one category of interest. In some examples, the input image may not include any target. In this case, a judgment can be made on an input image in which no target is present.

[0182] In some examples, in step S240, an input image can be input to the network module 20 to obtain a first output and a second output. In some examples, the first output can include the probability that each pixel in the input image belongs to each category. In some examples, the second output can include the offset between the position of each pixel in the input image and the tight bounding box of the target of each category. In some examples, the offset in the second output can be used as the target offset. In some examples, the network module 20 can include a backbone network 21, a segmentation network 22, and a regression network 23. In some examples, the segmentation network 22 can be based on weakly supervised learning for image segmentation. In some examples, the regression network 23 can be based on bounding box regression. In some examples, the backbone network 21 can be used to extract a feature map of the input image. In some examples, the segmentation network 22 can use the feature map as input to obtain the first output, and the regression network 23 can use the feature map as input to obtain the second output. In some examples, the resolution of the feature map can be consistent with that of the input image. For details, see the relevant description of the network module 20.

[0183] In some examples, in step S260, the target can be identified based on the first and second outputs to obtain tight bounding boxes for each category of the target. This allows for subsequent accurate measurement of the target based on the tight bounding boxes. As described above, the first output can include the probability that each pixel in the input image belongs to each category, and the second output can include the offset between the position of each pixel in the input image and the tight bounding box for each category of the target. In some examples, based on the first output, the target offset corresponding to the category of the pixel at the corresponding position can be selected from the second output, and the tight bounding boxes for each category of the target can be obtained based on the target offset.

[0184] In some examples, the position of the pixel point with the highest local probability belonging to each category can be obtained from the first output as the first position, and the tight frame mark of the target of each category can be obtained based on the position corresponding to the first position and the target offset of the corresponding category in the second output. In this case, one or more targets of each category can be identified. In some examples, the first position can be obtained using the non-maximum suppression method (NMS). In some examples, the number of first positions corresponding to each category can be greater than or equal to 1. However, the examples of the present disclosure are not limited to this. For input images with only one target in each category, in some examples, the position of the pixel point with the highest probability belonging to each category can be obtained from the first output as the first position, and the tight frame mark of the target of each category can be obtained based on the position corresponding to the first position and the target offset of the corresponding category in the second output. That is, the first position can be obtained using the maximum value method. In some examples, the first position can also be obtained using the smooth maximum value suppression method.

[0185] In some examples, the tight bounding box of each class of objects can be obtained based on the first position and the object offset. In some examples, the first position and the object offset can be substituted into formula (1) to infer the tight bounding box of the object. Specifically, the first position can be used as the position (x, y) of the pixel point in formula (1) and the object offset can be used as the offset t to obtain the tight bounding box b of the object.

[0186] FIG8( b) is a flowchart illustrating another example of a measurement method based on deep learning of tight-framed landmarks according to examples of the present disclosure. In some examples, as shown in FIG8( b), the measurement method may further include measuring the size of each target based on the target's tight-framed landmark (step S280). Thus, the target can be accurately measured based on the target's tight-framed landmark. In some examples, the target's size may be the width and height of the target's tight-framed landmark.

[0187] The following describes in detail, with reference to the accompanying drawings, the measurement device 100 for tight-framed deep learning, as disclosed herein. The measurement device 100 may also be referred to as an identification device, a tight-framed measurement device, a tight-framed identification device, an automatic measurement device, an auxiliary measurement device, or the like. The measurement device 100 disclosed herein is used to implement the aforementioned measurement method. Figure 9(a) is a block diagram illustrating the measurement device 100 for tight-framed deep learning, as disclosed herein.

[0188] As shown in FIG. 9( a ), in some examples, the measurement device 100 may include an acquisition module 10 , a network module 20 , and an identification module 30 .

[0189] In some examples, acquisition module 10 may be configured to acquire an input image. In some examples, the input image may include at least one object. In some examples, the at least one object may belong to at least one category of interest. For details, see the relevant description in step S220.

[0190] In some examples, the network module 20 can be configured to receive an input image and obtain a first output and a second output based on the input image. In some examples, the first output can include the probability that each pixel in the input image belongs to each category. In some examples, the second output can include the offset between the position of each pixel in the input image and the tight bounding box of the target of each category. In some examples, the offset in the second output can be used as the target offset. In some examples, the network module 20 can include a backbone network 21, a segmentation network 22, and a regression network 23. In some examples, the segmentation network 22 can be based on weakly supervised learning for image segmentation. In some examples, the regression network 23 can be based on bounding box regression. In some examples, the backbone network 21 can be used to extract a feature map from the input image. In some examples, the segmentation network 22 can use the feature map as input to obtain the first output, and the regression network 23 can use the feature map as input to obtain the second output. In some examples, the resolution of the feature map can be consistent with that of the input image. For details, please refer to the relevant description of the network module 20.

[0191] In some examples, the recognition module 30 may be configured to recognize the target based on the first output and the second output to obtain tight-box labels of targets of various categories. For details, see the relevant description in step S260.

[0192] Figure 9(b) is a block diagram illustrating another example of a measurement device 100 based on deep learning with tight frame labels according to an example of the present disclosure. Figure 9(c) is a block diagram illustrating another example of a measurement device 100 based on deep learning with tight frame labels according to an example of the present disclosure.

[0193] As shown in FIG9( b ) and FIG9( c ), in some examples, the measuring device 100 may further include a measuring module 40. The measuring module 40 may be configured to measure the size of each object based on the tight frame mark of the object. For details, see the relevant description in step S280.

[0194] The measurement method and measurement device 100 disclosed herein construct a network module 20 comprising a backbone network 21, a segmentation network 22 for image segmentation based on weakly supervised learning, and a regression network 23 based on bounding box regression. The network module 20 is trained based on the tight bounding box of the target. The backbone network 21 receives an input image (e.g., a fundus image) and extracts a feature map with the same resolution as the input image. The feature map is input into the segmentation network 22 and the regression network 23 to obtain a first output and a second output, and then the tight bounding box of the target in the input image is obtained based on the first output and the second output to achieve measurement. In this case, the network module 20 trained based on the tight bounding box of the target can accurately predict the tight bounding box of the target in the input image, and thus can accurately measure the tight bounding box of the target. In addition, by predicting the normalized offset through the regression network 23, the accuracy of identifying or measuring targets with little size change can be improved. In addition, by using the expected intersection-over-union ratio to screen the pixels used to optimize the regression network 23, the adverse effect of pixels far from the center on the optimization of the regression network 23 can be reduced and the efficiency of the optimization of the regression network 23 can be improved. In addition, the regression network 23 predicts the offset of a clear category, which can further improve the accuracy of target recognition or measurement.

[0195] Although the present disclosure has been described in detail above with reference to the accompanying drawings and examples, it will be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope of the present disclosure.

Claims

1. A training method based on tight frame markers, characterized in that: A training method for training a network module based on a tight bounding box of an object, wherein the tight bounding box is a minimum bounding rectangle of the object. The training method includes: constructing a training sample including input image data and label data, wherein the input image data includes multiple images to be trained, the multiple images to be trained include images containing objects belonging to at least one category, and the label data includes a gold standard of the category to which the object belongs and a gold standard of the tight bounding box of the object; inputting the training sample into the network module to obtain prediction data, wherein the network module includes a segmentation network for image segmentation and a regression network based on bounding box regression, and the prediction data includes predicted segmentation data output by the segmentation network and predicted offsets output by the regression network, wherein the predicted segmentation data includes a probability that each pixel in the image to be trained belongs to each category, and the predicted offset includes an offset between the position of each pixel in the image to be trained and the tight bounding box of each category of the object; and determining a training loss for the network module based on the label data, the predicted segmentation data, and the predicted offsets, and optimizing the network module based on the training loss.

2. The training method according to claim 1, wherein: The network module also includes a backbone network, which is used to extract a feature map of the image to be trained. The segmentation network uses the feature map as input to output the predicted segmentation data, and the regression network uses the feature map as input to output the predicted offset.

3. The training method according to claim 2, characterized in that: The backbone network includes an encoding module and a decoding module, wherein the encoding module is configured to extract image features at different scales, and the decoding module is configured to map the image features extracted at different scales back to the resolution of the input image to output the feature map.

4. The training method according to claim 1, wherein: The predicted offset is the offset normalized based on the average size of objects in each category.

5. The training method according to claim 1, wherein: The training loss of the network module is determined based on the label data, the predicted segmentation data, and the predicted offset, including: obtaining the segmentation loss of the segmentation network based on the predicted segmentation data and the label data corresponding to the training sample; obtaining the regression loss of the regression network based on the predicted offset corresponding to the training sample and the true offset corresponding to the label data, wherein the true offset is the offset between the position of the pixel point of the image to be trained and the gold standard of the tight frame mark of the target in the label data; and obtaining the training loss of the network module based on the segmentation loss and the regression loss.

6. The training method according to claim 5, characterized in that: Using multi-instance learning, multiple to-be-trained bags are obtained based on the gold standard of the tight-frame labels of the targets in each to-be-trained image by category, and the segmentation loss is obtained based on the multiple to-be-trained bags of each category, wherein the multiple to-be-trained bags include multiple positive bags and multiple negative bags, and all pixel points on each of the multiple straight lines connecting the two opposite sides of the gold standard of the tight-frame label of the target are divided into a positive bag, the multiple straight lines include at least one group of mutually parallel first parallel lines and mutually parallel second parallel lines respectively perpendicular to each group of first parallel lines, and the negative bag is a single pixel point in the area outside the gold standard of the tight-frame labels of all targets of a category.

7. The training method according to claim 5, characterized in that: Pixels that fall within a gold standard of at least one tight frame mark of an object are selected from the image to be trained as positive samples to optimize the regression network.

8. The training method according to claim 7, characterized in that: Pixel points that fall within the gold standard of the tight frame mark of at least one target are selected from the image to be trained as positive samples of each category, and matching tight frame marks corresponding to the positive samples are obtained to screen the positive samples of each category based on the matching tight frame marks. The regression network is then optimized using the screened positive samples of each category, wherein the matching tight frame mark is the gold standard of the tight frame mark that has the smallest actual offset relative to the position of the positive sample among the gold standard of the tight frame mark within which the positive sample falls.

9. The training method according to claim 8, characterized in that: The regression network is optimized by filtering out pixels of the image to be trained from the pixels of the image to be trained according to categories and using the expected intersection-and-union (IoU) corresponding to the pixels of the image to be trained, wherein a plurality of bounding boxes of different sizes are constructed with the pixel of the image to be trained as the center point, and the maximum value of the IoU of the plurality of bounding boxes with the matching tight frame mark of the pixel is obtained and used as the expected IoU, wherein the matching tight frame mark is the gold standard of the tight frame mark in which the pixel of the image to be trained falls, and the tight frame mark has the smallest actual offset relative to the position of the pixel.

10. A measurement device based on a tight frame marker, characterized in that: The invention comprises an acquisition module, a network module trained by the training method according to any one of claims 1 to 9, and a recognition module, wherein the acquisition module is configured to acquire an input image including at least one target, wherein the at least one target belongs to at least one category of interest; the network module is configured to receive the input image and obtain, based on the input image, a first output output by a segmentation network and a second output output by a regression network, wherein the first output includes the probability that each pixel in the input image belongs to each category, and the second output includes the offset between the position of each pixel in the input image and the tight bounding box of the target of each category; and the recognition module is configured to identify the target based on the first output and the second output to obtain the tight bounding box of the target of each category.

Images