Semantic segmentation method and system for near-infrared image

Abstract

The invention discloses a semantic segmentation method and system for a near-infrared image. The method is realized through a BiSeNetV2-based bilateral network architecture. And training a BiSeNetV2 bilateral network to obtain an initial semantic image segmentation model. And recognizing the current near-infrared image to be recognized through the initial semantic segmentation model to obtain a current segmented image. And evaluating the initial semantic image segmentation model according to the mean value accuracy and the cardinality difference of the current segmented image so as to obtain a final semantic image segmentation model. The problems of semantic information loss and inaccurate evaluation when high-resolution near-infrared images are processed by near-infrared visual data are solved. By introducing the evaluation method, the segmentation precision and robustness of the near-infrared visual data are improved, and a reliable segmentation result can be obtained in a complex environment. Through multi-level feature extraction and an optimized segmentation and evaluation method, the recognition capability of the model in a low-contrast and high-noise environment is remarkably improved, so that the segmentation result is accurate and reliable.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and specifically to a semantic segmentation method and system for near infrared images. Background Art

[0002] Near-infrared imaging technology has been studied since the 1980s. With the rapid development of artificial intelligence, near-infrared visual data is showing increasingly important application value. In 2020, the global near-infrared vision application market is estimated to be US$206 million, and is expected to grow at a compound annual growth rate of 3.9% by 2030. In industrial inspection, near-infrared visual data can be used to detect the internal structure and defects of materials; in the field of security monitoring, near-infrared cameras can present clear images in low-light environments; in the medical field, near-infrared technology is used in non-invasive diagnosis and research such as vascular imaging and tissue detection; in the agricultural field, near-infrared visual data is widely used to analyze crop quality and monitor environmental factors. With the expansion of these applications and the increase in data volume, existing processing technologies face new challenges.

[0003] Near-infrared visual data belongs to the non-visible light band. Traditional near-infrared image processing methods usually rely on traditional image processing algorithms, such as edge detection, threshold segmentation, and region growing. These methods often find it difficult to achieve ideal segmentation results when processing near-infrared images with low contrast, high noise, or complex backgrounds, and have certain limitations when evaluating target structures. Therefore, although traditional image processing technology can process this type of image data, it has shortcomings in terms of accuracy, robustness, and processing efficiency.

[0004] In recent years, the development of deep learning technology has provided new solutions for the processing of near-infrared visual data. Commonly used segmentation networks include Unet and SegNet. Unet is a convolutional neural network widely used in image segmentation. Its structure mainly consists of two parts: encoder and decoder. The encoder extracts high-level semantic features, and then the decoder gradually restores spatial details. At the same time, it uses cross-neural network layer connections to fuse features at different levels to improve segmentation accuracy. The encoder of SegNet is similar to Unet, containing a series of convolutional layers and maximum pooling layers for feature extraction.

[0005] Each upsampling step of the decoder directly uses the maximum pooling index of the corresponding pooling step of the encoder to restore the feature map to high resolution. The maximum pooling index is used to guide the upsampling process, ensuring that the decoder can more accurately restore spatial details. These two typical segmentation networks process visual data through downsampling and upsampling steps, but cannot take into account both spatial details and semantic information at the same time, resulting in good performance in some visible light visual data, but still have accuracy and robustness problems in processing low-contrast and high-noise near-infrared data.

[0006] In addition, the existing methods lack structural evaluation of segmented near-infrared visual data, which makes it difficult to ensure the reliability of segmentation results in low-contrast and high-noise environments. This brings new challenges to near-infrared image processing and evaluation, including how to more effectively fuse feature information, improve segmentation accuracy in complex scenes, and achieve highly reliable structural evaluation.

[0007] The existing methods have the following problems: First, semantic information loss: Existing segmentation networks such as Unet and SegNet often process high-resolution visual data through downsampling and upsampling steps. This method leads to the loss of semantic information, making it impossible to accurately capture important semantic features when processing near-infrared data, especially in complex scenes, where subtle structures and boundaries are difficult to identify;

[0008] Secondly, the performance in low-contrast and high-noise environments is insufficient: Existing technologies do not perform well in low-contrast and high-noise environments. In such environments, the target objects in the visual data are difficult to be effectively segmented and identified, affecting the overall processing effect. The low contrast and signal-to-noise ratio of near-infrared images themselves make the problem more prominent;

[0009] Third, there is a lack of accurate structural evaluation methods: Existing methods lack accuracy and stability when evaluating the structure of segmented near-infrared visual data. The reliability of the evaluation results is poor in low-contrast and high-noise environments. This is because the existing technology is not optimized for structural evaluation after segmentation, and does not conduct in-depth analysis of the relationship between prediction reliability and classification output. Summary of the invention

[0010] The purpose of the present invention is to provide a semantic segmentation method and system for near-infrared images, which solves the problem of semantic information loss and inaccurate evaluation when near-infrared visual data is processed in high-resolution near-infrared images. The reason is that the spatial details and semantic information cannot be effectively integrated, resulting in insufficient segmentation accuracy. In addition, the existing methods lack the optimization design of structural evaluation after segmentation, which makes the segmentation results unstable in complex environments.

[0011] A first aspect of the present invention provides a semantic segmentation method for near infrared images, comprising:

[0012] The semantic segmentation method of near infrared images is implemented by a network architecture based on BiSeNetV2 bilateral. The network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch. A backbone network is set in the first network branch. A plurality of subdivision neural networks are set in the second network branch. The output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image.

[0013] The near-infrared image to be trained is input into the first network branch and the second network branch, and the network based on BiSeNetV2 bilateral is trained by forward propagation and back propagation methods to obtain the initial semantic image segmentation model. The current near-infrared image to be identified is identified by the initial semantic segmentation model to obtain the current segmented image.

[0014] According to the mean precision and cardinality difference of the current segmented image, it is determined whether the initial semantic image segmentation model needs to continue training until the mean precision and cardinality difference of the current segmented image reach the threshold, and the current semantic image segmentation model is obtained as the final semantic image segmentation model.

[0015] In one embodiment of the semantic segmentation method of near infrared images of the present invention, the input ends of the first network branch and the second network branch are connected to a normalization processing unit. The backbone network is a ResNeXt50 residual network. The multiple subdivision neural networks are multiple arranged convolutional network units.

[0016] The first network branch aggregates data with the second network branch through upsampling / downsampling. The second network branch aggregates data with the first network branch through downsampling / upsampling. The first network branch adopts a context-embedded text representation method and sets an activation function.

[0017] In another embodiment of the semantic segmentation method of near infrared images of the present invention, the method comprises:

[0018] In the training of the initial semantic image segmentation model, the parameters of the near infrared image to be trained are initialized. The near infrared image to be trained after the parameters are initialized is enhanced by the set data enhancement method. The enhanced near infrared image to be trained is trained, a forward propagation prediction value is obtained, and forward training is performed according to the forward propagation prediction value.

[0019] The near infrared image to be trained after training enhancement is obtained, and the backward propagation prediction value is obtained, and the backward training is performed according to the backward propagation prediction value. In the backward propagation training, the training is performed by dynamically adjusting the learning rate and the weight decay coefficient.

[0020] In another embodiment of the semantic segmentation method of near infrared images of the present invention, the method comprises: in the training of the initial semantic image segmentation model, determining whether the training times have been reached, if so, terminating the training, if not, continuing the training until the training times have been reached. The data enhancement methods include: adding random noise to the image, adjusting the contrast of the image, and rotating the image.

[0021] In another embodiment of the semantic segmentation method of the near infrared image of the present invention, the current segmented image is segmented into a plurality of connected domains. The plurality of connected domains are subjected to edge effect processing. The edge processed image is subjected to Gaussian filtering processing to eliminate edge burr errors. The edge effects include: edge expansion, edge corrosion and edge smoothing processing.

[0022] Get the direction and length and width of the minimum enclosing rectangle in the image. According to the direction and length and width of the minimum enclosing rectangle, get the minimum enclosing rectangle for intersection and union. According to the intersection and union of the minimum enclosing rectangle, get the mean accuracy of the current segmented image. Determine whether the mean accuracy is greater than the set value. If so, continue training. If not, end training. According to the direction and length and width of the minimum enclosing rectangle, get the cardinality difference. Determine whether the cardinality difference is greater than the set value. If so, continue training. If not, end training.

[0023] In a second aspect of the present invention, a semantic segmentation system for near infrared images is provided, which comprises: a network construction unit, a training unit, a segmentation unit and an evaluation unit, wherein:

[0024] A network construction unit is configured such that a semantic segmentation system for near-infrared images is implemented by a network architecture based on BiSeNetV2 bilateral. The network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch. A backbone network is set in the first network branch. A plurality of subdivision neural networks are set in the second network branch. The output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image.

[0025] A training unit is configured to input the near infrared image to be trained into the first network branch and the second network branch, and train the BiSeNetV2 bilateral network based on the forward propagation and back propagation system to obtain an initial semantic image segmentation model.

[0026] The segmentation unit is configured to identify the current near-infrared image to be identified through an initial semantic segmentation model to obtain a current segmented image.

[0027] The evaluation unit is configured to determine whether the initial semantic image segmentation model needs to continue training according to the mean precision and cardinality difference of the current segmented image, until the mean precision and cardinality difference of the current segmented image reach a threshold, and obtain the current semantic image segmentation model as the final semantic image segmentation model.

[0028] In another embodiment of the semantic segmentation system for near infrared images of the present invention, the network construction unit is further configured to connect the input ends of the first network branch and the second network branch to the normalization processing unit.

[0029] The backbone network is a ResNeXt50 residual network. The multiple subdivision neural networks are multiple arranged convolutional network units. The first network branch aggregates the data of the second network branch by upsampling / downsampling. The second network branch aggregates the data of the first network branch by downsampling / upsampling. The context-embedded text representation system is used in the first network branch and the activation function is set.

[0030] In another embodiment of the semantic segmentation system for near infrared images of the present invention, the training unit is further configured to include:

[0031] In the training of the initial semantic image segmentation model, the parameters of the near infrared image to be trained are initialized. The near infrared image to be trained after the parameters are initialized is enhanced by the set data enhancement method. The enhanced near infrared image to be trained is trained, a forward propagation prediction value is obtained, and forward training is performed according to the forward propagation prediction value.

[0032] The near infrared image to be trained after training enhancement is obtained, and the backward propagation prediction value is obtained, and the backward training is performed according to the backward propagation prediction value. In the backward propagation training, the training is performed by dynamically adjusting the learning rate and the weight decay coefficient.

[0033] In another embodiment of the semantic segmentation system of near infrared images of the present invention, the system comprises: a training unit, which is further configured to determine whether the number of training times has been reached during the training of the initial semantic image segmentation model, and if so, terminate the training, and if not, continue the training until the number of training times has been reached. The data enhancement method comprises: adding random noise to the image, adjusting the contrast image, and rotating the image.

[0034] In another embodiment of the semantic segmentation system for near infrared images of the present invention, the evaluation unit is further configured to segment the current segmented image into a plurality of connected domains and perform edge effect processing on the plurality of connected domains.

[0035] Perform Gaussian filtering on the edge processed image to eliminate edge burr errors. Edge effects include: edge expansion, edge erosion and edge smoothing. Obtain the direction and length and width of the minimum enclosing rectangle in the image.

[0036] According to the direction, length and width of the minimum enclosing rectangle, the minimum enclosing rectangle is obtained for intersection and union. According to the intersection and union of the minimum enclosing rectangle, the mean accuracy of the current segmented image is obtained. Determine whether the mean accuracy is greater than the set value. If so, continue training. If not, end training. According to the direction, length and width of the minimum enclosing rectangle, the cardinality difference is obtained. Determine whether the cardinality difference is greater than the set value. If so, continue training. If not, end training.

[0037] The following will further explain the characteristics, technical features, advantages and implementation methods of the semantic segmentation method and system for near-infrared images in a clear and easy-to-understand manner with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart for illustrating a semantic segmentation method of a near-infrared image in one embodiment of the present invention.

[0039] Figure 2 It is a schematic diagram for illustrating the overall process of image processing and evaluation based on the BiSeNetV2 segmentation model in one embodiment of the present invention.

[0040] Figure 3 It is a schematic diagram for illustrating the composition of an improved BiSeNetV2 segmentation model in another embodiment of the present invention.

[0041] Figure 4 It is a control diagram for illustrating the training process of a network based on BiSeNetV2 bilateral in another embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram for explaining the composition of a precise structure evaluation module based on segmentation results in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to have a clearer understanding of the technical features, purposes and effects of the invention, the specific embodiments of the present invention are now described with reference to the accompanying drawings. The same reference numerals in the drawings represent components with the same structure or similar structures but the same functions.

[0044] In this article, "schematic" means "serving as an example, instance or illustration", and any diagram or implementation described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, each figure only schematically shows the parts related to the exemplary embodiment, and they do not represent the actual structure and true proportion of the product.

[0045] In order to solve these problems, this paper proposes a near-infrared semantic segmentation and evaluation method based on improved BiSeNetV2. This method first uses ResNeXt50 as the neural network backbone for feature extraction, and uses its powerful feature representation ability to effectively capture the key information in near-infrared images.

[0046] Secondly, BiSeNetV2 is used as the main architecture, combined with downsampling and upsampling branches to achieve efficient and accurate segmentation of visual data such as images. In this design, the near-infrared image is input into the backbone module containing detail branch and semantic branch. The detail branch contains 3 channels, corresponding to different levels of feature extraction, which is used to capture low-level, mid-level and high-level detail information in the image. The semantic branch reduces the model complexity through lightweight processing and outputs the context embedding block in the last stage to extract semantic information.

[0047] Finally, in the aggregation module, a bilateral aggregation layer is used to fuse the features extracted from different paths. In order to improve the adaptability of the model to low-contrast and high-noise near-infrared data, a variety of data enhancement techniques and regularization methods are introduced during the training process to improve the robustness and accuracy of the model in complex environments.

[0048] In addition to optimizing the segmentation process, the present invention also developed a set of instance-based accurate structure evaluation methods. This method extracts and analyzes geometric features, covers the steps of connected domain expansion, feature calculation, and applicability judgment, and determines the position and angle of the structure based on the minimum enclosing rectangle, thereby ensuring the accuracy of the segmentation result measurement.

[0049] The evaluation of the measurement results uses two indicators, Mean Average Precision (mAP) and Cardinality Difference (CD), which are obtained by matching the overlapping areas (Intersection over Union, IoU) of the true value partition and the predicted partition. This series of steps and evaluation indicators jointly ensure the reliability and accuracy of the evaluation results.

[0050] like Figure 1 As shown, the first aspect of the present invention provides a semantic segmentation method for near infrared images, which comprises:

[0051] Step S101, the semantic segmentation method of the near infrared image is implemented by a network architecture based on BiSeNetV2 bilateral. The network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch. A backbone network is set in the first network branch. A plurality of subdivision neural networks are set in the second network branch. The output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image.

[0052] Step S102, inputting the near infrared image to be trained into the first network branch and the second network branch, training the BiSeNetV2 bilateral network by forward propagation and back propagation methods, and obtaining an initial semantic image segmentation model.

[0053] Step S103, identifying the current near-infrared image to be identified by using the initial semantic segmentation model to obtain a current segmented image.

[0054] Step S104, judging whether the initial semantic image segmentation model needs to continue training according to the mean precision and cardinality difference of the current segmented image, until the mean precision and cardinality difference of the current segmented image reach a threshold, and obtaining the current semantic image segmentation model as the final semantic image segmentation model.

[0055] In another embodiment of the semantic segmentation method of near infrared images described in the present invention, Figure 2 As shown in the figure, the image processing process based on the BiSeNetV2 segmentation model is demonstrated. First, the bilateral segmentation network BiSeNetV2 segmentation model is built and improved. Then, the model is trained to improve the segmentation ability. Then the model inference phase is carried out. Finally, the segmentation result is output through precise structural evaluation.

[0056] In one embodiment of the semantic segmentation method of near infrared images of the present invention, the input ends of the first network branch and the second network branch are connected to a normalization processing unit. The backbone network is a ResNeXt50 residual network. The multiple subdivision neural networks are multiple arranged convolutional network units.

[0057] The first network branch aggregates data with the second network branch through upsampling / downsampling. The second network branch aggregates data with the first network branch through downsampling / upsampling. The first network branch adopts a context-embedded text representation method and sets an activation function.

[0058] In another embodiment of the semantic segmentation method for near infrared images described in the present invention, the improved BiSeNetV2 segmentation model is as follows: Figure 3 As shown. The present invention combines the bilateral network structure of BiSeNetV2 with the feature extraction capability of ResNeXt50, aiming to improve the accuracy and efficiency of image segmentation. The specific implementation steps include preprocessing of input images, feature extraction, feature fusion design, etc. The technical solution and implementation process of each step will be described in detail below.

[0059] First, the input image is normalized by module 101 to map the pixel values ​​to the range of [0,1]. Normalization improves the stability and training efficiency of the model. The preprocessed image is passed to the detail branch and semantic branch of the BiSeNetV2 segmentation model.

[0060] The detail branch 102 includes C1, C2 and C3 channels, which are used to extract low-dimensional spatial detail information in the image. The channels at each stage can be lightweight processed by the factor λ (λ<1). Its design structure is shallow and the stride is small, which can process information quickly and efficiently. The feature representation in the detail branch has a large spatial size and wide channels, which can capture a large amount of information in the image. To avoid increasing memory access costs and reducing speed, the detail branch does not use residual connections, that is, it allows information to bypass the connections of certain layers in the neural network.

[0061] The semantic branch 103 improves the level of feature representation through a fast downsampling strategy and embeds global context responses through global average pooling. Compared with the detail branch that captures low-level spatial details, the semantic branch is responsible for capturing high-level semantics and adopts the ResNeXt50 backbone network, which is an image classification network architecture based on the concept of aggregated residual transformation, which aggregates a group of transformations with the same topological structure by repeating a building block.

[0062] In the backbone network, a single neuron performs an inner product, which is a weighted sum of the input vector x. The inner product can be viewed as a form of aggregation transformation, where the weights of the filter are multiplied by the input vector and summed. The operation of a single neuron can be divided into three steps: splitting, transforming, and embedding. In the splitting step, the input vector x is cut into a low-dimensional embedding, i.e., a one-dimensional subspace x i In the transformation step, by using the filter weights w i x i The low-dimensional representation is transformed by scaling the low-dimensional representation. In the embedding step 104, the transformations in all embeddings are aggregated over all dimensions of the input vector. The overall process can be expressed as Formula 1:

[0063]

[0064] The above function is context-embedded, and the general unit of the neural network after execution is expressed as Formula 2:

[0065]

[0066] where τ i (x) can be any arbitrary function. This contextual embedding is extended along the new dimension, and the size of the set of transformations to be embedded is denoted by C and is called the cardinality.

[0067] In the aggregation step 105, in order to solve the problem that the feature representations of the detail branch and the semantic branch are complementary but do not know each other's information, a bilateral guided fusion technology is designed. The residual function executed by the aggregation layer involves branching and concatenation. Multiple paths share the same topology, and less additional computing power is required when designing each path. Multi-scale information is sampled after the Sigmoid activation function, and the formula description is as shown in Formula 3:

[0068]

[0069] The downsampling operation is recorded as Down, and the upsampling operation is recorded as Up. represents the Sigmoid function, Represents element-wise multiplication.

[0070] In the training optimization stage, the design of the loss function takes into account the characteristics of the image segmentation task and combines the cross entropy loss L c and boundary loss L b , α and β are weighted coefficients, which can better capture the details and boundary information in the image. The loss function is defined as shown in Formula 4:

[0071] L=α·L c +β·L b Formula 4

[0072] In addition, a loss function based on the Dice coefficient is used to further improve the performance of the model on imbalanced data. The Dice loss function is defined as follows:

[0073]

[0074] Among them, P represents the predicted segmentation result and G represents the actual segmentation label.

[0075] In order to control the update speed of model parameters during training, a learning rate decay strategy is also introduced. The learning rate update formula is as shown in Formula 6:

[0076]

[0077] Among them, η t represents the learning rate at the tth iteration, η0 represents the initial learning rate, and T represents the total number of training times.

[0078] In another embodiment of the semantic segmentation method of near infrared images of the present invention, the method comprises:

[0079] In the training of the initial semantic image segmentation model, the parameters of the near infrared image to be trained are initialized. The near infrared image to be trained after the parameters are initialized is enhanced by the set data enhancement method. The enhanced near infrared image to be trained is trained, a forward propagation prediction value is obtained, and forward training is performed according to the forward propagation prediction value.

[0080] The near infrared image to be trained after training enhancement is obtained, and the backward propagation prediction value is obtained, and the backward training is performed according to the backward propagation prediction value. In the backward propagation training, the training is performed by dynamically adjusting the learning rate and the weight decay coefficient.

[0081] The training process of the segmentation model is as follows Figure 4 As shown. First, the model parameters are initialized. Next, the data is divided into small batches for processing, and the input data is forward propagated through the calculations of each layer of the neural network to obtain the prediction results. Then, the loss between the predicted value and the true value is calculated. Next, the loss is passed back through the network through back propagation, the gradient of each parameter is calculated, and these gradients are used to update the parameters. The system checks whether the predetermined number of training times has been reached. If not, it returns to continue training. This process is repeated until the predetermined number of training times is reached, thereby improving the segmentation ability of low-contrast and high-noise near-infrared visual data.

[0082] In order to improve the segmentation ability of the model in low-contrast and high-noise near-infrared visual data, the following three main technical means are adopted: adjusting the training strategy, regularization and introducing data enhancement technology. First, module 201 data enhancement technology can significantly improve the robustness of the model.

[0083] By introducing methods such as random noise addition, contrast adjustment, image rotation and flipping, we generate diverse training samples and enhance the model's adaptability to different environments. Secondly, through a multi-stage training strategy, we use a standard dataset for pre-training in the initial stage, and then gradually introduce low-contrast and high-noise data. This progressive training method enables the model to better adapt to complex environments.

[0084] During the training process, module 202 dynamically adjusts the learning rate and weight decay coefficient to ensure the stability and adaptability of the model at each stage.

[0085] As shown in Formula 7, where L total is the total loss, y i is the true value, is the predicted value, λ is the regularization parameter, and w j are the weights of the model.

[0086]

[0087] As shown in Formula 8, the L2 regularization method of module 203 effectively prevents the model from overfitting, further improving the stability and robustness of the model.

[0088]

[0089] In another embodiment of the semantic segmentation method of near infrared images of the present invention, the method comprises: in the training of the initial semantic image segmentation model, determining whether the training times have been reached, if so, terminating the training, if not, continuing the training until the training times have been reached. The data enhancement methods include: adding random noise to the image, adjusting the contrast of the image, and rotating the image.

[0090] In another embodiment of the semantic segmentation method of the near infrared image of the present invention, the current segmented image is segmented into a plurality of connected domains. The plurality of connected domains are subjected to edge effect processing. The edge processed image is subjected to Gaussian filtering processing to eliminate edge burr errors. The edge effects include: edge expansion, edge corrosion and edge smoothing processing.

[0091] Get the direction and length and width of the minimum enclosing rectangle in the image. According to the direction and length and width of the minimum enclosing rectangle, get the minimum enclosing rectangle for intersection and union. According to the intersection and union of the minimum enclosing rectangle, get the mean accuracy of the current segmented image. Determine whether the mean accuracy is greater than the set value. If so, continue training. If not, end training. According to the direction and length and width of the minimum enclosing rectangle, get the cardinality difference. Determine whether the cardinality difference is greater than the set value. If so, continue training. If not, end training.

[0092] Accurate structural evaluation based on segmentation results is of great significance to machine vision tasks. By performing detailed structural analysis on the segmentation results, the accuracy of image segmentation can be improved, thereby providing a reliable foundation for subsequent high-level visual tasks. The specific process is as follows: Figure 5 As shown, the following is a detailed description of the method and evaluation metrics:

[0093] First, module 301 performs a n Dilation, erosion, smoothing operations are performed, and Gaussian filtering is performed to eliminate edge burr errors and improve segmentation accuracy. Then, module 302 extracts the edge points of each connected domain and calculates its roundness. i ) represents the area of ​​the connected domain, P(X i ) represents the perimeter of the connected domain, and the calculation formula of the roundness is formula 9:

[0094]

[0095] Next, the length and width of the minimum circumscribed rectangle are calculated. If the width is narrow, the connected domain is ignored. The specific steps of module 303 include: First, the centroid is calculated, that is, the average position of all points, as shown in formula 10:

[0096]

[0097] Where p represents X i The point in the x and p y They represent the horizontal and vertical coordinates of point p, respectively, and n represents X i The total number of midpoints.

[0098] Second, X iAll points are translated relative to the centroid so that the centroid becomes the origin of the coordinates. Third, for the translated point set, the PCA algorithm is applied to calculate its covariance matrix. Fourth, the covariance matrix is ​​decomposed to obtain the eigenvectors and eigenvalues. Fifth, the eigenvector corresponding to the larger eigenvalue in the eigenvector is the direction of the fitting rectangle. Sixth, the direction of the fitting rectangle is rotated back to the original coordinate system. Seventh, according to the rotated coordinate system, the side length of the fitting rectangle is calculated, which is the size of the minimum circumscribed rectangle.

[0099] The example-based evaluation method mainly includes two evaluation indicators: Mean Average Precision (mAP) of module 304 and Cardinality Difference (CD) of module 305. For mean precision, it is necessary to calculate the Intersection over Union (IoU) of the minimum bounding rectangle of each predicted partition calculated above and the minimum bounding rectangle of all true value segmentation results one by one. The IoU threshold t ranges from 0.5 to 1, and the step size is 0.05.

[0100] If the intersection and union ratio of the minimum enclosing rectangle in a certain prediction result and the minimum enclosing rectangle in the true value result is greater than a specific threshold, it is considered a match. Among them, True Positive (TP) represents the number of partition pairs whose minimum enclosing rectangles in the true and predicted results are correctly matched. False Positive (FP) represents the number of partitions that are not correctly matched in the predicted segmentation result. False Negative (FN) represents the number of partitions that are not correctly matched in the true value segmentation result. At each threshold, an accuracy value is calculated based on the number of TP, FP and FN generated by comparing the minimum enclosing rectangles of the predicted segmentation partition result with the partition result of the true value segmentation, and then divided by the number of thresholds to obtain the mean accuracy. Finally, the mean accuracy of all images is calculated, and then the average is calculated, as shown in Formula 11:

[0101]

[0102] For the calculation of cardinality difference, the number of instances in the true value result and the predicted result is determined according to the minimum enclosing rectangle of the partition, as shown in Formula 12:

[0103] CD=Num T -Num P formula 12

[0104] Among them, Num T Represents the number of instances in the actual result, Num PRepresents the number of instances in the prediction result. If CD is a positive number, it indicates under-segmentation; if CD is a negative number, it indicates over-segmentation. Through this evaluation method, the accuracy and reliability of the segmentation results can be comprehensively evaluated, thereby improving the applicability of the image segmentation algorithm.

[0105] A network construction unit is configured such that a semantic segmentation system for near-infrared images is implemented by a network architecture based on BiSeNetV2 bilateral. The network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch. A backbone network is set in the first network branch. A plurality of subdivision neural networks are set in the second network branch. The output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image.

[0106] A training unit is configured to input the near infrared image to be trained into the first network branch and the second network branch, and train the BiSeNetV2 bilateral network based on the forward propagation and back propagation system to obtain an initial semantic image segmentation model.

[0107] The segmentation unit is configured to identify the current near-infrared image to be identified through an initial semantic segmentation model to obtain a current segmented image.

[0108] The evaluation unit is configured to determine whether the initial semantic image segmentation model needs to continue training according to the mean precision and cardinality difference of the current segmented image, until the mean precision and cardinality difference of the current segmented image reach a threshold, and obtain the current semantic image segmentation model as the final semantic image segmentation model.

[0109] In another embodiment of the semantic segmentation system for near infrared images of the present invention, the network construction unit is further configured to connect the input ends of the first network branch and the second network branch to the normalization processing unit.

[0110] The backbone network is a ResNeXt50 residual network. The multiple subdivision neural networks are multiple arranged convolutional network units. The first network branch aggregates the data of the second network branch by upsampling / downsampling. The second network branch aggregates the data of the first network branch by downsampling / upsampling. The first network branch adopts a context-embedded text representation system and sets an activation function.

[0111] In another embodiment of the semantic segmentation system of the near infrared image of the present invention, the training unit is further configured to initialize the parameters of the near infrared image to be trained in the training of the initial semantic image segmentation model, enhance the near infrared image to be trained after the parameter initialization through a set data enhancement method, train the enhanced near infrared image to be trained, obtain the forward propagation prediction value, and perform forward training according to the forward propagation prediction value.

[0112] The near infrared image to be trained after training enhancement is obtained, and the backward propagation prediction value is obtained, and the backward training is performed according to the backward propagation prediction value. In the backward propagation training, the training is performed by dynamically adjusting the learning rate and the weight decay coefficient.

[0113] In another embodiment of the semantic segmentation system of near infrared images of the present invention, the system comprises: a training unit, which is further configured to determine whether the number of training times has been reached during the training of the initial semantic image segmentation model, and if so, terminate the training, and if not, continue the training until the number of training times has been reached. The data enhancement method comprises: adding random noise to the image, adjusting the contrast image, and rotating the image.

[0114] In another embodiment of the semantic segmentation system for near infrared images of the present invention, the evaluation unit is further configured to segment the current segmented image into a plurality of connected domains and perform edge effect processing on the plurality of connected domains.

[0115] Perform Gaussian filtering on the edge processed image to eliminate edge burr errors. Edge effects include: edge expansion, edge erosion and edge smoothing. Obtain the direction and length and width of the minimum enclosing rectangle in the image.

[0116] According to the direction, length and width of the minimum enclosing rectangle, the minimum enclosing rectangle is obtained for intersection and union. According to the intersection and union of the minimum enclosing rectangle, the mean accuracy of the current segmented image is obtained. Determine whether the mean accuracy is greater than the set value. If so, continue training. If not, end training. According to the direction, length and width of the minimum enclosing rectangle, the cardinality difference is obtained. Determine whether the cardinality difference is greater than the set value. If so, continue training. If not, end training.

[0117] The present invention modifies the MobileNetV2 or Xception network used in the semantic branch of BiSeNetV2 into ResNeXt50, uses ResNeXt50 to perform feature extraction optimization operations, and combines the downsampling and upsampling branch structures of BiSeNetV2 to achieve efficient and accurate image segmentation. This combined design greatly improves the accuracy and efficiency of image segmentation while keeping the model lightweight.

[0118] The present invention improves the segmentation capability of low-contrast and high-noise near-infrared visual data. Technical effect: The present invention improves the segmentation capability of the model in processing near-infrared visual data in low-contrast and high-noise environments by optimizing the training method of the segmentation model. By adjusting the training strategy, introducing data enhancement technology and regularization methods, the model can better understand and adapt to semantic information in complex environments, further improving the accuracy and robustness of segmentation tasks.

[0119] Based on the segmentation results, the present invention can achieve high-precision measurement of the target structure through geometric feature extraction and analysis. The evaluation method used combines the mean precision and cardinality difference, and judges the accuracy of the segmentation effect by matching the overlapping area of ​​the true value partition and the predicted partition. This method can not only comprehensively evaluate the accuracy of the segmentation results, but also help understand and improve the applicability of different image segmentation methods and ensure the reliability of the algorithm.

[0120] The present invention improves the segmentation accuracy and robustness of near-infrared visual data by optimizing the segmentation model and introducing an evaluation method, ensuring that reliable segmentation results can still be obtained in complex environments. Through multi-level feature extraction and optimized segmentation and evaluation methods, the performance of the model in low-contrast and high-noise environments is significantly improved, making the segmentation results more accurate and reliable. The method proposed in the present invention can promote the development of near-infrared visual data processing technology and provide efficient visual processing solutions for future smart medical care, industrial detection, intelligent monitoring and other fields.

[0121] In one embodiment of the present invention, the segmentation model can be further improved by using other deep learning network architectures, such as using DenseNet (densely connected network) to enhance feature reuse through dense connections between layers, which can improve the ability to capture details and semantic information in segmentation tasks. Replacing the semantic branch network in BiSeNetV2 with DenseNet can achieve an effect similar to ResNeXt50 and improve the accuracy and efficiency of image segmentation.

[0122] EfficientNet achieves a good balance between performance and efficiency through a compound scaling method. Replacing the semantic branch network of BiSeNetV2 with EfficientNet can improve segmentation accuracy and computational efficiency.

[0123] Further improve the segmentation ability in low-contrast and high-noise environments: Use other optimization training methods. For example, through self-supervised learning methods, use a large amount of unlabeled data for pre-training to improve the model's feature representation ability in low-contrast and high-noise environments. Self-supervised learning can improve the model's segmentation ability in near-infrared visual data.

[0124] Adversarial training is used to generate adversarial samples to enhance the robustness and generalization ability of the model. In low-contrast and high-noise environments, the accuracy and stability of the segmentation model can be improved by introducing adversarial training.

[0125] Furthermore, it is also possible to evaluate the precise structure based on the segmentation result, for example, by using other evaluation methods. The precise structure is evaluated using the Hausdorff distance, which is a metric for measuring the distance between the farthest points between two sets of points. In the evaluation of the segmentation result, the use of the Hausdorff distance can more accurately reflect the degree of match between the segmentation result and the true value.

[0126] It should be understood that although this specification is described according to various implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0127] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic segmentation method for near infrared images, characterized in that: It includes: The semantic segmentation method of the near infrared image is implemented by a network architecture based on BiSeNetV2 bilateral; The network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch; a backbone network is set in the first network branch; a plurality of subdivision neural networks are set in the second network branch; the output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image; Inputting the near infrared image to be trained into the first network branch and the second network branch, training the BiSeNetV2 bilateral network by forward propagation and back propagation methods to obtain an initial semantic image segmentation model; Recognize the current near-infrared image to be recognized by using the initial semantic segmentation model to obtain a current segmented image; According to the mean precision and cardinality difference of the current segmented image, it is determined whether the initial semantic image segmentation model needs to continue training until the mean precision and cardinality difference of the current segmented image reach a threshold, and the current semantic image segmentation model is obtained as the final semantic image segmentation model.

2. The semantic segmentation method of near infrared images according to claim 1, characterized in that: The input ends of the first network branch and the second network branch are connected to a normalization processing unit; The backbone network is a ResNeXt50 residual network; the multiple subdivision neural networks are multiple arranged convolutional network units; The first network branch is aggregated by upsampling / downsampling and data of the second network branch; The second network branch aggregates data of the first network branch by downsampling / upsampling; A context-embedded text representation method is adopted in the first network branch and an activation function is set.

3. The semantic segmentation method of near infrared images according to claim 1 or 2, characterized in that: include: In the training of the initial semantic image segmentation model, parameters are initialized for the near infrared image to be trained; Enhance the near infrared image to be trained after the parameters are initialized by a set data enhancement method; Training the enhanced near-infrared image to be trained, obtaining a forward propagation prediction value, and performing forward training according to the forward propagation prediction value; The enhanced near-infrared image to be trained is trained, a backward propagation prediction value is obtained, and backward training is performed according to the backward propagation prediction value; in the backward propagation training, training is performed by dynamically adjusting the learning rate and the weight attenuation coefficient.

4. The semantic segmentation method of near infrared images according to claim 3, characterized in that: include: In the training of the initial semantic image segmentation model, determining whether a training number is reached, if so, ending the training, if not, continuing the training until the training number is reached; The data enhancement method includes: adding random noise to the image, adjusting the contrast of the image and rotating the image.

5. The semantic segmentation method of near infrared images according to claim 1, characterized in that: Segmenting the current segmented image into a plurality of connected domains; Process edge effects on several connected domains; Performing Gaussian filtering on the edge-processed image to eliminate edge burr errors; the edge effects include: edge expansion, edge corrosion and edge smoothing; Get the direction, length and width of the minimum enclosing rectangle in the image; According to the direction, length and width of the minimum enclosing rectangle, the minimum enclosing rectangle is obtained for intersection and union ratio; according to the intersection and union ratio of the minimum enclosing rectangle, the mean accuracy of the current segmented image is obtained; Determine whether the mean accuracy is greater than a set value, if so, continue training, if not, end training; The cardinality difference is obtained according to the direction, length and width of the minimum circumscribed rectangle; and it is determined whether the cardinality difference is greater than a set value, if so, the training continues, and if not, the training ends.

6. A semantic segmentation system for near infrared images, characterized in that: It includes: Network construction unit, training unit, segmentation unit and evaluation unit, where: The network construction unit is configured such that the semantic segmentation system of the near-infrared image is implemented by a network architecture based on BiSeNetV2 bilateral; the network architecture based on BiSeNetV2 bilateral includes a first network branch and a second network branch; a backbone network is set in the first network branch; a plurality of subdivided neural networks are set in the second network branch; the output of the first network branch is a main semantic segmentation image, and the output of the second network branch is a detail semantic segmentation image; The training unit is configured to input the near infrared image to be trained into the first network branch and the second network branch, and train the BiSeNetV2 bilateral network through a forward propagation and a back propagation system to obtain an initial semantic image segmentation model; The segmentation unit is configured to identify the current near-infrared image to be identified through the initial semantic segmentation model to obtain a current segmented image; The evaluation unit is configured to determine whether the initial semantic image segmentation model needs to continue training according to the mean precision and cardinality difference of the current segmented image, until the mean precision and cardinality difference of the current segmented image reach a threshold, and obtain the current semantic image segmentation model as the final semantic image segmentation model.

7. The semantic segmentation system for near infrared images according to claim 6, characterized in that: The network construction unit is further configured to connect the input ends of the first network branch and the second network branch to a normalization processing unit; The backbone network is a ResNeXt50 residual network; the multiple subdivision neural networks are multiple arranged convolutional network units; The first network branch is aggregated by upsampling / downsampling and data of the second network branch; The second network branch aggregates data of the first network branch by downsampling / upsampling; A context-embedded text representation system is adopted in the first network branch and an activation function is set.

8. The semantic segmentation system for near infrared images according to claim 6 or 7, characterized in that: The training unit is further configured to include: In the training of the initial semantic image segmentation model, parameters are initialized for the near infrared image to be trained; Enhance the near infrared image to be trained after the parameters are initialized by a set data enhancement method; Training the enhanced near-infrared image to be trained, obtaining a forward propagation prediction value, and performing forward training according to the forward propagation prediction value; The enhanced near-infrared image to be trained is trained, a backward propagation prediction value is obtained, and backward training is performed according to the backward propagation prediction value; in the backward propagation training, training is performed by dynamically adjusting the learning rate and the weight attenuation coefficient.

9. The semantic segmentation system for near infrared images according to claim 8, characterized in that: include: The training unit is further configured to determine whether a training number is reached during the training of the initial semantic image segmentation model, and if so, terminate the training; if not, continue the training until the training number is reached; The data enhancement method includes: adding random noise to the image, adjusting the contrast of the image and rotating the image.

10. The semantic segmentation system for near infrared images according to claim 6, characterized in that: The evaluation unit is further configured to segment the current segmented image into a plurality of connected domains; Process edge effects on several connected domains; Performing Gaussian filtering on the edge-processed image to eliminate edge burr errors; the edge effects include: edge expansion, edge corrosion and edge smoothing; Get the direction, length and width of the minimum enclosing rectangle in the image; According to the direction, length and width of the minimum enclosing rectangle, the minimum enclosing rectangle is obtained for intersection and union ratio; according to the intersection and union ratio of the minimum enclosing rectangle, the mean accuracy of the current segmented image is obtained; Determine whether the mean accuracy is greater than a set value, if so, continue training, if not, end training; The cardinality difference is obtained according to the direction, length and width of the minimum circumscribed rectangle; and it is determined whether the cardinality difference is greater than a set value, if so, the training continues, and if not, the training ends.

Images