An image analysis method and system based on multi-branch networks

Abstract

This invention provides an image analysis method and system based on a multi-branch network, relating to the field of smart healthcare technology. It includes: acquiring and annotating multi-task data; constructing a multi-branch image recognition network model, which includes a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch; and training the multi-branch image recognition network model using a morphological perception loss function. Compared to a two-stage network that first performs object detection and then image segmentation or keypoint recognition, this method has a faster computation speed. That is, without increasing the number of networks or significantly increasing recognition time, it uses a single neural network to complete the analysis of multiple visual recognition tasks without compromising the accuracy of the recognition results.

Description

Technical Field

[0001] This invention relates to the field of remotely operated robot system control technology, and in particular to an image analysis method and system based on multi-branch networks. Background Technology

[0002] In the task of intracytoplasmic sperm injection (ICSI) microscopic image analysis, it is necessary to use artificial intelligence and computer vision technology to automatically, accurately and efficiently detect and identify target areas such as injection needles, eggs and sperm in microscopic images, so as to provide positioning support for the characterization of the sperm injection process and the automated control of the injection device.

[0003] However, in practical applications, it is necessary to simultaneously perform target detection of sperm, image segmentation of needles, eggs, and polar bodies, and keypoint detection for needle tip localization. These multi-task image recognition requirements pose a challenge to the real-time performance of algorithms. Traditional methods such as Faster-RCNN and Mask-RCNN, which first perform target detection and then semantic segmentation and keypoint detection within the target bounding box, require multiple stages of image analysis, thus prolonging the actual analysis time and failing to meet the real-time image analysis requirements of practical applications.

[0004] Furthermore, in intracytoplasmic sperm injection (ICSI) microscopy, the accuracy of needle tip identification directly affects the operation of automated injection devices, making needle tip identification even more important. However, needle tip identification has not been optimized in the past.

[0005] Therefore, how to improve the network structure and optimize the algorithm convergence method to meet the above-mentioned multi-task image recognition requirements while maintaining high efficiency, and to improve the accuracy of the needle tip recognition algorithm, is an important research problem in the task of intracytoplasmic sperm injection microscopic image analysis of oocytes. Summary of the Invention

[0006] This invention proposes an image analysis method based on multi-branch networks, thereby solving the problem of how to improve the network structure and optimize the algorithm convergence method, so as to meet the above-mentioned multi-task image recognition requirements with high efficiency and improve the accuracy of the pinhead recognition algorithm.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] This invention provides an image analysis method based on multi-branch networks, applicable to electronic devices, and the method steps include:

[0009] S1. Acquire multi-task data and label it;

[0010] S2. Construct a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch;

[0011] S3. Based on multi-task data and annotations, a multi-branch image recognition network model is trained using the morphological perception loss function to obtain the trained model.

[0012] S4. Input the image to be analyzed into the trained model to obtain analysis data and complete the image analysis based on the multi-branch network.

[0013] Preferably, in S1, acquiring multi-task data and annotations includes:

[0014] The original image is labeled according to the labeling method of each task in the multi-task requirement;

[0015] The original images and multi-task labeled data are divided into training set, validation set and test set according to a preset ratio;

[0016] The annotation methods include: for image segmentation tasks, dividing the foreground and background of the original image to obtain the mask annotation of the original image; for object detection tasks, using bounding boxes to annotate the objects to be identified in the original image, and concentrating all the bounding boxes of a single original image into a single text; for key point detection tasks, using point annotation methods to annotate the key points to be identified in the original image, and concentrating all the point annotations of a single original image into a single text.

[0017] Preferably, in S2, a multi-branch image recognition network model is constructed. This multi-branch image recognition network model includes: a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch, comprising:

[0018] The original image is input into the main branch of feature extraction to obtain the high-dimensional features corresponding to the original image.

[0019] Input the high-dimensional features into the keypoint extraction branch to obtain the coordinates of M keypoints. i∈[0,M-1];

[0020] High-dimensional features are input into the image segmentation branch to obtain an image segmentation confidence map corresponding to the original image size. The image segmentation confidence map has dimensions H, W, and C, where H represents the height, W represents the width, and C represents the number of categories.

[0021] The high-dimensional features are input into the target detection branch to obtain the position parameters of q targets to be detected. in These are the top-left corner coordinates, top-right corner coordinates, width, height, category, and confidence level of the target q to be detected, where q∈[0,Q-1].

[0022] Preferably, the main branch for image feature extraction includes one or more of the following networks: VGG, DenseNet, and ResNet.

[0023] Preferably, the main branch of feature extraction from the original image input is used to obtain the high-dimensional features corresponding to the original image, including:

[0024] The original image with size H×W was downsampled to three sizes: H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, resulting in three different scale feature maps.

[0025] Using variants of Feature Pyramid Network (FPN), Adaptive Feature Pyramid Network (PAN), or combinations thereof, feature maps of different scales are upsampled or downsampled to achieve the same size H / 8×W / 8.

[0026] By adding the values ​​at the same position in feature maps of the same size after upsampling or downsampling, or by merging them along the channel dimension, high-dimensional features can be obtained through feature fusion.

[0027] Preferably, in S3, based on multi-task data and annotations, a multi-branch image recognition network model is trained using a morphology-aware loss function to obtain the trained model, including:

[0028] Construct a shape-aware loss function for multi-branch tasks;

[0029] Based on multi-task data and annotations, a multi-branch image recognition network model is trained using the morphological perception loss function of multi-branch tasks.

[0030] Preferably, the form-aware loss function for multi-branch tasks includes:

[0031] The morphological perception loss function L for multi-branch tasks is constructed according to the following formula (1):

[0032] L = L seg +L obj +L point (1)

[0033] Where L seg L obj , and L point These are the loss functions for the image segmentation branch, the object detection branch, and the keypoint extraction branch, respectively. obj L is the sum of the IOU loss between the predicted coordinates and the true coordinates, and the BCE loss between the predicted confidence and the true label. pointThe BCE loss is used to predict the coordinates of keypoints and the actual keypoint locations.

[0034] Preferably, the definition of the image segmentation branch includes:

[0035] L is defined according to the following formula (2). seg :

[0036]

[0037] Where: N is the number of pixels in the image; l(x) is the true classification corresponding to pixel x. p represents the weight of the loss function corresponding to pixel x. l(x) (x) refers to the probability that pixel x belongs to its true category in the detection result, and Ω represents the pixel space.

[0038] Preferably, the definition of the image segmentation branch includes:

[0039] The weights consist of three weight loss terms: category loss term, class loss term, and class loss term. Morphological loss term Key point localization loss term w KPL (x), calculated according to the following formula (3). Weight:

[0040]

[0041] Where λ1, λ2, and λ3 are three weighted hyperparameters used to control the proportion of the three weight items; T is the category in which the target key point is located;

[0042] The category loss term is calculated according to the following formula (4):

[0043]

[0044] Where, num k C is the number of pixels in category k; C is the number of categories including the background; and i is the category weight channel number, which takes a value from the set {0, ..., C}.

[0045] The morphological loss term is calculated according to the following formula (5):

[0046]

[0047] Where d(x) is the nearest distance from pixel x to the edge of its foreground category l(x), x i Let l(x) be the mapping of pixel x to class weight channel i. i ) is x i The corresponding category; when i = 0, This represents the morphological loss value from the background pixel to the foreground region; when i≠0, This represents the shape loss value from the foreground pixel to its corresponding foreground region boundary;

[0048] The key point localization loss term is calculated according to the following formula (6):

[0049]

[0050] Where A is the location of the key target point in the image, and d A (x) represents the distance from pixel x in the foreground region to the target region A at the keypoint. The target distance loss w is calculated for each pixel x. KPL (x), forming the target distance loss term w KPL .

[0051] An image analysis system based on multi-branch networks, the system being used in the aforementioned image analysis method based on multi-branch networks, the system comprising:

[0052] The data acquisition block is used to acquire and label multi-task data.

[0053] The model building module is used to build a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch.

[0054] The model training module is used to train the multi-branch image recognition network model based on the multi-task data and annotations using the morphological perception loss function, and obtain the trained model.

[0055] The image analysis module is used to input the image to be analyzed into the trained model, obtain analysis data, and complete image analysis based on a multi-branch network.

[0056] On one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described image analysis method based on a multi-branch network.

[0057] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described image analysis method based on a multi-branch network.

[0058] The above technical solution has at least the following advantages compared with the existing technology:

[0059] The above-described solution, provided by this invention, is an image analysis method based on a multi-branch network. Addressing the shortcomings of existing technologies, it acquires multi-task data and annotations; constructs a multi-branch image recognition network model, which includes a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch; and trains the multi-branch image recognition network model using a morphological perception loss function. Compared to two-stage networks that first perform object detection and then image segmentation or keypoint recognition, this method offers faster computation speed. Specifically, it completes the analysis of multiple visual recognition tasks using a single neural network without increasing the number of networks or significantly increasing recognition time, and without compromising the accuracy of the recognition results. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 The flowchart illustrates an image analysis method based on a multi-branch network provided by this invention.

[0062] Figure 2 This is a schematic diagram of the network structure provided in an embodiment of the present invention;

[0063] Figure 3 A shape-aware weighted graph provided for embodiments of the present invention;

[0064] Figure 4 This is a schematic diagram of an image analysis system based on a multi-branch network provided by the present invention.

[0065] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0067] This invention addresses the limitation of existing general object detection networks in simultaneously reasoning about multiple categories of computer vision recognition tasks. It proposes a scheme that integrates multiple deep learning detection tasks into a single network. This invention can complete the analysis of multiple visual recognition tasks using a single neural network without increasing the number of networks or significantly increasing recognition time, and without compromising the accuracy of the recognition results.

[0068] Figure 1 This is a flowchart of an image analysis method based on a multi-branch network according to the present invention. The method can be implemented by an electronic device. The method is used in an image analysis system based on a multi-branch network and includes:

[0069] S101. Obtain multi-task data and label it.

[0070] In one feasible implementation, acquiring multi-task data and annotations includes:

[0071] The original image is labeled according to the labeling method of each task in the multi-task requirement;

[0072] The original images and multi-task labeled data are divided into training set, validation set and test set according to a preset ratio;

[0073] The annotation methods include: for image segmentation tasks, dividing the foreground and background of the original image to obtain the mask annotation of the original image; for object detection tasks, using bounding boxes to annotate the objects to be identified in the original image, and concentrating all the bounding boxes of a single original image into a single text; for key point detection tasks, using point annotation methods to annotate the key points to be identified in the original image, and concentrating all the point annotations of a single original image into a single text.

[0074] S102. Construct a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch.

[0075] In one feasible implementation, as in this embodiment, such as Figure 2 As shown, the overall structure of the deep learning-based multi-task recognition network with three types of branches consists of Backbone, Neck, and Head.

[0076] The backbone includes: a Focus Block that processes image channels; three CC3 Blocks that downsample and convolve features; a feature convolution pooling module CSC3 Block; and three skip connections that output to the Neck. Each Focus Block contains a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. After the input feature map, the convolutional layer uses a method called "straddle convolution" to transform the original feature map into a smaller, lower-resolution feature map. Then, the Batch Normalization layer and LeakyReLU activation function further process the feature map to extract important feature information. The CC3_Block mainly includes a downsampling convolutional module Conv_Block and a multiple convolutional module C3_Block. When CC3_Block is used outside the Backbone, a Concat module is added between the two modules mentioned earlier as an additional input for skip connections. The Conv_Block module includes a convolutional layer, a Batch Normalization layer, and a ReLU activation function. The C3_Block module consists of three convolutional layers and a residual block. Each convolutional layer contains a 3x3 convolutional kernel, a Batch Normalization layer, and a ReLU activation function. Its main function is to enhance the network's non-linearity and improve its feature representation capabilities, thereby enabling the network to detect and recognize targets more accurately. The CSC3_Block mainly includes a Conv_Block, an SPP_Block, and a C3_Block. The SPP_Block module is a feature pyramid pooling structure, which is used to extract and fuse feature information at different scales to improve the accuracy and robustness of target detection.

[0077] The Neck comprises: four CUC3_Block modules that perform bottom-up upsampling and skip-layer connections; two CC3_Block modules that perform top-down feature downsampling and accept skip-layer connections for feature fusion; three skip-layer connections that accept input from the Backbone; and one skip-layer connection for internal feature fusion. Each CUC3_Block module mainly includes a Conv_Block module, an Upsample_Block module, and a C3_Block module. When feature input from additional branches is required, a Concat module is added between the Upsample_Block and C3_Block modules for feature merging. The two CC3_Block modules need to accept skip-layer connections for feature fusion, therefore both include a Concat module.

[0078] The Header includes network heads for three tasks: image segmentation, keypoint detection, and object detection. The image segmentation task network head uses 1x1 convolutional layers, and the output result maintains the same spatial scale as the original image. The number of channels corresponds to the number of objects to be identified (including background) in the image segmentation. Specifically, it represents the prediction of each category at each pixel. After a Softmax operation, a result mask is obtained. The keypoint detection task network head uses 1x1 convolutional modules, and the output result has a spatial scale of 1 / 4 of the original image. The number of channels corresponds to the number of objects to be identified. It represents the probability of a target point existing at each location on a feature map that is 1 / 4 the size of the original image. After a Softmax operation in the spatial dimension, a normalized Heatmap is obtained. The heatmap image is then processed by Soft-argmax to obtain the coordinates of the key points to be identified. The network head of the object detection task has three output heads, each consisting of three 1x1 convolutional modules. In addition to the usual batch dimension, channel dimension, and spatial dimension, the output adds a dimension corresponding to the number of anchors in each layer, representing the prediction based on different anchors in that layer. The channel dimension is 5 plus the sum of the number of target categories to be identified, which means the prediction of the target bounding box X, Y, W, H, the prediction of whether there is a target at that location, and the prediction of the specific class of the target at that location. The spatial dimensions are 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. The number of channel dimensions is equal to the class number plus 5, and the number of anchor dimensions is 3 for each.

[0079] In this embodiment, a multi-task recognition network based on deep learning and employing three types of branch heads is constructed, as follows: Figure 1 As shown. The specific workflow of the multi-task branch network includes:

[0080] Convert the image using Letterbox so that the longest side is a multiple of 640 and the shortest side is a multiple of 32 (padded with 0s if necessary).

[0081] The transformed image is then fed into the Backbone network. Specifically, it is first fed into the Focus_Block_1 module to obtain feature F10 (Focus_Block_1 has a 3*3 kernel size, a downsampling factor of 2, and 3-64 input / output channels). Feature F10 is then fed into the CC3_Block_1 module to obtain feature F11 (in the CC3_Block_1 module, the Conv and C3 modules both have a 3*3 kernel size, strides of 2 and 1 respectively, and 64-128 and 128-128 input / output channels respectively). Feature F11 is then fed into the CC3_Block_2 module to obtain feature F12 (in the CC3_Block_2 module, the Conv and C3 modules both have a 3*3 kernel size, strides of 2 and 1 respectively, and 64-128 and 128-128 input / output channels respectively). The features F12 (128-256, 256-256) are input into the CC3_Block_3 module to obtain feature F13 (in the CC3_Block_3 module, the kernel size of the Conv module and the C3 module is 3*3, the stride is 2 and 1 respectively, and the input and output channels are 256-512 and 512-512 respectively). Feature F13 is then input into the CSC3_Block_4 module to obtain feature F14 (in the CSC3_Block_4 module, the kernel size of the Conv module and the C3 module is 3*3, the stride is 2 and 1 respectively, and the input and output channels of the Conv module, the Spatial Pyramid Pooling (SPP) module, and the C3 module are 512-1024, 1024-1024, and 1024-1024 respectively). Thus, the Backbone network outputs four feature maps to the Neck network: F11, F12, F13, and F14.

[0082] In the Neck network, feature F14 is renamed FN4. The specific operations for F11, F12, F13, and F14 are as follows: Feature FU4 (original feature F14) is input into the CUC3_Block_1* module, along with feature F13. The two are fused and upsampled in the CUC3_Block_1* module to obtain feature FU3 (a Concat module is inserted between the Upsample module and the C3 module in the CUC3_Block_1* module. The convolution kernel sizes of the Conv module and the C3 module are 1*1 and 3*3, respectively, with a stride of 1 for both, and input and output channels of 1024-1024 and 1024-512, respectively). Feature FU3 is then input into the CUC3_Block_1* module. The CUC3_Block_2* module takes feature F12 as input, and the two are fused and upsampled in the CUC3_Block_2* module to obtain feature FU2 (a Concat module is inserted between the Upsample module and the C3 module in the CUC3_Block_2* module; the convolution kernel sizes of the Conv module and the C3 module are 1*1 and 3*3 respectively, the stride is 1, and the input and output channels are 512-512 and 512-256 respectively); feature FU2 is input into the CUC3_Block_3* module, and feature F11 is input simultaneously. The two are fused and upsampled in the CUC3_Block_3* module to obtain feature FU1 (CUC3_Block_2* module). In the ck_3* module, a Concat module is inserted between the Upsample module and the C3 module. The convolution kernel sizes of the Conv module and the C3 module are 1*1 and 3*3, respectively, with a stride of 1, and input and output channels of 256-256 and 256-128, respectively. Feature FU1 is input into the CUC3_Block_4* module to obtain feature FU0 (in the CUC3_Block_4 module, the convolution kernel sizes of the Conv module and the C3 module are 1*1 and 3*3, respectively, with a stride of 1, and input and output channels of 128-128 and 128-64, respectively). Feature FU0 is input into the CUC3_Block_5* module to obtain feature FH0 (CUC3_Block_5* module). In the ock_5 module, the convolutional kernel sizes of the Conv module and the C3 module are 1*1 and 3*3, respectively, with a stride of 1, and input and output channels of 64-64 and 64-32, respectively. Feature FU2 is input into the CC_Block_4* module, and feature FU3 is input at the same time. The two are fused and downsampled in the CC3_Block_4* module to obtain feature FD4 (in the CC3_Block_4* module, a Concat module is inserted between the Conv_Block module and the C3_Block module. The convolutional kernel sizes of the Conv module and the C3 module are 3*3, with strides of 2 and 1, respectively, and input and output channels of 256-512 and 512-512, respectively).Feature FD4 is input into the CC_Block_4* module, along with feature FU4. These two features are then fused and downsampled in the CC3_Block_5* module to obtain feature FH4. (In the CC3_Block_5* module, a Concat module is inserted between the Conv_Block and C3_Block modules. The convolution kernel size of both the Conv and C3 modules is 3*3, with strides of 2 and 1 respectively, and input / output channels of 512-1024 and 1024-1024 respectively.) The Neck part provides the Head part with five feature maps: feature FH4, feature FH3 (original feature FD4), feature FH2 (original feature FU2), feature FH1 (original feature FU1), and feature FH0.

[0083] The Head section is divided into five branches, corresponding to three recognition tasks. The image segmentation task occupies one Segment Head branch, consisting of a Conv_Block_5 module (its own convolutional kernel size is 1*1, stride is 1, and input / output channels are 32-CLASS_NUM). The feature map FH0 of the Neck network is input to obtain the image segmentation result. The keypoint detection task occupies one KeyPoint Head branch, consisting of one Conv_Block_4 module (its own convolutional kernel size is 1*1, stride is 1, and input / output channels are 128-CLASS_NUM). The feature map FH1 of the Neck network is input to obtain the image keypoint detection result. The object detection task occupies three Detection Head branches. The Head branch consists of three Conv_Blocks: Conv_Block_3, Conv_Block_2, and Conv_Block_1 (all have a kernel size of 1*1, a stride of 1, and input and output channels of 256-3*(5+CLASS_NUM), 512-3*(5+CLASS_NUM), and 1024-3*(5+CLASS_NUM) respectively).

[0084] In this embodiment, the three-task branch network can fuse image feature information from different levels (high and low levels) according to the needs of different recognition tasks, thereby improving the network convergence speed and the performance of the corresponding recognition task.

[0085] In one feasible implementation, a multi-branch image recognition network model is constructed. This model includes: a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch.

[0086] The original image is input into the main branch of feature extraction to obtain the high-dimensional features corresponding to the original image.

[0087] Input the high-dimensional features into the keypoint extraction branch to obtain the coordinates of M keypoints. i∈[0,M-1];

[0088] High-dimensional features are input into the image segmentation branch to obtain an image segmentation confidence map corresponding to the original image size. The image segmentation confidence map has dimensions H, W, and C, where H represents the height, W represents the width, and C represents the number of categories.

[0089] The high-dimensional features are input into the target detection branch to obtain the position parameters of q targets to be detected. in These are the top-left corner coordinates, top-right corner coordinates, width, height, category, and confidence level of the target q to be detected, where q∈[0,Q-1].

[0090] In one feasible implementation, the main branch for image feature extraction includes one or more of the following networks: VGG, DenseNet, and ResNet.

[0091] In one feasible implementation, the main branch of feature extraction from the original image input is used to obtain the high-dimensional features corresponding to the original image, including:

[0092] The original image with size H×W was downsampled to three sizes: H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, resulting in three different scale feature maps.

[0093] Using variants of Feature Pyramid Network (FPN), Adaptive Feature Pyramid Network (PAN), or combinations thereof, feature maps of different scales are upsampled or downsampled to achieve the same size H / 8×W / 8.

[0094] By adding the values ​​at the same position in feature maps of the same size after upsampling or downsampling, or by merging them along the channel dimension, high-dimensional features can be obtained through feature fusion.

[0095] In one feasible implementation, the keypoint extraction branch needs to upsample the high-dimensional features of H / 8×W / 8 to a size of H / 4×W / 4, then perform a 1×1 convolution transformation to obtain a keypoint confidence map, and obtain the coordinates of M keypoints with the highest confidence based on the keypoint confidence map.

[0096] The image segmentation branch continuously upsamples the high-dimensional features of H / 8×W / 8 to the original image size H×W, and then performs a 1×1 convolution transformation to obtain the image segmentation confidence map.

[0097] The target detection branch transforms the high-dimensional features of H / 8×W / 8 through 1×1 convolution to obtain the category and location information of the target to be detected.

[0098] S103. Based on multi-task data and annotations, a multi-branch image recognition network model is trained using a morphological perception loss function to obtain the trained model.

[0099] In one feasible implementation, the training dataset is input into the multi-task branch network, and the fundus network is trained using a stochastic gradient descent optimizer until the validation set evaluation metric reaches convergence, thus obtaining the trained multi-task branch network. Here, a shape-aware loss function can be used to calculate the error of the multi-branch task.

[0100] In one feasible implementation, a multi-branch image recognition network model is trained using a morphology-aware loss function based on multi-task data and annotations to obtain the trained model, including:

[0101] Construct a shape-aware loss function for multi-branch tasks;

[0102] Based on multi-task data and annotations, a multi-branch image recognition network model is trained using the morphological perception loss function of multi-branch tasks.

[0103] In one feasible implementation, the form-aware loss function for multi-branch tasks includes:

[0104] The morphological perception loss function L for multi-branch tasks is constructed according to the following formula (1):

[0105] L = L seg +L obj +L point (1)

[0106] Where L seg L obj , and L point These are the loss functions for the image segmentation branch, the object detection branch, and the keypoint extraction branch, respectively. obj L is the sum of the IOU loss between the predicted coordinates and the true coordinates, and the BCE loss between the predicted confidence and the true label. point The BCE loss is used to predict the coordinates of keypoints and the actual keypoint locations.

[0107] In one feasible implementation, the definition of the image segmentation branch includes:

[0108] L is defined according to the following formula (2). seg :

[0109]

[0110] Where: N is the number of pixels in the image; l(x) is the true classification corresponding to pixel x. p represents the weight of the loss function corresponding to pixel x. l(x) (x) refers to the probability that pixel x belongs to its true category in the detection result, and Ω represents the pixel space.

[0111] In one feasible implementation, the definition of the image segmentation branch includes:

[0112] The weights consist of three weight loss terms: category loss term, class loss term, and class loss term. Morphological loss term Key point localization loss term w KPL (x), weighted graph as follows Figure 3 As shown, calculate according to the following formula (3). Weight:

[0113]

[0114] Where λ1, λ2, and λ3 are three weighted hyperparameters used to control the proportion of the three weight items; T is the category in which the target key point is located;

[0115] The category loss term is calculated according to the following formula (4):

[0116]

[0117] Where, num k C is the number of pixels in category k; C is the number of categories including the background; and i is the category weight channel number, which takes a value from the set {0, ..., C}.

[0118] The morphological loss term is calculated according to the following formula (5):

[0119]

[0120] Where d(x) is the nearest distance from pixel x to the edge of its foreground category l(x), x i Let l(x) be the mapping of pixel x to class weight channel i. i ) is x i The corresponding category; when i = 0, This represents the morphological loss value from the background pixel to the foreground region; when i≠0, This represents the shape loss value from the foreground pixel to its corresponding foreground region boundary;

[0121] The key point localization loss term is calculated according to the following formula (6):

[0122]

[0123] Where A is the location of the key target point in the image, and d A (x) represents the distance from pixel x in the foreground region to the target region A at the keypoint. The target distance loss w is calculated for each pixel x. KPL (x), forming the target distance loss term w KPL .

[0124] In one feasible implementation, by using L seg and L point The loss optimization for key point detection is incorporated into all models, which can improve the accuracy of key point recognition.

[0125] S104. Input the image to be analyzed into the trained model to obtain analysis data, complete image recognition based on a multi-branch network, and realize the output of multiple image task results by a single network.

[0126] This invention provides an image analysis method based on a multi-branch network. Addressing the shortcomings of existing technologies, it acquires multi-task data and annotations; constructs a multi-branch image recognition network model, which includes a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch; and trains the multi-branch image recognition network model using a morphological perception loss function. Compared to a two-stage network that performs object detection followed by image segmentation or keypoint recognition, this method offers faster computation speed. Specifically, it completes the analysis of multiple visual recognition tasks using a single neural network without increasing the number of networks or significantly increasing recognition time, and without compromising the accuracy of the recognition results.

[0127] like Figure 4 As shown, this embodiment of the invention provides an image analysis system 200 based on a multi-branch network, which can be implemented by an electronic device. Figure 4 The diagram shown illustrates an image analysis system 200 based on a multi-branch network. The system 200 includes:

[0128] Data acquisition block 210 is used to acquire multi-task data and annotations;

[0129] The model building module 220 is used to build a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch.

[0130] Model training module 230 is used to train the multi-branch image recognition network model based on the multi-task data and annotations using a morphological perception loss function to obtain the trained model.

[0131] The image analysis module 240 is used to input the image to be analyzed into the trained model, obtain analysis data, and complete image analysis based on a multi-branch network.

[0132] Preferably, the data acquisition block 210 is further used to annotate the original image according to the annotation method of each task in the multi-task requirement;

[0133] The original images and multi-task labeled data are divided into training set, validation set and test set according to a preset ratio;

[0134] The annotation methods include: for image segmentation tasks, dividing the foreground and background of the original image to obtain the mask annotation of the original image; for object detection tasks, using bounding boxes to annotate the objects to be identified in the original image, and concentrating all the bounding boxes of a single original image into a single text; for key point detection tasks, using point annotation methods to annotate the key points to be identified in the original image, and concentrating all the point annotations of a single original image into a single text.

[0135] Preferably, the model building module 220 is further used to extract the main branch of the original image input features to obtain the high-dimensional features corresponding to the original image;

[0136] Input the high-dimensional features into the keypoint extraction branch to obtain the coordinates of M keypoints. i∈[0,M-1];

[0137] High-dimensional features are input into the image segmentation branch to obtain an image segmentation confidence map corresponding to the original image size. The image segmentation confidence map has dimensions H, W, and C, where H represents the height, W represents the width, and C represents the number of categories.

[0138] The high-dimensional features are input into the target detection branch to obtain the position parameters of q targets to be detected. in These are the top-left corner coordinates, top-right corner coordinates, width, height, category, and confidence level of the target q to be detected, where q∈[0,Q-1].

[0139] Preferably, the main branch for image feature extraction includes one or more of the following networks: VGG, DenseNet, and ResNet.

[0140] Preferably, the main branch of feature extraction from the original image input is used to obtain the high-dimensional features corresponding to the original image, including:

[0141] The original image with size H×W was downsampled to three sizes: H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, resulting in three different scale feature maps.

[0142] Using variants of Feature Pyramid Network (FPN), Adaptive Feature Pyramid Network (PAN), or combinations thereof, feature maps of different scales are upsampled or downsampled to achieve the same size H / 8×W / 8.

[0143] By adding the values ​​at the same position in feature maps of the same size after upsampling or downsampling, or by merging them along the channel dimension, high-dimensional features can be obtained through feature fusion.

[0144] Preferably, the model training module 230 is further used to construct a morphological awareness loss function for multi-branch tasks;

[0145] Based on multi-task data and annotations, a multi-branch image recognition network model is trained using the morphological perception loss function of multi-branch tasks.

[0146] Preferably, the form-aware loss function for multi-branch tasks includes:

[0147] The morphological perception loss function L for multi-branch tasks is constructed according to the following formula (1):

[0148] L = L seg +L obj +L point (1)

[0149] Where L seg L obj , and L point These are the loss functions for the image segmentation branch, the object detection branch, and the keypoint extraction branch, respectively. obj L is the sum of the IOU loss between the predicted coordinates and the true coordinates, and the BCE loss between the predicted confidence and the true label. point The BCE loss is used to predict the coordinates of keypoints and the actual keypoint locations.

[0150] Preferably, the definition of the image segmentation branch includes:

[0151] L is defined according to the following formula (2). seg :

[0152]

[0153] Where: N is the number of pixels in the image; l(x) is the true classification corresponding to pixel x. p represents the weight of the loss function corresponding to pixel x. l(x) (x) refers to the probability that pixel x belongs to its true category in the detection result, and Ω represents the pixel space.

[0154] Preferably, the definition of the image segmentation branch includes:

[0155] The weights consist of three weight loss terms: category loss term, class loss term, and class loss term. Morphological loss term Key point localization loss term w KPL (x), calculated according to the following formula (3). Weight:

[0156]

[0157] Where λ1, λ2, and λ3 are three weighted hyperparameters used to control the proportion of the three weight items; T is the category in which the target key point is located;

[0158] The category loss term is calculated according to the following formula (4):

[0159]

[0160] Where, num k C is the number of pixels in category k; C is the number of categories including the background; and i is the category weight channel number, which takes a value from the set {0, ..., C}.

[0161] The morphological loss term is calculated according to the following formula (5):

[0162]

[0163] Where d(x) is the nearest distance from pixel x to the edge of its foreground category l(x), x i Let l(x) be the mapping of pixel x to class weight channel i. i ) is x i The corresponding category; when i = 0, This represents the morphological loss value from the background pixel to the foreground region; when i≠0, This represents the shape loss value from the foreground pixel to its corresponding foreground region boundary;

[0164] The key point localization loss term is calculated according to the following formula (6):

[0165]

[0166] Where A is the location of the key target point in the image, and d A (x) represents the distance from pixel x in the foreground region to the target region A at the keypoint. The target distance loss w is calculated for each pixel x. KPL (x), forming the target distance loss term w KPL .

[0167] This invention provides an image analysis method based on a multi-branch network. Addressing the shortcomings of existing technologies, it acquires multi-task data and annotations; constructs a multi-branch image recognition network model, which includes a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch; and trains the multi-branch image recognition network model using a morphological perception loss function. Compared to a two-stage network that performs object detection followed by image segmentation or keypoint recognition, this method offers faster computation speed. Specifically, it completes the analysis of multiple visual recognition tasks using a single neural network without increasing the number of networks or significantly increasing recognition time, and without compromising the accuracy of the recognition results.

[0168] Figure 5 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present invention. The electronic device 300 can vary considerably due to differences in configuration or performance. It may include one or more central processing units (CPUs) 301 and one or more memories 302. The memories 302 store at least one instruction, which is loaded and executed by the processors 301 to implement the steps of the following image analysis method based on multi-branch networks:

[0169] S1. Acquire multi-task data and label it;

[0170] S2. Construct a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch;

[0171] S3. Based on multi-task data and annotations, a multi-branch image recognition network model is trained using the morphological perception loss function to obtain the trained model.

[0172] S4. Input the image to be analyzed into the trained model to obtain analysis data and complete the image analysis based on the multi-branch network.

[0173] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the aforementioned image analysis method based on a multi-branch network. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0174] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

Claims

1. An image analysis method based on multi-branch networks, characterized in that, The method steps include: S1. Acquire multi-task data and label it; S2. Construct a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch; S3. Based on the multi-task data and annotations, train the multi-branch image recognition network model using the morphological perception loss function to obtain the trained model. S4. Input the image to be analyzed into the trained model to obtain analysis data and complete the image analysis based on the multi-branch network. In step S2, a multi-branch image recognition network model is constructed. This model includes: a main branch for feature extraction, a keypoint extraction branch, an image segmentation branch, and an object detection branch. The original image is input into the main branch of feature extraction to obtain the high-dimensional features corresponding to the original image; The high-dimensional features are input into the keypoint extraction branch to obtain the coordinates of M keypoints. The high-dimensional features are input into the image segmentation branch to obtain an image segmentation confidence map corresponding to the original image size. The image segmentation confidence map has dimensions H, W, and C, where H represents height, W represents width, and C represents the number of categories. The high-dimensional features are input into the target detection branch to obtain the position parameters of q targets to be detected. The above These are the top-left corner coordinates, top-right corner coordinates, width, height, category, and confidence level of the target q to be detected, where q∈[0,Q-1]. The definition of the image segmentation branch includes: L is defined according to the following formula (2). seg : Where: N is the number of pixels in the image; l(x) is the true classification corresponding to pixel x. p represents the weights of the loss function corresponding to pixel x. l(x) (x) refers to the probability that pixel x belongs to its true category in the detection result, and Ω represents the pixel space.

2. The method according to claim 1, characterized in that, In step S1, acquiring multi-task data and annotations includes: The original image is labeled according to the labeling method of each task in the multi-task requirement; The original images and multi-task labeled data are divided into training set, validation set and test set according to a preset ratio; The annotation methods include: for image segmentation tasks, dividing the foreground and background of the original image to obtain the mask annotation of the original image; for object detection tasks, using bounding boxes to annotate the objects to be identified in the original image, and concentrating all the bounding boxes of a single original image into a single text; for key point detection tasks, using point annotation methods to annotate the key points to be identified in the original image, and concentrating all the point annotations of a single original image into a single text.

3. The method according to claim 1, characterized in that, The main branch for image feature extraction includes one or more of the following networks: VGG, DenseNet, and ResNet.

4. The method according to claim 3, characterized in that, The step of extracting the main branch of the input feature from the original image to obtain the high-dimensional features corresponding to the original image includes: The original image with size H×W is downsampled to three sizes: H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, resulting in three different scale feature maps. Using variants of Feature Pyramid Network (FPN), Adaptive Feature Pyramid Network (PAN), or combinations thereof, feature maps of different scales are upsampled or downsampled to achieve the same size H / 8×W / 8. By adding the values ​​at the same position in feature maps of the same size after upsampling or downsampling, or by merging them along the channel dimension, high-dimensional features can be obtained through feature fusion.

5. The method according to claim 4, characterized in that, In step S3, based on the multi-task data and annotations, the multi-branch image recognition network model is trained using a morphology-aware loss function to obtain the trained model, including: Construct a shape-aware loss function for multi-branch tasks; Based on the multi-task data and annotations, the multi-branch image recognition network model is trained using the morphological perception loss function of the multi-branch task.

6. The method according to claim 5, characterized in that, The form-aware loss function for multi-branch tasks includes: The morphological perception loss function L for multi-branch tasks is constructed according to the following formula (1): L=L seg +L obj +L point (1) Where L seg L obj , and L point These are the loss functions for the image segmentation branch, the object detection branch, and the key point extraction branch, respectively; L obj L is the sum of the IOU loss between the predicted coordinates and the true coordinates, and the BCE loss between the predicted confidence and the true label. point BCEloss is used to predict the coordinates of keypoints and the actual locations of keypoints.

7. The method according to claim 1, characterized in that, The definition of the image segmentation branch includes: The weights consist of three weight loss terms: category loss term, class loss term, and class loss term. Morphological loss term Key point localization loss term w KPL (x), calculated according to the following formula (3). Weight: Where λ1, λ2, and λ3 are three weighted hyperparameters used to control the proportion of the three weight items; T is the category in which the target key point is located; The category loss term is calculated according to the following formula (4): Where, num k C is the number of pixels in category k; C is the number of categories including the background; and i is the category weight channel number, which takes a value from the set {0,…,C}. The morphological loss term is calculated according to the following formula (5): Where d(x) is the nearest distance from pixel x to the edge of its foreground category l(x), x i Let l(x) be the mapping of pixel x to class weight channel i. i ) is x i The corresponding category; when i = 0, This represents the morphological loss value from the background pixel to the foreground region; when i≠0, This represents the shape loss value from the foreground pixel to its corresponding foreground region boundary; The key point localization loss term is calculated according to the following formula (6): Where A is the location of the key target point in the image, and d A (x) represents the distance from pixel x in the foreground region to the target region A of the key point. The target distance loss w is calculated for each pixel x. KPL (x), forming the target distance loss term w KPL .

8. An image analysis system based on a multi-branch network, characterized in that, The system includes: The data acquisition block is used to acquire and label multi-task data. The model building module is used to build a multi-branch image recognition network model, which includes: a feature extraction main branch, a key point extraction branch, an image segmentation branch, and an object detection branch. The model training module is used to train the multi-branch image recognition network model based on the multi-task data and annotations using the morphological perception loss function, and obtain the trained model. The image analysis module is used to input the image to be analyzed into the trained model, obtain analysis data, and complete image analysis based on a multi-branch network. The model building module is used for: The original image is input into the main branch of feature extraction to obtain the high-dimensional features corresponding to the original image; The high-dimensional features are input into the keypoint extraction branch to obtain the coordinates of M keypoints. The high-dimensional features are input into the image segmentation branch to obtain an image segmentation confidence map corresponding to the original image size. The image segmentation confidence map has dimensions H, W, and C, where H represents height, W represents width, and C represents the number of categories. The high-dimensional features are input into the target detection branch to obtain the position parameters of q targets to be detected. The above These are the top-left corner coordinates, top-right corner coordinates, width, height, category, and confidence level of the target q to be detected, where q∈[0,Q-1]. The definition of the image segmentation branch includes: L is defined according to the following formula (2). seg : Where: N is the number of pixels in the image; l(x) is the true classification corresponding to pixel x. p represents the weights of the loss function corresponding to pixel x. l(x) (x) refers to the probability that pixel x belongs to its true category in the detection result, and Ω represents the pixel space.

Images