Segmentation network training and target segmentation methods and related equipment

Abstract

The present application discloses a training method for a segmentation network, a target segmentation method, an electronic device, and a computer-readable storage medium. The training method comprises: using a segmentation network to process a training image containing at least one target to obtain a first joint point topology map of the main target, where the main target is a target among at least one target; using a segmentation network to fuse the first joint point topology map and a common feature map to obtain a fused feature map; using a segmentation network to perform target segmentation on the fused feature map to obtain a first target segmentation result; obtaining a first loss based on the first target segmentation result; and adjusting the parameters of the segmentation network at least based on the first loss. In the above manner, the accuracy of the subsequent segmentation network for segmenting the main target in the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a segmentation network training method, a target segmentation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Target segmentation technology can be applied to scenarios such as behavior analysis and gait recognition. Target segmentation technology is used to segment the target area and background area in a single image or video. The existing target segmentation method is to extract the features of the image, perform target segmentation on the features of the image, and obtain the area where the target is located. When there is only a single target in the image, the existing target segmentation method has a good segmentation effect. However, when there are multiple targets in the image, the target may be occluded in the image, and then the main target in the image may be occluded, affecting the segmentation effect of the main target, and then affecting the subsequent behavior analysis, gait recognition and other processes. Summary of the invention

[0003] The present application provides a training method for a segmentation network, a target segmentation method, an electronic device and a computer-readable storage medium, which can improve the effect of segmenting the main target in an image.

[0004] In order to solve the above technical problems, a technical solution adopted by the present application is: to provide a training method for a segmentation network. The training method includes: using the segmentation network to process a training image containing at least one target to obtain a first joint point topology map of the main target, where the main target is a target among at least one target; using the segmentation network to fuse the first joint point topology map and the common feature map to obtain a fused feature map; using the segmentation network to perform target segmentation on the fused feature map to obtain a first target segmentation result; obtaining a first loss based on the first target segmentation result; and adjusting the parameters of the segmentation network based at least on the first loss.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide a target segmentation method, the method comprising: obtaining a target image; inputting the target image into a segmentation network to obtain a segmentation result of the main target in the target image; wherein the target segmentation network is trained based on the aforementioned training method.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium storing program instructions, which can implement the above method when executed.

[0008] Through the above method, the segmentation network of the present application does not directly perform target segmentation on the common feature map, but fuses it with the first joint feature map of the main target to obtain a fused feature map, and then performs target segmentation on the fused feature map. Since the fusion process enhances the semantic information of the part corresponding to the main target (joint feature map) in the common feature map, the final first target segmentation result of the main target is more accurate. Furthermore, based on the segmentation network trained by the training method of the present application, in the subsequent use process, even if there are multiple targets in the target image and there are occlusions, the segmentation network can accurately segment the main target. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of Embodiment 1 of the training method for segmentation network of the present application;

[0010] Figure 2 is a schematic diagram of the first segmentation result;

[0011] Figure 3 It is a flow chart of Embodiment 2 of the training method for segmentation network of the present application;

[0012] Figure 4 It is a schematic diagram of the first key point heat map of this application;

[0013] Figure 5 is a schematic diagram of one of the second joint point topology diagrams;

[0014] Figure 6 It is a schematic diagram of another second joint point topology;

[0015] Figure 7 yes Figure 3 The figure is a detailed flow chart of 113;

[0016] Figure 8 is a schematic diagram of the topological diagram of the first joint point;

[0017] Fig. 9 This is a schematic diagram of the structure of the segmented network of the present application;

[0018] Fig.10 It is a flowchart of Embodiment 3 of the training method for segmentation network of the present application;

[0019] Fig.11 is a schematic diagram of the second target segmentation result;

[0020] Fig.12 This is another schematic diagram of the structure of the segmented network of the present application;

[0021] Fig.13It is a flowchart of Embodiment 4 of the training method for segmentation network of the present application;

[0022] Fig.14 This is another structural diagram of the segmentation network of this application;

[0023] Fig.15 It is a flowchart of Embodiment 5 of the training method for segmentation network of the present application;

[0024] Fig.16 yes Fig.15 The specific process diagram of S41;

[0025] Fig.17 This is another structural diagram of the segmentation network of the present application;

[0026] Fig.18 It is a flowchart of an embodiment of the target segmentation method of the present application;

[0027] Fig.19 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0028] Fig. 20 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The terms "first", "second", and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first", "second", and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0032] Figure 1 is a flow chart of the first embodiment of the training method for segmenting the network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment may include:

[0033] S11: Using a segmentation network to process a training image containing at least one target, to obtain a first joint point topology map of the main target.

[0034] The main target is a target in at least one target. And the main target is the target that needs to be segmented from the training image by the segmentation network. The first joint point topology map of the main target is a topology map composed of the key points of the main target in the training image. The key points of the target mentioned in this article can include but are not limited to the key points of the target's head, neck, left and right elbows, left and right shoulders, left and right hips, left and right knees, left and right ankles, left and right soles, etc.

[0035] There are multiple training images. The target involved in this article can be a living body, such as a human, various animals (including but not limited to pigs, cows, sheep, chickens, cats or dogs, etc.), and the following article only uses the example of a human as the target for explanation.

[0036] The size of the image fed into the segmentation network is fixed. Therefore, as an embodiment, before feeding the training image into the segmentation network, the training image may be scaled to unify the scale. Alternatively, in order to speed up the training, the training image may be normalized before feeding the training image into the segmentation network.

[0037] S12: Using the segmentation network, the first joint topology map and the common feature map of the training image are fused to obtain a fused feature map.

[0038] The fusion of the first joint point topology map and the common feature map can enhance the semantic information of the part of the common feature map corresponding to the first joint point topology map.

[0039] S13: Use the segmentation network to perform target segmentation on the fused feature map to obtain a first target segmentation result.

[0040] The first segmentation result is the area corresponding to the main target in the training image. Figure 2 is a schematic diagram of the first segmentation result.

[0041] S14: Obtain a first loss based on the first target segmentation result.

[0042] The first loss can be, but is not limited to, the mIoU indicator. Taking the mIoU indicator as an example, the first loss constructed can be as follows:

[0043]

[0044] Among them, the training image includes N pixels, q j represents the segmentation result of the jth pixel in the training image, p j Represents the label of the j-th pixel in the training image.

[0045] S15: Adjust parameters of the segmentation network based at least on the first loss.

[0046] Through the implementation of this embodiment, the segmentation network of the present application does not directly perform target segmentation on the common feature map, but fuses it with the first joint feature map of the main target to obtain a fused feature map, and then performs target segmentation on the fused feature map. Since the fusion process enhances the semantic information of the part corresponding to the main target (joint feature map) in the common feature map, the final first target segmentation result of the main target is more accurate. Furthermore, based on the segmentation network trained by the training method of the present application, in the subsequent use process, even if there are multiple targets in the target image and there are occlusions, the segmentation network can accurately segment the main target.

[0047] Figure 3 is a flow chart of the second embodiment of the training method for segmenting the network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 3 The process sequence shown is limited. This embodiment is a further extension of the above S11. Figure 3 As shown, this embodiment may include:

[0048] S111: Use the segmentation network to extract features from the training image to obtain a common feature map.

[0049] S112: Use a segmentation network to extract key points from the common feature map to obtain a first key point heat map, and generate a second joint point topology map of at least one target based on the first key point heat map.

[0050] The first key point heat map includes the key points of at least one target (all targets) in the training image, and the key points can be connected to form a second joint point topology map. Ideally, in the connected second joint point topology map, the key points belonging to the same target should be connected to form a sub-joint point topology map of the corresponding target, and the connection relationship between the key points in the sub-joint point topology map of the target is correct, that is, the same as the actual connection relationship of the corresponding part.

[0051] For example, the training image includes two targets, and the second joint point topology map includes sub-joint point topology maps of the two targets. The key points belonging to the same are connected to form the sub-joint point topology map of the corresponding target. In addition, the key points of the head belonging to the same target are connected to the key points of the neck; the key points of the neck are connected to the key points of the left and right shoulders respectively; the key points of the left shoulder are connected to the key points of the left elbow, and the key points of the right shoulder are connected to the key points of the right elbow; the key points of the left shoulder are connected to the key points of the left hip, and the key points of the right shoulder are connected to the key points of the right hip; the key points of the left hip are connected to the key points of the left knee, and the key points of the right hip are connected to the key points of the right knee; the key points of the left knee are connected to the key points of the left ankle, and the key points of the right knee are connected to the key points of the right ankle; the key points of the left ankle are connected to the key points of the left sole, and the key points of the right ankle are connected to the key points of the right sole.

[0052] Since different key points are connected based on the distance between each key point, the second joint point topology map is generated. Therefore, in actual situations, if the training image includes at least two targets, there may be connection errors, that is, the sub-joint point topology map that constitutes one target is mixed with key points belonging to another target, or in other words, the key points belonging to one target are in the sub-joint point topology map of another target. Therefore, when the training image includes at least two targets, there may be multiple second joint point topology maps generated according to the first key point heat map.

[0053] For example, Figure 4 is a schematic diagram of the first key point heat map of this application, including target 1 (left side) and target 2 (right side). Figure 4 There are two types of second joint point topology maps generated by the first key point heat map shown. Figure 5 This is a schematic diagram of one of the second joint topology diagrams. Figure 6 It is a schematic diagram of another second joint point topology diagram. Figure 5 In the figure, A is the sub-joint point topology map of target 1, and B is the sub-joint point topology map of target 2. All key points of target 1 are included in A, and the connection relationship between the key points in A is correct; all key points of target 2 are included in B, and the connection relationship between the key points in B is correct. Figure 6 A' is the sub-joint topology map of target 1, and B' is the sub-joint topology map of target 2. Some key points of target 1 (such as the key points of the right shoulder and the key points of the right elbow) are included in B', and some key points of target 2 (such as the key points of the right shoulder and the key points of the right elbow) are included in A', and there are errors in the connection relationship of some key points in A' and B'.

[0054] S113: Using the segmentation network to process the second joint point topology map to obtain the first joint point topology map.

[0055] The main target can be regarded as a target whose corresponding key point ratio is greater than a ratio threshold, and / or the main target can be regarded as a target whose corresponding key point distribution has a similarity greater than a similarity threshold with a preset distribution. The number of main targets may be one or more. Thus, the first joint point topology map can be obtained based on the ratio and / or distribution of key points of the same target in the joint point topology map of at least one target.

[0056] Among them, the distribution of the key points of the target can be reflected as the distance relationship between the key points of the target in the actual situation, and the preset distribution can be reflected as the distance relationship between the key points in the ideal situation. Therefore, the difference between the distribution of the key points of the target and the preset distribution can be reflected as the difference between the distance relationship between the key points in the actual situation and the ideal situation. The smaller the difference in distance relationship, the lower the probability of error in the connection relationship between the key points and the higher the similarity. For example, ideally, the distance between the head and the left shoulder and the head and the right shoulder should be similar / equal. If in actual situation, the key point of the head and the key point of the left shoulder are the first distance, and the key point of the head and the key point of the right shoulder are the second distance, and the difference between the first distance and the second distance is greater than the preset distance threshold, then it means that the distance relationship between the head and the left and right shoulders in the actual situation and the ideal situation is different.

[0057] Combined with reference Figure 7 , when the first joint point topology map is obtained based on the proportion and distribution of key points, S113 can be expanded into the following sub-steps:

[0058] S1131: Separate a first joint point topology map from a joint point topology map of at least one target using a segmentation network based on the proportion and layout of key points of the same target in the second joint point topology map.

[0059] Among them, the process of separating the first joint point topology map from the joint point topology map of at least one target can be called key point topology optimization. Figure 8 It is a schematic diagram of the topology diagram of the first joint point.

[0060] S1132: Use the segmentation network to perform feature extraction on the initial first joint point topology map to obtain a first feature extraction result, and use the first feature extraction result as a new first joint point topology map.

[0061] The first joint topology map has a different dimension from the common feature map. In order to unify the dimension with the common feature map and retain the semantic information therein, a deep feature extraction may be performed on it to obtain a first feature extraction result, and the first feature extraction result is used to update the first joint topology map.

[0062] The following is combined with Fig. 9The structure of the segmentation network shown is used to illustrate the training method provided in the above embodiment.

[0063] Example 1: Fig. 9 This is a schematic diagram of the network structure of this application. Fig. 9 As shown, the segmentation network includes a feature extraction module, a key point extraction module, an object segmentation module, a key point topology optimization module and a feature fusion module.

[0064] 1) Feed the training image into the segmentation network.

[0065] 2) The feature extraction module extracts features from the training image to obtain a common feature map.

[0066] 3) The key point extraction module extracts key points from the common feature map to obtain a first key point heat map, and generates a second joint point topology map of at least one target based on the first key point heat map.

[0067] 4) The key point topology optimization module separates the first joint point topology map from the second joint point topology map.

[0068] 5) The feature extraction module performs feature extraction on the first joint point topology map to obtain a first feature extraction result, and uses the first feature extraction result as a new first joint point topology map.

[0069] 6) Use the feature fusion module to fuse the first joint topology map and the common feature map to obtain a fused feature map.

[0070] 7) Use the target segmentation module to perform target segmentation on the fused feature map to obtain the first target segmentation result.

[0071] 8) Constructing a first loss based on the first target segmentation result to adjust the parameters of each module of the segmentation network.

[0072] The feature extraction modules involved in 2) and 5) in the above example 1 may be the same or different. In the case of the sameness, the parameters of the segmentation network are relatively fewer; in the case of differentness, the processing accuracy of the segmentation network is relatively higher.

[0073] In addition, in order to further improve the training effect, the above embodiment can be further expanded into embodiment 3. The specific implementation can be as follows:

[0074] Fig.10 is a flow chart of the third embodiment of the training method for segmenting the network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Fig.10 The process sequence shown is limited. S21-S23 are steps that can be included before S12, and S24 is a further extension of S12. The same parts of this embodiment as the previous embodiment are not repeated. Fig.10 As shown, this embodiment may include:

[0075] S21: Use the segmentation network to perform target segmentation on the common feature map to obtain a second target segmentation result.

[0076] The second target segmentation result is the area corresponding to all targets in the training image. Fig.11 It is a schematic diagram of the second target segmentation result.

[0077] S22: Use the segmentation network to splice the second target segmentation result and the first joint point topology map to obtain a spliced ​​feature map.

[0078] S23: Use the segmentation network to perform feature extraction on the initial splicing feature map to obtain a second feature extraction result, and use the second feature extraction result as a new splicing feature map.

[0079] The dimensions of the spliced ​​feature map obtained in S22 are different from those of the common feature map. In order to unify the dimensions of the spliced ​​feature map and the common feature map while retaining the semantic information in the spliced ​​feature map, a deep feature extraction may be performed on the spliced ​​feature map to obtain a second feature extraction result, and the second feature extraction result is used to update the spliced ​​feature map.

[0080] S24: Use the segmentation network to fuse the concatenated feature map and the common feature map to obtain a fused feature map.

[0081] Combine the following Fig.12 The segmented network structure shown is used as an example to illustrate the third embodiment.

[0082] Example 2:

[0083] 1) Feed the training image into the segmentation network.

[0084] 2) The feature extraction module extracts features from the training image to obtain a common feature map.

[0085] 3) The key point extraction module extracts key points from the common feature map to obtain a first key point heat map, and generates a second joint point topology map of at least one target based on the first key point heat map; the key point topology optimization module separates the first joint point topology map from the second joint point topology map; the feature extraction module extracts features from the first joint point topology map to obtain a first feature extraction result, and uses the first feature extraction result as a new first joint point topology map; and 4): using the target segmentation module to perform target segmentation on the common feature map to obtain a second target segmentation result.

[0086] 5) Using the feature extraction module, extract features from the splicing result (splicing feature map) of the first joint point topology map and the second segmentation result to obtain a second feature extraction result, and use the second feature extraction result as a new splicing feature map.

[0087] 6) Use the feature fusion module to fuse the spliced ​​feature map and the common feature map to obtain a fused feature map.

[0088] 7) Use the target segmentation module to perform target segmentation on the fused feature map to obtain the first target segmentation result.

[0089] 8) Constructing a first loss based on the first target segmentation result to adjust the parameters of each module of the segmentation network.

[0090] The target segmentation modules in 4) and 7) of the above example 2 can be the same or different. In the case of the same, the parameters of the segmentation network are relatively fewer; in the case of different, the processing accuracy of the segmentation network is relatively higher.

[0091] Through the implementation of this embodiment, the training effect of the segmentation network can be further improved, so that the target segmentation results obtained in the subsequent application process are more accurate.

[0092] In addition, in order to further improve the training effect of the segmentation network, before S15, the segmentation network can be used to extract key points from the fused feature map to obtain a second key point heat map, and a third joint point topology map of the main target can be generated based on the second key point heat map, and the third joint point topology map can be applied to the subsequent embodiments.

[0093] Therefore, the fourth embodiment obtained by expanding on the basis of the above embodiment can be specifically as follows:

[0094] Fig.13 is a flow chart of the fourth embodiment of the training method for segmenting the network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Fig.13 The process sequence shown is limited. In this embodiment, S31 is a step that can be included before S15, and S33 is a further extension of S15. Fig.13 As shown, this embodiment may include:

[0095] S31: Construct a second loss based on the third joint point topology map.

[0096] The second loss includes but is not limited to the OKS indicator. Taking the OKS indicator as an example, the constructed second loss can be as follows:

[0097]

[0098] Among them, d pirepresents the Euclidean distance between the main object p and the i-th key point of the predicted main object p, S p represents the scale factor of p, σ i represents the normalization factor of the i-th key point, v pi Indicates whether the i-th key point of p is visible, and δ is a function that selects visible key points for calculation.

[0099] S32: Adjust parameters of the segmentation network based on the first loss and the second loss.

[0100] The loss used to adjust the parameters of the segmentation network can be expressed as l loss = l oks +l mask .

[0101] If this embodiment is an extension of the above-mentioned embodiment 3, the structure of the segmentation network can be as follows: Fig.14 As shown. That is, relative Fig.12 , a key point extraction module is connected after the feature fusion module. The feature extraction module used to obtain the second key point heat map can be the same as or different from the feature extraction module used to obtain the first key point heat map. In the case of the same, the parameters of the segmentation network are relatively fewer; in the case of different, the processing accuracy of the segmentation network is relatively higher.

[0102] The fifth embodiment expanded on the basis of the above embodiment can be specifically as follows:

[0103] Fig.15 is a flow chart of the fifth embodiment of the training method for segmenting the network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Fig.15 The process sequence shown is limited. S41 is a step that can be included before S15, and S42 is a further extension of S15. Fig.15 As shown, this embodiment may include:

[0104] S41: Construct a third loss based on the relationship between the third joint point topology map and the first target segmentation result.

[0105] It is understandable that the third joint point topology obtained by the key point extraction task in actual situations Figure 1The first target segmentation result must be included in the first target segmentation result, and the first target segmentation result obtained by the target segmentation task must be enclosed outside the third joint point topology map. That is, there is coupling between the key point extraction and the target segmentation tasks, or in other words, there is a correlation between the third joint point topology map and the first target segmentation result. Therefore, during the training process, the key point extraction and target segmentation tasks can supervise and promote each other. Therefore, this embodiment constructs a third loss to train the segmentation network based on the relationship between the third joint point topology map and the first target segmentation result, which can promote mutual learning between the key point extraction and target segmentation tasks in the segmentation network, and further improve the training accuracy of the segmentation network.

[0106] Combined with reference Fig.16 In one implementation, S41 may include the following sub-steps:

[0107] S411: Determine a neighborhood corresponding to each key point in the third joint point topology graph in the first target segmentation result.

[0108] One implementation method of this step may be: determining the key points of reference, the key points of reference including the key points in the joint point topology map that are closest to each pixel point in the first target segmentation result; determining the pixel point in the first target segmentation result that is closest to the same reference key point as the pixel point within the neighborhood of the same reference key point.

[0109] That is, each keypoint is determined as a reference keypoint, and for each keypoint, its neighborhood consists of the pixels that take it as the closest keypoint.

[0110] S412: Construct a third loss based on the distribution of pixels in the neighborhood of each key point.

[0111] The third loss constructed in this step can be as follows:

[0112]

[0113] Where N represents the number of pixels, d 0j represents the distance between the jth pixel and the 0th key point, min{d 0j , d 1j ...d ij} represents the distance between the jth pixel and its nearest key point n, d pos(n) represents the set of distances between key point n and each pixel in its neighborhood, max{d pos(n)} represents the maximum distance in the set.

[0114] The l constructed in the above way like It can be used to measure the concentration of pixels in each neighborhood, based on llike The segmentation network is trained in a direction where the concentration is sufficiently low.

[0115] S42: Adjust parameters of the segmentation network based on the first loss and the third loss.

[0116] Therefore, the loss function based on which the parameters of the segmentation network are adjusted can be expressed as l loss = l like +l mask .

[0117] The different embodiments of the above training method can be combined with each other if they do not conflict with each other. For example, by combining the above embodiment 4 and embodiment 5, the loss function for training the segmentation network can be obtained as l loss = l like +l oks +l mask In this case, the training effect of the segmentation network can be further improved.

[0118] Combine the following Fig.17 , a specific description is given of the training method provided in this application.

[0119] Example 3:

[0120] 1) The input image (training image) is sent to the segmentation network, and the feature extraction module is used to extract the common features of the input image.

[0121] 2) Using the key point extraction module and the key point topology optimization module in turn to process the common feature map to obtain a first joint point topology map, and using the portrait segmentation module to process the common feature map to obtain a second target segmentation result.

[0122] 3) Splicing the first joint point topology map and the second target segmentation result, and performing feature extraction on the splicing result to obtain a splicing feature map.

[0123] 4) Fuse the spliced ​​feature map and the common feature map.

[0124] 5) The fusion results are processed using the key point extraction module and the portrait segmentation module respectively to obtain the key point extraction results and the portrait segmentation results.

[0125] 6) Construct the first, second and third losses based on the key point extraction results and portrait segmentation results to train the segmentation network.

[0126] The segmentation network trained by any of the above embodiments can be applied to the segmentation of the target image.

[0127] Fig.18FIG. 1 is a flow chart of an embodiment of the target segmentation method of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Fig.18 The process sequence shown is limited. Fig.18 As shown, this embodiment may include:

[0128] S51: Acquire a target image.

[0129] S52: Input the target image into the segmentation network to obtain the segmentation result of the main target in the target image.

[0130] The structure of the segmentation network in the application phase corresponds to that in the training phase. In addition, since the segmentation network obtained through training tends to extract the joint point topology map of the main target in the common feature map during the application process, even if the key point topology optimization module included in the segmentation network is not used, the interference of other targets can be eliminated to obtain the segmentation result of the main target. Therefore, the key point topology optimization module can be omitted in the application phase (i.e., the step of separating the first joint point topology map of the main target from the second joint point topology map of at least one target is omitted), thereby reducing the parameters of the segmentation network in the application phase.

[0131] In addition, in other embodiments, a key point topology optimization module may also be used in the application stage to improve the accuracy of the segmentation result of the main target.

[0132] In addition, since the second target segmentation result in the training phase is used to construct the third loss, the step of using the key point extraction module to perform target segmentation on the common feature map to obtain the second target segmentation result is omitted in the application phase.

[0133] For other detailed descriptions of this embodiment, please refer to the previous embodiments, which will not be repeated here.

[0134] In the above manner, the segmentation network can be used to segment the main target in the target image to obtain the segmentation result of the main target.

[0135] Fig.19 Schematic diagram of the structure of an embodiment of the electronic device of the present application. Fig.19 As shown, the electronic device may include a processor 61 and a memory 62 coupled to the processor 61 .

[0136] The memory 62 stores program instructions for implementing the method of any of the above embodiments; the processor 61 is used to execute the program instructions stored in the memory 62 to implement the steps of the above method embodiments. The processor 61 may also be called a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor 61 may also be any conventional processor, etc.

[0137] Fig. 20 Schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application. Fig. 20 As shown, the computer-readable storage medium 70 of the embodiment of the present application stores program instructions 71, and the program instructions 71 are executed to implement the method provided in the above embodiment of the present application. Among them, the program instructions 71 can form a program file and be stored in the above computer-readable storage medium 70 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 70 includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0139] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A training method for a segmentation network, characterized in that: include: Processing a training image containing at least one target using a segmentation network to obtain a first joint point topology map of a main target, wherein the main target is a target among the at least one target; Using the segmentation network to fuse the first joint point topology map and the common feature map of the training image to obtain a fused feature map; Using the segmentation network to perform target segmentation on the fused feature map to obtain a first target segmentation result; Obtaining a first loss based on the first target segmentation result; adjusting parameters of the segmentation network based at least on the first loss; The method of using a segmentation network to process a training image containing at least one target to obtain a first joint point topology map of the main target includes: Extracting features from the training image using a segmentation network to obtain the common feature map; Extracting key points from the common feature map using the segmentation network to obtain a first key point heat map, and generating a second joint point topology map of the at least one target based on the first key point heat map; Separating the first joint point topology map from the second joint point topology map using the segmentation network based on the proportion and / or distribution of key points of the same target in the second joint point topology map; The segmentation network is used to perform feature extraction on the first joint point topology map to obtain a first feature extraction result, and the first feature extraction result is used as a new first joint point topology map.

2. The method according to claim 1, characterized in that The method further comprises: The segmentation network is used to extract key points from the fused feature map to obtain a second key point heat map, and a third joint point topology map of the main target is generated based on the second key point heat map.

3. The method according to claim 2, characterized in that The method further comprises: Constructing a second loss based on the third joint point topology graph; The adjusting the parameters of the segmentation network based at least on the first loss comprises: Parameters of the segmentation network are adjusted based on the first loss and the second loss.

4. The method according to claim 2, characterized in that: The method further comprises: Constructing a third loss based on the relationship between the third joint point topology map and the first target segmentation result; The adjusting the parameters of the segmentation network based at least on the first loss comprises: Parameters of the segmentation network are adjusted based on the first loss and the third loss.

5. The method according to claim 4, characterized in that The constructing a third loss based on the relationship between the third joint point topology map and the first target segmentation result includes: Determine a neighborhood corresponding to each key point in the third joint point topology graph in the first target segmentation result; Based on the distribution of pixels in the neighborhood of each key point, a third loss is constructed.

6. The method according to claim 5, characterized in that The determining of a neighborhood corresponding to each key point in the third joint point topology graph in the first target segmentation result includes: Determine a reference key point, wherein the reference key point includes a key point in the third joint point topology map that is closest to each pixel point in the first target segmentation result; The pixel point in the first target segmentation result that is closest to the key point of the same reference is determined as the pixel point in the neighborhood of the key point of the same reference.

7. The method according to claim 5, characterized in that The formula for constructing the third loss based on the distribution of pixels in the neighborhood of each key point is as follows: Wherein, N represents the number of pixels, d 0j represents the distance between the jth pixel and the 0th key point, min{d 0j ,d 1j …d ij } represents the distance between the jth pixel and its nearest key point n, d pos(n) represents the set of distances between the key point n and each pixel in its neighborhood, max{d pos(n) } represents the maximum distance in the set.

8. The method according to claim 1, characterized in that Before the first joint point topology map and the common feature map are fused by the segmentation network to obtain a fused feature map, the method further includes: Using the segmentation network to perform target segmentation on the common feature map to obtain a second target segmentation result; Using the segmentation network, the second target segmentation result and the first joint point topology map are spliced ​​to obtain a spliced ​​feature map; Using the segmentation network to perform feature extraction on the initial splicing feature map to obtain a second feature extraction result, and using the second feature extraction result as a new splicing feature map; The step of fusing the first joint point topology map and the common feature map using the segmentation network to obtain a fused feature map includes: The segmentation network is used to fuse the concatenated feature map and the common feature map to obtain the fused feature map.

9. A target segmentation method, characterized in that: include: Get the target image; Inputting the target image into a segmentation network to obtain a segmentation result of a main target in the target image; Wherein, the target segmentation network is trained based on the method described in any one of claims 1-8.

10. An electronic device, characterized in that: The invention comprises a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed, the method according to any one of claims 1 to 9 is implemented.

Images