Multi-person resolution method and device based on instance decoupling and multi-feature point fusion

Abstract

The present invention provides a multi-person analysis method and device based on instance decoupling and multi-feature point fusion, the method comprising: obtaining an image to be tested, inputting the image to be tested into a trained feature pyramid network to obtain multiple feature maps, connecting the obtained multiple feature maps to obtain a feature total map; inputting the feature total map into a trained multi-feature fusion model, extracting features from the instance center based on the multi-feature fusion model to obtain a central feature, extracting features from the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature, and fusing the central feature and the offset feature to obtain each instance feature; inputting the feature total map and the instance features into a trained global decoupling model, obtaining an instance perception map based on the global decoupling model, and obtaining a heat map of each instance based on the instance perception map. The present invention can improve the accuracy of multi-person analysis results.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-person analysis method and device based on instance decoupling and multi-feature point fusion. Background Art

[0002] Multi-person parsing not only needs to distinguish the partial instance labels of each pixel, but also needs to determine the subordinate relationship between human body parts and human body instances; there are three main existing multi-person parsing methods, including top-down multi-person parsing method, bottom-up multi-person parsing method and single-stage multi-person parsing method. The top-down multi-person parsing method is to crop different spatial regions according to the bounding box, separate the people in the image, and then perform single-person parsing, using a fully convolutional network to classify and segment the target pixel by pixel, and each segmentation result is output as a different instance; in addition, the detector and personnel analysis module can be jointly optimized to obtain better human parsing results.

[0003] The bottom-up approach first identifies instance-independent clues, such as key point locations or segmentation masks, and then uses grouping operations to combine these clues into instance-aware results. The grouping operation is a key step in the bottom-up approach. For segmentation tasks, grouping is achieved by combining other contextual information such as edges. At present, this method can be applied to design an additional instance-aware edge detection branch based on PGN for grouping, and then use row decoding to generate instance-aware masks for separating partial segmentation results.

[0004] Unlike top-down and bottom-up methods, the one-stage method tends to directly regress instance-level results in one stage. It mainly relies on the supervision of instance mask annotations and can directly segment instances without bbox detection or additional pixel pairing relationships. This method integrates object detection or instance segmentation tasks into a single model, reducing complexity and computational cost; compared with the traditional two-stage method, it simplifies the process.

[0005] Although the currently commonly used top-down multi-person analysis method, bottom-up multi-person analysis method and single-stage multi-person analysis method can achieve multi-person analysis, they have the disadvantage of low accuracy of analysis results. Therefore, how to improve the accuracy of multi-person analysis results is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a multi-person resolution method and device based on instance decoupling and multi-feature point fusion to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a multi-person resolution method based on instance decoupling and multi-feature point fusion, the method comprising the following steps:

[0008] Acquire an image to be tested, input the image to be tested into a trained feature pyramid network to obtain multiple feature maps, and connect the multiple feature maps to obtain a feature overall map;

[0009] The feature map is input into a trained multi-feature fusion model, based on the multi-feature fusion model, feature extraction is performed on the instance center to obtain a center feature, feature extraction is performed on the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature, and feature fusion is performed on the center feature and the offset feature to obtain each instance feature;

[0010] The feature total map and the instance features are input into a trained global decoupling model, an instance perception map is obtained based on the global decoupling model, and a heat map of each instance is obtained based on the instance perception map.

[0011] In some embodiments of the present invention, the acquired multiple feature graphs are connected to obtain a feature overall graph, including:

[0012] Resize multiple feature maps to the same size based on bilinear interpolation;

[0013] Connect multiple feature maps of the same size to obtain the total feature map.

[0014] In some embodiments of the present invention, based on the multi-feature fusion model, feature extraction is performed on the instance center to obtain a center feature, and feature extraction is performed on the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature, including:

[0015] Determine the instance center coordinates based on the multi-feature fusion model, and obtain the central features of each instance based on the instance center coordinates;

[0016] The position coordinates of the center of each part of each instance are determined based on the multi-feature fusion model, the offset of each part is determined based on the position coordinates of the center of each part of each instance and the instance center coordinates, and the offset feature is obtained based on the offset of each part.

[0017] In some embodiments of the present invention, the central feature and the offset feature are subjected to feature fusion to obtain each instance feature, including:

[0018] Performing average pooling on the offset features;

[0019] Based on the residual network, the center feature and the offset feature after average pooling are fused to obtain each instance feature.

[0020] In some embodiments of the present invention, the feature total map and the instance features are input into a trained global decoupling model, an instance perception map is obtained based on the global decoupling model, and a heat map of each instance is obtained based on the instance perception map, including:

[0021] Based on the mapping relationship between each instance feature and the feature master map, spatially recalibrate each instance feature to obtain a first feature perception map;

[0022] Based on the mapping relationship between each instance feature and the feature overall map, recalibrate the channel of each instance feature to obtain a second feature perception map;

[0023] The first feature perception map and the second feature perception map are fused to obtain an instance perception map.

[0024] In some embodiments of the present invention, the method further comprises:

[0025] Construct an initial multi-feature fusion model and an initial global decoupling model;

[0026] Pre-training the initial multi-feature fusion model to obtain a trained multi-feature fusion model;

[0027] The initial global decoupling model is pre-trained to obtain a trained global decoupling model.

[0028] In some embodiments of the present invention, pre-training the initial multi-feature fusion model to obtain a trained multi-feature fusion model includes: constructing first sample data and a first loss function, and pre-training the initial multi-feature fusion model based on the first sample data and the first loss function to obtain a trained multi-feature fusion model; and / or

[0029] Pre-training an initial global decoupling model to obtain a trained global decoupling model includes: constructing second sample data and a second loss function, and pre-training the initial global decoupling model based on the second sample data and the second loss function to obtain a trained global decoupling model.

[0030] In some embodiments of the present invention, the first sample data includes a predicted instance center and an actual instance center, a predicted instance part center and an actual instance part center, a predicted part offset and an actual part offset, and the first loss function includes Offset i,j,p =(c px ―i,c py ―j), where Offset i,j,p represents the offset loss function, (c px , c py ) means c pThe position coordinates of the part center, (i, j) represents the grid coordinates on the image, i∈[0,S-1], j∈[0,S-1], S represents the size of the feature map after resizing multiple feature maps by bilinear interpolation;

[0031] The second sample data includes predicted instance mask information and actual instance mask information. The second loss function includes Dice loss and L1 correlation loss. The Dice loss is The L1 correlation loss is Among them, T (i) Represents the predicted instance mask information, T (i)* represents the actual instance mask information, N represents the total number of instance parts, and diff i Represents the difference between the prediction tensor and the target tensor.

[0032] According to another aspect of the present invention, a multi-person analysis system based on instance decoupling and multi-feature point fusion is also disclosed, including a processor, a memory, and a computer program stored in the memory, wherein the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0033] According to another aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0034] The multi-person analysis method and device based on instance decoupling and multi-feature point fusion of the present invention extracts central features and offset features based on a multi-feature fusion model, that is, the multi-person analysis method uses instance decoupling combined with central points and offsets based on central points to understand the possibility of the human body, thereby performing multi-person analysis, realizing an end-to-end multi-person analysis framework, eliminating the time-consuming grouping process, improving the analysis efficiency during the multi-person analysis process, and improving the accuracy of the multi-person analysis results.

[0035] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0036] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0038] Figure 1 The figure is a flowchart of a multi-person resolution method based on instance decoupling and multi-feature point fusion in one embodiment of the present invention.

[0039] Figure 2 It is a flowchart of a multi-person resolution method based on instance decoupling and multi-feature point fusion in another embodiment of the present invention.

[0040] Figure 3 The figure is a flowchart of a multi-feature fusion process of a multi-person analysis method in one embodiment of the present invention.

[0041] Figure 4 The figure is a flowchart of the global decoupling process of the multi-person resolution method in one embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0043] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0044] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0045] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0046] The bottom-up method commonly used in multi-person parsing methods usually segments or semantically segments the human body parts in the image, and then combines the parts belonging to the same human instance to finally parse each human instance. This method can gradually analyze the various parts of the human body from the image and associate them in subsequent steps to obtain a complete multi-person parsing result. The top-down multi-person parsing method first detects the human instances in the image, and then parses the parts of each human instance one by one; this method starts from the overall, first finds the location and bounding box of the human instance, and then further parses each detected instance, such as posture estimation, key point detection, etc.; the advantage of the top-down method is that it can directly locate the human instance and parse it individually, but there are also challenges such as the possibility of being limited by the accuracy of the detector and difficulty in handling overlapping instances. Compared with the above two-stage method, the single-stage multi-person parsing method simplifies the process and improves efficiency.

[0047] At present, there are also existing technologies that use CID context instance decoupling methods for instance-level human analysis. The CID context instance decoupling method uses a single-stage method to decouple people in the image into multiple instance-aware feature maps, where CID uses CNN to extract feature maps, instance information abstraction (IIA) extracts positions and features to represent each person, and global feature decoupling (GFD) modulates the original feature map to generate instance-aware feature maps, and feeds each feature map to a task-specific head to estimate a person's mask. This existing method does not need to rely on instance bounding boxes to distinguish people, but instead adopts a context instance decoupling strategy to decouple instances in the image into multiple feature maps to achieve human body analysis; and compared with bounding box detection, this method is differentiable and robust to detection errors; however, this existing method may fail in scenes where multiple human instances are extremely crowded. Therefore, when building a single-stage multi-person parsing framework, how to efficiently and accurately achieve multi-person human parsing remains a challenging problem that needs further exploration.

[0048] In this application, we mainly draw on the single-stage multi-person parsing method and explore a method to parse the possibility of human body by decoupling the instance into feature maps and combining the center point and the offset based on the center point to achieve multi-person parsing. At present, there are many target foreground mask segmentation methods based on full convolutional networks, such as the method of fixed grids representing the instances to be detected in SOLO and the method of using dynamic lightweight kernels to generate foreground masks in CondInst; in general, the head network design of these existing methods is task-oriented, and the application to different tasks may lead to an increase in parameter size and computational cost, which brings challenges to lightweight design. The multi-person parsing method of this application simplifies the complexity of key point or mask grouping, making it easier to process, and can access larger context clues outside the bounding box, improving accuracy while avoiding the difficulties of long-distance regression and mask encoding and decoding faced by single-stage methods. In the following, instance decoupling refers to decoupling the human body into different feature maps, and the center-based offset refers to the offset from the center of the human body to the center of each part of the human body.

[0049] In this application, the image is sent to a Feature Pyramid Network (FPN) to generate feature maps of different sizes; then, a multi-feature fusion model is used to extract the location and features of the instance, and the feature map is processed using a global decoupling model to generate an instance-aware feature map; finally, by utilizing the output of global feature decoupling, an instance-aware multi-person parsing result can be obtained.

[0050] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0051] Figure 1 FIG. 1 is a flow chart of a multi-person resolution method based on instance decoupling and multi-feature point fusion according to an embodiment of the present invention. Figure 1 As shown, the method at least includes steps S10 to S30.

[0052] Step S10: obtaining an image to be tested, inputting the image to be tested into a trained feature pyramid network to obtain a plurality of feature maps, and connecting the plurality of feature maps to obtain a feature overall map.

[0053] In this step, the image to be tested is sent to the feature pyramid network to generate multiple feature maps, and the multiple feature maps are further connected to form a feature total map. Specifically, for a given image to be tested i, the backbone network and the FPN network extract the semantic features of the image to be tested i. Among them, the sizes of the multiple feature maps obtained based on the FPN network are different, and in order to connect multiple feature maps of different sizes, the sizes of each feature map are first adjusted so that the sizes of the multiple feature maps are the same. Exemplarily, the multiple feature maps obtained are connected to obtain a feature total map, including: adjusting the multiple feature maps to the same size based on the bilinear interpolation method; connecting the multiple feature maps of the same size to obtain the feature total map. In this embodiment, the multi-scale feature maps are adjusted to feature maps of the same size (S×S) based on the bilinear interpolation method, and multiple S×S feature maps are connected together to obtain a feature total map.

[0054] Step S20: Input the feature map into a trained multi-feature fusion model, extract features from the instance center based on the multi-feature fusion model to obtain a central feature, extract features from the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature, and fuse the central feature and the offset feature to obtain features of each instance.

[0055] This step is to enhance the representation of instance features. Based on the feature map obtained in step S10, the center feature and offset feature are obtained through a multi-feature fusion model; that is, the mapping relationship is estimated by predicting the offset from the center of the human body to the center of the corresponding part instance, and then the center feature is fused with the offset feature based on the center point, so that the matching between the human body and the parts can be completed without grouping. Compared with the method of representing each instance based on the center of the instance alone, this method enriches the representation of instance features, thereby ensuring the accuracy of multi-person parsing results.

[0056] In one embodiment, based on the multi-feature fusion model, feature extraction is performed on the instance center to obtain a center feature, and feature extraction is performed on the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature. Specifically, it may include: determining the instance center coordinates based on the multi-feature fusion model, and obtaining the center feature of each instance based on the instance center coordinates; determining the position coordinates of the center of each part of each instance based on the multi-feature fusion model, determining the offset of each part based on the position coordinates of the center of each part of each instance and the instance center coordinates, and obtaining the offset feature based on the offset of each part. The obtained center feature can be expressed as Where S×S is the size of the feature map after resizing the multi-scale feature map. In this embodiment, the coordinate convolution (CoordConv) can be used to obtain the position information of the center of the instance and the center of each part of the instance, and then the decoupled structure of AdaptBlock is used to predict the offset of each part of the human body. Finally, all the branch outputs of the decoupling are connected together to obtain the offset map Among them C P Indicates the total number of types of parts in the instance. The pixels near the center of the human body instance in the offset map include C P The regression vector points to the pixel C P The center position of the part.

[0057] Furthermore, in order to ensure that the size of the offset feature is the same as that of the center feature, the offset feature (offset map) is further average pooled, and finally, the center feature and the offset feature are fused using a residual module to obtain an enhanced instance feature. Specifically, in this embodiment, the center feature and the offset feature are feature fused to obtain each instance feature, including: average pooling the offset feature; and fusing the center feature and the average pooled offset feature based on a residual network to obtain each instance feature.

[0058] Figure 3 FIG. 1 is a flow chart of a multi-feature fusion process of a multi-person analysis method in an embodiment of the present invention. Figure 3 As shown, when performing multi-feature fusion, the center point features of each person are first extracted based on the feature map, and then the offset features of each person are extracted. Finally, the center point features and offset features of each person are fused to obtain the instance features of the corresponding person.

[0059] Step S30: input the feature total map and the instance features into a trained global decoupling model, obtain an instance perception map based on the global decoupling model, and obtain a heat map of each instance based on the instance perception map.

[0060] The global decoupling model aims to decouple the human body into different feature maps, while allowing the interference from other people to be separated and exploring contextual clues at a scale larger than the bounding box size. In this step, the output results of the multi-feature fusion model and the feature map obtained in step S10 are input into the trained global decoupling model, and the attention mechanism is used to guide the model to learn the feature heat map processing in space and channels, and each output heat map contains the prediction result of an instance.

[0061] In one embodiment, the feature map and the instance features are input into a trained global decoupling model, an instance perception map is obtained based on the global decoupling model, and a heat map of each instance is obtained based on the instance perception map, including: based on the mapping relationship between each instance feature and the feature map, spatially recalibrating each instance feature to obtain a first feature perception map; based on the mapping relationship between each instance feature and the feature map, channel recalibrating each instance feature to obtain a second feature perception map; fusing the first feature perception map and the second feature perception map to obtain an instance perception map. In this embodiment, based on the mapping relationship between each person's instance feature and the feature map, each person's instance feature is recalibrated from a spatial dimension, and each person's instance feature is recalibrated from a channel dimension.

[0062] Specifically, spatial recalibration is used to separate each person into different spatial positions, with the goal of increasing the weight of features on the foreground of the instance and reducing the weight of other features; spatial recalibration generates a spatial mask to represent the foreground of the instance, and then calculates the spatial recalibration of the instance to obtain new feature information. Channel recalibration is used to divide instances into different channels of feature maps; first, the original feature map in the channel dimension is reweighted with the instance features (implemented by multiplying the two features) to generate a conditional feature map; the principle is: if a feature is multiplied with a similar feature, it will be enhanced, and if multiplied with a different feature, it will be suppressed. Finally, the spatial recalibration result and the channel recalibration result are fused to obtain the final instance perception map. This step can be achieved through Conv convolution. For example Among them, i represents the i-th instance, M i represents the instance perception map of the i-th instance, O i represents the relative coordinate graph of the i-th instance, Represents the pixel-level feature similarity of the i-th instance.

[0063] Figure 4 FIG. 1 is a flow chart of a multi-person resolution method in an embodiment of the present invention during a global decoupling process. Figure 4 As shown in the figure, in the global decoupling process, the instance features are recalibrated in the spatial dimension based on the feature total map and the instance features to obtain a spatial mask, and a new feature map (which can be called the first feature perception map) is obtained. In addition, the feature map (which can be called the second feature perception map) is recalibrated in the channel dimension based on the feature total map and the instance features. Finally, the new feature map is fused with the feature map to obtain the final instance mask.

[0064] In some embodiments of the present invention, in order to obtain a trained multi-feature fusion model and a trained global decoupling model, an initial multi-feature fusion model and an initial global decoupling model are first constructed, and then the initial multi-feature fusion model is pre-trained to obtain a trained multi-feature fusion model, and the initial global decoupling model is pre-trained to obtain a trained global decoupling model.

[0065] Exemplarily, pre-training the initial multi-feature fusion model to obtain a trained multi-feature fusion model includes: constructing first sample data and a first loss function, and pre-training the initial multi-feature fusion model based on the first sample data and the first loss function to obtain a trained multi-feature fusion model. Similarly, pre-training the initial global decoupling model to obtain a trained global decoupling model includes: constructing second sample data and a second loss function, and pre-training the initial global decoupling model based on the second sample data and the second loss function to obtain a trained global decoupling model.

[0066] The losses corresponding to the multi-feature fusion model may include offset loss, center positioning loss, and each part positioning loss. For the two positioning losses, the centroid (c x ,c y ) and the height h and width w of the instance, the centroid is expanded to a rectangular area, which is the central area (c x ,c y ,∈ h ,∈ w ), where ∈ represents a hyperparameter used to control the size of the region (∈ can be set to 0.2). If the center of the actual instance falls in the center of the grid in the rectangular area, it is a positive sample. For offset loss, offset prediction can be regarded as a regression problem, and the offset vector on each grid is obtained by subtracting the grid coordinates from the center coordinates of the corresponding part. The grid mentioned above represents a grid unit on the image, which is usually used to describe the discretization representation of the image; in deep learning, images are usually divided into small square grids, each of which is called a "grid"; these grids are usually used to segment, process and analyze images. In this step, Smooth L1 Loss is used to solve the regression problem, that is, when calculating the offset loss, each grid is multiplied by a weight related to scaling to balance the impact of scaling.

[0067] The losses corresponding to the global decoupling model include Dice loss and L1 correlation loss. The Dice loss is used to measure the similarity between the model's predicted segmentation results and the actual segmentation results. The L1 correlation loss is used to calculate the mean absolute error after calculating the similarity between the predicted tensor and the target tensor.

[0068] In a specific embodiment, the first sample data includes a predicted instance center and an actual instance center, a predicted instance part center and an actual instance part center, a predicted part offset and an actual part offset, and the first loss function includes Offset i,j,p =(c px ―i,c py ―j), where Offset i,j,p represents the offset loss function, (c px , c py ) means c p The position coordinates of the part center, p∈[0,c p -1] is the part category label; (i, j) is the grid coordinate, i∈[0,S-1], j∈[0,S-1], S represents the size of the feature map after resizing multiple feature maps by bilinear interpolation. Specifically, the size of the resized feature map is S×S. The grid mentioned above represents a grid unit on the image, which is usually used to describe the discretization representation of the image; in deep learning, images are usually divided into small square grids, each of which is called a "grid"; these grids are usually used to segment, process and analyze images.

[0069] The second sample data includes predicted instance mask information and actual instance mask information. The second loss function includes Dice loss and L1 correlation loss. The Dice loss is The L1 correlation loss is Among them, T (i) Represents the predicted instance mask information, that is, the segmentation result predicted by the model, which is an n-channel mask, where n represents the number of body parts in the instance; T (i)* represents the actual instance mask information, N represents the total number of body parts of the instance, and diff i Represents the difference between the prediction tensor and the target tensor, where the prediction tensor refers to the predicted instance segmentation result and the target tensor refers to the actual instance segmentation result.

[0070] In the above embodiment, the goal of multi-person resolution is achieved by decoupling instances into feature maps and combining center points and offsets based on center points. Figure 2Specifically, the mask of the human body part instance is first generated using the feature of the instance center, and then the offset from the instance center to the center of each part is used to unify the human body instance. Then, the center feature and the offset feature are fused to characterize each instance. Finally, the original feature map is modulated using global feature decoupling to generate an instance-aware feature map for estimating each person's mask. This multi-person parsing method based on instance decoupling and multi-feature point fusion adds a multi-feature point fusion (MPF) method on the basis of global feature decoupling (GFD), that is, the center feature is fused with the offset feature based on the center point to achieve the extraction of the human body instance position and features, and enhance the representation of instance features. This method not only retains the robustness of the existing technology, but also improves the relevant performance; and the simple end-to-end model structure significantly improves the efficiency of model reasoning, making the reasoning faster and more accurate, which is helpful for people-oriented research fields.

[0071] Corresponding to the above method, the present invention also provides a multi-person analysis system based on instance decoupling and multi-feature point fusion, including a processor, a memory and a computer program stored in the memory, the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0072] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0073] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0074] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0075] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-person parsing method based on instance decoupling and multi-feature point fusion, characterized in that: The method comprises the following steps: Acquire an image to be tested, input the image to be tested into a trained feature pyramid network to obtain multiple feature maps, and connect the multiple feature maps to obtain a feature overall map; The feature map is input into a trained multi-feature fusion model, based on the multi-feature fusion model, feature extraction is performed on the instance center to obtain a center feature, feature extraction is performed on the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain an offset feature, and feature fusion is performed on the center feature and the offset feature to obtain each instance feature; Inputting the feature total map and the instance features into a trained global decoupling model, obtaining an instance perception map based on the global decoupling model, and obtaining a heat map of each instance based on the instance perception map; Among them, based on the multi-feature fusion model, feature extraction is performed on the instance center to obtain the center feature, and feature extraction is performed on the offset between the center of each part of each instance and the instance center of the corresponding instance to obtain the offset feature, including: Determine the instance center coordinates based on the multi-feature fusion model, and obtain the central features of each instance based on the instance center coordinates; Determine the position coordinates of the center of each part of each instance based on the multi-feature fusion model, determine the offset of each part based on the position coordinates of the center of each part of each instance and the instance center coordinates, and obtain the offset feature based on the offset of each part; Based on coordinate convolution, the position coordinates of the instance center and the center of each part are obtained.

2. The multi-person resolution method based on instance decoupling and multi-feature point fusion according to claim 1 is characterized in that: Connecting the obtained multiple feature maps to obtain a feature overall map includes: Resize multiple feature maps to the same size based on bilinear interpolation; Connect multiple feature maps of the same size to obtain the total feature map.

3. The multi-person parsing method based on instance decoupling and multi-feature point fusion according to claim 1, characterized in that: The center feature and the offset feature are fused to obtain each instance feature, including: Performing average pooling on the offset features; Based on the residual network, the center feature and the offset feature after average pooling are fused to obtain each instance feature.

4. The multi-person resolution method based on instance decoupling and multi-feature point fusion according to claim 1, characterized in that: Inputting the feature total map and the instance features into a trained global decoupling model, obtaining an instance perception map based on the global decoupling model, and obtaining a heat map of each instance based on the instance perception map, including: Based on the mapping relationship between each instance feature and the feature master map, spatially recalibrate each instance feature to obtain a first feature perception map; Based on the mapping relationship between each instance feature and the feature overall map, recalibrate the channel of each instance feature to obtain a second feature perception map; The first feature perception map and the second feature perception map are fused to obtain an instance perception map.

5. The multi-person resolution method based on instance decoupling and multi-feature point fusion according to claim 2 is characterized in that: The method further comprises: Construct an initial multi-feature fusion model and an initial global decoupling model; Pre-training the initial multi-feature fusion model to obtain a trained multi-feature fusion model; The initial global decoupling model is pre-trained to obtain a trained global decoupling model.

6. The multi-person resolution method based on instance decoupling and multi-feature point fusion according to claim 5, characterized in that: Pre-training the initial multi-feature fusion model to obtain a trained multi-feature fusion model, including: constructing first sample data and a first loss function, and pre-training the initial multi-feature fusion model based on the first sample data and the first loss function to obtain a trained multi-feature fusion model; and / or Pre-training an initial global decoupling model to obtain a trained global decoupling model includes: constructing second sample data and a second loss function, and pre-training the initial global decoupling model based on the second sample data and the second loss function to obtain a trained global decoupling model.

7. The multi-person resolution method based on instance decoupling and multi-feature point fusion according to claim 6, characterized in that: The first sample data includes a predicted instance center and an actual instance center, a predicted instance part center and an actual instance part center, a predicted part offset and an actual part offset, and the first loss function includes an offset i,j,p =(c px -i,c py -j), where Offset i,j,p represents the offset loss function, (c px , c py ) means c p The position coordinates of the part center, (i, j) represents the grid coordinates on the image, i∈[0,S-1], j∈[0,S-1], S represents the size of the feature map after resizing; The second sample data includes predicted instance mask information and actual instance mask information. The second loss function includes Dice loss and L1 correlation loss. The Dice loss is The L1 correlation loss is Among them, T (i) Represents the predicted instance mask information, T (i)* represents the actual instance mask information, N represents the total number of instance parts, and diff i Represents the difference between the prediction tensor and the target tensor.

8. A multi-person analysis system based on instance decoupling and multi-feature point fusion, comprising a processor, a memory, and a computer program stored in the memory, characterized in that: The processor is used to execute the computer program. When the computer program is executed, the system implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images