Cartoon analysis method based on semantic consistency and structural correlation

Abstract

The invention relates to the technical field of computer vision, and discloses a cartoon analysis method based on semantic consistency and structural correlation, which is used for cartoon analysis based on semantic consistency and structural correlation. The problems of low cartoon analysis accuracy and the like in the prior art are solved.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a cartoon parsing method based on semantic consistency and structural relevance. Background Art

[0002] Character parsing has made remarkable progress in the field of computer vision. Existing character parsing methods have achieved promising results in human body part recognition. However, applying character parsing methods to cartoon parsing faces challenges due to the differences between cartoons and real people. Cartoon images are complex and abstract, and their features vary greatly depending on the creator and cultural background. Character parsing models are designed on real-world human image datasets, which makes them unsuitable for cartoon parsing because cartoon characters often have non-traditional body proportions and joint structures. Therefore, cartoon parsing requires dedicated research and specialized algorithms to cope with its unique characteristics, emphasizing accuracy and effectiveness in handling the diversity and complexity of cartoon images.

[0003] In recent years, cartoon parsing tasks have received increasing attention due to their importance in a variety of applications. Researchers have proposed a variety of methods to cope with the challenges posed by the diverse and complex appearance and structure of cartoons. Early work in this field focused on rule-based methods and handcrafted feature extraction techniques. Although these methods can capture some features of cartoon structures, they often have difficulty in dealing with the diversity and complexity of cartoon images. With the emergence of deep learning techniques, such as convolutional neural networks, cartoon parsing has made significant progress. Although convolutional network-based methods have shown good performance in recognizing cartoon characters and their body parts, dealing with the diversity and complexity of cartoon characters remains a challenge. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a cartoon parsing method based on semantic consistency and structural relevance, which solves the problems of low accuracy of cartoon parsing in the prior art.

[0005] The technical solution adopted by the present invention to solve the above problems is:

[0006] A cartoon parsing method based on semantic consistency and structural relevance is proposed to perform cartoon parsing based on semantic consistency and structural relevance.

[0007] As a preferred technical solution, the following steps are included:

[0008] S1, memory bank construction: building the memory bank;

[0009] S2, semantic consistency learning: semantic consistency learning of cartoon characters’ body parts;

[0010] S3, Structural Association Learning: Structural association learning of body parts of cartoon characters.

[0011] As a preferred technical solution, in step S2, semantic consistency learning includes representation consistency learning: integrating and learning the consistency of the feature representation of the current sample with the sample representation stored in the memory bank.

[0012] As a preferred technical solution, when performing integration and consistency learning, the current sample is spliced ​​and fused with the sample stored in the memory bank, and the fused features are input into the multi-head self-attention network. The current sample is associated with the sample stored in the memory bank through the self-attention network to obtain the consistent feature F rc .

[0013] As a preferred technical solution, in step S2, semantic consistency learning includes classification consistency learning: achieving semantic consistency between the current sample and the sample stored in the memory bank through loss weighting.

[0014] As a preferred technical solution, multiple samples associated with the current sample are obtained through a memory selection operation, and then the loss of each body part of each sample is calculated, and then the losses of the same body part of different samples are accumulated, and then the accumulated losses are normalized to obtain the loss weight; then the loss is calculated for each part of the current sample, and the loss weight is weighted to each part of the current sample.

[0015] As a preferred technical solution, in step S3, the feature F after consistency processing is rc After being processed by the PCL pixel association module, the PCL pixel association features are obtained, and then the PCL pixel association features are processed into graph structure features, and then the graph structure features are processed by the subject-guided graph attention network to obtain the subject-perceived structure association features; then the PCL pixel association features are input into the PCM structure association module to obtain the PCM structure association features; then the subject-perceived structure association features and the PCM structure association features are spliced ​​to obtain the fusion feature F for the final prediction final , F final After being processed by the classifier composed of convolutional layers, the final prediction result is obtained.

[0016] As a preferred technical solution, each node in the graph structure represents each body part, the connection between nodes represents the connection between body parts, and the connection relationship of the nodes is constructed through the adjacency matrix.

[0017] As a preferred technical solution, in step S1, a memory bank including a prediction column and a representation column is constructed, and then the cartoon image is encoded into a feature composed of a two-dimensional feature map through a convolutional network, the two-dimensional feature map is mapped to a prediction feature using a convolutional layer, and the two-dimensional feature map is processed using a global pooling operation to obtain a one-dimensional representation feature, the prediction feature is stored in the prediction column, and the representation feature is stored in the representation column.

[0018] As a preferred technical solution, in step S1, the constructed memory library includes a label column.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) This paper proposes a novel cartoon parsing method, which designs a semantic consistency method based on memory structure and a subject structure correlation perception learning strategy to address the challenges of diverse appearances and complex structures in cartoon parsing;

[0021] (2) This paper proposes a learning strategy based on memory structure to facilitate appearance encoding, storage, selection and consistency learning. This method aims to solve the diversity problem of cartoon images by capturing the inconsistency of appearance features of different cartoon characters and adapting to the changes in color, texture and details;

[0022] (3) This paper proposes a subject structure correlation perception method that can recognize cartoon characters with complex structural features. By using a graph attention network to model and associate subjects, this method solves the problem of complex cartoon image structures of anthropomorphic and animalized cards;

[0023] (4) On the cartoon parsing dataset, the proposed method achieves state-of-the-art performance, which demonstrates the effectiveness of the proposed semantic consistency and structural correlation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a cartoon parsing method based on semantic consistency and structural relevance according to the present invention;

[0025] Figure 2 for Figure 1 One of the partial enlarged pictures;

[0026] Figure 3 for Figure 1 The second partial enlarged image. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0028] Example 1

[0029] like Figures 1 to 3 As shown, the present invention proposes a memory-based learning structure to facilitate appearance encoding, storage, selection and consistency learning. This method aims to solve the diversity problem of cartoon images by capturing the inconsistency of appearance features of different cartoon characters and adapting to the changes in color, texture and details.

[0030] This paper proposes a subject-aware structure association method that can recognize cartoon characters with complex structural features. By using a graph attention network to model and associate subject structures, this method can recognize complex anthropomorphic and animalized cartoon images.

[0031] This paper proposes a novel cartoon parsing method, which aims to identify and segment the parts of cartoon characters. It addresses the visual diversity and structural complexity of cartoon images through semantic consistency learning and structural correlation modeling. First, a semantic consistency learning module based on memory structure is designed to encode and learn the diverse appearances displayed by cartoon characters. By storing, selecting and associating different cartoon images, the module learns the consistency of body parts belonging to the same category but from different cartoon characters. In order to identify the complex and irregular structures present in cartoon images, a structural correlation module is proposed. Using graph attention network and subject perception mechanism, this method is able to perform structural learning and association, thereby identifying and parsing cartoon images with significant complexity. Experiments conducted on cartoon parsing datasets demonstrate the effectiveness of the method.

[0032] The purpose of this solution is to solve complex and diverse cartoon parsing and segmentation problems. The solution mainly consists of two modules: semantic consistency learning based on memory structure and structural association learning based on subject structure. The overall architecture is as follows Figure 1 shown.

[0033] 1. Memory Bank Construction

[0034] Existing cartoon parsing methods only consider multi-scale learning and pixel and body part correlation of a single cartoon character. These methods have difficulty in recognizing diverse cartoon images. Therefore, the first important component of the proposed method is to encode and learn the visual appearance of diverse cartoon characters. In order to address the problems brought by visual diversity, a memory structure-based learning strategy is proposed to achieve semantic consistency between similar body parts. The memory structure-based consistency learning includes two parts: representation consistency learning and classification consistency learning. Representation consistency aims to learn the consistency of body parts from different cartoon characters at the high-dimensional semantic feature level; classification consistency is to learn the consistency of body parts from various cartoon characters at the semantic recognition level, and use the constraint information provided by the labels to achieve consistency.

[0035] Given a cartoon image, it is encoded into features through a convolutional network. After processing, the features are obtained to obtain prediction features and representation features, which are stored in the prediction column and representation column of the memory bank. In addition, the memory bank has a label column for storing labels. In the initial stage, the three columns of the memory bank are initialized to zero. During the training process, the image features of each iteration are stored in the memory bank. When the memory bank is full, the earliest stored features will be removed to ensure that the memory bank has space to store the features of the new iteration, so that the memory bank is continuously updated throughout the training process.

[0036] In each iteration, features are not only stored in the memory bank, but also further forwarded. In addition, a memory selection operation is performed in each iteration to select some samples from the memory structure. These selected samples have a certain similarity with the current sample and are backtracked to achieve semantic consistency between different samples, thereby enhancing the network's semantic reasoning ability for targets with similar features. Specifically, when a new sample is input, the network selects similar samples from the memory structure. By leveraging the knowledge gained from these learned samples, the network can accurately infer and identify the current sample. When selecting similar samples, it is mainly achieved through Euclidean distance calculation.

[0037] 2. Representation Consistency Learning

[0038] The problem of inconsistent appearance between different cartoon images poses a great challenge to cartoon parsing. In order to achieve consistent learning of body parts belonging to the same category but from different samples, the network initially performs consistency learning at the representation level. It integrates the feature representation of the current sample with the feature representation of related samples retrieved from the memory bank and achieves consistency learning. In order to achieve consistent learning between body parts from different samples, a multi-head self-attention mechanism is used to achieve consistency between the current sample and the samples stored in the memory structure at the feature representation level.

[0039] Specifically, in order to learn representation consistency, the current sample and the sample of the selected memory structure are concatenated and fused, and then the fused features are input into the multi-head self-attention network. The current sample and other samples taken out of the memory bank are associated through the association ability of the self-attention network, so as to achieve representation consistency learning and obtain the representation consistent feature F rc .

[0040] 3. Classification Consistency Learning

[0041] In order to achieve semantic consistency, supervision of different body parts is required during training. Therefore, in addition to representation consistency learning, classification consistency learning is also proposed. Classification consistency learning is performed through loss weighting to achieve semantic consistency between the current sample and the sample retrieved from the memory bank. The labels of the current sample and the retrieved sample are used to provide supervision information.

[0042] Specifically, through the memory selection operation, multiple samples associated with the current sample can be obtained, and the loss of these selected samples is calculated respectively, and the loss of each body part of each sample is calculated. Then, the losses of the same body part in these samples are aggregated, that is, the losses of the same body part of different samples are accumulated. Then, the aggregated losses are normalized to become weights. After that, the loss is calculated for each part of the current sample, and the loss weight of each part aggregated from the selected multiple samples is weighted to each part of the current sample, so as to use the recognition results of the samples in the memory structure to calibrate the recognition results of the current sample.

[0043] 4. Structural Association Learning

[0044] In addition to appearance learning, structure learning is also crucial for cartoon parsing. Previous methods considered the associations between different parts, but ignored the learning of strongly correlated local subject structures. Local subject structures mainly refer to some local structures that show high correlation, such as face, hair, and hat. Similarly, thighs, calves, and shoes, or hands, upper arms, and forearms, also have strong correlations. Therefore, the proposed method introduces a subject structure-aware recognition strategy, focusing on the structural modeling of strongly correlated structures to explore the complex structure of cartoon images.

[0045] Specifically, given the feature F representing the consistency process rc, which is processed by PCL pixel correlation structure (PCL, PixelCorrelation Learning) to obtain pixel correlation features. The pixel correlation features are then processed into features of a graph structure, in which each node represents each body part, and the connections between nodes represent the connections between body parts. The connection relationship of the nodes is mainly constructed through the adjacency matrix. After that, the graph structure is processed by a graph attention network to achieve structural association. Unlike the general graph attention network, the constructed graph attention network is hierarchical and aims to model the feature representation of global and local subject structures. First, the graph attention network layer is used to model the constructed graph structure, update all node data and return the results. Subsequently, a subgraph structure is constructed by merging strongly correlated nodes in the nodes, and then the graph attention network is applied to model these strongly correlated nodes and return the results. The network promotes the recognition of specific areas by focusing on the main body structures. After that, the learning results of each main body structure are aggregated and fused with the modeling results from the global body structure to produce subject-perceived structural features. The subject-perceived structural features are spliced ​​with the output of the PCM structural association module to obtain the feature F final , F final That is, the feature used for the final prediction, F final After being processed by the classifier composed of convolutional layers, the final prediction result is obtained.

[0046] V. Experimental Verification and Analysis

[0047] In order to verify the effectiveness of the proposed method, it was verified on the CartoonSet dataset. The evaluation indicator used is the Mean IoU, which calculates the average IoU between the predicted body parts and the true annotations. In addition, the Mean Acc. is used to calculate the average accuracy of all categories, and the Pixel Acc. is used to calculate the accuracy of correctly predicted pixels.

[0048] To evaluate the performance of the proposed method, the present invention is compared with the state-of-the-art human parsing methods and cartoon parsing methods. The results are listed in Table 1. Existing methods have limited performance due to ignoring the diverse, complex, and abstract characteristics of cartoon images. Compared with previous methods, the proposed method achieves the best results on CartoonSet. The semantic consistency learning module based on memory structure looks back on past experience to improve generalization ability. The main structure perception mechanism focuses on important local areas to improve structural correlation learning. The proposed method conducts in-depth exploration of diverse and complex cartoon images and solves the challenges brought by cartoon characters of various styles.

[0049] Table 1 Performance comparison on CartoonSet

[0050]

[0051] As described above, the present invention can be preferably implemented.

[0052] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0053] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A cartoon parsing method based on semantic consistency and structural relevance, It is characterized in that Cartoon parsing based on semantic consistency and structural relevance.

2. A cartoon parsing method based on semantic consistency and structural relevance according to claim 1, It is characterized in that The following steps are involved: S1, memory bank construction: building the memory bank; S2, semantic consistency learning: semantic consistency learning of cartoon characters’ body parts; S3, Structural Association Learning: Structural association learning of body parts of cartoon characters.

3. A cartoon parsing method based on semantic consistency and structural relevance according to claim 2, It is characterized in that In step S2, semantic consistency learning includes representation consistency learning: integrating and learning the consistency of the feature representation of the current sample with the sample representation stored in the memory bank.

4. A cartoon parsing method based on semantic consistency and structural relevance according to claim 3, It is characterized in that When performing integration and consistency learning, the current sample is concatenated with the sample stored in the memory bank, and the fused features are input into the multi-head self-attention network. The self-attention network is used to associate the current sample with the sample stored in the memory bank to obtain the consistent feature F. rc .

5. A cartoon parsing method based on semantic consistency and structural relevance according to claim 4, It is characterized in that In step S2, semantic consistency learning includes classification consistency learning: achieving semantic consistency between the current sample and the sample stored in the memory bank through loss weighting.

6. A cartoon parsing method based on semantic consistency and structural relevance according to claim 5, It is characterized in that Through the memory selection operation, multiple samples associated with the current sample are obtained, and then the loss of each body part of each sample is calculated. The losses of the same body part of different samples are accumulated, and then the accumulated losses are normalized to obtain the loss weight; then the loss is calculated for each part of the current sample, and the loss weight is weighted to each part of the current sample.

7. A cartoon parsing method based on semantic consistency and structural relevance according to claim 4, It is characterized in that In step S3, the feature F after consistency processing is rc After being processed by the PCL pixel association module, the PCL pixel association features are obtained, and then the PCL pixel association features are processed into graph structure features, and then the graph structure features are processed by the subject-guided graph attention network to obtain the subject-perceived structure association features; then the PCL pixel association features are input into the PCM structure association module to obtain the PCM structure association features; then the subject-perceived structure association features and the PCM structure association features are spliced ​​to obtain the fusion feature F for the final prediction final , F final After being processed by the classifier composed of convolutional layers, the final prediction result is obtained.

8. A cartoon parsing method based on semantic consistency and structural relevance according to claim 7, It is characterized in that Each node in the graph structure represents each body part, the connection between nodes represents the connection between body parts, and the connection relationship between nodes is constructed through the adjacency matrix.

9. A cartoon parsing method based on semantic consistency and structural relevance according to any one of claims 2 to 8, It is characterized in that In step S1, a memory bank including a prediction column and a representation column is constructed, and then the cartoon image is encoded into a feature composed of a two-dimensional feature map through a convolutional network. The two-dimensional feature map is mapped to a prediction feature using a convolutional layer, and the two-dimensional feature map is processed using a global pooling operation to obtain a one-dimensional representation feature. The prediction feature is stored in the prediction column, and the representation feature is stored in the representation column.

10. A cartoon parsing method based on semantic consistency and structural relevance according to claim 9, It is characterized in that In step S1, the constructed memory library includes a label column.

Images