Corrected interactive human parsing method, system, medium, and device

Abstract

The present disclosure relates to a method, system, medium and device for interactive human body parsing based on correction, the method comprising: encoding user clicks to generate a positioning map, cascading the positioning map with a human body RGB image as an input of a model; constructing a neural network with a convolution layer having a plurality of selectable convolution parameters, and obtaining a deep feature of the human body RGB image through the neural network; wherein the neural network is used to parse different categories of interactive samples; constructing an online updating memory bank to store correction-mis-prediction pairs that have appeared in history; wherein the correction-mis-prediction pair is the difference between the semantic cognition of the human body part of the labeler and the model; using the neural network to update the content of the online updating memory bank within a preset iteration period, and updating and replacing the memory prototypes that are not commonly used in the online updating memory bank; adding correction clicks to the error area through user iteration to perfect the segmentation result until the human body image segmentation is completed.

Description

Technical Field

[0001] This disclosure relates to the field of image segmentation technology, and more specifically, to a correction-based interactive human body parsing method, system, medium, and device. Background Technology

[0002] In recent years, human body analysis has received considerable attention due to its wide range of applications, such as human behavior and attribute analysis, autonomous driving, and smart healthcare. Excellent human body analysis models typically require a large amount of finely labeled, pixel-level data for training. More labeled data often leads to better performance. However, the number of publicly available human body analysis datasets is limited, and constructing more fine-grained training data is both time-consuming and laborious.

[0003] Due to the efficiency of interactive segmentation in labeled image data, this research area is attracting increasing attention. Existing interactive segmentation methods typically aim for higher efficiency by mining the potential information from user clicks or exploring efficient interaction forms and strategies. However, these works neglect to explore the semantic correlation between user corrections and model mispredictions when facing interactive human body parsing tasks, thus exhibiting two drawbacks. First, similar prediction errors frequently occur in practical applications, requiring users to repeatedly correct their predictions, resulting in redundant interactive information. Second, different human body part classes have varying levels of interaction difficulty across different images, but existing methods use a single convolution parameter setting for all images, leading to poor robustness when parsing human body part classes with high interaction difficulty. Summary of the Invention

[0004] The main purpose of this invention is to solve the technical problem that existing technology models have poor robustness in parsing human body parts with high interaction difficulty.

[0005] To achieve the above technical objectives, this disclosure provides an interactive human body parsing method based on correction and misprediction of semantic relevance, the method comprising:

[0006] The user clicks to generate a location map, which is then concatenated with a human RGB image as input to the model.

[0007] A neural network with multiple selectable convolutional parameters is constructed, and the depth features of the human RGB target image are obtained through the convolutional neural network; wherein, the neural network is used to parse interactive samples of different categories;

[0008] An online update memory is constructed to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts;

[0009] The neural network is used to update the contents of the online update memory within a preset iteration period, and unused memory prototypes in the online update memory are updated and replaced.

[0010] The segmentation results are improved by adding correction clicks to erroneous areas through user iterations until the human image segmentation is completed.

[0011] Furthermore, the construction of an online update memory for storing historically occurring correction-misprediction pairs specifically includes:

[0012] A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

[0013] Furthermore, the neural network is obtained by modifying the DeepLab V3+ network model as the base model.

[0014] Furthermore, the neural network has an online learning loss function, which is expressed as:

[0015]

[0016] in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks;

[0017] γ is the cost of changing network parameters; γ is the strength of parameter regularization; λ is the loss coefficient.

[0018] To achieve the above technical objectives, this disclosure also provides an interactive human body parsing system based on correction and misprediction of semantic relevance, the system comprising:

[0019] The input module is used to encode user clicks to generate a location map, and concatenates the location map with a human RGB image as the input to the model;

[0020] The model building module is used to construct a neural network with multiple selectable convolutional parameters and to obtain the depth features of the human RGB target image through the convolutional neural network; wherein, the neural network is used to parse interactive samples of different categories;

[0021] An online update memory is constructed to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts;

[0022] An iterative module is used to update the contents of the online update memory bank using the neural network within a preset iteration period, and to update and replace the memory prototypes that are not frequently used in the online update memory bank.

[0023] The image segmentation module is used to improve the segmentation results by adding correction clicks to erroneous areas through user iterations until the human image segmentation is completed.

[0024] Furthermore, the construction of an online update memory for storing historically occurring correction-misprediction pairs specifically includes:

[0025] A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

[0026] Furthermore, the neural network is obtained by modifying the DeepLab V3+ network model as the base model.

[0027] Furthermore, the neural network has an online learning loss function, which is expressed as:

[0028]

[0029] in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks;

[0030] γ is the cost of changing network parameters; γ is the strength of parameter regularization; λ is the loss coefficient.

[0031] To achieve the above-mentioned technical objectives, this disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the steps of the interactive human body parsing method based on correction and misprediction of semantic relevance described above.

[0032] To achieve the above-mentioned technical objectives, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the interactive human body parsing method based on correction and misprediction of semantic relevance described above.

[0033] The beneficial effects of this disclosure are:

[0034] This disclosure presents a simple and effective online learning scheme for interactive human body parsing, namely the Correction-Misprediction Association Mining Model (CM). 2This method helps to effectively improve interaction efficiency by reducing similar mispredictions and enhancing the model's ability to handle complex interaction part classes.

[0035] This CM is disclosed 2 The model achieves leading segmentation performance on mainstream human body parsing datasets, demonstrating its effectiveness. Meanwhile, the CM disclosed in this publication... 2 The model also helps improve the efficiency of interactive object segmentation and maintains its effectiveness in scene parsing tasks, demonstrating its versatility. Attached Figure Description

[0036] Figure 1 The CM shown is an embodiment of the present disclosure. 2 Network structure diagram;

[0037] Figure 2 A schematic diagram of the CMM network structure of Embodiment 1 of this disclosure is shown;

[0038] Figure 3 A schematic diagram of the CACL structure according to Embodiment 1 of this disclosure is shown;

[0039] Figure 4 A flowchart illustrating the method of Embodiment 1 of this disclosure is shown;

[0040] Figure 5 A schematic diagram of the system structure according to Embodiment 2 of this disclosure is shown;

[0041] Figure 6 A schematic diagram of the structure of Embodiment 4 of this disclosure is shown. Detailed Implementation

[0042] Embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0043] The accompanying drawings illustrate various structural schematics according to embodiments of the present disclosure. These drawings are not to scale, and some details have been enlarged and may have been omitted for clarity. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0044] Example 1:

[0045] like Figure 4 As shown:

[0046] This disclosure provides an interactive human body parsing method based on correction and misprediction of semantic relevance, the method comprising:

[0047] S101: The user clicks to generate a location map, and the location map is concatenated with a human RGB image as input to the model;

[0048] S102: Construct a neural network with multiple selectable convolutional layers, and obtain the depth features of the human RGB target image through the convolutional neural network; wherein, the neural network is used to parse interactive samples of different categories;

[0049] S103: Construct an online update memory to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts;

[0050] S104: The neural network is used to update the contents of the online update memory bank within a preset iteration period, and the infrequently used memory prototypes in the online update memory bank are updated and replaced.

[0051] S105: Improve the segmentation results by adding correction clicks to the error areas through user iterations until the human image segmentation is completed.

[0052] Furthermore, the construction of an online update memory for storing historically occurring correction-misprediction pairs specifically includes:

[0053] A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

[0054] Furthermore, the neural network is obtained by modifying the DeepLab V3+ network model as the base model.

[0055] Furthermore, the neural network has an online learning loss function, which is expressed as:

[0056]

[0057] in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks;

[0058] γ is the cost of changing network parameters; γ is the strength of parameter regularization; λ is the loss coefficient.

[0059] like Figures 1-3As shown, this disclosure elaborates in detail the definition and characteristics of “correction-misprediction pairs”, as well as a method for measuring the difficulty of human body part interactions.

[0060] The "correction-misprediction pair" essentially reflects the difference in semantic understanding between the annotator and the model regarding human body parts. Let C be the number of categories in the dataset, and this disclosure defines a confusion category g. t ∈G, where Represented as a pair of different body part classes, for example (i,j), it means that in the t-th round of interaction, the model predicts i and the user corrects the category to j (i≠j).

[0061] Therefore, we can obtain |G| = C × (C-1). The confusion category represents the cognitive difference between the user and the model at the clicked pixel, considering the limited generalization of features of a single pixel in representing confusion categories. Therefore, this disclosure applies to each g... t We construct a feature vector called the confusion prototype to better represent the different confusion categories.

[0062] Specifically, for image I, let R t For the prediction of human body parts after the t-th round of interaction, and during the (t+1)-th round of interaction, add a single click to the largest error region to generate the confusion category g. t+1 CM 2 The model is then updated accordingly to generate the predicted R. t+1 Next, this disclosure constructs a set of pixels S, whose predicted changes from round t to round t+1 are related to g. t+1 Consistency:

[0063]

[0064] Among them, R t p R represents t The predicted category of the middle pixel p.

[0065] Next, the average feature of S in round t is calculated as g. t+1 The obfuscated prototype.

[0066] To better capture the characteristics of confusion categories, multiple confusion prototypes are generated and saved for each confusion category, and similar prediction errors are corrected using CMM.

[0067] The "correction-misprediction pair" can also reflect the semantic interaction difficulty of the model in different human body part classes. This disclosure uses the number of correction clicks to measure the semantic interaction difficulty of mispredicted regions in each human body part class; categories with more correction clicks are more interaction-difficult. Simultaneously, this disclosure observes that correction clicks tend to cluster in specific human body part classes within each image. Therefore, this disclosure infers that in most images, there exists a category with significantly higher semantic interaction difficulty than other human body parts. Based on this analysis, in the process of learning and parsing interaction information, the model should focus on this human body part class with the highest semantic interaction difficulty (SClg-cls). It should be noted that SClg-cls varies from image to image. Therefore, this disclosure designs a convolutional layer (CACL) whose parameters can be selected according to the interaction difficulty. By selecting convolutional parameters corresponding to SClg-cls, it replaces the use of traditional convolutional layers for feature parsing, thereby better learning and processing SClg-cls in different image samples.

[0068] This disclosure uses DeepLab V3+ as the backbone network, which has performed well in several previous interactive segmentation works. It also adopts the method proposed by Theodora et al., using clicks as labels for online learning. This method simply and effectively improves the efficiency of interaction. Furthermore, in practical applications, there is a certain domain difference between the image to be labeled (test image) and the training set images; the online learning-based algorithm can effectively alleviate this problem. This disclosure uses the online learning loss function from CA-learning, which can be expressed as:

[0069]

[0070] in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks;

[0071] γ is the cost of changing network parameters; γ is the strength of parameter regularization; λ is the loss coefficient.

[0072] To overcome the limitation of insufficient robustness of monotonic convolution parameters in parsing human body parts with high semantic interaction difficulty, and to better handle SClg-cls of different samples, this disclosure sets the feature fusion layer of the ASPP module as a convolutional layer of CACL, and selects the convolution parameters according to the number of corrected clicks for each human body part class. Furthermore, to reduce errors of the same confusion category, the features extracted by CACL are input into CMM, and a memory is built to store the confusion prototypes that have appeared. CM2 compares the image features with the features stored in the memory to identify regions that may have similar prediction errors, and generates a confusion guidance matrix to guide feature optimization.

[0073] Confusion Memory Module. To save and record erroneous predictions and their correction information, this disclosure constructs a memory for storing confusion prototypes, where each confusion category has its corresponding confusion prototype (a set of feature vectors). Confusion prototypes for each category are stored independently in the memory, with a maximum of five prototypes per category. During each round of interaction, prototypes highly similar to image features are "activated".

[0074] After every s rounds of interaction, the CMM updates its memory, generating new memory prototypes to replace infrequently activated prototypes in each category. Specifically, this disclosure employs a momentum parameter update strategy to maintain the consistency of the obfuscated prototypes in the memory. Let w t Let w be the model parameters for the t-th online training step, then the momentum parameter w m The calculation is as follows:

[0075]

[0076] Where w0 represents the parameters learned from the training set. This disclosure sets δ to 0.999. Experiments demonstrate that the momentum parameter update strategy has minimal impact on model computation time in practical applications.

[0077] Based on the obfuscation prototype features stored in the memory, this disclosure constructs a Comparative Model (CMM) to reduce interaction redundancy caused by clicks on the same obfuscation category. "F" and "F'" represent the input and output features, respectively; "C" represents the total number of categories; and "D", "H", and "W" represent the feature's dimension, height, and width, respectively. The CMM calculates the cosine similarity between the image feature F and the g obfuscation prototypes stored in the memory. For each f∈F, the similarity can be expressed as:

[0078]

[0079] Among them O u g(u∈{1,...,5}) is the confusion prototype of category g in the memory. This disclosure uses and filters the maximum similarity within the same confusion category as the similarity of that confusion category. Then, the CMM constructs a confusion guidance matrix X by filtering regions with high similarity. Each element in X has the following representation:

[0080]

[0081] Where, x g f Let θ represent the similarity preserved by feature f on the confusing category g. θ is the threshold hyperparameter. The dimension of X is [C×(C-1), H, W], where C is the number of categories, and H and W are the height and width of feature map F, respectively. That is, each element x∈X is represented by a C×(C-1) dimensional vector to represent the possible confusing categories.

[0082] Finally, the CMM maps the similarity in X to the feature space F through convolutional layers and fuses it with F, thereby helping the model reduce potential mispredictions. This fusion process can be represented as follows:

[0083]

[0084] in, For the Hadamard product operation, Conv is a convolutional layer with a 1×1 kernel used for feature dimension transformation, and its output dimension is the same as F. F′ is the output feature of the CMM. During online learning, the parameters of Conv are initialized to zero in this disclosure.

[0085] Semantic interaction difficulty adaptive convolution. In practical applications, the high-difficulty interaction part classes (SClg-cls) vary from image to image, and interactions within the SClg-cls region are usually less efficient. Unlike ordinary image segmentation tasks, in interactive segmentation, the semantic interaction difficulty can be measured by the number of corrected clicks for different human body part classes. This allows for the identification of SClg-cls, the construction and selection of specific parameters for classes with higher interaction difficulty, and the enhancement of robustness to difficult interaction regions while reducing interference from other classes on parameter learning.

[0086] The CACL structure disclosed herein is as follows: Figure 3As shown, "Corr queue" represents the number of corrected clicks in different human body part classes, "C" represents the total number of classes, and "Init" represents the initial network model parameters. CACL constructs a learnable list of convolutional parameters to store the convolutional layer parameters corresponding to each human body part class. Specifically, in the initialization phase without corrected clicks, CACL constructs a separate set of parameters for feature parsing. In each round of interaction, CACL first counts the number of corrected clicks for each class to determine the SClg-cls in the image. Since the metric based on the number of corrected clicks may be biased in some cases (such as annotation errors, multiple human body part classes having high interaction difficulty at the same time), it cannot accurately reflect the interaction difficulty of human body parts. Therefore, this disclosure constructs a class probability distribution based on the number of corrected clicks and determines SClg-cls by sampling, adding a certain degree of randomness to the selection of convolutional parameters and making CACL more robust. Let Nc be the number of corrected clicks for class c∈{1,…,C}, then SClg-cls Mcls can be expressed as:

[0087] Mcls~Cat(P1,...,Pc),with

[0088] Here, Cat(·) represents the category probability distribution that follows the corrected click count. CACL selects parameters corresponding to the category Mcls from the parameter list and performs forward / backward propagation. In particular, to enable CACL to be directly applied to the online learning process, the feature fusion layer of ASPP is replaced with CACL, and its parameters are retained to initialize CACL.

[0089] Example 2:

[0090] like Figure 5 As shown:

[0091] To achieve the above technical objectives, this disclosure also provides an interactive human body parsing system based on correction and misprediction of semantic relevance, the system comprising:

[0092] Input module 201 is used to encode user clicks to generate a positioning map, and concatenate the positioning map with a human RGB image as input to the model;

[0093] The model building module 202 is used to build a neural network with multiple selectable convolutional parameters and obtain the depth features of the human RGB target image through the convolutional neural network; wherein, the neural network is used to parse interactive samples of different categories;

[0094] An online update memory is constructed to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts;

[0095] The iteration module 203 is used to update the contents of the online update memory bank within a preset iteration period using the neural network, and to update and replace the memory prototypes that are not frequently used in the online update memory bank.

[0096] The image segmentation module 204 is used to improve the segmentation results by adding correction clicks to the error areas through user iterations until the human image segmentation is completed.

[0097] Furthermore, the construction of an online update memory for storing historically occurring correction-misprediction pairs specifically includes:

[0098] A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

[0099] Furthermore, the neural network is obtained by modifying the DeepLab V3+ network model as the base model.

[0100] Furthermore, the neural network has an online learning loss function, which is expressed as:

[0101]

[0102] in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks;

[0103] γ is the cost of changing network parameters; γ is the strength of parameter regularization; λ is the loss coefficient.

[0104] Example 3:

[0105] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the steps of the interactive human body parsing method based on correction and misprediction of semantic relevance described above.

[0106] The computer storage medium disclosed herein can be implemented using semiconductor memory, magnetic core memory, magnetic drum memory, or disk memory.

[0107] Semiconductor memory, primarily used in computers, mainly consists of two types of semiconductor storage elements: MOSFETs and bipolar transistors. MOSFETs offer high integration density and simple manufacturing processes but are relatively slow. Bipolar transistors have complex manufacturing processes, high power consumption, and low integration density but are fast. The advent of NMOS and CMOS technologies led to MOSFETs becoming the dominant type of semiconductor memory. NMOS is fast; for example, Intel's 1K-bit static random access memory (SRAM) has an access time of 45ns. CMOS, on the other hand, consumes less power; a 4K-bit CMOS SRAM has an access time of 300ns. The semiconductor memories mentioned above are all random access memories (RAM), meaning they can be randomly read from and written to during operation. Semiconductor read-only memories (ROMs), however, can be randomly read from but not written to during operation; they are used to store pre-programmed programs and data. ROMs are further divided into two types: non-rewritable fuse-type ROMs (PROMs) and rewritable EPROMs (EPROMs).

[0108] Magnetic core memory is characterized by low cost and high reliability, and has over 20 years of practical application experience. Before the mid-1970s, magnetic core memory was widely used as main memory. Its storage capacity could reach 10 bits or more, with the fastest access time being 300 ns. Typical international magnetic core memory capacities ranged from 4 MS to 8 MB, with access cycles of 1.0 to 1.5 μs. Even after the rapid development of semiconductor memory replaced magnetic core memory as the main memory, magnetic core memory can still be used as a large-capacity expansion memory.

[0109] Magnetic drum memory is a type of external storage device that records magnetic data. Due to its fast data access speed and stable, reliable operation, although its capacity is relatively small and it is gradually being replaced by disk storage, it is still used as external storage for real-time process control computers and medium- to large-scale computers. To meet the needs of small and microcomputers, ultra-miniature magnetic drums have emerged, which are small in size, lightweight, highly reliable, and easy to use.

[0110] Disk storage is a type of external storage device that records magnetic data. It combines the advantages of magnetic drums and magnetic tapes: its storage capacity is larger than that of magnetic drums, its access speed is faster than that of magnetic tapes, and it can be stored offline. Therefore, disks are widely used as high-capacity external storage in various computer systems. Disks are generally divided into two main categories: hard disks and floppy disks.

[0111] There are many types of hard disk storage devices. Structurally, they are divided into two types: replaceable and fixed. Replaceable disks have interchangeable platters, while fixed disks have fixed platters. Both replaceable and fixed disks have multi-platter and single-platter structures, and can be further divided into fixed-head and movable-head types. Fixed-head disks have smaller capacities, lower recording densities, and higher access speeds, but are more expensive. Movable-head disks have higher recording densities (up to 1000-6250 bits / inch), resulting in larger capacities, but their access speeds are relatively lower than fixed-head disks. Disk products can have storage capacities of several hundred megabytes, with a bit density of 6250 bits / inch and a track density of 475 tracks / inch. Multi-platter replaceable disk storage devices, due to their replaceable platters, offer very large independent capacity, high speed, and are widely used in online information retrieval systems and database management systems for storing large amounts of information.

[0112] Example 4:

[0113] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the interactive human body parsing method based on correction and misprediction of semantic relevance described above.

[0114] Figure 6 This is a schematic diagram of the internal structure of an electronic device in one embodiment. For example... Figure 6 As shown, the electronic device includes a processor, storage medium, memory, and network interface connected via a system bus. The storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement an interactive human body parsing method based on correcting and mispredicting semantic relevance. The processor provides computational and control capabilities to support the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to implement an interactive human body parsing method based on correcting and mispredicting semantic relevance. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] This electronic device includes, but is not limited to, smartphones, computers, tablets, wearable smart devices, artificial intelligence devices, and power banks.

[0116] In some embodiments, the processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory (e.g., executing remote data read / write programs) and calls data stored in the memory to perform various functions of the electronic device and process data.

[0117] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory and at least one processor, etc.

[0118] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0119] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0120] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0121] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0122] Furthermore, the computer's usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, applications required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0123] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0124] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0126] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An interactive human body parsing method based on correction and misprediction of semantic relevance, characterized in that, include: The user clicks to generate a location map, which is then concatenated with a human RGB image as input to the model. A neural network with multiple selectable convolutional layers is constructed, and the depth features of the human RGB image are obtained through the neural network; wherein, the neural network is used to parse interactive samples of different categories; An online update memory is constructed to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts; The neural network is used to update the contents of the online update memory within a preset iteration period, and unused memory prototypes in the online update memory are updated and replaced. The segmentation results are improved by adding correction clicks to erroneous areas through user iterations until the human image segmentation is completed.

2. The method according to claim 1, characterized in that, The construction of an online update memory bank, which stores historically occurring correction-misprediction pairs, specifically includes: A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

3. The method according to any one of claims 1 or 2, characterized in that, The neural network is obtained by modifying the DeepLabV3+ network model as the base model.

4. The method according to claim 3, characterized in that, The neural network has an online learning loss function, which is expressed as: in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks; The cost of changing network parameters is calculated using γ, where γ is the strength of parameter regularization and λ is the loss coefficient.

5. An interactive human body parsing system based on correction and misprediction of semantic relevance, characterized in that, include: The input module is used to encode user clicks to generate a location map, and concatenates the location map with a human RGB image as the input to the model; The model building module is used to construct a neural network with multiple selectable convolutional parameters and to obtain the depth features of the human RGB image through the neural network; wherein, the neural network is used to parse interactive samples of different categories; An online update memory is constructed to store historically occurring correction-misprediction pairs; wherein, the correction-misprediction pairs are the differences between the annotator and the model in semantic cognition of human body parts; An iterative module is used to update the contents of the online update memory bank within a preset iteration period using the neural network, and to update and replace the memory prototypes that are not frequently used in the online update memory bank. The image segmentation module is used to improve the segmentation results by adding correction clicks to erroneous areas through user iterations until the human image segmentation is completed.

6. The system according to claim 5, characterized in that, The construction of an online update memory for storing historically occurring correction-misprediction pairs specifically includes: A set of feature vectors is constructed for each confusion category and stored in the online update memory respectively.

7. The system according to claim 5 or 6, characterized in that, The neural network is obtained by modifying the DeepLab V3+ network model as the base model.

8. The system according to claim 7, characterized in that, The neural network has an online learning loss function, which is expressed as: in, It is the standard cross-entropy loss calculated using the label gt, where the label gt takes init and corr, where init is the initialization mask and corr is a sparse label containing only the correction clicks; The cost of changing network parameters is calculated using γ, where γ is the strength of parameter regularization and λ is the loss coefficient.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps corresponding to the interactive human body parsing method based on correction and misprediction of semantic relevance as described in any one of claims 1 to 4.

10. A computer storage medium storing computer program instructions thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the steps of the interactive human body parsing method based on correction and misprediction of semantic relevance as described in any one of claims 1 to 4.

Images