A multi-prototype representation enhanced visual pattern recognition method for robots

Abstract

The application provides a robot-oriented multi-prototype characterization enhanced visual pattern recognition method, which can be applied to the fields of artificial intelligence, image processing and signal processing technology. The method comprises the following steps: using a prototype integrated classifier to perform feature extraction on an image training data set for a robot visual pattern recognition task, and obtaining a predicted classification result of a sample image by calculating the pattern similarity between the extracted sample image features and each prototype in a multi-prototype set and by mapping the sample image features; using a label-aware multi-prototype updater to perform target prototype recognition on image data collected in real time by the robot based on the visual pattern recognition task, obtaining a prototype to be updated from the multi-prototype set, performing prototype movement quantitative updating on the prototype to be updated based on the predicted classification result of the sample image, obtaining an updated prototype, and obtaining a classification result of the image data based on the updated prototype.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, image processing and signal processing, and in particular to a multi-prototype representation-enhanced visual pattern recognition method, electronic equipment and storage medium for robots. Background Art

[0002] Visual pattern recognition plays a crucial role in robotics and automated societies, relying on representation learning. However, existing representation learning methods often overlook two key issues: diversity in within-class representations and insufficient label utilization, particularly negative feedback during training. Representation learning and prototype learning methods are commonly used in robotic visual pattern recognition tasks. Existing representation learning methods face the degradation of quality associated with the explosive growth of data volumes. Noise (e.g., outliers) interferes with model training, reducing the reliability of learned feature representations. Furthermore, existing representation learning methods struggle to effectively capture complex distributions within the same category, such as color and texture, leading to insufficient generalization of feature representations. Furthermore, existing prototype learning methods mostly rely on gradient backpropagation, which is prone to small update step sizes during the update process. This reduces the ability of multiple prototypes to maintain diversity during learning, hindering their full potential and failing to fully exploit the potential advantages of the multi-prototype mechanism. Furthermore, existing target-based or conditional prototype update strategies are indirect in receiving prediction feedback and cannot accurately and timely adjust prototypes based on the model's predictions. Moreover, a single prototype for each category directly ignores the rich variations of the data when dealing with intra-class diversity in real-world datasets and cannot comprehensively represent various patterns within the class. Summary of the Invention

[0003] In view of the above problems, the present invention provides a robot-oriented multi-prototype representation enhanced visual pattern recognition method, electronic device and storage medium.

[0004] According to a first aspect of the present invention, a visual pattern recognition method with multi-prototype representation enhancement for robots is provided, comprising: utilizing a prototype ensemble classifier to perform feature extraction on an image training data set used for a robot visual pattern recognition task, and obtaining a predicted classification result of the sample image by calculating the pattern similarity between the extracted sample image features and each prototype in a multi-prototype set and by mapping the sample image features; utilizing a label-aware multi-prototype updater to perform target prototype recognition on image data collected in real time by the robot based on a visual pattern recognition task, obtaining a prototype to be updated from the multi-prototype set, performing a prototype movement quantization update on the prototype to be updated based on the predicted classification result of the sample image to obtain an updated prototype, and obtaining a classification result of the image data based on the updated prototype.

[0005] According to an embodiment of the present invention, the above-mentioned multi-prototype representation enhanced visual pattern recognition method for robots also includes: deriving initial prototypes from an image training data set using a multi-prototype construction function, and constructing a multi-prototype set based on the category of the initial prototype and the intra-class pattern of each initial prototype.

[0006] According to an embodiment of the present invention, the above-mentioned multi-prototype representation enhanced visual pattern recognition method for robots also includes: using a mean square error loss function with a contrast loss regularization term to supervise the data processing process of the prototype ensemble classifier and the data processing process of the label-aware multi-prototype updater to achieve inter-class separation and intra-class diversity of multiple prototypes.

[0007] According to an embodiment of the present invention, the above-mentioned method of obtaining the predicted classification result of the sample image by calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and by mapping the sample image features includes: extracting the sample image features of each sample image from the image training data set using a preset neural network in the prototype ensemble classifier; calculating the inverse of the Euclidean similarity between the sample image features and each prototype in the multi-prototype set to obtain the Euclidean distance; and performing normalized activation processing on the Euclidean distance to obtain the first predicted classification result of the sample image.

[0008] According to an embodiment of the present invention, the above-mentioned method of obtaining the predicted classification result of the sample image by calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and mapping the sample image features also includes: using the fully connected layer classifier in the prototype integration classifier to map the sample image features to obtain a second predicted classification result of the sample image, wherein the fully connected layer classifier is cascaded with the feature extractor of the directly preset neural network; using adjustable parameters to operate on the first predicted classification result and the second predicted classification result of the sample image to obtain the predicted classification result of the sample image, wherein the adjustable parameters are updated by the multi-layer perceptron in the prototype integration classifier.

[0009] According to an embodiment of the present invention, the above-mentioned preset neural networks include a deep convolutional neural network with residual connections, an image classification neural network with a transformer module and a convolution module, a graph structured convolutional neural network, a graph embedding neural network based on random walk and natural language processing technology, and a graph neural network model based on an attention mechanism.

[0010] According to an embodiment of the present invention, the above-mentioned label-aware multi-prototype updater is used to perform target prototype recognition on image data collected in real time by a robot based on a visual pattern recognition task, and the prototype to be updated is obtained from the multi-prototype set, including: constructing an irregular multi-prototype class space based on the categories of the prototypes in the multi-prototype set, and using a regularization term based on unsupervised contrastive learning to calculate the similarity between each prototype in the multi-prototype class space to achieve inter-class separation, and obtaining a multi-prototype set after comparison optimization; using the label-aware multi-prototype updater to select the prototype closest to the instance represented by the image data from the multi-prototype class space as the prototype to be updated; using the memory-based update counter in the label-aware multi-prototype updater to track the number of instances allocated to each prototype in the multi-prototype class space; when the number of instances allocated to the current prototype by the memory-based update counter is less than a preset threshold, the current prototype is set to an outdated prototype, and the current prototype is reintegrated into the representation space.

[0011] According to an embodiment of the present invention, the above-mentioned prototype movement quantization update based on the predicted classification result of the sample image to be updated to obtain the updated prototype includes: when the predicted classification result of the sample image is represented by a negative feedback label, the preset control update amount and the preset adjustment movement scale are used to perform a prototype movement quantization update based on the moving average on the prototype to be updated and the sample image features corresponding to the sample image to obtain the updated prototype; when the predicted classification result of the sample image is represented by a positive feedback label, the preset control update amount is used to perform a prototype movement quantization update based on the moving average on the prototype to be updated and the sample image features corresponding to the sample image to obtain the updated prototype.

[0012] A second aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0013] The third aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0014] The robot-oriented multi-prototype representation-enhanced visual pattern recognition method provided by the present invention utilizes a multi-prototype set to achieve irregular but compact class boundaries, improve intra-class diversity, and solve the technical problems that existing representation learning ignores intra-class representation diversity and existing single-prototype learning methods are unable to adapt to intra-class representation diversity. At the same time, the present invention utilizes a label-aware multi-prototype updater to separate representation learning and prototype updating, realizing a label-guided adaptive update strategy. In addition, the present invention accelerates the convergence speed of the training process and optimizes feature extraction through a prototype ensemble classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0016] Figure 1 is a schematic illustration of intra-class diversity according to an embodiment of the present invention;

[0017] Figure 2 is a flow chart of a visual pattern recognition method enhanced by multiple prototype representations for robots according to an embodiment of the present invention;

[0018] Figure 3 2 is a framework diagram of label-aware multi-prototype learning according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of a prototype update according to an embodiment of the present invention;

[0020] Figure 5 2. It is a schematic diagram showing the effect of the number of prototypes on the classification results according to an embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of prototyping visualization of ResNet-50 on CIFAR-10 according to an embodiment of the present invention;

[0022] Figure 7 4 is a block diagram of an electronic device suitable for implementing a multi-prototype representation enhanced visual pattern recognition method for robots according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] Commonly used methods for robot visual pattern recognition tasks include representation learning and prototype learning. To better understand the method provided by the present invention, the following describes the representation learning method and the prototype learning method, respectively, and also explains the technical problems existing in existing representation learning methods and prototype learning methods.

[0028] Representation learning refers to how to convert or encode raw data (such as image pixels, text words, and sensor readings) into a new form. Its core purpose is to automatically discover the optimal feature representation of the data. In AI applications, neural network-based representation learning plays a crucial role, enabling high-quality recognition and pattern extraction in graphics and image signal processing. By learning from large amounts of data, neural networks attempt to transform complex graphics and image information into feature representations that computers can understand and process, providing strong support for subsequent tasks such as classification and recognition.

[0029] However, model performance isn't solely determined by a carefully designed neural architecture. Data quality plays a crucial role. With technological advancements, data volumes have exploded. Unfortunately, this increase in data volume hasn't been accompanied by a simultaneous improvement in data quality. On the contrary, excessive data places a heavy computational burden on model training and introduces a significant amount of noise, which can interfere with the model's learning process, compromising its accuracy and stability.

[0030] Against this backdrop, emerging real-world datasets exhibit three notable characteristics. First, there's the explosion of data volume: the rapid expansion of data scale is exceeding the processing capabilities of traditional models. Second, there's outlier noise: the presence of abnormal data within the dataset that significantly deviates from the overall data distribution, misleading model learning. Finally, there's intra-class pattern diversity: even within the same category, patterns can vary widely. For example, in image datasets, different images of the same object category can exhibit significant differences in color, shape, texture, and other aspects. These characteristics collectively constitute inherent challenges facing learning models.

[0031] Most existing representation learning methods focus on modifying the learning objective. For example, the combination of cross-entropy and contrastive loss promotes intra-class diversity, allowing the model to focus on the different features of data within the same category during learning, thereby better accommodating intra-class differences. Contrastive loss, on the other hand, simultaneously expands inter-class diversity and minimizes intra-class diversity, allowing data of different categories to be more separated in the feature space and data of the same category to be more clustered, which helps improve the model's classification accuracy. Furthermore, there is a triplet loss design that learns representations in a compact Euclidean space through instance-level similarity. This approach leverages the similarities and differences between data instances to guide the model to learn more discriminative feature representations. These well-learned representations have been widely used in a wide range of tasks, from recognition and verification to clustering.

[0032] Prototype learning methods, among others, achieve classification or other tasks by finding typical representatives (called "prototypes") for each category, rather than directly learning complex decision boundaries. Given the complex nature of real-world datasets, prototype learning has emerged to effectively address the adverse effects of exploding data volumes and prominent outliers on model learning. Essentially, the core goal of prototype learning is to select samples with a high signal-to-noise ratio within each data category that represent the typical characteristics of that category. The commonalities among these samples are then summarized and summarized to form a representative representation. In this way, prototype learning effectively filters out noise in the data, allowing the model to focus on more valuable information during the learning process, thereby improving learning and inference efficiency.

[0033] In recent years, researchers have discovered through in-depth studies of prototype learning that the functions of prototypes are not limited to improving learning efficiency and refining data. They can also play a positive role in model design, such as helping to achieve extremely challenging tasks such as few-shot learning. In few-shot learning scenarios, due to the very limited available training data, traditional learning methods often fail to achieve ideal results. Prototype learning, by summarizing the common features of categories, enables the model to quickly capture key information even with a small number of samples, thereby achieving accurate classification and recognition of new samples. Based on these advantages, incorporating prototypes into deep models has become an effective way to improve representation quality.

[0034] In practical applications, prototype learning can be divided into two main directions: single prototype and multiple prototypes. The single prototype method was first proposed, and its original intention was to summarize the commonalities of instances within a class and realize dynamic prototype updates. In practical applications, the single prototype method does provide a feasible way to effectively capture more robust patterns, while also reducing memory usage during inference. For example, in some simple image classification tasks, the single prototype method can quickly determine the typical features of a category of images, thereby improving classification efficiency. However, when faced with the complex intra-class diversity in real-world datasets, the single prototype strategy exposes inherent flaws. Due to the existence of multiple different patterns within the same category of data, it is difficult for a single prototype to fully cover these differences, resulting in poor performance in forming a reasonable but irregular feature space to adapt to intra-class diversity.

[0035] To overcome the shortcomings of single-prototype approaches, an emerging research area has proposed a multi-prototype solution. The core idea of ​​this approach is to assign multiple prototypes to each category, using multiple prototypes to represent different patterns within the same category, thereby promoting diversity within the category. For example, in a dataset containing images of various dog breeds, multiple prototypes can represent the characteristics of each breed separately, thereby more comprehensively covering the various variations within the category.

[0036] In terms of prototype updating, there are currently two commonly used methods. One is to develop a conditional and rule-based method to regulate the update process, and by setting specific conditions and rules, determine under what circumstances and how the prototype should be updated. The other is to design various optimization objectives to guide the update, and formulate corresponding optimization objectives according to different task requirements and data characteristics, so that the prototype is updated in a direction that is more conducive to task completion. The concept of prototype learning originally originated from Learning Vector Quantization (LVQ) based on neural networks. The core of LVQ is to classify input data by adjusting a set of prototype vectors. These prototype vectors represent different categories, and their position and distribution in the feature space determine the boundaries of the classification. During the training process, LVQ finds the prototype vector (i.e., the winning neuron) closest to the input vector based on the distance between the input vector and the prototype vector, and then adjusts the position of the prototype vector based on the true category of the input vector and the category relationship between the winning prototype vector. For each input vector, calculate the distance between it and all prototype vectors. Euclidean distance is usually used to measure the similarity between vectors, and the formula is ,in is the input vector, It is prototype vectors, and They are respectively A quantity, Is the dimension of the vector. Update the prototype vector according to the true category of the input vector and the category relationship of the winning prototype vector. If the category of the input vector is the same as the category of the winning prototype vector, move the winning prototype vector towards the direction of the input vector. , making it closer to the input vector, the update formula is shown as (1):

[0037] (1), where Indicates the training round.

[0038] If the two categories are different, the winning prototype vector is moved in the direction away from the input vector. The update formula is shown in (2):

[0039] (2). Here The learning rate decreases gradually as training progresses, controlling the step size of each update and ensuring gradual convergence of the algorithm. By calculating distances, the prototype vector closest to the input vector is found, representing the winning neuron. Prototype learning, developed based on LVQ, derives prototypes from high-signal-to-noise ratio samples, distilling the commonalities of high-quality samples into representative representations. This representation not only filters out noise but also greatly facilitates model inference.

[0040] Figure 1 is a schematic diagram illustrating intra-class diversity according to an embodiment of the present invention.

[0041] The following is combined with Figure 1 The technical problems of existing prototype learning methods are explained.

[0042] In the big framework of representation learning, there are two key issues that seriously hinder the model from achieving better representation effects. These two issues are representation diversity and insufficient label utilization.

[0043] In terms of representation diversity, intra-class diversity is a common and unavoidable problem in real-world datasets. Figure 1 For example, the three dog photos in the image above show distinct patterns in shape and color, even though they all belong to the dog category. This intra-class variation presents a significant challenge for model learning. If the model doesn't fully consider the diversity of intra-class representations during learning, it won't be able to form irregular yet compact boundary shapes that can accommodate the differences between previously seen instances of the same category. This means that when faced with new, intra-class variations, the model is prone to misjudgment and cannot accurately identify the sample's category.

[0044] In terms of label utilization, although various label-aware and label distribution learning methods have been developed by inserting label learning objectives, these methods still have significant shortcomings. The current objective-based optimization method gradually adjusts the learnable parameters in small steps, which makes the model update process slow and is easily affected by the randomness of gradient descent. In the actual learning process, according to the law of natural learning, negative feedback can leave a deeper impression on the learner, prompting them to learn lessons and make more effective improvements. However, existing label utilization methods do not fully utilize this law and lack a more direct label utilization method. In particular, the emphasis on negative feedback is insufficient, resulting in an inability to effectively promote the correction of model updates, which in turn affects the improvement of model performance.

[0045] While multi-prototype solutions offer new hope for addressing representation learning challenges, several challenges remain in practical applications. Research results indicate that existing multi-prototype solutions fail to fully exploit the potential advantages of the multi-prototype mechanism. A thorough analysis reveals that this is primarily due to the reliance of current prototype updating methods on gradient backpropagation, which can easily become trapped in small update steps during the update process. This reduces the ability of multiple prototypes to maintain diversity during learning and hinders their full potential. Furthermore, existing target-based or conditional prototype updating strategies are indirect in receiving prediction feedback and cannot promptly and accurately adjust prototypes based on the model's predictions. Furthermore, a single prototype per class directly ignores the rich variations in the data when addressing the intra-class diversity found in real-world datasets, failing to fully represent the various patterns within the class. In summary, while multi-prototype solutions hold great potential and promise for achieving representation diversity and high-quality label utilization, they face numerous challenges in practical applications, requiring further research and improvement.

[0046] The present invention proposes a multi-prototype learning method based on label perception - LamPro, which improves the representation quality by integrating label perception into the formation and update of prototypes. For robots, in complex working environments, accurately identifying different objects and scenes is a key prerequisite for their efficient execution of tasks. In order to achieve the separation between categories during robot visual recognition, the present invention draws on the idea of ​​contrastive learning and constructs a contrastive loss regularization term in the loss function. In terms of maintaining intra-class diversity, considering that objects of the same category faced by the robot may have multiple differences in appearance and form, the present invention first randomly initializes multiple prototypes for each category, and proposes a simple and effective update strategy to improve the prototype granularity from the category level to the pattern level. The main technical problems to be solved by the present invention are as follows:

[0047] (1) In the process of robot visual pattern recognition, how to use prototypes to construct irregular but compact class boundaries with inter-class separation and intra-class diversity, so that the robot can accurately distinguish objects of different categories and accommodate the various morphological differences of objects of the same category;

[0048] (2) How to make full use of labels to establish an active feedback mechanism for the robot so as to update the prototype in a timely manner and effectively adjust the model, so that the robot can continue to maintain high-accuracy visual recognition capabilities when facing a dynamically changing environment.

[0049] Figure 2 4 is a flowchart of a multi-prototype representation enhanced visual pattern recognition method for robots according to an embodiment of the present invention.

[0050] like Figure 2As shown, the above-mentioned multi-prototype representation enhanced visual pattern recognition method for robots includes operations S210 to S220.

[0051] In operation S210, a prototype ensemble classifier is used to extract features from an image training dataset for a robot visual pattern recognition task, and a predicted classification result of the sample image is obtained by calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and by mapping the sample image features.

[0052] In operation S220, a label-aware multi-prototype updater is used to perform target prototype recognition on image data collected in real time by the robot based on a visual pattern recognition task, and a prototype to be updated is obtained from a multi-prototype set. The prototype to be updated is subjected to prototype movement quantization update based on the predicted classification result of the sample image to obtain an updated prototype, and a classification result of the image data is obtained based on the updated prototype.

[0053] According to an embodiment of the present invention, the above-mentioned multi-prototype representation enhanced visual pattern recognition method for robots also includes: deriving initial prototypes from an image training data set using a multi-prototype construction function, and constructing a multi-prototype set based on the category of the initial prototype and the intra-class pattern of each initial prototype.

[0054] According to an embodiment of the present invention, the above-mentioned multi-prototype representation enhanced visual pattern recognition method for robots also includes: using a mean square error loss function with a contrast loss regularization term to supervise the data processing process of the prototype ensemble classifier and the data processing process of the label-aware multi-prototype updater to achieve inter-class separation and intra-class diversity of multiple prototypes.

[0055] The robot-oriented multi-prototype representation-enhanced visual pattern recognition method provided by the present invention utilizes a multi-prototype set to achieve irregular but compact class boundaries, improve intra-class diversity, and solve the technical problems that existing representation learning ignores intra-class representation diversity and existing single-prototype learning methods are unable to adapt to intra-class representation diversity. At the same time, the present invention utilizes a label-aware multi-prototype updater to separate representation learning and prototype updating, realizing a label-guided adaptive update strategy. In addition, the present invention accelerates the convergence speed of the training process and optimizes feature extraction through a prototype ensemble classifier.

[0056] The following is a detailed description of the embodiments and the attached Figure 3 The above-mentioned robot-oriented multi-prototype representation enhanced visual pattern recognition method provided by the present invention is further described in detail.

[0057] Figure 3 4 is a framework diagram of label-aware multi-prototype learning according to an embodiment of the present invention.

[0058] The present invention implements a label-aware multi-prototype learning architecture for image classification tasks. Given a training set ,in It's an image. is the category label. is an unseen set reserved for testing (a set of data samples that the model has never learned). The present invention designs a multi-prototype construction function , to export a prototype set ,in is the total number of categories in the training set, is the number of prototypes assigned to each category, Indicates the The first category Prototype, and a label-aware update strategy (or label-aware update function) , in order to dynamically obtain the latest prototype. Finally, the present invention can distribute Get the prediction results The process can be expressed by formula (3):

[0059] (3),

[0060] in, Represents the composite operation of functions, Represents a multi-prototype construction function Parameters, Represents a label-aware update strategy Parameters.

[0061] To simultaneously model intra-class diversity and achieve label awareness in representation learning, we propose LamPro, a label-aware multi-prototype learning strategy. Figure 3 An overview of the proposed LamPro is shown. Figure 3 As shown, LamPro of the present invention mainly consists of two components: a prototype ensemble classifier , for explicitly incorporating prototype-based solutions into the final prediction; and a multi-prototype optimization module for fine-grained and timely prototype updating.

[0062] According to an embodiment of the present invention, the above-mentioned method of obtaining the predicted classification result of the sample image by calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and by mapping the sample image features includes: extracting the sample image features of each sample image from the image training data set using a preset neural network in the prototype ensemble classifier; calculating the inverse of the Euclidean similarity between the sample image features and each prototype in the multi-prototype set to obtain the Euclidean distance; and performing normalized activation processing on the Euclidean distance to obtain the first predicted classification result of the sample image.

[0063] According to an embodiment of the present invention, the above-mentioned steps of calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and mapping the sample image features to obtain the predicted classification result of the sample image further include:

[0064] Mapping the sample image features using the fully connected layer classifier in the prototype ensemble classifier to obtain a second predicted classification result of the sample image, wherein the fully connected layer classifier is cascaded with a feature extractor of a directly preset neural network;

[0065] The first prediction classification result and the second prediction classification result of the sample image are calculated using the adjustable parameters to obtain the prediction classification result of the sample image, wherein the adjustable parameters are updated by the multi-layer perceptron in the prototype ensemble classifier.

[0066] According to an embodiment of the present invention, the above-mentioned preset neural networks include a deep convolutional neural network with residual connections, an image classification neural network with a transformer module and a convolution module, a graph structured convolutional neural network, a graph embedding neural network based on random walk and natural language processing technology, and a graph neural network model based on an attention mechanism.

[0067] The prototype ensemble classifier provided by the present invention is further described in detail below through specific embodiments.

[0068] To model intra-class diversity, the present invention utilizes a multi-prototype scheme to accommodate different patterns within the same class. However, in prototype learning, the initialized prototypes serve only as placeholders for pattern learning and will be gradually updated through subsequent prototype optimization. Therefore, the present invention utilizes an ensemble classifier consisting of a prototype-driven classifier and a classifier driven by a Convolutional Neural Network (CNN) feature extractor, such as Figure 3 As shown in Figure 2, the final prediction is achieved jointly, which can maximize the capabilities of prototypes and CNN-based representations. Next, these two branches are elaborated in detail.

[0069] Assume that in a given task there are categories, each of which has Mode, first initialize prototypes, which will be gradually updated using a preset strategy. In fact, the CNN-based classifier and prototype initialization are interdependent. Given an image sample , the present invention first uses a CNN-based extractor (in, express Parameters, Representing image samples) extracts its feature representation, and then uses the extractor Specifically, the present invention selects Euclidean distance as affinity metric, calculates the distance by calculating the inverse of the Euclidean similarity between the extracted feature and all prototypes, and then selects the prototype with the closest distance to the extracted feature as the most likely prototype, and its corresponding category is the predicted category. In order to obtain the predicted probability, the present invention further calculates the derived distance Apply the Softmax function to normalize the probability to The specific implementation is shown in formulas (4) and (5):

[0070] (4),

[0071] (5),

[0072] in, Represents an image sample The probability of being classified into a category by the prototype-driven classifier. Defined as the prototype of the true label ( The first category prototype), it can be concluded that the learning objectives of the present invention will force Minimize and make Maximization. Notably, this process is very clever, enabling iterative mutual reinforcement. Specifically, the prototype is determined and optimized by the CNN extractor, and during the learning process, the feature representation tends to approach the prototype. Subsequently, thanks to the prototype, the present invention can summarize representations from different samples into a compact prototype, thereby significantly filtering out noise and significantly improving the representation effect. Therefore, the present invention can fully utilize the prototype to enhance CNN-based representations.

[0073] Even so, if the model is completely guided by the prototype in the initialization phase, the results may not be ideal because the prototype has not been well updated at this time. To avoid this, the present invention integrates a classifier based on a fully connected layer, which is directly cascaded with the CNN-based extractor and maps the representation to a probability value. In this way, the learning process of the present invention becomes an integrated classifier, in which the importance of the two classifiers is calculated by Fusion is performed as shown in formula (6):

[0074] (6),

[0075] in, is the predicted probability given by the prototype, and is the prediction result of CNN-Multi-layer Perceptron (MLP) classifier. The present invention first gives it a close and let it gradually increase during the training process. near When the initial prototype is not perfect and reliable, the CNN-based classifier can dominate the final result. As the model is trained, the prototype becomes reliable, and the present invention further increases , so that the model gradually benefits from the advantages of prototype learning. In the final stage of training, the present invention reduces the scale of the MLP classifier to zero and When prototype prediction is fully utilized, both inter-class separation and intra-class diversity can be fully achieved.

[0076] Using MLP to adjust the parameters The technical advantages of the adjustment are: first, the present invention can use MLP to accelerate the model convergence speed and obtain better feature representation extraction; second, When is small, the model of the present invention (including the prototype ensemble classifier and the label-aware multi-prototype updater) has buffer time to gradually update the prototype to a reliable template.

[0077] According to an embodiment of the present invention, the above-mentioned label-aware multi-prototype updater is used to perform target prototype recognition on image data collected in real time by a robot based on a visual pattern recognition task, and the prototype to be updated is obtained from the multi-prototype set, including: constructing an irregular multi-prototype class space based on the categories of the prototypes in the multi-prototype set, and using a regularization term based on unsupervised contrastive learning to calculate the similarity between each prototype in the multi-prototype class space to achieve inter-class separation, and obtaining a multi-prototype set after comparison optimization; using the label-aware multi-prototype updater to select the prototype closest to the instance represented by the image data from the multi-prototype class space as the prototype to be updated; using the memory-based update counter in the label-aware multi-prototype updater to track the number of instances allocated to each prototype in the multi-prototype class space; when the number of instances allocated to the current prototype by the memory-based update counter is less than a preset threshold, the current prototype is set to an outdated prototype, and the current prototype is reintegrated into the representation space.

[0078] According to an embodiment of the present invention, the above-mentioned prototype movement quantization update based on the predicted classification result of the sample image to be updated to obtain the updated prototype includes: when the predicted classification result of the sample image is represented by a negative feedback label, the preset control update amount and the preset adjustment movement scale are used to perform a prototype movement quantization update based on the moving average on the prototype to be updated and the sample image features corresponding to the sample image to obtain the updated prototype; when the predicted classification result of the sample image is represented by a positive feedback label, the preset control update amount is used to perform a prototype movement quantization update based on the moving average on the prototype to be updated and the sample image features corresponding to the sample image to obtain the updated prototype.

[0079] The following is a detailed description of the embodiments and the attached Figure 4 The multi-prototype optimization process provided by the present invention is further described in detail.

[0080] Figure 4 2 is a schematic diagram of a prototype update according to an embodiment of the present invention.

[0081] in, Figure 4 (a) in the example represents the case of an outdated prototype. Figure 4 (b) in the figure shows the schematic diagram of the adjustment process of the outdated prototype; Figure 4 The middle circle represents the sample image feature representation, the asterisk represents the prototype, the triangle represents the new representation instance, and the asterisk with an exclamation mark in the upper right corner indicates that the prototype representation is outdated.

[0082] In fact, most data in nature obey Gaussian distribution, with most data concentrated near the expected value and only a few data scattered at the edge. Therefore, in classification tasks, the virtual class boundary is often determined by a few edge data far away from the concentration point. Therefore, how to construct an irregular class space from a complex data distribution has always been a challenge. In this work, the present invention utilizes the summarization and noise resistance of prototypes to construct multiple prototypes for each category to formalize the irregular class space. Then, the present invention elaborates on the solution of dynamically updating the prototype by receiving real-time input representation, where the core goal of the prototype representation is to achieve inter-class separation and intra-class diversity. In fact, the prototype optimization process of the present invention can generally be divided into three parts: contrast-based optimization, determining the updated prototype, and calculating the movement amount of each updated prototype. Contrast-based inter-class separation: First, in order to achieve inter-class separation, the present invention draws on the idea of ​​unsupervised contrastive learning and proposes a regularization term to maintain this separation, as shown in formulas (7) and (8):

[0083] (7),

[0084] (8),

[0085] in, is a small batch size, is an indicator function, i.e., if and only if the sample and When belonging to the same label, the indicator function is 1, otherwise it is 0. Specifically, represents the similarity between representations, is the temperature hyperparameter, Represents an image sample The extracted features, Represents an image sample The extracted features, Represents an image sample The true label, Represents an image sample The true label of the dataset is obtained from the dataset. This contrast-based regularization ensures that the representations of different classes are as separate as possible, while the representations of the same class are sufficiently compact. Multi-Prototype Updater: We propose a label-aware multi-prototype updater that adaptively optimizes multiple prototypes for each class to address the intra-class diversity problem. The updater assigns new instances to prototypes and operates in two stages: target prototype identification and prototype shift quantization.

[0086] Target Prototype Identification: When a new instance arrives, the updater selects the closest prototype to update to maintain intra-class diversity. However, inactive prototypes may become outdated and deviate from the representation space. To prevent this, the present invention uses a memory-based counter to track the number of instances assigned to each prototype. The number of instances is less than the threshold The prototypes of are considered outdated and are reintegrated into the representation space until they regain sufficient sample representation, such as Figure 4 shown.

[0087] Prototype movement quantification: In this stage, the present invention quantifies the amount that each prototype should move at each update. The present invention emphasizes the importance of negative feedback (misclassification) in forming class boundaries, giving greater weight to misclassified instances. This label-aware approach ensures that the prototype is adjusted based on correct and incorrect predictions, optimizing class boundaries without affecting the classifier. When the prototype-based classifier makes a correct decision, the present invention will perform a moving average-based update process on the corresponding prototype. Given the first The first in the category Prototype and image samples (or instances) Extracted features , the present invention can obtain a prototype and The linear combination of is shown in formula (9):

[0088] (9),

[0089] in is the control update amount. When the prototype-based classifier makes an incorrect decision, this not only indicates that the quality of the existing prototype representation is low, but also means that the representation space of the category is poor and unreasonable. Fortunately, the first problem can be solved by utilizing gradient descent through the learning objective of the present invention. For the second problem, the present invention uses the representation of misclassified samples to adjust the category representation space, which emphasizes the error feedback and narrows the gap between the unreasonable prototype and the corresponding category representation space. Specifically, the present invention pushes the prototype away from the sample representation while bringing the prototype closer to the correct category representation space. The present invention can modify formula (10) to formalize this update process:

[0090] (10),

[0091] in is a misclassified prototype, This invention specifically introduces a positive integer To adjust the movement scale, Push the prototype significantly out of the wrong representation space, where the parameters 、 satisfy Through this solution, the present invention can place more emphasis on the update amount of misclassified samples to correct the classification while pushing away the wrong prototype.

[0092] LamPro, provided by the present invention, is applied to classification tasks, aiming to achieve higher classification accuracy and lower classification loss. Here, the present invention selects the commonly used mean square error (MSE) as the loss function and adds a contrastive loss regularization term to achieve inter-class separation and intra-class compactness. For a total of N samples, the final loss function of the present invention is shown in Equations (11) and (12):

[0093] (11),

[0094] (12).

[0095] In order to better illustrate the advantages of LamPro provided by the present invention, specific experiments are performed below to verify it.

[0096] The present invention uses six datasets, including graphs and images, which are common signals in robotic vision tasks, and selects basic models for image and graph learning as baselines.

[0097] Regarding datasets, since visual signals in the open world can generally be divided into graphs and images, we evaluate our method on both image and graph datasets. Three mainstream image datasets for recognition tasks include CIFAR-10, CIFAR100, and Caltech-101. Three widely used real-world graph datasets (graph datasets with node, edge, content features, and label information) include the Cora Graph Dataset, the Citeseer Graph Dataset, and the Pubmed Graph Dataset. Detailed statistical information is shown in Table 1.

[0098] Table 1: Dataset statistics

[0099]

[0100] Regarding baselines and backbone networks, we selected multiple backbone networks to evaluate our solution. We employed two classic deep learning architectures for image classification and three graph-based learning frameworks for graph-level classification. We combined our solution with these backbone networks, analyzed performance changes, and conducted ablation studies.

[0101] ResNet (Residual Network): A type of deep CNN model with residual connections. In this work, we used ResNet-18 and ResNet50 as the backbone networks for evaluation (where 18 and 50 represent the total number of layers between convolutional and fully connected layers in the residual network).

[0102] CvT (Convolutional Vision Transformers): A transformer-based architecture that combines transformer blocks with convolutional blocks for image classification. Here, CvT-7 is used.

[0103] GCN (Graph Convolution Neural Networks): A classic baseline for modeling graph-structured data that uses GCN layers to aggregate information about a node’s local neighborhood and update node embeddings.

[0104] DeepWalk: A node embedding strategy that samples nodes by using random walks in a neural network and treats the walks as sentences, learning distributed node representations via a skip-gram model.

[0105] GAT (Cross Attention Transformer): It incorporates the self-attention mechanism into GCN to learn node embeddings. It uses the relationship between nodes to assign attention scores to each neighbor for information aggregation.

[0106] Results: The performance of different methods on image datasets is summarized in Table 2, and the performance on graphic datasets is summarized in Table 3. Due to space limitations, this paper only reports The reported results clearly show the three backbone networks with and without prototype updates. When it is simplified to a single prototype, the present invention has the following three observations.

[0107] Table 2: Performance results on image datasets (best results are in bold)

[0108]

[0109] Table 3: Performance results on graph datasets

[0110]

[0111] Performance comparison: When comparing baselines with and without prototypes, prototype-based representation learning can generally outperform the baseline without prototypes by 1.64%-4.41%. Specifically, the present invention finds that the performance of multiple prototypes has significant advantages over a single prototype, for example, The performance is better than This demonstrates the robustness of the intra-class diversity and multi-prototype solution. ResNet-18 achieves a nearly 3.5% performance improvement on CIFAR-10, with improvements of approximately 2% common across other benchmarks. This not only confirms the existence of multiple patterns within the same class, but also demonstrates the reliability of the proposed multi-prototype update strategy.

[0112] Figure 5 Schematic diagram of the effect of the number of prototypes on classification results according to an embodiment of the present invention.

[0113] in, Figure 5 (a) shows the effect of the number of prototypes of the image classification model on the classification results. Figure 5 (b) in the figure shows the effect of the number of prototypes on the classification results of the graph node classification model.

[0114] different (Right now Figure 5 The impact of K in the figure (representing the number of prototypes): The present invention allows different numbers of prototypes to be The parameters of the same model are varied and Figure 5 The present invention gradually increases The performance is improved accordingly. Generally, the present invention selects ResNet-18 and ResNet50 on the CIFAR-10 dataset, and GCN and GAT on the Cora dataset for visualization. The present invention finds that on the image dataset, when the number of prototypes reaches 7, the improvement of model performance reaches its limit, while in the graph node classification, when When the value is greater than 13, the performance improvement rate slows down.

[0115] Differences between the two types of tasks: The reason behind this difference is attributed to the presence of more complex patterns in the graph.

[0116] For example, the Cora graph dataset is a citation network consisting of scientific publications from various disciplines. It contains 2,708 scientific papers, each represented as a binary bag-of-words feature vector indicating the presence or absence of a word in the corresponding document. This complexity can be reflected by the local neighborhood aggregation of GNNs, which includes neighbors and the node itself. As a result, the graph does have more diverse patterns.

[0117] Figure 6 3 is a schematic diagram of prototyping visualization of ResNet-50 on CIFAR-10 according to an embodiment of the present invention.

[0118] In terms of representation visualization: This paper uses dimensionality reduction and visualization technology to map the prototype into a two-dimensional space for display. This paper selects a typical task, that is, using ResNet-50 to perform representation learning on CIFAR-10. The results are as follows Figure 6 As shown in the figure, the present invention clearly shows that there is not only sufficient distance space between different categories, but also clear boundaries for each category. Compared with the representation based on a single prototype, the method of the present invention has more space to accommodate the diversity of samples within different classes. In this case, the diverse representation space provided by multiple prototypes can have stronger representation capabilities. The visualization results demonstrate the quality of the multi-prototype learning of the present invention and further confirm the interpretability of LamPro when applied to key visual recognition tasks of autonomous robots.

[0119] In terms of hyperparameter setting: for the update amount , the present invention conducted a series of analysis experiments, and the results are shown in Table 4, including the impact on images (Impacts on images) and the impact on graphs (Impacts on graphs). In the experimental setting of the present invention, the number of prototypes is uniformly set to 20, and five different The experimental results show that the present invention finds that A value that is too large or too small will not achieve optimal performance. Specifically, when When is too large, the prototype is overly dependent on the representation of subsequent instances, resulting in insufficient retention of previous representation information and poor generalization ability. On the contrary, when If it is too small, the prototype will not be updated enough, resulting in slow update speed and unsatisfactory performance. Finally, the present invention selects an intermediate value and sets In the settings, the hyperparameters Searching between 10 and 25, the present invention observed that when The present invention also sets a threshold for the number of samples of outdated prototypes. Set multiple test values ​​and find out through experiments Optimal performance.

[0120] Table 4: Impact on performance

[0121]

[0122] The present invention proposes a multi-prototype representation-enhanced visual pattern recognition method for robots. For each newly arrived instance, the geometrically closest prototype is typically updated to maximize the preservation of the original shape of the overall class boundary, thereby suppressing interference between distant instances within the same class. In this way, well-learned prototypes within each class can collectively establish an irregular yet compact class boundary. Secondly, to achieve efficient updates and emphasize negative feedback, the present invention utilizes informative labels and proposes a label-guided adaptive update strategy. First, the present invention separates representation learning and prototype updating to avoid the unwieldy gradient propagation caused by discrete prototype labels and ensures more flexible and direct predictive feedback for the model. To determine whether the model is optimizing in the right direction, the present invention refines the update strategy based on different predictive feedback. When the label identifies positive feedback, the present invention updates the closest prototype to the newly arrived instance, which can be considered a beneficial prototype that dominates the results. However, this closest update strategy can introduce a bias problem: due to imperfect initialization and uneven feature distribution of incoming samples, the prototype may deviate from its intrinsic representation space. The present invention refers to such deviated prototypes as outdated prototypes and designs a memory-based update counter to alternately update these outdated prototypes until they are no longer in this state. When negative feedback arrives, the present invention uses labels to identify the correct prototype and selects the geometrically closest prototype for corresponding instance-level update, thereby achieving efficient and negative feedback-aware updates. Extensive experiments on six datasets show that LamPro achieves competitive performance compared to baseline methods, demonstrating its superiority in graphic and image visual recognition and providing support for robotic automation.

[0123] The robot-oriented multi-prototype representation-enhanced visual pattern recognition method provided by the present invention has the following technical advantages:

[0124] (1) Solve the problem of intra-class diversity: Existing representation learning often ignores the diversity of intra-class representations, and the single prototype method is difficult to adapt. This paper randomly initializes multiple prototypes for each category, constructs irregular but compact class boundaries, and improves intra-class diversity. Experiments show that on multiple datasets, multiple prototypes ( ) compared to the single prototype ( ) The performance is significantly improved, such as ResNet-18 on CIFAR-10, The accuracy is 91.25±0.88 when K=20, and it increases to 93.56±0.37 when K=20, proving that the multi-prototype strategy can effectively accommodate intra-class differences.

[0125] (2) Optimizing label utilization: Traditional methods do not fully utilize labels and are affected by the randomness of gradient descent. This paper proposes a label-guided adaptive update strategy that separates representation learning and prototype updating, updates prototypes based on predicted feedback, and emphasizes negative feedback. When the label identifies positive feedback, the most recent prototype is updated; when negative feedback is identified, the correct prototype is updated. A memory-based update counter is also designed to handle outdated prototypes, making model updates more efficient and reasonable.

[0126] (3) Improving model performance and generalization: This paper combines contrastive learning to construct a loss function and adds a contrastive loss regularization term to achieve inter-class separation and intra-class compactness. Experiments on six datasets show that prototype-based representation learning outperforms the prototype-free baseline by 1.64%-4.41%. At the same time, by adjusting the number of prototypes K, it is found that performance improves as K increases. In the image and graph node classification tasks, the performance improvement tends to be stable when K is 7 and greater than 13, respectively. This proves that this method performs well on different types of data and has strong generalization ability.

[0127] (4) Accelerate model convergence and optimize feature extraction: Design a prototype ensemble classifier, consisting of a prototype-driven classifier and a CNN feature extractor-driven classifier, and fuse the capabilities of both through the hyperparameter α. In the early stages of training, when α is close to 0, the CNN-based classifier dominates the results. As training increases, α is increased to fully utilize the advantages of prototype learning. This not only accelerates model convergence but also obtains better feature representation extraction.

[0128] The advantages and effectiveness of the LamPro provided by the present invention are further verified in detail through specific experiments below.

[0129] In terms of model construction and training: In the image classification task, given a training set ,in It's an image. is the category label, and there is an unseen set left for testing Obtain relevant image and graphic data from actual robot application scenarios, such as image data of robots identifying parts in industrial production environments, or monitoring image data in intelligent security scenarios, and organize them according to the above format.

[0130] LamPro mainly consists of prototype ensemble classifiers and a multi-prototype optimization module (including a label-aware multi-prototype updater).

[0131] Prototype ensemble classifier: Assume that there are categories, each with Mode, initialization prototypes. Taking the robot visual recognition task as an example, if we want to identify different types of tools (such as wrenches, screwdrivers, etc.), the number of tool categories is used as , determined by the different appearance, size, etc. of the tool Value. Given an input image , through a convolutional neural network (CNN) based extractor Extract feature representation and calculate the inverse of the Euclidean similarity between the extracted feature and all prototypes to get the distance , select the nearest prototype to determine the predicted category, and then normalize the probability to The predicted probability is obtained from the interval At the same time, a classifier based on a fully connected layer is integrated and cascaded with a CNN-based extractor, and the hyperparameters The prediction results of the two classifiers are fused as shown in formula (6):

[0132] (6).

[0133] In the early stages of training, Close to 0, the CNN-based classifier dominates the results; gradually increases as training progresses , in the later stages of training When fully utilizing prototype prediction.

[0134] Multi-prototype optimization module: Based on contrast optimization, drawing on the idea of ​​unsupervised contrastive learning, it achieves inter-class separation through formulas (13) and (14), ensuring that different categories are represented separately and the same category is represented compactly:

[0135] (13),

[0136] (14).

[0137] In the target prototype identification phase, when a new instance arrives, the nearest prototype is selected for update, and a memory-based counter is used to track the number of instances assigned to each prototype. The number of instances is less than the threshold. The prototype is considered outdated and reintegrated into the representation space. In the robot visual recognition task, if the number of object instances of a certain category recognized by the robot at a certain moment is too small, the corresponding prototype may be judged as outdated and needs to be readjusted. In the prototype movement quantization stage, when the prototype-based classifier makes a correct decision, the formula Update the prototype; when making a wrong decision, use the formula Adjust the prototype, where satisfy , to emphasize the update amount of misclassified samples and optimize the class boundaries.

[0138] The mean square error (MSE) is selected as the loss function, and the contrast loss regularization term is added. The final loss function is During the training process, the model is trained on different datasets, such as image datasets such as CIFAR-10 and CIFAR100, and graph datasets such as Cora and Citeseer, and the model parameters are continuously adjusted to minimize the loss function and improve the classification accuracy of the model.

[0139] Experimental validation includes dataset selection, baseline and backbone network settings, representation visualization, and hyperparameter settings.

[0140] Dataset Selection: We selected common image and graph datasets used in robot vision tasks, including three image datasets: CIFAR-10 (60,000 32×32 pixel, 3-channel images, 10 categories), CIFAR100 (60,000 32×32 pixel, 3-channel images, 100 categories), and Caltech-101 (9,146 300×200 pixel, 3-channel images, 101 categories); and three graph datasets: Cora (2,708 nodes, 5,249 edges, 1,433-dimensional features, 7 categories), Citeseer (3,327 nodes, 3,703 edges, 3,703-dimensional features, 6 categories), and Pubmed (19,717 nodes, 44,338 edges, 500-dimensional features, 3 categories). These datasets cover data of varying sizes and types, enabling comprehensive validation of the proposed method's performance in robot vision recognition.

[0141] Baseline and backbone networks: Six backbone networks were selected for comparison, including ResNet-18, ResNet50, and CvT-7 for image classification, and GCN, DeepWalk, and GAT for graph-level classification. LamPro was combined with these backbone networks to analyze performance changes. For example, when using ResNet-18 as the backbone network, the performance was compared with and without prototype updates and with different numbers of prototypes. The classification accuracy is 90.00±0.63% when ResNet-18 is used on CIFAR-10 without prototypes. When is 91.25±0.88, At the same time, the number of different prototypes was analyzed The impact on model performance is found that The performance improves accordingly, but in image datasets (such as CIFAR-10) and graph node classification tasks (such as Cora dataset), the performance is improved by The performance improvement rate slows down when it is 7 or greater than 13. These experimental results verify the effectiveness of the multi-prototype strategy and label-aware update strategy in the present invention.

[0142] Representation visualization: Dimensionality reduction and visualization techniques are used to map the prototype to a two-dimensional space for display. Taking the representation learning of ResNet-50 on CIFAR-10 as an example, the results show that there is sufficient distance space between different categories, the boundary of each category is clear, and the diverse representation space provided by multiple prototypes has stronger representation capabilities, further confirming the interpretability and effectiveness of LamPro in robot visual recognition tasks.

[0143] Hyperparameter setting: update amount , hyperparameters and the sample size threshold for obsolete prototypes Conduct experimental analysis. Set the number of prototypes to 20 and study different value The impact on performance was found Too large or too small can not achieve the best performance, the final choice Search between 10 and 25 ,Discover The performance is best when The performance is best when . The determination of these hyperparameters provides an important basis for model optimization.

[0144] Figure 7 The block diagram of an electronic device suitable for implementing a multi-prototype representation enhanced visual pattern recognition method for robots according to an embodiment of the present invention is schematically shown.

[0145] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 702 or programs loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0146] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations according to the method flow of the embodiment of the present invention by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also perform various operations according to the method flow of the embodiment of the present invention by executing the programs stored in the one or more memories.

[0147] According to an embodiment of the present invention, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0148] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0149] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above, and / or one or more memories other than ROM 702 and RAM 703.

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0152] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A multi-prototype representation-enhanced visual pattern recognition method for robots, characterized by: The method comprises: Using a prototype ensemble classifier to extract features from an image training dataset for a robot visual pattern recognition task, and obtaining a predicted classification result of the sample image by calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and by mapping the sample image features; Using a label-aware multi-prototype updater to perform target prototype recognition on image data collected in real time by a robot based on a visual pattern recognition task, obtaining a prototype to be updated from the multi-prototype set, performing prototype movement quantization update on the prototype to be updated based on the predicted classification result of the sample image to obtain an updated prototype, and obtaining a classification result of the image data based on the updated prototype; The label-aware multi-prototype updater is used to perform target prototype recognition on image data collected in real time by the robot based on the visual pattern recognition task, and the prototype to be updated obtained from the multi-prototype set includes: An irregular multi-prototype class space is constructed based on the categories of the prototypes in the multi-prototype set, and the similarity between each prototype in the multi-prototype class space is calculated using a regularization term based on unsupervised contrastive learning to achieve inter-class separation, thereby obtaining a multi-prototype set after contrast optimization; Selecting, by using a label-aware multi-prototype updater, a prototype closest in distance to the instance represented by the image data from the multi-prototype class space as the prototype to be updated; Tracking the number of instances assigned to each prototype in the multi-prototype class space using a memory-based update counter in a label-aware multi-prototype updater; When the number of instances allocated to the current prototype of the memory-based update counter is less than a preset threshold, setting the current prototype as an obsolete prototype and reintegrating the current prototype into the representation space; The prototype to be updated is subjected to prototype movement quantization updating based on the predicted classification result of the sample image to obtain the updated prototype, which includes: When the predicted classification result of the sample image is characterized as a negative feedback label, a preset control update amount and a preset adjustment movement scale are used to perform a prototype movement quantization update based on a moving average on the prototype to be updated and the sample image features corresponding to the sample image, to obtain the updated prototype; When the predicted classification result of the sample image is characterized as a positive feedback label, the preset control update amount is used to perform prototype moving quantization update based on moving average on the prototype to be updated and the sample image features corresponding to the sample image to obtain the updated prototype.

2. The method according to claim 1, further comprising: Initial prototypes are derived from the image training data set using a multi-prototype construction function, and the multi-prototype set is constructed based on the category of the initial prototype and the intra-class mode of each initial prototype.

3. The method according to claim 1, characterized in that Also includes: The data processing process of the prototype ensemble classifier and the data processing process of the label-aware multi-prototype updater are supervised by using a mean square error loss function with a contrastive loss regularization term to achieve inter-class separation and intra-class diversity of multiple prototypes.

4. The method according to claim 1, wherein By calculating the pattern similarity between the extracted sample image features and each prototype in the multi-prototype set and mapping the sample image features, the predicted classification result of the sample image is obtained, including: Extracting sample image features of each sample image from the image training data set using a preset neural network in the prototype ensemble classifier; Calculating the inverse of the Euclidean similarity between the sample image feature and each prototype in the multi-prototype set to obtain a Euclidean distance; Normalization activation processing is performed on the Euclidean distance to obtain a first predicted classification result of the sample image.

5. The method according to claim 4, characterized in that Also includes: Mapping the sample image features using a fully connected layer classifier in the prototype ensemble classifier to obtain a second predicted classification result of the sample image, wherein the fully connected layer classifier is cascaded with a feature extractor of the preset neural network; The first predicted classification result and the second predicted classification result of the sample image are calculated using adjustable parameters to obtain the predicted classification result of the sample image, wherein the adjustable parameters are updated by a multi-layer perceptron in the prototype ensemble classifier.

6. The method according to claim 4, characterized in that The preset neural networks include a deep convolutional neural network with residual connections, an image classification neural network with a transformer module and a convolution module, a graph structured convolutional neural network, a graph embedding neural network based on random walk and natural language processing technology, and a graph neural network model based on an attention mechanism.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images