A processing method, apparatus and electronic device

Abstract

The application discloses a processing method, device and electronic equipment. The method comprises the following steps: obtaining a target object to be processed, determining an object feature of the target object, and determining the correlation between the object feature of the target object and the class features of a plurality of different first categories respectively, wherein the first category is a category for object classification currently contained in a processing model; then, according to the correlation and an abnormality detection rule, it is determined whether the target object belongs to each first category; if it is determined that the target object does not belong to each first category, the processing model is updated based on at least one target object that does not belong to each first category under the condition that an update condition is met.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a processing method, apparatus and electronic device. Background Technology

[0002] Edge AI (Edge Artificial Intelligence) models operate in an open-set environment. The sample data used in the initial training phase typically doesn't cover all data distributions, potentially leading to the introduction of new, uncovered data categories during model execution. Traditional Cloud AI solutions, on the one hand, rely on experts to discover and collect these new categories, consuming significant manpower and time. On the other hand, the collected new category data needs to be uploaded back to the cloud and retrained along with the original training set, resulting in low training efficiency, long training cycles, and potential privacy and data security issues. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A processing method, the method comprising:

[0005] Obtain the target object to be processed;

[0006] Determine the object characteristics of the target object;

[0007] Determine the correlation between the object features and category features of multiple different first categories; the first category is the category currently included in the processing model for object classification;

[0008] Based on the correlation and anomaly detection rules, determine whether the target object belongs to each of the first categories;

[0009] If it is determined that the target object does not belong to any of the first categories, the processing model is updated based on at least one target object that does not belong to any of the first categories, provided that the update conditions are met.

[0010] Optional:

[0011] The process of obtaining the target object to be processed includes:

[0012] Obtain the image to be processed, which will be the target object to be processed;

[0013] Determining the object characteristics of the target object includes:

[0014] Extract the local image features corresponding to different local regions of the image to be processed.

[0015] Optionally, updating the processing model based on at least one target object that does not belong to each of the first categories when the update conditions are met includes:

[0016] Determine the number of target images that have been labeled with categories among multiple images to be processed that do not belong to each of the first categories;

[0017] When the number of target images with completed category labeling reaches the target number, the processing model is updated based on each target image with completed category labeling.

[0018] Optionally, updating the processing model based on at least one target object that does not belong to each of the first categories includes:

[0019] Obtain the labeled categories of the current target image;

[0020] If the labeled category does not belong to the second category, the labeled category is added to the processing model, and a memory module is added to store the first target feature in the local image features corresponding to different local regions of the target image that satisfies the second correlation condition with the labeled category into the added memory module, as the category feature of the labeled category; wherein, the second category includes the category added to the processing model based on the preceding target image of the target image;

[0021] If the labeled category belongs to the second category, the second target feature that satisfies the third correlation condition with the labeled category among the local image features corresponding to different local regions of the target image is added as one of the category features of the labeled category to the memory module corresponding to the labeled category.

[0022] Optionally, the second relevant condition includes: the distance between the local image features and each of the first category and the second category is greater than a first distance, or the similarity is less than a first similarity.

[0023] The third relevant condition includes: the distance between the local image features and each of the first and second categories other than the labeled categories is greater than the second distance or the similarity is less than the second similarity, and the distance between the local image features and the labeled categories is less than the third distance or the similarity is greater than the third similarity.

[0024] Optionally, the object feature is an object sub-feature corresponding to multiple local objects of the target object, and the category feature of the first category is multiple category sub-features; determining the correlation between the object feature and the category features of multiple different first categories includes:

[0025] Determine the distance or similarity between multiple object sub-features corresponding to the multiple local objects and multiple category sub-features of each first category, and obtain feature distance distribution data or feature similarity distribution data between the target object and each first category, so as to characterize the correlation based on the feature distance distribution data or the feature similarity distribution data.

[0026] Optionally, the processing model includes multiple memory modules that correspond one-to-one with different first categories and are used to store category features of the corresponding first category; the category features stored in each memory module include: multiple local image features that satisfy a first correlation condition with the corresponding first category, obtained by filtering each image sample of the processing model based on at least a portion of the training and usage phases of the processing model.

[0027] Determining the correlation between the object features and category features of multiple different first categories includes:

[0028] The local image features corresponding to different local regions of the image to be processed are determined, and the distance or similarity between them and the multiple local image features stored in each memory module is obtained. The feature distance distribution data or feature similarity distribution data between the image to be processed and each first category are obtained, and the correlation is characterized based on the feature distance distribution data or the feature similarity distribution data.

[0029] Optionally, determining whether the target object belongs to an unknown category distinct from each of the first categories based on the correlation and anomaly detection rules includes:

[0030] Based on the feature distance distribution data or the feature similarity distribution data, determine the distance or similarity between the target object and each of the first categories;

[0031] Based on the distance or similarity between the target object and each of the first categories, and the anomaly detection rules related to the meta-memory model, it is determined whether the target object belongs to an unknown category that is different from each of the first categories;

[0032] The meta-memory model is a model obtained by modeling the feature distance distribution data or feature similarity distribution data between object samples under each first category and each first category. The meta-memory model includes the distance distribution features or similarity distribution features between object samples under each first category and each first category.

[0033] Optionally, the distance distribution feature or similarity distribution feature includes: clustering the distance or similarity between object samples under each first category and each first category, and obtaining at least one cluster center corresponding to the distance or similarity respectively;

[0034] The distance or similarity between the object samples under each first category and each first category is the result obtained by processing the feature distance distribution data or feature similarity distribution data between the object samples under each first category and each first category.

[0035] A processing apparatus, comprising:

[0036] The acquisition unit is used to obtain the target object to be processed.

[0037] The first determining unit is used to determine the object characteristics of the target object;

[0038] The second determining unit is used to determine the correlation between the object features and category features of multiple different first categories; the first category is the category currently included in the processing model for object classification;

[0039] The third determining unit is used to determine whether the target object belongs to each of the first categories based on the correlation and anomaly detection rules.

[0040] An update unit is configured to update the processing model based on at least one target object that does not belong to any of the first categories, provided that update conditions are met, if it is determined that the target object does not belong to any of the first categories.

[0041] An electronic device, comprising:

[0042] Memory, used to store at least one set of computer instructions;

[0043] A processor for implementing the processing method described in any of the preceding descriptions by invoking and executing the instruction set stored in the memory.

[0044] As can be seen from the above scheme, the processing method, apparatus and electronic device disclosed in this application obtains the target object to be processed, determines the object characteristics of the target object, and determines the correlation between the object characteristics of the target object and the category characteristics of multiple different first categories, wherein the first category is the category currently included in the processing model for object classification; then, according to the correlation and anomaly detection rules, it is determined whether the target object belongs to each of the first categories; if it does not belong to each of the first categories, the processing model is updated based on at least one target object that does not belong to each of the first categories, provided that the update conditions are met.

[0045] This application, on the processing model side, performs correlation detection between the target object and each existing category (first category) of the processing model based on the object features of the target object to be processed. Based on the correlation, it determines whether the target object belongs to each known category of the processing model. If it does not belong, it updates the processing model using the target object in combination with the set update conditions. This allows for the automatic identification of new category samples and dynamic model updates based on new category samples directly on the edge side for the Edge AI runtime environment. It eliminates the need for experts to manually discover and collect new category samples and eliminates the need to send new category samples back to the cloud for model retraining, saving manpower and time costs, improving model training efficiency, and avoiding the security risks associated with manually collecting and sending back new category samples, thus improving data security on the edge side. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating one of the processing methods provided in this application;

[0048] Figure 2 This is an example image of the patch features of the image provided in this application;

[0049] Figure 3 This is an example diagram illustrating the working principle of incremental update of model categories based on the method of this application;

[0050] Figure 4 This is another flowchart illustrating the processing method provided in this application;

[0051] Figure 5 This is a structural diagram of the processing apparatus provided in this application;

[0052] Figure 6 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Edge AI operates in an open-set environment. Taking quality inspection in the field of intelligent manufacturing as an example, the data in the initial stage of model training cannot cover all data distributions. Due to machine aging, the different optical properties of complex objects being inspected, changes in production batches of various products, and similar materials from different suppliers, completely new sample defects that have never been seen before may appear in the production environment. Identifying these open-set data environments will seriously affect the model's performance. The traditional method of relying on experts to discover and collect new categories of data and then sending it back to the cloud for model retraining has drawbacks such as low model training efficiency, long cycle, and potential privacy protection issues.

[0055] To address the aforementioned issues, this application discloses a processing method, apparatus, and electronic device. This method primarily involves adding an open-set testing plugin to the edge-side processing model (such as an AI model) to identify new category samples. This enables the automatic collection of new category data at the edge side and the automatic dynamic updating of the model directly at the edge side based on the collected new category data. However, edge-side samples often suffer from limited quantity and difficulty in detection. To address this issue, this application employs category incremental learning based on a multi-memory update mechanism to achieve dynamic model updates at the edge side.

[0056] The processing method disclosed in this application can be applied, but is not limited to, to edge devices on a production line, such as personal computers and server computers on the production line. See also Figure 1 The flowchart shown illustrates the processing method, which may include the following processing steps:

[0057] Step 101: Obtain the target object to be processed.

[0058] The target object to be processed can be, but is not limited to, the image to be processed. Taking the application scenario of identifying product defect categories by processing online product image information to achieve product quality inspection as an example, the target object to be processed can specifically be the image on the production line that needs to be detected for defect categories.

[0059] This application embodiment will primarily use the image to be processed as the target object for explanation. Accordingly, in this step, the image to be processed can be obtained as the target object.

[0060] Step 102: Determine the object characteristics of the target object.

[0061] Optionally, the object features of the target object can be the object sub-features corresponding to multiple local objects of the target object. Multiple local objects of the target object refer to the different parts corresponding to different parts of the target object. Taking the image to be processed as the target object, the multiple local objects of the target object can refer to multiple image regions of the image to be processed. Correspondingly, the local image features corresponding to different local regions of the image to be processed can be extracted as the object features of the target object, i.e., the image to be processed.

[0062] Preferably, the extracted local image features are the image's patch features.

[0063] The applicant has found that the number of new category samples accumulated on edge devices is limited, and the number of new category samples is small. In order to enable the model to learn to identify new categories with a small number of samples, and to avoid catastrophic forgetting problems, the embodiments of this application preferably use a model based on metric learning as the processing model for processing the target object to be processed.

[0064] In particular, metric-based classification structures typically perform worse than end-to-end neural networks. To overcome this problem and improve the classification performance of metric-based structures, this application employs finer-grained features, specifically by extracting patch features from the image to be processed using a feature extractor. Figure 2 As shown, each small square represents a patch feature of the image to be processed. One image corresponds to multiple patches. Each patch represents the local features of the corresponding local region of the image, including but not limited to the features of the corresponding local image in terms of color, texture, semantics, etc.

[0065] Optionally, a feature extractor based on a deep network structure, such as a CNN (Convolutional Neural Networks) feature extractor, can be constructed for patch feature extraction of the image to be processed.

[0066] Step 103: Determine the correlation between the object features of the target object and the category features of multiple different first categories; the first category is the category currently included in the processing model for object classification.

[0067] As is easy to understand, each first category is a known category of the processing model. The processing model uses these categories to classify and process each input object, such as classifying defects in each input image to identify its corresponding defect category.

[0068] The first category's category features specifically refer to the multiple category sub-features of the first category.

[0069] After determining the object features of the target object, that is, after determining the object sub-features corresponding to the multiple local objects of the target object, the distance or similarity between the multiple object sub-features corresponding to the multiple local objects of the target object and the multiple category sub-features of each first category can be further determined. This yields feature distance distribution data or feature similarity distribution data between the target object and each first category. Based on the obtained feature distance distribution data or feature similarity distribution data, the correlation between the object features of the target object and the category features of multiple different first categories can be characterized. This correlation is essentially the correlation between the target object (such as the image to be processed) and different first categories.

[0070] The processing model includes multiple memory modules, each corresponding one-to-one with a different first category, for storing category features of the corresponding first category. Taking an image processing model as an example, the category features stored in each memory module of the model may include: multiple local image features that satisfy a first correlation condition with the corresponding first category, obtained by filtering each image sample based on the processing model during at least a portion of the training and usage phases of the processing model.

[0071] The first relevant condition is a condition that can be used to characterize the high correlation between the local image features of an image sample and the corresponding first category. It can be set as follows: the distance between the local image features of the image sample and the category features of the corresponding first category is lower than a set distance threshold, or the similarity is higher than a set similarity threshold.

[0072] Correspondingly, the distance or similarity between the local image features corresponding to different local regions of the image to be processed and the multiple local image features stored in each memory module can be determined, thereby obtaining the feature distance distribution data or feature similarity distribution data between the image to be processed and each first category. Based on the feature distance distribution data or feature similarity distribution data, the correlation between the image features of the image to be processed and the category features of multiple different first categories can be characterized, that is, the correlation between the image to be processed and multiple different first categories.

[0073] For images where local features are patch features, the memory module corresponding to each first category in the processing model stores category features for that first category. Essentially, these features are selected from a series of historical image samples and satisfy a first correlation condition (e.g., patch features that satisfy a first correlation condition with a defect category). Patch features that do not satisfy the first correlation condition (e.g., patch features of the background in the image) are discarded. Furthermore, the distance or similarity between each patch feature of the image to be processed and each patch feature in each memory module can be specifically determined to obtain feature distance distribution data or feature similarity distribution data between the image to be processed and each first category, thereby characterizing the correlation between the image to be processed and each first category.

[0074] As mentioned earlier, in order to enable the model to learn to recognize new classes with a small number of samples without catastrophic forgetting, this application preferably uses a metric learning-based model as the processing model for processing the target objects. Specifically, this application uses common deep network structures, such as a feature extractor built on CNNs, to extract object features in the model, and incorporates a set of memory modules into the network to achieve incremental learning of sample characteristics.

[0075] See Figure 3 The provided diagram illustrates the working principle of incremental update of model categories based on the method of this application. Taking the image to be processed as the target object, this embodiment uses a metric learning-based image recognition model as its core. It adds an open-set testing plugin (open-set testing module) to identify new category samples and updates the model using a few-sample category update module. The metric learning-based image recognition model is... Figure 3 The part corresponding to the solid line connection includes a feature extractor for feature extraction, a set of memory modules (Memory Class1, 2...n) for storing the category features of each first category, and a classification module (Classifier) ​​for classification. The open set testing plugin is... Figure 3 The section corresponding to the dashed connecting line includes an outlier detector module for detecting new class samples (such as images), and also has a corresponding database of unknown samples (Samples of New Classes) to store new class samples detected by the outlier detector module (such as images). Figure 3 Outliers in the data ("Outlier"), the few sample class update module is Figure 3The part corresponding to the double solid lines is used to provide the function of dynamically updating the model based on the labeled new category samples (Labeled Data). Specifically, this can be done by labeling the new category samples in the unknown sample database using appropriate methods, such as manual methods (e.g., ...). Figure 3 After labeling the new category samples, the processing model is dynamically updated based on the labeled Labels.

[0076] The memory module can be viewed as a set of prototype vectors, connected to the main network body via the read / write head. Specifically, one end of the memory module connects to the feature extractor, and the other end connects to the classification module. It can perform read operations through an attention mechanism. That is, for each input image, the memory module uses the attention mechanism to obtain the similarity between the image's patched features and the features stored in each category's memory module. This ultimately yields the feature distance distribution data / feature similarity distribution data (Metrics in the figure) from the image to each category. Finally, this distribution data is input into the classification module to obtain the final category recognition result.

[0077] Specifically, when the image to be processed is input into the model (such as...) Figure 3 After inputting the image (Image), the feature extractor extracts the features of each patch of the image to be processed and sends each extracted patch feature to a memory module. Each memory module, through an attention mechanism, determines the distance or similarity between each patch feature of the image and the corresponding patch features of the first category stored in the memory module. This yields the feature distance distribution data or feature similarity distribution data between each patch feature of the image to be processed and the corresponding first category of each memory module. Assuming the image to be processed has 9*9 (81) patch features, and the processing model currently has 3 categories (3 in the first category), and each memory module for the first category stores 100 patch features as the feature vector set for that category, then by determining the distance or similarity between each patch feature of the image to be processed and the patch features in each memory module, three 81*100 (81 rows 100 columns) feature distance distribution matrices or feature similarity distribution matrices can be obtained. Figure 3 The metrics in the matrix allow us to characterize the correlation between the image to be processed and each first category based on the three feature distance distribution matrices / feature similarity distribution matrices obtained.

[0078] Step 104: Based on the correlation and anomaly detection rules, determine whether the target object belongs to each of the first categories.

[0079] After determining the correlation between the object features of the target object and the category features of multiple different first categories, it is possible to further determine whether the target object belongs to each first category based on the correlation and the preset anomaly detection rules. This process can be implemented as follows:

[0080] 11) Based on the feature distance distribution data or the feature similarity distribution data, determine the distance or similarity between the target object and each of the first categories;

[0081] Taking the image to be processed as the target object as an example, the distance or similarity between the image to be processed and each first category can be determined based on the feature distance distribution data or feature similarity distribution data between the image to be processed and each first category. Optionally, in one embodiment, based on a preset fusion rule, each feature distance / feature similarity in the feature distance distribution data / feature similarity distribution data between the image to be processed and the first category can be fused, and the resulting fused distance / fused similarity can be used as the distance or similarity between the image to be processed and the first category.

[0082] In other embodiments, a certain feature distance / feature similarity can be selected from the feature distance distribution data / feature similarity distribution data between the image to be processed and the first category based on a preset filtering rule, and used as the distance or similarity between the image to be processed and the first category. For example, the minimum distance can be taken from each row of the feature distance distribution matrix between the image to be processed and a certain first category, and then the maximum distance can be taken from the minimum distances of each row (here, the purpose of taking the maximum distance is to avoid the value being too idealized and causing the category recognition to be incorrect), and used as the distance between the image to be processed and the first category, etc.

[0083] 12) Based on the distance or similarity between the target object and each of the first categories, and the anomaly detection rules related to the meta-memory model, determine whether the target object belongs to an unknown category that is different from each of the first categories.

[0084] In this embodiment of the application, the anomaly detection rules are rules related to the meta-memory model.

[0085] The meta-memory model is a model obtained by modeling the feature distance distribution data or feature similarity distribution data between object samples under each first category and each first category. The meta-memory model includes the distance distribution features or similarity distribution features between object samples under each first category and each first category.

[0086] Taking image processing as an example, the applicant discovered that for images of known categories, their feature distance distribution data or feature similarity distribution data, such as Figure 3Metrics in a dataset exhibit certain patterns. For example, feature distances are typically smaller or feature similarities are higher in one category, while feature distances are larger or feature similarities are lower in other categories. Based on this, to better define these patterns and thus accurately identify samples of unknown categories, the anomaly detection module models the metrics corresponding to all samples of known categories. This involves constructing a meta-memory model of metrics to depict their distribution.

[0087] Optionally, the method for constructing the meta-memory model of metrics can be implemented through clustering algorithms, including but not limited to k-means, k-center and other clustering algorithms.

[0088] Correspondingly, the distance distribution features or similarity distribution features in the meta-memory model can further include: the distance or similarity corresponding to at least one cluster center obtained by clustering the object samples under each first category with each first category and the distance or similarity between them.

[0089] Here, the distance or similarity between each object sample in the first category and the first category itself is the result of processing the feature distance distribution data or feature similarity distribution data between the object samples in the first category and the first category. For example, based on the above filtering rules, a certain feature distance or feature similarity is selected from the feature distance distribution matrix / feature similarity distribution matrix between the image samples in the first category and the first category, and used as the distance or similarity between the image samples in the first category and the first category.

[0090] After determining the distance or similarity between the target object (e.g., the image to be processed) and each of the first categories through step 11), specifically, the difference between this distance / similarity and the distance / similarity corresponding to each cluster center in the meta-memory model can be determined. If the difference meets the set condition for representing an abnormal distance / similarity (e.g., the difference between this distance / similarity and the distance / similarity corresponding to each cluster center in the meta-memory model is greater than a set threshold), then it indicates that the target object does not belong to any of the first categories, that is, the target object does not belong to any of the known categories of the processing model. For the processing model, the category of the target object is unknown. Otherwise, it indicates that it belongs to a certain first category.

[0091] Corresponding to Figure 3Each time the metrics of the input image to be processed are obtained, the metrics are input into the anomaly detection module to calculate the distance or similarity with the meta-memory. If the distance to each cluster center in the meta-memory is greater than the set threshold (or the similarity is less than the set threshold), the image to be processed is determined to be an unknown category (new category) and stored in the unknown sample database; otherwise, the image to be processed is determined to be a known category and is not stored in the unknown sample database.

[0092] Step 105: If it is determined that the target object does not belong to any of the first categories, and the update conditions are met, update the processing model based on at least one target object that does not belong to any of the first categories.

[0093] Taking image processing as an example, the update condition can be set to any of the following conditions, but is not limited to:

[0094] Condition 1: The time interval since the last model update reaches the set duration;

[0095] Condition 2: The number of unknown category images that have been labeled has reached the target number.

[0096] Optionally, the detected images of unknown categories (i.e., images of new categories that do not belong to any of the first categories) can be labeled with categories by means of manual annotation.

[0097] In this application embodiment, preferred condition two is used as the update condition. Accordingly, the number of target images that have completed category labeling among multiple images to be processed that do not belong to each first category can be determined, and when the number of target images that have completed category labeling reaches the target number, the processing model is updated based on each target image that has completed category labeling.

[0098] The process of updating the processing model based on each target image with completed category labeling can be further implemented as follows:

[0099] 21) Obtain the labeled category of the current target image.

[0100] Optionally, manual annotation can be used, whereby a person combines the objective image information of the target image to assign it an actual category, and obtains the category manually labeled for the target image as its labeled category.

[0101] 22) If the labeled category does not belong to the second category, add the labeled category to the processing model and add a memory module to store the first target feature in the local image features corresponding to different local regions of the target image that satisfies the second correlation condition with the labeled category into the added memory module, as the category feature of the labeled category.

[0102] The second category includes categories added to the processing model based on the target image that do not belong to the respective first categories.

[0103] It's easy to understand that when the number of target images with completed category labeling (i.e., images of unknown / new categories detected by the processing model) reaches the target number, when updating the processing model based on the target images with completed category labeling, for the current target image, a second category may have already been added to the processing model based on a series of preceding target images that do not belong to the respective first categories. Furthermore, the labeled category corresponding to the current target image may or may not belong to the second category.

[0104] Specifically, for cases where the current target image does not belong to the second category, the labeled category of the current target image is added to the processing model, and a new memory module is added to the memory module group, such as... Figure 3 The Memory Class(n+1) is used to store the category features of the labeled category. The category features of the labeled category can be obtained by filtering from the local image features (such as patch features) of the current target image based on the second correlation condition mentioned above.

[0105] Specifically, the second relevant condition includes: the distance between the local image features and each of the first category and the second category is greater than the first distance, or the similarity is less than the first similarity.

[0106] In other words, if the labeled category corresponding to the current target image is a new category compared to the known categories of the processing model (referring to each of the first category + the second category, or only the first category if no second category is added), then the labeled category is added to the processing model. Simultaneously, a new memory is created for it, and based on the second relevance condition, patch features that are relatively far from all existing memories are preferentially added to the new memory as its corresponding category features. Alternatively, from a similarity perspective, patch features that are relatively low in similarity to all existing memories are preferentially added as their corresponding category features.

[0107] The distance / similarity between each patch feature of the current target image and a certain memory refers to the distance / similarity between each patch feature of the current target image and the category features stored in that memory.

[0108] 23) If the labeled category belongs to the second category, the second target feature that satisfies the third correlation condition with the labeled category among the local image features corresponding to different local regions of the target image is added as one of the category features of the labeled category to the memory module corresponding to the labeled category.

[0109] If the current target image's labeled category belongs to the second category, it means that the current target image belongs to a known category of the processing model, and its corresponding memory module already exists in the processing model. In this case, the category features corresponding to the labeled category in the processing model can be dynamically updated based on the local features of the current target image (such as patch features), so that the second target features related to the labeled category in the image features of the current target image are included in the corresponding memory module, thereby improving the model's subsequent recognition accuracy of objects of the labeled category.

[0110] Specifically, the second target feature can be obtained by filtering from the local image features (such as patch features) of the current target image based on the third relevant conditions.

[0111] The third relevant condition includes: the distance between the local image features and each of the first and second categories other than the labeled categories is greater than the second distance or the similarity is less than the second similarity, and the distance between the local image features and the labeled categories is less than the third distance or the similarity is greater than the third similarity.

[0112] In other words, if the labeled category of the current target image is known to the processing model and does not belong to an unknown new category, then its corresponding memory module already exists in the model and does not need to be created.

[0113] In this scenario, we can consider the distances of each patch feature of the current target image to its corresponding target memory and to non-target memories. Based on the third correlation condition, we can prioritize adding features of the image that are close to its corresponding target memory and far from non-target memories to the target memory corresponding to the current image to be processed. Alternatively, from a similarity perspective, we can prioritize adding features that are highly similar to the corresponding target memory and low similarity to each non-target memory to dynamically update the category features in the corresponding target memory.

[0114] It should be noted that in practical applications, for a target image whose category is unknown (i.e., a new category) determined by the processing model, its actual corresponding category may be a known category in the processing model. In other words, the processing model may misclassify the category of the image due to insufficient recognition accuracy. In this case, the labeled category of the target image is actually a first category of the processing model. Accordingly, the features selected from each patch feature of the target image based on the third correlation condition can be added to the memory of the first category that the labeled category actually corresponds to, so as to dynamically update the category features of the first category. This allows the recognition accuracy of the first category image to be further improved based on the updated category features of the first category.

[0115] In summary, the method of this application embodiment, by performing correlation detection between the target object and each existing category (first category) of the processing model based on the object features of the target object to be processed, determining whether the target object belongs to each known category of the processing model based on the correlation, and updating the processing model using the target object in combination with the set update conditions, enables the automatic identification of new category samples and dynamic model updates based on new category samples directly on the edge side for the Edge AI runtime environment. This eliminates the need for experts to manually discover and collect new category samples, and eliminates the need to send new category samples back to the cloud for model retraining, saving manpower and time costs, improving model training efficiency, and avoiding the security risks caused by manually collecting and sending back new category samples, thus improving data security on the edge side.

[0116] In one embodiment, see Figure 4 The flowchart of the processing method shown in this application may include the following processing after step 103:

[0117] Step 106: Based on the correlation between the object features of the target object and the category features of multiple different first categories, classify the target object.

[0118] The embodiments of this application characterize the correlation between the object features of the target object and the category features of multiple different first categories based on the feature distance distribution data or feature similarity distribution data between the target object and each first category.

[0119] After obtaining the feature distance distribution data or feature similarity distribution data between the target object and each first category, the probability of the target object corresponding to each first category can be determined based on the feature distance distribution data or feature similarity distribution data between the target object and each first category. Finally, based on the probability of the target object corresponding to each first category, the target category to which the target object belongs in each first category can be identified. Specifically, the category with the highest probability can be identified as the target category to which the target object belongs.

[0120] like Figure 3 In this process, when dealing with an image to be processed, after the memory module outputs the distance distribution data / feature similarity distribution data (Metrics) between the image and each of the first categories in the processing model, the Metrics are not only sent to the anomaly detection module for new category sample detection, but also sent to the classification module (Classifier). The Classifier determines the matching probability between the image to be processed and each of the existing first categories in the model based on the Metrics, and identifies the category with the highest probability as the target category to which the image to be processed belongs, and outputs it, such as... Figure 3 The PredictedLabel (predicted label, which represents the target category to which the image to be processed) is output by the Classifier module.

[0121] In this embodiment, as new category samples are continuously detected based on the open set testing plugin, and the model is dynamically updated based on the new category samples, the memory module of the processing model and the category features contained therein become more and more complete. Consequently, the model can continuously match the recognition of various newly added category samples, thereby improving the model's recognition accuracy and the comprehensiveness of matching the continuously added samples on the edge side.

[0122] This application also discloses a processing apparatus, see [link to relevant documentation] Figure 5 The structural diagram of the processing device shown includes:

[0123] Acquisition unit 501 is used to acquire the target object to be processed;

[0124] The first determining unit 502 is used to determine the object characteristics of the target object;

[0125] The second determining unit 503 is used to determine the correlation between the object features and category features of multiple different first categories; the first category is the category currently included in the processing model for object classification;

[0126] The third determining unit 504 is used to determine whether the target object belongs to each of the first categories based on the correlation and anomaly detection rules.

[0127] The update unit 505 is configured to update the processing model based on at least one target object that does not belong to any of the first categories if it is determined that the target object does not belong to any of the first categories, provided that the update conditions are met.

[0128] In one embodiment, the acquisition unit 501 is specifically used for:

[0129] Obtain the image to be processed, which will be the target object to be processed;

[0130] The first determining unit 502 is specifically used for:

[0131] Extract the local image features corresponding to different local regions of the image to be processed.

[0132] In one embodiment, the updating unit 505 is specifically used for:

[0133] Determine the number of target images that have been labeled with categories among multiple images to be processed that do not belong to each of the first categories;

[0134] When the number of target images with completed category labeling reaches the target number, the processing model is updated based on each target image with completed category labeling.

[0135] In one embodiment, the updating unit 505, when updating the processing model based on at least one target object that does not belong to each of the first categories, is specifically used for:

[0136] Obtain the labeled categories of the current target image;

[0137] If the labeled category does not belong to the second category, the labeled category is added to the processing model, and a memory module is added to store the first target feature in the local image features corresponding to different local regions of the target image that satisfies the second correlation condition with the labeled category into the added memory module, as the category feature of the labeled category; wherein, the second category includes the category added to the processing model based on the preceding target image of the target image;

[0138] If the labeled category belongs to the second category, the second target feature among the local image features corresponding to different local regions of the target image that satisfies the third correlation condition with the labeled category is added as one of the category features of the labeled category to the memory module corresponding to the labeled category. In one embodiment, the second correlation condition includes: the distance between the local image feature and each of the first category and the second category is greater than a first distance, or the similarity is less than a first similarity.

[0139] The third relevant condition includes: the distance between the local image features and each of the first and second categories other than the labeled categories is greater than the second distance or the similarity is less than the second similarity, and the distance between the local image features and the labeled categories is less than the third distance or the similarity is greater than the third similarity.

[0140] In one embodiment, the object feature is an object sub-feature corresponding to multiple local objects of the target object, and the category feature of the first category is multiple category sub-features;

[0141] The second determining unit 503 is specifically used for:

[0142] Determine the distance or similarity between multiple object sub-features corresponding to the multiple local objects and multiple category sub-features of each first category, and obtain feature distance distribution data or feature similarity distribution data between the target object and each first category, so as to characterize the correlation based on the feature distance distribution data or the feature similarity distribution data.

[0143] In one embodiment, the processing model includes multiple memory modules that correspond one-to-one with different first categories and are used to store category features of the corresponding first categories; the category features stored in each memory module include: multiple local image features that satisfy a first correlation condition with the corresponding first category, obtained by filtering each image sample of the processing model based on at least a portion of the training and usage phases of the processing model.

[0144] The second determining unit 503 is specifically used for:

[0145] The local image features corresponding to different local regions of the image to be processed are determined, and the distance or similarity between them and the multiple local image features stored in each memory module is obtained. The feature distance distribution data or feature similarity distribution data between the image to be processed and each first category are obtained, and the correlation is characterized based on the feature distance distribution data or the feature similarity distribution data.

[0146] In one embodiment, the third determining unit 504 is specifically used for:

[0147] Based on the feature distance distribution data or the feature similarity distribution data, determine the distance or similarity between the target object and each of the first categories;

[0148] Based on the distance or similarity between the target object and each of the first categories, and the anomaly detection rules related to the meta-memory model, it is determined whether the target object belongs to an unknown category that is different from each of the first categories;

[0149] The meta-memory model is a model obtained by modeling the feature distance distribution data or feature similarity distribution data between object samples under each first category and each first category. The meta-memory model includes the distance distribution features or similarity distribution features between object samples under each first category and each first category.

[0150] In one embodiment, the distance distribution feature or similarity distribution feature includes: clustering the distance or similarity between object samples under each first category and each first category, and obtaining at least one cluster center corresponding to the distance or similarity respectively;

[0151] The distance or similarity between the object samples under each first category and each first category is the result obtained by processing the feature distance distribution data or feature similarity distribution data between the object samples under each first category and each first category.

[0152] In one embodiment, the above-described apparatus further includes an identification unit, configured to identify the category of the target object based on the correlation between the object features of the target object and the category features of a plurality of different first categories.

[0153] The processing apparatus disclosed in this application corresponds to the processing method disclosed in the above method embodiments, so the description is relatively simple. For related similarities, please refer to the description of the above method embodiments, which will not be described in detail here.

[0154] This application also discloses an electronic device, which may be, but is not limited to, a device in a variety of general or special computing device environments or configurations, such as: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0155] The composition and structure of electronic devices, such as Figure 6 As shown, it includes at least:

[0156] Memory 10 is used to store the computer instruction set;

[0157] Computer instruction sets can be implemented in the form of computer programs.

[0158] Processor 20 is configured to implement the processing method disclosed in any of the above method embodiments by executing a computer instruction set.

[0159] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices.

[0160] Electronic devices have a display device and / or have a display interface and can connect to an external display device.

[0161] Optionally, the electronic device may also include a camera assembly, and / or be connected to an external camera assembly.

[0162] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0163] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0164] In summary, the processing method, apparatus, and electronic device provided in this application have at least the following technical advantages compared to known technologies:

[0165] a. Open set detection mechanism based on multi-memory distribution: An open set testing plugin based on the model inference state has been added to the edge side, which enables the automatic detection of new category samples based on the intermediate inference results (Metrics) of the model at the edge side.

[0166] b. Patch memory update method for category increment: A prototype update method based on patch features and multiple memory modules is proposed. This method can realize dynamic incremental update of small sample categories while avoiding catastrophic forgetting.

[0167] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0168] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0169] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0170] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A processing method, the method comprising: Obtain the target object to be processed; Determine the object characteristics of the target object; Determine the correlation between the object features and category features of multiple different first categories; The first category is the category currently included in the processing model for object classification; Based on the correlation and anomaly detection rules, it is determined whether the target object belongs to each of the first categories; wherein, the anomaly detection rules are determination rules based on the meta-memory model, used to distinguish between known and unknown categories; If it is determined that the target object does not belong to any of the first categories, and the update conditions are met, the processing model is incrementally updated based on at least one target object that does not belong to any of the first categories by adding a memory module to the processing model; wherein, the update conditions include: the number of target objects that have been categorized and do not belong to any of the first categories reaches a target number, or the time interval since the last model update reaches a set duration.

2. The method according to claim 1, wherein: The process of obtaining the target object to be processed includes: Obtain the image to be processed, which will be the target object to be processed; Determining the object characteristics of the target object includes: Extract the local image features corresponding to different local regions of the image to be processed.

3. The method according to claim 2, wherein, under the condition of satisfying the update, incrementally updating the processing model based on at least one target object that does not belong to each of the first categories includes: Determine the number of target images that have been labeled with categories among multiple images to be processed that do not belong to each of the first categories; When the number of target images with completed category labeling reaches the target number, the processing model is updated based on each target image with completed category labeling.

4. The method according to claim 3, wherein incrementally updating the processing model based on at least one target object that does not belong to each of the first categories comprises: Obtain the labeled categories of the current target image; If the labeled category does not belong to the second category, the labeled category is added to the processing model. The first target feature that satisfies the second correlation condition with the labeled category in the local image features corresponding to different local regions of the target image is stored in the added memory module as the category feature of the labeled category. The second category includes categories added to the processing model based on the preceding target image of the target image. If the labeled category belongs to the second category, the second target feature that satisfies the third correlation condition with the labeled category among the local image features corresponding to different local regions of the target image is added as one of the category features of the labeled category to the memory module corresponding to the labeled category.

5. The method according to claim 4, wherein the second relevant condition includes: The distance between local image features and each of the first and second categories is greater than a first distance, or the similarity is less than a first similarity. The third relevant condition includes: the distance between the local image features and each of the first and second categories other than the labeled categories is greater than the second distance or the similarity is less than the second similarity, and the distance between the local image features and the labeled categories is less than the third distance or the similarity is greater than the third similarity.

6. The method according to claim 1, wherein the object feature is an object sub-feature corresponding to a plurality of local objects of the target object, and the category feature of the first category is a plurality of category sub-features; determining the correlation between the object feature and the category features of a plurality of different first categories includes: Determine the distance or similarity between multiple object sub-features corresponding to the multiple local objects and multiple category sub-features of each first category, and obtain feature distance distribution data or feature similarity distribution data between the target object and each first category, so as to characterize the correlation based on the feature distance distribution data or the feature similarity distribution data.

7. The method according to claim 2, wherein the processing model comprises a plurality of memory modules, each corresponding one-to-one with a different first category, for storing category features of the corresponding first category; The category features stored in each memory module include: multiple local image features that satisfy a first correlation condition with the corresponding first category, obtained by filtering each image sample of the processing model during at least a portion of the training and usage phases of the processing model. Determining the correlation between the object features and category features of multiple different first categories includes: The local image features corresponding to different local regions of the image to be processed are determined, and the distance or similarity between them and the multiple local image features stored in each memory module is obtained. The feature distance distribution data or feature similarity distribution data between the image to be processed and each first category are obtained, and the correlation is characterized based on the feature distance distribution data or the feature similarity distribution data.

8. The method according to claim 6 or 7, wherein determining whether the target object belongs to an unknown category distinct from each of the first categories based on the correlation and anomaly detection rules includes: Based on the feature distance distribution data or the feature similarity distribution data, determine the distance or similarity between the target object and each of the first categories; Based on the distance or similarity between the target object and each of the first categories, and the anomaly detection rules related to the meta-memory model, it is determined whether the target object belongs to an unknown category that is different from each of the first categories; The meta-memory model is a model obtained by modeling the feature distance distribution data or feature similarity distribution data between object samples under each first category and each first category. The meta-memory model includes the distance distribution features or similarity distribution features between object samples under each first category and each first category.

9. The method according to claim 8, wherein, The distance distribution feature or similarity distribution feature includes: clustering the distance or similarity between object samples under each first category and each first category, and the distance or similarity corresponding to at least one cluster center; The distance or similarity between the object samples under each first category and each first category is the result obtained by processing the feature distance distribution data or feature similarity distribution data between the object samples under each first category and each first category.

10. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor for implementing the processing method as described in any one of claims 1-9 by invoking and executing the instruction set stored in the memory.

Images