A detection method, apparatus, device, and storage medium

Abstract

The embodiment of the application discloses a detection method, device, equipment and storage medium, the method comprises: obtaining a to-be-migrated model; wherein the to-be-migrated model comprises a memory feature set, and the memory feature set represents a feature vector set having an association relationship with an application scenario corresponding to the to-be-migrated model; at least one sample data of a target scene is respectively measured and calculated with the memory feature set, and at least one measurement calculation result is obtained; the memory feature set is updated according to the at least one measurement calculation result, and a target memory feature set is obtained; wherein the target memory feature set represents a feature vector set having an association relationship with the target scene; the target memory feature set is used to replace the memory feature set in the to-be-migrated model, and a target detection model is obtained.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a detection method, apparatus, device, and storage medium. Background Technology

[0002] In actual production operations, there is often a need to migrate models to different scenarios. At this time, the following problems are often encountered: for new scenarios, it is difficult to generate enough sample data for model training in a short period of time; and even if there is a lot of sample data, supervised training of the model requires a lot of data annotation, annotation verification and other work, resulting in high time and manpower costs. Summary of the Invention

[0003] This application discloses a detection method, apparatus, equipment, and storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a detection method, including:

[0006] Obtain the model to be transferred; wherein, the model to be transferred includes a memory feature set, which represents a set of feature vectors that are related to the application scenario corresponding to the model to be transferred;

[0007] Perform metric calculations on at least one sample data of the target scene with the memory feature set to obtain at least one metric calculation result;

[0008] The memory feature set is updated based on at least one metric calculation result to obtain the target memory feature set; wherein, the target memory feature set represents the set of feature vectors that are related to the target scene;

[0009] The target detection model is obtained by replacing the memory feature set in the model to be transferred with the target memory feature set.

[0010] Secondly, embodiments of this application provide a detection device, including:

[0011] The acquisition unit is configured to acquire the model to be transferred; wherein the model to be transferred includes a set of memory features, and the memory features are adapted to the application scenario corresponding to the model to be transferred;

[0012] The update unit is configured to perform metric calculations on at least one sample data of the target scene and the memory feature set respectively to obtain at least one metric calculation result, and update the memory feature set according to the at least one metric calculation result to obtain a target memory feature set adapted to the target scene; and replace the memory feature set in the model to be transferred with the target memory feature set to obtain the target detection model.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein,

[0014] The memory is used to store computer programs that can run on the processor;

[0015] The processor is configured to execute the detection method as described in the first aspect when running the computer program.

[0016] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when executed by at least one processor, implements the detection method as described in the first aspect.

[0017] This application provides a detection method, apparatus, device, and storage medium that acquires a model to be transferred. The model includes a memory feature set, which represents a set of feature vectors related to an application scenario corresponding to the model. At least one sample data from a target scenario is used to perform metric calculations with the memory feature set to obtain at least one metric calculation result. The memory feature set is then updated based on the at least one metric calculation result to obtain a target memory feature set, which represents a set of feature vectors related to a target scenario. The target memory feature set is then used to replace the memory feature set in the model to be transferred, resulting in a target detection model. Thus, when transferring the model to be transferred to a target scenario, based on the metric calculations between features, only the sample data from the target scenario needs to be used to update and replace the memory feature set of the model to be transferred. This ensures that the updated target memory feature set is suitable for the target scenario, enabling the updating and transfer of the model to be transferred even with a small amount of sample data. The update speed is fast, and this method is also applicable to the migration and updating of models that have undergone pruning and quantization, making it widely applicable. Attached Figure Description

[0018] Figure 1 A schematic flowchart of a detection method provided in an embodiment of this application;

[0019] Figure 2 A schematic diagram of the flow architecture of a detection method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the composition structure of a detection device provided in an embodiment of this application;

[0021] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0022] Figure 5This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be noted that the terms "first, second, and third" used in the embodiments of this application are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] It should be understood that in actual production operations, there are often needs to migrate models to new production lines (e.g., migrating a model from an old production line to a new production line due to production expansion). In such cases, the following problems will be encountered: For new scenarios, it is difficult to generate enough sample data for model training in a short period of time; Even if there is a lot of sample data, supervised training of the model requires a lot of data annotation and annotation verification, resulting in extremely high time and manpower costs; In addition, in many cases, in order to improve the inference efficiency of the model, especially for models deployed on lightweight edge devices, the model is usually pruned and quantized before deployment, which makes it impossible to update the model on edge devices.

[0028] Based on this, this application provides a detection method, the basic idea of ​​which is: to obtain a model to be transferred; wherein, the model to be transferred includes a memory feature set, which represents a set of feature vectors that are related to the application scenario corresponding to the model to be transferred; to perform measurement calculations on at least one sample data of the target scenario with the memory feature set respectively, and obtain at least one measurement calculation result; to update the memory feature set according to the at least one measurement calculation result, and obtain a target memory feature set; wherein, the target memory feature set represents a set of feature vectors that are related to the target scenario; and to replace the memory feature set in the model to be transferred with the target memory feature set, and obtain the target detection model. In this way, when transferring the model to be transferred to the target scenario, based on the measurement calculation between features, only the sample data of the target scenario needs to be used to update and replace the memory feature set of the model to be transferred, so that the updated target memory feature set is suitable for the target scenario. Thus, it is possible to update and transfer the model to be transferred with a small number of sample data, and its update speed is fast, saving time and labor costs. In addition, this method is also applicable to the transfer and update of models that have undergone pruning and quantization, and has a wide range of applications.

[0029] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0030] In one embodiment of this application, see [link to embodiment]. Figure 1 The diagram illustrates a flowchart of a detection method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0031] S101: Obtain the model to be transferred; wherein, the model to be transferred includes a set of memory features, which represents a set of feature vectors that are related to the application scenario corresponding to the model to be transferred.

[0032] It should be noted that the detection method provided in this application embodiment can be applied to a detection device or an electronic device integrating the updating device. Here, the electronic device can be such as a computer, smartphone, tablet computer, laptop computer, handheld computer, personal digital assistant (PDA), navigation device, server, etc., and this application embodiment does not specifically limit it.

[0033] It should also be noted that the detection method provided in the embodiments of this application is particularly suitable for small-sample model updates on edge devices with weak computing power when there are few training samples. Therefore, electronic devices can also refer to edge computing boxes and other edge devices, where the central processing unit (CPU) of the edge device can be an Advanced RISC Machine (ARM) architecture, or a lightweight edge device such as Jetson NX if there is a graphics processing unit (GPU).

[0034] It should also be noted that the embodiments of this application can be applied to the model migration process. Migration can mean updating the model from one application scenario to another. Here, the model can refer to a model for object detection problems, used to detect objects. Therefore, the method provided in the embodiments of this application can be called a detection method. In the following description, the implementation of the embodiments of this application will be illustrated by taking the model as an object detection model and the object to be detected as an example.

[0035] The model to be transferred represents the model before scenario transfer. At this stage, the model to be transferred includes a set of memory features. Specifically, the set of memory features in the model to be transferred represents a set of feature vectors that are related to the application scenario corresponding to the model.

[0036] For example, if the application scenario of the model to be transferred is scenario A, then the memory feature set represents the set of feature vectors that are related to scenario A. Specifically, if the detection target of the model to be transferred in scenario A is target A, then the memory feature set represents the set of feature vectors of target A. Here, the feature vectors mainly refer to the prototype vectors of target A; that is, the memory features are a set of prototype vectors of the target object.

[0037] Furthermore, for the model to be transferred, in some embodiments, the model to be transferred may include a memory module, and the memory module stores a set of memory features. The method may also include:

[0038] Obtain the initial training set, which includes training samples corresponding to several tasks;

[0039] The model is trained based on the initial training set to build an initial model including the memory module;

[0040] Acquire real image data and use it to adjust the initial model to obtain the model to be transferred.

[0041] It should be noted that the model architecture of the model to be transferred may include a memory module, which is used to store the memory feature set. In this embodiment, when the method is applied to edge devices, the limited computing power of edge devices may lead to excessive memory consumption when training a large number of training samples. Therefore, when training the initial model, multi-task model training can be performed on a cloud device to establish an initial model including the memory module.

[0042] It should also be noted that the embodiments of this application realize the migration update of the model to be transferred based on the replacement of the memory feature set. At this time, it is necessary to use the initial model obtained by multi-task pre-training to realize the scenario adaptation in the model transfer process.

[0043] When training a model using an initial training set to obtain an initial model, the initial training set includes training samples corresponding to several tasks. These tasks are multiple tasks that are quite similar to the target task of the model to be transferred. Typically, the initial training set contains a large number of training samples, thus enabling the initial model to achieve more accurate detection results. This multi-task model training process can be considered a meta-learning process, through which an initial model including a memory module is established.

[0044] It should also be noted that during the training of the multi-task model, the model's detection results are determined using a full metric decoding mechanism. In other words, for the initial model, and for the transfer model and object detection model obtained by updating the initial model, the model's detection results are only related to the metric calculation results, and do not depend on the input training samples or the image to be detected itself.

[0045] For example, the initial model may include an encoder, a memory module, and a decoder. The encoder extracts features from training samples (such as sample images) to obtain deep feature information. The memory module stores a memory feature set corresponding to each task. It performs a metric calculation on each deep feature of the training sample and each feature vector in the corresponding task's memory feature set, obtaining the metric calculation result for each deep feature of the training sample and the corresponding feature vector in the memory feature set. The obtained metric calculation result is input into the decoder, which decodes the data to obtain the detection result. This result is then compared with the pre-labeled true result, and iterative calculations are performed based on the loss function until the initial model is obtained. In other words, training the initial model in the cloud primarily trains the decoder within the model to obtain accurate detection results.

[0046] It should also be noted that the measurement calculation for deep feature information and feature vectors in the memory feature set is as follows: For example, if the memory feature set includes feature vector 1, feature vector 2, and feature vector 3, then when extracting features from the training samples, the obtained deep feature information can include deep feature 1', deep feature 2', and deep feature 3'. Here, deep feature 1' is the deep feature information corresponding to feature vector 1 in the training samples, deep feature 2' is the deep feature information corresponding to feature vector 2 in the training samples, and deep feature 3' is the deep feature information corresponding to feature vector 2 in the training samples. During the measurement calculation, the measurement results for feature vector 1 and deep feature 1', feature vector 2 and deep feature 2', and feature vector 3 and deep feature 3' are calculated separately, resulting in three measurement results. These three measurement results are then input into the decoder for decoding.

[0047] After obtaining the initial model, since the memory feature set stored in the memory module corresponds to the memory feature sets of several tasks, these tasks may not necessarily match the target task when the model is actually used. Therefore, the memory feature set in the initial model can be cleared. Then, a corresponding memory feature set can be generated based on the model's real-world usage scenario.

[0048] The initial model is deployed on an edge device and fine-tuned based on real image data to obtain the model to be transferred. Real image data refers to sample image data of the actual application scenario corresponding to the model's deployment on the edge device. Fine-tuning the model involves generating a memory feature set of the current real application scenario and storing it in the model's memory module.

[0049] For example, the memory feature set can be generated as follows: extract the feature vector of the target object to be detected from each real image data; for each feature vector, average it over all real image data to obtain the average of each feature vector. The average of the feature vectors is also called the prototype vector. The averages of these feature vectors constitute the memory feature set for the real application scenario. This method can also be used to determine the feature vector set for the aforementioned tasks.

[0050] In short, for n real image data, if the feature vector of the target object includes at least vector A, then for vector A, a feature vector corresponding to vector A can be extracted from each real image data, that is, n feature vectors corresponding to vector A. The average of these n feature vectors is calculated to obtain vector A in the memory feature set.

[0051] S102: Perform metric calculations on at least one sample data of the target scene with the memory feature set to obtain at least one metric calculation result.

[0052] S103: Update the memory feature set based on at least one metric calculation result to obtain the target memory feature set; wherein, the target memory feature set represents the set of feature vectors that are related to the target scene.

[0053] It should be noted that at least one sample data point from the target scene is used as the sample dataset for updating the model to be transferred. Metrics are performed on at least one sample data point from the target scene and the memory feature set to obtain the metric calculation result of at least one sample data point and the memory feature set. Based on the obtained metric calculation result, the memory feature set is updated to obtain the target memory feature set. At this point, the target memory feature set is a set of feature vectors that are related to the target scene.

[0054] For example, if the target scene is scene B, then the memory feature set represents the set of feature vectors that are related to scene B. Specifically, if the detection target of the model to be transferred is target B in scene B, then the memory feature set represents the set of feature vectors of target B, where the feature vectors mainly refer to the prototype vectors of target B.

[0055] For sample data, in some embodiments, before performing metric calculations on at least one sample data point of the target scene with the memory feature set, the method may further include:

[0056] Identify the target scenario;

[0057] Based on the target scenario, obtain at least one initial sample data for the target scenario;

[0058] At least one initial sample data is labeled to obtain at least one sample data of the target scene.

[0059] It's important to note that when determining sample data for the target scene, one can first obtain at least one initial sample data point for the target scene, and then label this initial sample data. For example, if the initial sample data is an image, the type or bounding box of the target object in the image can be labeled to obtain the sample data. This allows for accurate extraction of depth feature information from the sample data during metric calculations. Furthermore, if backpropagation training is performed on the model to be transferred, supervised training updates can be performed on the model based on the labeled sample data.

[0060] It should be noted that metric calculations can include, but are not limited to, Euclidean distance calculation, Mahalanobis distance calculation, Manhattan distance calculation, Minkowski distance calculation, Hamming distance calculation, Jaccard correlation coefficient calculation, cosine similarity calculation, Chebyshev distance calculation, or Pearson correlation coefficient calculation. Since metric calculations typically involve calculating the distance between two individuals, the result of such calculations can also be referred to as a distance value.

[0061] The degree of similarity between individuals can be determined by measurement calculations. Generally speaking, the smaller the value of the measurement calculation result, the higher the similarity between individuals, and the larger the value of the measurement calculation result, the greater the difference between individuals.

[0062] The method for measuring at least one sample data point in the target scene against the memory feature set can be as follows: For each sample data point (taking an image as an example), firstly, feature extraction is performed on each sample data point to obtain at least one depth feature information. For any given depth feature information, the average depth feature information is calculated across all sample data points to obtain the average depth feature information. Then, the feature vector corresponding to this average depth feature information in the memory feature set is measured against this average depth feature information to obtain the measurement result. For feature vectors whose measurement result is greater than a measurement threshold, the feature vector is updated using the average depth feature information, or replaced using the average depth feature information. For feature vectors whose measurement result is not greater than the measurement threshold, no update is required. Here, the measurement threshold is a limit value used to characterize the degree of difference between two individuals.

[0063] Another way to measure and calculate the target scene at least one sample data with the memory feature set is as follows: First, extract features from at least one sample data to obtain several deep feature information. For any one deep feature information, measure and calculate it with each feature vector in the memory feature set, so as to obtain the measurement and calculation result of the deep feature information with each feature vector in the memory feature set. The measurement and calculation result with the smallest value is taken as the measurement and calculation result of the deep feature information with the memory feature set.

[0064] In this way, for each deep feature information, a metric calculation result can be obtained between it and the memory feature set. Then, the average of all metric calculation results is calculated. This average represents the degree of difference between the sample dataset consisting of at least one sample data and the memory feature set. The larger the average, the greater the difference between the sample dataset and the memory feature set, that is, there is a large difference between the target scene and the scene of the model to be transferred; the smaller the average, the smaller the difference between the sample dataset and the memory feature set, that is, there is a small difference between the target scene and the scene of the model to be transferred.

[0065] At this point, a metric threshold can still be set. This metric threshold is a threshold value used to characterize the degree of difference between the two scenarios. If the average value is greater than the metric threshold, it means that there is a large difference between the target scenario and the scenario of the model to be transferred. In this case, a brand new target memory feature set is generated based on the sample data. If the average value is not greater than the metric threshold, it means that there is a small difference between the target scenario and the scenario of the model to be transferred. In this case, only a partial update of the memory feature set is needed. For example, only the features whose metric calculation results are greater than the metric threshold are updated.

[0066] In other words, at least one sample data of the target scene is measured and calculated with the memory feature set to obtain at least one measurement result. The similarity between the target scene and the application scene of the model to be transferred can be determined by the at least one measurement result. If the similarity is high, only some feature vectors in the memory feature set need to be updated, while the other part of the feature vectors remain in the original memory feature set. If the similarity is low, a brand new target memory feature set is generated for incremental learning.

[0067] In addition to the two methods listed above, those skilled in the art can also use other methods based on the measurement to determine the similarity between the application scenario of the model to be transferred and the target scenario, so as to determine the target memory feature set. These methods will not be listed here.

[0068] Here, a simple example is used to illustrate the target scenario, the application scenario of the model to be transferred, and the updating of the memory feature set. If the model to be transferred is a target detection model for red apples, that is, the application scenario of the model to be transferred is to detect fruit in the image to determine whether the fruit in the image is a red apple and / or to determine the bounding box of a red apple in the image, then the memory feature set represents the feature vector set of red apples, that is, the prototype vector of red apples. If the target scenario is to determine whether the fruit in the image is a green apple and / or to determine the bounding box of a green apple in the image, then, after measurement calculation, it can be determined that red apples and green apples have a very high similarity. Therefore, only a part of the feature vectors in the memory feature set needs to be updated, while the other part remains unchanged, to obtain the target memory feature set. If the target scenario is to determine whether the fruit in the image is a star fruit and / or to determine the bounding box of a star fruit in the image, then, after measurement calculation, it can be determined that red apples and star fruit have a low similarity. Therefore, it is necessary to generate a completely new target memory feature set based on the sample data of star fruit.

[0069] In addition, for some simpler scenarios where the degree of similarity can be easily judged manually, the developers can manually determine the degree of update of the memory feature set instead of relying on the measurement calculation results to determine the target memory feature set.

[0070] S104: Replace the memory feature set in the model to be transferred with the target memory feature set to obtain the target detection model.

[0071] It should be noted that after obtaining the target memory feature set, the memory feature set in the model to be transferred is replaced with the target memory feature set. The resulting model is the object detection model, thus completing the transfer update of the model to be transferred. Furthermore, if there is a need for model transfer after obtaining the object detection model, the object detection model is used as the model to be transferred, and the transfer update is performed according to the aforementioned method.

[0072] In some embodiments, if the model to be transferred is a model that has not undergone pruning and quantization, the method may further include:

[0073] At least one sample of data from the target scene is augmented to obtain an augmented dataset;

[0074] The model to be transferred is trained by backpropagation using an expanded dataset, so as to update the model parameters in the model to be transferred, except for the memory feature set.

[0075] It should be noted that if the model to be transferred is a pruned quantized model, then because unimportant channels in the model are deleted and the floating-point weights and biases in the model are approximated with low-precision integers, the model can occupy less memory and has a faster computation speed. However, this model is no longer suitable for calculating parameter gradients. Therefore, when updating the pruned quantized model to be transferred, the memory feature set replacement method provided in the embodiments of this application can be used.

[0076] For models that have not undergone pruning and quantization, in addition to updating the memory feature set, backpropagation training can be used to iteratively update other parameters of the model, such as network weights.

[0077] When the number of sample data in at least one of the target scenarios is small, backpropagation training may lead to model overfitting. Therefore, the sample data of the target scenario can be augmented to obtain a larger number of sample data to form an augmented dataset. The augmented dataset can then be used to perform backpropagation training on the model to be transferred, and iterative updates can be performed to update other model parameters of the model to be transferred.

[0078] Data augmentation methods can include mirroring, rotating (e.g., rotating a sample image by a certain angle), scaling (e.g., changing the resolution of an image), extracting (e.g., extracting a portion of a sample image as a new sample image), color dithering (e.g., adding slight noise), and so on.

[0079] In this way, the resulting object detection model not only updates the memory module, but also updates the other parameters. That is to say, if backpropagation training is not involved, the embodiments of this application mainly update the memory module when updating the model to be transferred. This makes the update faster and can be completed without a large amount of sample data.

[0080] Furthermore, in some embodiments, when using an object detection model to detect an image, the method may further include:

[0081] Acquire the image to be detected;

[0082] The target detection model is used to perform measurement calculations on the image to be detected and the target memory feature set to obtain the target measurement calculation results;

[0083] The target detection result is determined based on the target metric calculation result; wherein, the target memory feature set includes at least one memory feature information.

[0084] It should be noted that, taking the image to be detected as the object to be detected as an example, when detecting the image to be detected, after obtaining the image to be detected, the image to be detected can be input into the target detection model, and the image to be detected and the target memory feature set can be metrically calculated to obtain the target metric calculation result, and the target detection result can be determined based on the target metric calculation result.

[0085] Furthermore, in some embodiments, the target detection model is used to perform measurement calculations on the image to be detected and the target memory features to obtain the target measurement calculation result, which may include:

[0086] Feature extraction is performed on the image to be detected to obtain at least one depth feature information;

[0087] The target measurement result is obtained by measuring at least one deep feature and at least one memory feature.

[0088] It should be noted that the target feature set includes at least one memory feature, which is the aforementioned feature vector that is related to the target scene, i.e., the prototype vector of the target object that needs to be detected in the target scene.

[0089] The measurement and calculation of the memory features of the image to be detected mainly refers to the measurement and calculation of the depth feature information and memory feature information of the image to be detected. Therefore, feature extraction is first performed on the image to be detected. Here, the encoder in the object detection model can be used to extract features of the object to be detected, and at least one depth feature information is obtained. This at least one depth feature information is mainly the part of the feature information in the image to be detected that corresponds to at least one memory feature information.

[0090] A measurement calculation is performed between at least one deep feature information and at least one memory feature information to obtain at least one measurement calculation result, namely the target measurement calculation result.

[0091] After obtaining the target metric calculation result, the target detection result is determined based on the target metric calculation result.

[0092] In some embodiments, determining the target detection result based on the target metric calculation result may include:

[0093] The target measurement calculation results are decoded to obtain the target detection results of the image to be detected.

[0094] It should be noted that when determining the object detection result, the object metric calculation result can be decoded. This can be done using a decoder within the object detection model to obtain the object detection result. For example, decoding can determine the type of the target object in the image to be detected, or mark the bounding box of the region containing the target object in the image to be detected.

[0095] This application provides a detection method that involves obtaining a model to be transferred, wherein the model to be transferred includes a memory feature set, which represents a set of feature vectors related to the application scenario corresponding to the model to be transferred; measuring at least one sample data of the target scenario with the memory feature set to obtain at least one measurement calculation result; updating the memory feature set based on the at least one measurement calculation result to obtain a target memory feature set; wherein the target memory feature set represents a set of feature vectors related to the target scenario; and replacing the memory feature set in the model to be transferred with the target memory feature set to obtain a target detection model. Thus, since the detection result is determined by decoding based on the measurement calculation result between the image to be detected and the memory feature set when using the model for target detection, the detection result is only related to the measurement calculation result and does not depend on the depth feature information of the image to be detected itself. Therefore, when performing model transfer updates, only a small amount of sample data from the target scenario is needed to obtain the target memory feature set to replace the memory feature set in the memory module, without needing to retrain the model's decoder. In this way, on the one hand, for new scenarios with a small number of training samples, small-sample transfer updates of the model can be achieved; on the other hand, since the update of the memory feature set is actually a process of replacing the prototype vector, that is, only the memory feature set in the memory module needs to be replaced to complete the model transfer update, so that a large number of iterative calculations are not required during the model transfer update process, and the adaptation update can be completed in a short time (e.g., a few minutes), saving time and manpower costs; furthermore, since the model detection process based on the metric calculation results is a forward inference process that does not rely on backpropagation, there is no need to construct a loss function and calculate gradients when performing model transfer updates, and transfer updates can also be achieved for quantized models.

[0096] In another embodiment of this application, see Figure 2 This illustrates a schematic diagram of the flow architecture of a detection method provided in an embodiment of this application. Figure 2 As shown, the process architecture can mainly include the cloud preparation phase 201, the edge deployment phase 202, and the new scenario migration phase 203.

[0097] In the cloud preparation phase 201, meta-learning is mainly performed based on training samples from relevant task groups to build an initial model based on "memory modules". In the edge deployment phase 202, the initial model is fine-tuned based on business data from edge device application scenarios to obtain the corresponding detection model. In the new scenario migration phase, the model is migrated and updated based on business data from the new scenario when edge devices switch scenarios to obtain the corresponding detection model for the new scenario.

[0098] It should be noted that, as Figure 2 As shown in this embodiment, small-sample updates of the model can be achieved based on multi-task metric learning. The detection model can include three parts: an encoder, a decoder, and a memory module. During object detection, forward inference of the model is used. In simple terms, the inference principle and process of the detection model mainly include the following steps:

[0099] (1) Input the image to be detected into the encoder and obtain the depth feature information of the image to be detected through the encoder.

[0100] (2) The depth feature information of the image to be detected is measured and calculated with the memory feature information in the memory module to obtain the measurement calculation result.

[0101] (3) Input the measurement calculation result into the decoder to decode the detection result.

[0102] It should be noted that for this type of detection model based on "encoder-decoder-memory module", multi-task pre-training is usually required to build the initial model so that the detection model can be updated by updating the memory feature set in the memory module, thereby achieving scene adaptation.

[0103] It should also be noted that the relevant processes in the cloud preparation phase 201 are usually executed on cloud devices. The main content of this phase is to train the encoder and decoder of the model, so that the model can extract features from the input image and decode the detection results based on the metric calculation results.

[0104] Specifically, on cloud devices, meta-learning can be performed based on training samples from relevant task groups to build an initial model based on "memory modules." These relevant task groups typically consist of multiple tasks similar to the target task but with a larger amount of training data; they are generally determined manually by the model developers. Figure 2 Task 1, Task 2, ..., Task n in the model; the target task represents the specific task corresponding to the actual use of the model.

[0105] Taking object detection as an example, let's say the target task is to detect animal A. This involves determining whether an animal in an image is animal A, and / or selecting the region containing animal A within that image. Related task groups could include detection tasks for animal categories such as B, C, D, and E. In this case, the memory module can store memory feature sets for each of these categories. Meta-learning is then performed using these animal categories as training samples to obtain the initial model.

[0106] In the embodiments of this application, the meta-learning process can be referred to as a multi-task full metric learning process. Here, multi-task means that the training samples used to train the initial model come from a group of related tasks that are similar to the target task, and the number of training samples in the related task group is large; full metric learning means that for the model's decoder, the final detection result is only related to the metric calculation result between the depth feature information and memory feature information of the input image, and no longer depends on the original depth feature information of the input image.

[0107] In the multi-task full-metric learning process, firstly, sample images from relevant task groups are input into the decoder to obtain the depth feature information of the sample images. Specifically, the depth feature information of the sample images mainly includes partial features corresponding to the memory feature information in the memory module. Then, for each decoded depth feature information, a metric calculation is performed between it and the corresponding memory feature information in the memory module to obtain the metric calculation result for each depth feature information. Finally, the metric calculation result is input into the decoder for decoding to obtain the detection result. The detection result is compared with the true detection result of the sample image, and the current detection accuracy achieved by the model is determined based on the comparison result. The process continues to iterate with the next sample image until the detection accuracy of the model reaches the preset accuracy. Here, when judging the detection accuracy of the model, a loss function can be set. When the value of the loss function is less than the preset value, the detection accuracy of the model is determined to have reached the preset accuracy, thus obtaining the initial model. Alternatively, the initial model can be determined when the number of iterations reaches the preset value.

[0108] It should also be noted that the set of memory features stored in the initial model's memory module corresponds to the application scenario of the relevant task group, but not necessarily to the application scenario of the model on the edge device. Therefore, the memory module can be cleared. In other words, at this point, only an initial model with a memory module is obtained, but the memory features in the memory module need to be determined when deploying the model on the edge device.

[0109] After obtaining the initial model, it can be fine-tuned on the edge device to obtain a detection model that meets the application scenario requirements of the edge device. Here, fine-tuning the initial model means determining the set of memory features based on the current application scenario of the edge device to update the memory module.

[0110] For example Figure 2 As shown, the model can be deployed on multiple edge devices. For a new task A, based on the sample data of the new task A, the memory feature set corresponding to the application scenario of the new task A is determined by forward reasoning on the initial model, and the corresponding memory feature set is stored in the memory module; the same applies to the new task B.

[0111] In this way, when using the model for object detection, the encoder obtains the depth features of the image to be detected, and then measures and calculates the depth feature information and the memory feature information in the memory feature set respectively to obtain the measurement calculation results. After the measurement calculation results are input into the decoder, the decoder decodes them to obtain the final detection results.

[0112] Since edge devices typically have lower computing power and limited storage space, alternatively, during the edge deployment phase 202, the fine-tuned model can be pruned and quantized. This involves removing unnecessary channels or unimportant connections from the model and quantizing the model's weights to enable weight sharing, thereby reducing the model's memory consumption without sacrificing its detection accuracy. The model corresponding to the new task A is denoted as detection model A, and the model corresponding to the new task B is denoted as detection model B.

[0113] Furthermore, after detection model A is successfully deployed on the edge device, if it is necessary to switch the application scenario of the model, such as to use the model to perform a new detection task or to migrate the model to other edge devices, the memory module of detection model A can be updated using the sample data of the new task C based on the aforementioned update method, thereby obtaining detection model C.

[0114] It is important to note that if the detection model A is a pruned and quantized model, then usually only the memory module can be updated; however, if the detection model A is not pruned and quantized, then the backpropagation method can be used to update other parameters of the model.

[0115] It should also be noted that when using the backpropagation method to update the model, since the amount of sample data for the new task is usually small, data augmentation methods can be used to expand the small amount of sample data for the new task in order to achieve iterative training and updating of the model.

[0116] In short, the detection method provided in this application mainly includes the following steps:

[0117] (1) Cloud preparation stage: Based on the training sample data of relevant task groups, meta-learning is carried out to build an initial model based on "memory module".

[0118] (2) Edge deployment computing segment: Based on business data, the initial model is fine-tuned (optionally, after fine-tuning, the model can also be pruned and quantized), and the "memory module" of the business is obtained and stored through forward inference.

[0119] (3) New scene migration stage: When switching scenes, a small amount of new scene data is used to generate a new “memory module” through forward inference of the model, and the old “memory module” is replaced.

[0120] Optional: If the model has not been pruned or quantized, data augmentation methods can be used to augment a small amount of data, and then backpropagation can be used to update some of the model's parameters.

[0121] This application provides a detection method. The specific implementation of the aforementioned embodiments is described in detail through the above embodiments. It can be seen that this solution can construct an initial model based on multi-task full-metric learning. During model transfer, only a small amount of sample data is needed to update the model's memory module, enabling the model to quickly adapt to new application scenarios. Compared with related technologies, this method has at least the following advantages: Based on the full-metric decoding mechanism, the final prediction result of the detection model is only related to the metric calculation result between the depth feature information and memory feature information of the image to be detected. This makes the memory feature set strongly correlated with a specific scene, while the rest of the model is weakly correlated with the scene. Therefore, by replacing the model's memory feature set, the model can be updated using only a small number of labeled samples (e.g., tens to hundreds). Since updating the memory module is actually based solely on replacing the prototype vector, the model update process does not require a large number of iterations and can be completed in minutes, allowing for rapid model updates. Since the model update relies only on forward inference, there is no need to construct a loss function or calculate gradients. The quantized model is also applicable, thus the model update does not rely on backpropagation.

[0122] In another embodiment of this application, see [link to application]. Figure 3 This illustrates a schematic diagram of the composition of a detection device 30 provided in an embodiment of this application. Figure 3 As shown, the detection device 30 may include:

[0123] The acquisition unit 301 is configured to acquire the model to be transferred; wherein the model to be transferred includes a set of memory features, and the memory features are adapted to the application scenario corresponding to the model to be transferred.

[0124] The update unit 302 is configured to perform metric calculations on at least one sample data of the target scene and the memory feature set respectively to obtain at least one metric calculation result, and update the memory feature set according to the at least one metric calculation result to obtain a target memory feature set adapted to the target scene; and replace the memory feature set in the model to be transferred with the target memory feature set to obtain a target detection model.

[0125] In some embodiments, such as Figure 3 As shown, the detection device 30 may further include a calculation unit 303 and a determination unit 304, wherein,

[0126] The acquisition unit 301 is also configured to acquire the image to be detected;

[0127] The computing unit 303 is configured to use the target detection model to perform measurement calculations on the image to be detected and the target memory feature set to obtain the target measurement calculation result.

[0128] The determining unit 304 is configured to determine the target detection result based on the target metric calculation result; wherein the target memory feature set includes at least one memory feature information.

[0129] In some embodiments, the calculation unit 303 is specifically configured to perform feature extraction on the image to be detected to obtain at least one depth feature information; and to perform measurement calculation on the at least one depth feature information and at least one memory feature information to obtain a target measurement calculation result.

[0130] In some embodiments, the determining unit 304 is specifically configured to decode the target metric calculation result to obtain the target detection result of the image to be detected.

[0131] In some embodiments, the acquisition unit 301 is further configured to determine a target scene; acquire at least one initial sample data of the target scene based on the target scene; and perform annotation processing on the at least one initial sample data to obtain at least one sample data of the target scene.

[0132] In some embodiments, the model to be transferred includes a memory module, and the memory module stores a set of memory features. The acquisition unit 301 is also configured to acquire an initial training set, which includes training samples corresponding to several tasks.

[0133] The update unit 302 is also configured to train the model based on the initial training set to establish an initial model including a memory module; and to acquire real image data and use the real image data to adjust the initial model to obtain the model to be transferred.

[0134] In some embodiments, the steps of obtaining an initial training set and training the model based on the initial training set to build an initial model including a memory module are performed on a cloud device.

[0135] In some embodiments, the update unit 302 is further configured to augment at least one sample data of the target scene to obtain an augmented dataset when the model to be transferred has not undergone pruning and quantization processing; and to use the augmented dataset to perform backpropagation training on the model to be transferred in order to update the model parameters in the model to be transferred, excluding the memory feature set.

[0136] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0137] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] Therefore, this embodiment provides a computer storage medium storing a computer program that, when executed by a plurality of processors, implements the steps of any of the methods in the foregoing embodiments.

[0139] Based on the above-described composition of the detection device 30 and the computer storage medium, see [link to relevant documentation]. Figure 4This illustrates a schematic diagram of the hardware structure of an electronic device 40 provided in an embodiment of this application. For example... Figure 4 As shown, electronic device 40 may include: a communication interface 401, a memory 402, and a processor 403; the various components are coupled together via a bus device 404. It is understood that the bus device 404 is used to implement communication between these components. In addition to a data bus, the bus device 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 All buses are labeled as bus devices 404. Among them, the communication interface 401 is used for receiving and sending signals during information exchange with other external network elements.

[0140] Memory 402 is used to store computer programs that can run on processor 403;

[0141] Processor 403, when running the computer program, performs the following:

[0142] Obtain the model to be transferred; wherein, the model to be transferred includes a memory feature set, which represents a set of feature vectors that are related to the application scenario corresponding to the model to be transferred;

[0143] Perform metric calculations on at least one sample data of the target scene with the memory feature set to obtain at least one metric calculation result;

[0144] The memory feature set is updated based on at least one metric calculation result to obtain the target memory feature set; wherein, the target memory feature set represents the set of feature vectors that are related to the target scene;

[0145] The target detection model is obtained by replacing the memory feature set in the model to be transferred with the target memory feature set.

[0146] It is understood that the memory 402 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 402 of the systems and methods described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] The processor 403 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 403 or by instructions in software form. The processor 403 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 402, and the processor 403 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0148] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0149] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0150] Alternatively, as another embodiment, the processor 403 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0151] In another embodiment of this application, a schematic diagram of the composition of the model detection device 30 described above is provided (see attached diagram). Figure 5 This illustrates a schematic diagram of the structural composition of an electronic device 40 provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 40 includes at least one of the detection devices 30 in the foregoing embodiments.

[0152] For the electronic device 40, since it includes the detection device 30, when migrating the model to be transferred to the target scene, based on the measurement calculation between features, it is only necessary to use the sample data of the target scene to update and replace the memory feature set of the model to be transferred, so that the updated target memory feature set is suitable for the target scene. Thus, the update and migration of the model to be transferred can be achieved with a small number of sample data. The update speed is fast, saving time and labor costs. In addition, this method is also applicable to the migration and update of models that have undergone pruning and quantization, and has a wide range of applications.

[0153] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

[0154] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0155] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0157] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0158] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A detection method, the method comprising: Obtain the model to be transferred; wherein, the model to be transferred includes a memory feature set, the memory feature set representing a set of feature vectors that are related to the application scenario corresponding to the model to be transferred; At least one sample data of the target scene is measured and calculated with the memory feature set to obtain at least one measurement calculation result; The memory feature set is updated based on the calculation result of at least one metric to obtain a target memory feature set; wherein, the target memory feature set represents a set of feature vectors that are associated with the target scene; The target memory feature set is used to replace the memory feature set in the model to be transferred, thus obtaining the target detection model; Both the model to be transferred and the target detection model are used for image detection.

2. The method according to claim 1, further comprising: Acquire the image to be detected; The target detection model is used to perform measurement calculations on the image to be detected and the target memory feature set to obtain the target measurement calculation results. The target detection result is determined based on the target metric calculation result; wherein, the target memory feature set includes at least one memory feature information.

3. The method according to claim 2, wherein the step of using the target detection model to measure and calculate the image to be detected and the target memory features to obtain the target measurement calculation result includes: Feature extraction is performed on the image to be detected to obtain at least one depth feature information; The target measurement calculation result is obtained by measuring and calculating the at least one depth feature information and the at least one memory feature information.

4. The method according to claim 2, wherein determining the target detection result based on the target metric calculation result includes: The target metric calculation result is decoded to obtain the target detection result of the image to be detected.

5. The method according to claim 1, wherein before performing measurement calculations on at least one sample data of the target scene with the memory feature set, the method further comprises: Identify the target scenario; Based on the target scenario, obtain at least one initial sample data of the target scenario; The at least one initial sample data is labeled to obtain at least one sample data of the target scene.

6. The method according to claim 1, wherein the model to be transferred includes a memory module, and the memory module stores the memory feature set, and the method further includes: Obtain an initial training set, which includes training samples corresponding to several tasks; The model is trained based on the initial training set to establish an initial model including the memory module; Acquire real image data and use the real image data to adjust the initial model to obtain the model to be transferred.

7. The method according to any one of claims 1 to 6, wherein the method further comprises: Without pruning and quantization processing, at least one sample data of the target scene is augmented to obtain an augmented dataset. The expanded dataset is used to perform backpropagation training on the model to be transferred, so as to update the model parameters in the model to be transferred, except for the memory feature set.

8. A detection device, comprising: The acquisition unit is configured to acquire the model to be transferred; wherein the model to be transferred includes a set of memory features, and the memory features are adapted to the application scenario corresponding to the model to be transferred; The update unit is configured to perform metric calculations on at least one sample data of the target scene and the memory feature set respectively to obtain at least one metric calculation result, and update the memory feature set according to the at least one metric calculation result to obtain a target memory feature set adapted to the target scene; and replace the memory feature set in the model to be transferred with the target memory feature set to obtain a target detection model. Both the model to be transferred and the target detection model are used for image detection.

9. An electronic device, the electronic device comprising a memory and a processor, wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to execute the detection method as described in any one of claims 1 to 7 when running the computer program.

10. A computer storage medium storing a computer program that, when executed by at least one processor, implements the detection method as described in any one of claims 1 to 7.

Images