An image processing method, apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application disclose an image processing method and device, electronic equipment and storage medium. The method comprises: processing an input image to be processed based on a pre-trained feature extraction model to obtain feature sub-vectors corresponding to multiple model stages in the feature extraction model; if the receptive field of the neuron corresponding to each feature sub-vector is consistent with the pre-set region of interest, the elements matched with the region of interest are retained to obtain a target feature sub-vector corresponding to each feature sub-vector; the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library is determined, and the sample with a distance threshold less than the corresponding pre-set distance threshold is taken as a target similar sample of the image to be processed; and the classification result corresponding to the target similar sample is taken as the target classification result of the image to be processed. The problem that the system cannot process special samples is solved, and the processing of special samples is realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to image processing technology, and more particularly to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of the internet, digital image processing has become widespread in daily life, especially in applications such as medicine and intelligent robotics. The introduction of image information has greatly benefited users, and it can be said that the continuous innovation of image processing technology is an important part of modern life.

[0003] In image recognition systems, the image to be recognized is input into a general image recognition model, which then outputs a general result. However, when a specific sample is input, the general image recognition model may fail to process the specific sample or output a specific result. Therefore, existing models cannot process specific samples, and consequently, cannot obtain classification results corresponding to those specific samples, leading to technical problems related to data processing inconvenience. Summary of the Invention

[0004] This invention provides an image processing method, apparatus, electronic device, and storage medium to achieve the effect of processing special samples without retraining the image recognition model.

[0005] In a first aspect, embodiments of the present invention provide an image processing method, the method comprising:

[0006] The input image to be processed is processed based on a pre-trained feature extraction model to obtain feature sub-vectors corresponding to multiple model stages in the feature extraction model;

[0007] If the receptive field of the neuron corresponding to each feature sub-vector is consistent with the pre-defined region of interest, then the element that matches the region of interest is retained to obtain the target feature sub-vector corresponding to each feature sub-vector;

[0008] Determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed;

[0009] The classification results corresponding to the target similar samples are used as the target classification results of the image to be processed.

[0010] Secondly, embodiments of the present invention also provide an image processing apparatus, the apparatus comprising:

[0011] The feature vector acquisition module is used to process the input image to be processed based on a pre-trained feature extraction model to obtain feature vectors corresponding to multiple model stages in the feature extraction model.

[0012] The target feature vector acquisition module is used to retain the elements that match the region of interest if the receptive field of the neuron corresponding to each feature vector is consistent with a pre-defined region of interest, thereby obtaining the target feature vector corresponding to each feature vector.

[0013] The target similarity sample determination module is used to determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and to take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed.

[0014] The target classification result determination module is used to take the classification result corresponding to the target similar sample as the target classification result of the image to be processed.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, the device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of the present invention.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described in any of the embodiments of the present invention.

[0020] The technical solution of this invention extracts features from the image to be processed based on a pre-trained feature extraction model, obtaining feature sub-vectors corresponding to multiple model stages in the feature extraction model. If the receptive field of the neuron corresponding to each feature sub-vector matches a pre-defined region of interest, the element matching the region of interest is retained to obtain a target feature sub-vector corresponding to each feature sub-vector. A distance threshold is determined between each target feature sub-vector and the sample sub-vectors of each sample in the sample library, and samples with a distance threshold less than the preset distance threshold are taken as target similar samples of the image to be processed. The classification result corresponding to the target similar sample is taken as the target classification result of the image to be processed. This solves the problem that existing image recognition systems cannot process special samples, and achieves the effect of processing special samples without retraining the image recognition model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of exemplary embodiments of the present invention, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the drawings of the embodiments to be described in this invention, and not all of the drawings. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0022] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a schematic diagram of an image processing method provided in Embodiment 2 of the present invention;

[0024] Figure 3 This is a structural block diagram of an image processing device provided in Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0027] Example 1

[0028] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the processing of special samples and the determination of corresponding classification results. The method can be executed by the image processing device in this embodiment, which can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server. The device can be configured in a computing device. The image processing method provided in this embodiment specifically includes the following steps:

[0029] S110. Based on the pre-trained feature extraction model, the input image to be processed is processed to obtain the feature sub-vectors corresponding to multiple model stages in the feature extraction model.

[0030] The image to be processed is the image whose classification result needs to be determined. By inputting the image into the model, the predicted classification result output by the model can be obtained.

[0031] In this embodiment, the feature extraction model is divided into multiple different model stages. Each model stage can extract features from the image to be processed and output a corresponding feature vector. The feature vector output by each model stage is used as the feature sub-vector of that stage. That is, the neural network layer of a model is divided into multiple model stages.

[0032] Specifically, a feature extraction model is pre-trained using a large number of images. This model can process images and obtain classification results. At this stage, the model can process ordinary samples and determine the corresponding classification results. After the feature extraction model is trained, the image to be processed is input into the trained model. The model can extract features from the image in multiple stages and output feature vectors corresponding to each stage. The feature vectors output by each stage are then used as the feature sub-vectors corresponding to that stage.

[0033] Optionally, the feature extraction model includes at least three model stages. The step of processing the input image to be processed based on the pre-trained feature extraction model to obtain feature sub-vectors corresponding to multiple model stages in the feature extraction model includes: inputting the image to be processed into the feature extraction model, processing it sequentially through the at least three model stages, and obtaining feature sub-vectors matching each model stage.

[0034] Each model stage includes multiple neural network layers. The feature extraction model includes at least three model stage layers. Regardless of whether the feature extraction model contains three or more model stages, the processing method is the same. Taking a three-stage model as an example, the feature extraction model can be divided into the initial stage, the middle stage, and the advanced stage.

[0035] It should be noted that the initial, middle, and advanced stages of the model are all complete neural network layers, each capable of outputting feature vectors corresponding to the image to be processed.

[0036] Specifically, the image to be processed is input into the feature extraction model, which sequentially extracts features from the image through the initial, middle, and final stages of the model, obtaining feature vectors corresponding to each stage. That is, the initial stage extracts features from the image and outputs the feature vectors for that stage; the middle stage extracts features from the image and outputs the feature vectors for that stage; and the final stage extracts features from the image and obtains the feature vectors for that stage.

[0037] S120. If the receptive field of the neuron corresponding to each feature sub-vector is consistent with the pre-defined region of interest, then the element that matches the region of interest is retained to obtain the target feature sub-vector corresponding to each feature sub-vector.

[0038] In this embodiment, each model stage is a complete neural network layer. Each feature vector corresponds to the image to be processed, but the semantic depth of the feature vectors output by the primary model stage is lower than that of the feature vectors output by the intermediate model stage.

[0039] In this context, the pre-defined regions of interest (ROIs) are typical or important image regions pre-selected by the developers. Neurons correspond to the areas being tested, and their receptive fields can be the visible areas corresponding to those areas. For example, using a fundus image as the image to be processed, the image regions corresponding to the optic disc, macula, and blood vessels are designated as pre-defined ROIs. The receptive fields of neurons corresponding to the feature vectors of the fundus image are matched with the image regions corresponding to the optic disc, macula, and blood vessels. If the receptive field of a neuron matches the ROI, the array elements in the feature vector that match the ROI are retained, resulting in the target feature vector.

[0040] Specifically, the feature vector is consistent with the image to be processed, but the content in the image to be processed is not entirely within the predefined region of interest. In this case, algorithms can be used to calculate the similarity between the neuron's receptive field and the predefined region of interest. If the similarity reaches a certain threshold, the neuron's receptive field can be considered consistent with the predefined region of interest. Then, the array elements in the feature vector that match the region of interest are retained, resulting in the target feature vector.

[0041] For example, using a fundus image as the image to be processed, the images corresponding to the optic disc, macula, and blood vessels in the fundus image are important bases for medical diagnosis during medical image recognition. The image regions corresponding to the optic disc, macula, and blood vessels can be designated as pre-defined regions of interest (ROIs). The receptive fields of neurons corresponding to the feature vectors of the fundus image are matched with the image regions corresponding to the optic disc, macula, and blood vessels. If the receptive fields of neurons match the ROIs, the array elements in the feature vectors that match the ROIs are retained, resulting in the target feature vector.

[0042] Optionally, if the receptive field of the neuron corresponding to each feature sub-vector is consistent with a pre-defined region of interest, then the elements matching the region of interest are retained to obtain a target feature sub-vector corresponding to each feature sub-vector, including: for each feature sub-vector, performing masking processing on the sub-image corresponding to the current feature sub-vector, and using the unmasked region as the receptive field of the neuron; if the receptive field overlaps with the region of interest, then the elements corresponding to the unmasked region are retained; repeatedly performing masking processing on the sub-image corresponding to the current feature sub-vector to obtain a target feature sub-vector corresponding to the current feature sub-vector.

[0043] Masking refers to the process of removing or covering certain regions of an image. Unmasked regions refer to the areas in the image that are not removed or covered after masking.

[0044] Specifically, each feature vector has a corresponding sub-image. By performing a bitwise AND operation between each pixel in the sub-image corresponding to each feature vector and each corresponding pixel in the mask, the masked image can be obtained. The unmasked image region in the masked image is used as the receptive field of the neuron. Developers can pre-define the region of interest (ROI) and compare it with the receptive field. If there is an overlap between the receptive field and the ROI, it means that the image region corresponding to the receptive field is the region of interest to the user, and the array element corresponding to the unmasked region is retained. Since the image to be processed has corresponding feature vectors at different model stages, it is necessary to repeatedly perform sub-image masking for all feature vectors and obtain the array elements corresponding to the overlapping areas of the receptive field and ROI for each feature vector, and use these array elements as the target feature vector.

[0045] S130. Determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed.

[0046] Developers can pre-build a sample library containing many special samples. The sample library also stores the sample sub-vectors obtained after processing each special sample in different model stages. For example, if there is a special sample A in the sample library, sample A can be processed through three model stages in sequence to obtain the sample sub-vectors a1, a2, and a3 corresponding to the three model stages.

[0047] The preset distance threshold can be a threshold set by the developers in advance, which can be used to determine the similarity between the target feature vector and the sample vectors in the sample library.

[0048] Specifically, the distance threshold between the target feature vector and the sample vectors of each sample in the sample library is calculated using a cosine similarity algorithm or Euclidean distance. A pre-set distance threshold is used; if the distance threshold between the target feature vector and the sample vectors of one or more samples in the sample library is less than the pre-set threshold, the target feature vector is considered to have a high similarity to those samples, and those samples can be used as target similar samples for the image to be processed.

[0049] Optionally, determining the distance threshold between each target feature vector and the sample vector of each sample in the sample library includes: sequentially determining the cosine similarity value between the target feature vector in the first model stage and the sample vector of each sample in the sample library in the first model stage; normalizing the target feature vectors in the second and third model stages to determine the target Euclidean distance corresponding to each target feature vector; and determining the distance threshold between the target feature vector and the corresponding sample vector based on the target Euclidean distance and the sample Euclidean distance of the sample vectors in the sample library.

[0050] Optionally, determining the distance threshold between the target feature vector and the corresponding sample sub-vector based on each target Euclidean distance and the sample Euclidean distance of the sample sub-vectors in the sample library includes: determining the sample Euclidean distances corresponding to the sample sub-vectors in the second model stage and the third model in the sample library, respectively; and for sample sub-vectors in the same model stage, determining the distance threshold between the target feature vector and each sample sub-vector based on the target Euclidean distance of the current model stage and the magnitude of the target feature vector, as well as the sample sub-vectors in the current model stage and the corresponding sample Euclidean distances.

[0051] The feature extraction model comprises three stages: the first stage, the second stage, and the third stage. Each stage corresponds to a target feature vector. For example, the target feature vector for the first stage is x1, for the second stage it is x2, and for the third stage it is x3.

[0052] It should be noted that each sample also has a corresponding sample sub-vector in each model stage. For example, the sample sub-vector corresponding to sample A in the first model stage is a1, the sample sub-vector corresponding to sample A in the second model stage is a2, and the sample sub-vector corresponding to sample A in the third model stage is a3.

[0053] The target Euclidean distance refers to the Euclidean distance calculated using the Euclidean distance algorithm for the target feature vectors, and the calculated Euclidean distance is used as the target Euclidean distance. The sample Euclidean distance refers to the Euclidean distance calculated using the Euclidean distance algorithm for the sample vectors, and the calculated Euclidean distance is used as the sample Euclidean distance.

[0054] Specifically, for the target feature vector in the first model stage, the cosine similarity value between the sample subvector of each sample in the sample library in the first model stage and the target feature vector in the first model stage is calculated. For the second model stage, the target feature vector in this stage is first normalized. Then, the Euclidean distance of the target feature vector in this stage is calculated as the target Euclidean distance. Further, the Euclidean distance of the sample subvectors of each sample in the sample library in this stage is calculated as the sample Euclidean distance. Finally, the magnitude of the target feature vector in the second stage is calculated, and the target Euclidean distance is divided by this magnitude to obtain one result; the magnitude of the sample subvector in the second stage is calculated, and the sample Euclidean distance is divided by the sample magnitude to obtain another result. Adding the two results yields the distance threshold. The calculation method for the third model stage is the same as that for the second model stage and will not be repeated.

[0055] For example, the target feature vector is X, and the feature sub-vectors in the three model stages are x1, x2, and x3. A sample A in the sample library and its corresponding sample sub-vectors a1, a2, and a3.

[0056] For the first stage, the cosine similarity value between x1 and a1 is calculated. For the second stage, x2 is normalized, and its Euclidean distance is calculated as the target Euclidean distance; the Euclidean distance of a2 is calculated as the sample Euclidean distance. The target Euclidean distance is divided by the modulus of x2 (2) to obtain x2 / 2; the Euclidean distance of a2 is divided by the modulus of a2 (3) to obtain a2 / 3. Adding x2 / 2 to a2 / 3 yields the distance threshold between the target feature vector and the sample subvector A in the second stage model. For the third model stage, the distance threshold between x3 and a3 is calculated in the same way as in the second stage.

[0057] Optionally, the step of using samples whose distance threshold is less than a corresponding preset distance threshold as target similar samples of the image to be processed includes: for each sample, if the cosine similarity value of the current sample in the first model stage is greater than a preset similarity threshold, and the distance threshold of the current sample in the second and third model stages is less than the corresponding preset distance threshold, then the current sample is determined as a candidate similar sample; if the number of candidate similar samples is greater than a preset number threshold, then a target similar sample is determined from the candidate similar samples based on a pre-determined weight value corresponding to each model stage.

[0058] The preset similarity threshold is a pre-set value used to determine the matching degree between a sample and the target feature vector in the first model stage. The preset distance threshold is a pre-set value used to determine the matching degree between a sample and the target feature vector in the second and third model stages. The target similar sample is the sample in the sample library with the highest matching degree to the target feature vector.

[0059] Specifically, for all samples, if the cosine similarity between a sample's sub-vector and its target feature vector in the first model stage is greater than a preset similarity threshold, and the distance threshold between the sample's sub-vector and its target feature vector in the second model stage is less than a preset distance threshold, and the distance threshold between the sample's sub-vector and its target feature vector in the third model stage is less than a preset distance threshold, then the sample's sub-vector and target feature vector are considered to have a high similarity, and this sample can be selected as a candidate sample. Since there may be multiple candidate samples, a preset number threshold needs to be set. When the number of candidate similar samples exceeds the preset number threshold, the target similar sample is determined from the candidate similar samples according to the preset weights for each model stage.

[0060] For example, the similarity threshold between sample 1 and target feature vector 2 in the first model stage is D1, and the distance thresholds in the second and third model stages are D2 and D3, respectively. The preset similarity threshold is 0.9, and the preset distance thresholds are 0.05 and 1, respectively. When D1 > 0.9, D2 < 0.05, and D3 < 1, sample 1 is a candidate sample for the target feature vector. When the number of candidate similar samples exceeds the preset number threshold, the target similar sample needs to be determined from the candidate similar samples according to the pre-set weights of each model stage.

[0061] Optionally, determining the target similar sample from the candidate similar samples based on the pre-determined weight values ​​corresponding to each model stage includes: for each candidate similar sample, determining the similarity value of the candidate similar sample according to the cosine similarity value, distance threshold and corresponding weight value corresponding to each model stage; and taking the candidate similar sample with the highest similarity as the target similar sample.

[0062] Specifically, for each candidate similar sample, a weighted sum of the cosine similarity value, the distance threshold of the second model stage, and the distance threshold of the third model stage is calculated based on the pre-set weight values ​​of each stage. The negative of the weighted sum is used as the similarity value of each candidate sample. The candidate similar sample with the highest similarity value is selected as the target similar sample.

[0063] S140. The classification result corresponding to the target similar sample is used as the target classification result of the image to be processed.

[0064] It should be noted that the target similar sample is a special sample in the sample library, and this special sample has a corresponding specific classification result.

[0065] Specifically, the target similarity sample is the special sample that has the highest matching degree with the target feature sub-vector of the image to be processed. Therefore, the classification result corresponding to the special sample can be used as the classification result of the image to be processed and output.

[0066] For example, when the image to be processed is a medical image, the target classification result can be the result corresponding to whether there is a lesion and the type of lesion.

[0067] In practical applications, if the distance threshold between each target feature vector and the sample vector of each sample in the sample library is greater than a preset distance threshold, it indicates that there are no target-like samples similar to the target feature vectors. Therefore, the image to be processed can be considered not a special sample, but just an ordinary sample. In this case, the image to be processed can be input into a pre-trained target classification model to obtain the target classification result corresponding to the image to be processed.

[0068] The technical solution of this invention extracts features from the image to be processed based on a pre-trained feature extraction model, obtaining feature sub-vectors corresponding to multiple model stages in the feature extraction model. If the receptive field of the neuron corresponding to each feature sub-vector matches a pre-defined region of interest, the element matching the region of interest is retained to obtain a target feature sub-vector corresponding to each feature sub-vector. A distance threshold is determined between each target feature sub-vector and the sample sub-vectors of each sample in the sample library, and samples with a distance threshold less than the preset distance threshold are taken as target similar samples of the image to be processed. The classification result corresponding to the target similar sample is taken as the target classification result of the image to be processed. This solves the problem that existing image recognition systems cannot process special samples, and achieves the effect of processing special samples without retraining the image recognition model.

[0069] Example 2

[0070] Figure 2 This is a schematic diagram of an image processing method provided in Embodiment 2 of the present invention. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0071] like Figure 2 As shown, the method specifically includes:

[0072] 1. After the data enters the image processing model, it undergoes a feature extraction process. The resulting image features are then entered into the feature matching module, where they are matched with the corresponding features of a specific sample. If a match is found, the manually selected output result for that specific sample is output. If no match is found, the image features are input into the normal output segment of the model to obtain the model's predicted output result.

[0073] Here, "data" refers to the data of the image to be processed, and "image features" refers to the feature sub-vectors corresponding to multiple model stages in the feature extraction model.

[0074] 2. Image processing models need to have multi-layer feature processing structures. Conventional multi-layer convolutional neural networks for image recognition, such as ResNet, can be used. Image features are obtained from three different depths of the model's convolutional layers. The initial segment of the model before the first feature extraction point must contain at least four convolutional layers. The extraction points after the higher segments of the model are generally the last convolutional output before the fully connected output layer.

[0075] Among them, the image processing model is the feature extraction model, the initial stage of the model refers to the first model stage, and the advanced stage of the model refers to the third model stage.

[0076] 3. The special sample library stores the image features, masks (if the user needs to achieve partial region matching), and predetermined output results obtained from the same three extraction points for specific image samples.

[0077] 4. The feature matching module performs matching judgment according to the following steps: (1) Let the data labels of the features of the initial segment, middle segment, and high segment be F1, F2, and F3, which are all 3D arrays. When a mask is required, the receptive field of the neurons in the layer where the three segments are located is calculated. If the receptive field intersects with the area that the mask wants to focus on, the corresponding features are retained, and the remaining array elements are deleted. (2) The 3D array is transformed into 1D. The similarity between the F1, F2, and F3 of the input image and the F1, F2, and F3 of the sample library features is calculated. The calculation method uses cosine similarity on F1 to eliminate the influence of vector length, and normalized Euclidean distance on F2 and F3: the Euclidean distance between the feature vectors of the image and the sample library is divided by the sum of the feature vector magnitudes of the two, and the value range is [0,1]. Let the obtained values ​​be D1, D2, and D3. (3) For D1, D2, and D3, compare them with the threshold. Only when all three values ​​are less than the threshold are the input and sample library images considered similar (i.e., matched). This threshold can be set with different parameters for different types of images. For fundus images, values ​​of 0.2, 0.05, and 0.01 are recommended. (4) Compare between the input sample and each sample in the special case library. If multiple matches occur, the result with the highest matching degree is selected for output. The matching degree is the negative of the weighted sum of D1, D2, and D3, and the weight coefficients can be selected as 0.5, 0.5, and 1.

[0078] The input image's F1, F2, and F3 are the target feature sub-vectors corresponding to the image to be processed in the three model stages, while the sample library features F1, F2, and F3 are the sample sub-vectors corresponding to the samples in the three model stages. Matching degree is equivalent to similarity.

[0079] 5. The neural network model required by the system is first trained according to the task requirements, and then feature output points are added to calculate three feature quantities to obtain the pre-stored sample database data.

[0080] 6. For special samples requested by users during use, the system can add them with one click. The system calculates the F1, F2, and F3 of the model and then stores them as new entries in the sample library.

[0081] The technical solution of this invention extracts features from the image to be processed based on a pre-trained feature extraction model, obtaining feature sub-vectors corresponding to multiple model stages in the feature extraction model. If the receptive field of the neuron corresponding to each feature sub-vector matches a pre-defined region of interest, the element matching the region of interest is retained to obtain a target feature sub-vector corresponding to each feature sub-vector. A distance threshold is determined between each target feature sub-vector and the sample sub-vectors of each sample in the sample library, and samples with a distance threshold less than the preset distance threshold are taken as target similar samples of the image to be processed. The classification result corresponding to the target similar sample is taken as the target classification result of the image to be processed. This solves the problem that existing image recognition systems cannot process special samples, and achieves the effect of processing special samples without retraining the image recognition model.

[0082] Example 3

[0083] Figure 3 This is a structural block diagram of an image processing device provided in Embodiment 3 of the present invention. The device includes: a feature sub-vector acquisition module 310, a target feature sub-vector acquisition module 320, a target similar sample determination module 330, and a target classification result determination module 340.

[0084] The feature sub-vector acquisition module 310 is used to process the input image to be processed based on a pre-trained feature extraction model to obtain feature sub-vectors corresponding to multiple model stages in the feature extraction model.

[0085] The target feature vector acquisition module 320 is used to retain the elements that match the region of interest if the receptive field of the neuron corresponding to each feature vector is consistent with a pre-defined region of interest, thereby obtaining the target feature vector corresponding to each feature vector.

[0086] The target similarity sample determination module 330 is used to determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and to take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed.

[0087] The target classification result determination module 340 is used to take the classification result corresponding to the target similar sample as the target classification result of the image to be processed.

[0088] The technical solution of this embodiment extracts features from the image to be processed based on a pre-trained feature extraction model, obtaining feature sub-vectors corresponding to multiple model stages in the feature extraction model. If the receptive field of the neuron corresponding to each feature sub-vector matches a pre-defined region of interest, the element matching the region of interest is retained to obtain the target feature sub-vector corresponding to each feature sub-vector. A distance threshold is determined between each target feature sub-vector and the sample sub-vectors of each sample in the sample library, and samples with a distance threshold less than the preset distance threshold are taken as target similar samples of the image to be processed. The classification result corresponding to the target similar sample is taken as the target classification result of the image to be processed. This solves the problem that existing image recognition systems cannot process special samples, and achieves the effect of processing special samples without retraining the image recognition model.

[0089] Based on the above-mentioned device, optionally, the feature extraction model includes at least three model stages, and the feature sub-vector acquisition module 310 includes:

[0090] The feature extraction unit is used to input the image to be processed into the feature extraction model, and process it sequentially through the at least three model stages to obtain feature sub-vectors that match each model stage.

[0091] Optionally, based on the above-mentioned device, the target feature sub-vector acquisition module 320 includes:

[0092] The masking unit is used to perform masking processing on the sub-image corresponding to the current feature sub-vector for each feature sub-vector, and to use the unmasked area as the receptive field of the neuron.

[0093] The retention unit is used to retain the element corresponding to the unmasked region if the receptive field and the region of interest overlap.

[0094] The target feature vector acquisition unit is used to repeatedly perform sub-image mask processing on the current feature vector to obtain the target feature vector corresponding to the current feature vector.

[0095] Optionally, based on the above-described apparatus, the target similarity sample determination module 330 includes:

[0096] The cosine similarity value determination unit is used to sequentially determine the cosine similarity value between the target feature sub-vector in the first model stage and the sample sub-vector of each sample in the sample library in the first model stage.

[0097] The target Euclidean distance determination unit is used to normalize the target feature vectors in the second and third model stages and determine the target Euclidean distance corresponding to each target feature vector.

[0098] The distance threshold calculation unit is used to determine the distance threshold between the target feature vector and the corresponding sample vector based on the Euclidean distance of each target and the sample Euclidean distance of the sample subvectors in the sample library.

[0099] Optionally, based on the above-described device, the distance threshold calculation unit includes:

[0100] The sample Euclidean distance determination subunit is used to determine the sample Euclidean distances corresponding to the sample subvectors in the second and third models of the sample library, respectively.

[0101] The distance threshold calculation subunit is used to determine the distance threshold between the target feature vector and each sample subvector for the same model stage, based on the target Euclidean distance and the magnitude of the target feature vector in the current model stage, as well as the sample subvector in the current model stage and the corresponding sample Euclidean distance.

[0102] Optionally, based on the above-described device, the target similarity sample determination module 330 further includes:

[0103] The unit for determining similar samples to be selected is used to determine the current sample as a similar sample to be selected if the cosine similarity value of the current sample in the first model stage is greater than a preset similarity threshold, and the distance threshold of the current sample in the second model stage and the third model stage is less than the corresponding preset distance threshold.

[0104] The target similar sample determination unit is used to determine the target similar sample from the similar samples to be selected based on the pre-determined weight values ​​corresponding to each model stage if the number of similar samples to be selected is greater than a preset number threshold.

[0105] Optionally, based on the above-described apparatus, the target similarity sample determination unit includes:

[0106] The similarity value determination subunit is used to determine the similarity value of each sample to be selected based on the cosine similarity value, distance threshold and corresponding weight value corresponding to each model stage.

[0107] The target similar sample determination subunit is used to select the candidate similar sample with the highest similarity as the target similar sample.

[0108] The image processing apparatus provided in the embodiments of the present invention can execute the image processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0109] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0110] Example 4

[0111] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 4 A block diagram is shown of an exemplary electronic device 40 suitable for implementing embodiments of the present invention. Figure 4 The electronic device 40 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0112] like Figure 4 As shown, electronic device 40 is represented in the form of a general-purpose computing device. The components of electronic device 40 may include, but are not limited to: one or more processors or processing units 401, system memory 402, and bus 403 connecting different system components (including system memory 402 and processing unit 401).

[0113] Bus 403 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0114] Electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0115] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data media interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0116] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 407 typically perform the functions and / or methods described in the embodiments of the present invention.

[0117] Electronic device 40 can also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), and with one or more devices that enable a user to interact with the electronic device 40, and / or with any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 411. Furthermore, electronic device 40 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 412. As shown, network adapter 412 communicates with other modules of electronic device 40 via bus 403. It should be understood that, although... Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402, such as implementing the image processing method provided in the embodiments of the present invention.

[0119] Example 5

[0120] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image processing method. The method includes:

[0121] The input image to be processed is processed based on a pre-trained feature extraction model to obtain feature sub-vectors corresponding to multiple model stages in the feature extraction model;

[0122] If the receptive field of the neuron corresponding to each feature sub-vector is consistent with the pre-defined region of interest, then the element that matches the region of interest is retained to obtain the target feature sub-vector corresponding to each feature sub-vector;

[0123] Determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed;

[0124] The classification results corresponding to the target similar samples are used as the target classification results of the image to be processed.

[0125] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0126] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0127] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0128] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An image processing method, characterized in that, include: The input image to be processed is processed based on a pre-trained feature extraction model to obtain feature sub-vectors corresponding to the three model stages in the feature extraction model. The process includes: inputting the image to be processed into the feature extraction model and processing it sequentially through the at least three model stages to obtain feature sub-vectors that match each model stage; wherein the three model stages are the initial stage, the middle stage, and the high stage, and each model stage includes multiple neural network layers. If the receptive field of the neuron corresponding to each feature sub-vector matches a pre-defined region of interest, then the elements matching the region of interest are retained to obtain the target feature sub-vector corresponding to each feature sub-vector, including: For each feature sub-vector, a masking process is performed on the sub-image corresponding to the current feature sub-vector, and the unmasked region is used as the receptive field of the neuron; if the receptive field overlaps with the region of interest, the elements corresponding to the unmasked region are retained; the masking process for the sub-image corresponding to the current feature sub-vector is repeated to obtain the target feature sub-vector corresponding to the current feature sub-vector; Determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, including: sequentially determining the cosine similarity value between the target feature sub-vector in the first model stage and the sample sub-vector of each sample in the sample library in the first model stage; The target feature vectors in the second and third model stages are normalized to determine the target Euclidean distance corresponding to each target feature vector. Based on the Euclidean distance of each target and the Euclidean distance of the sample sub-vectors in the sample library, the distance threshold between the target feature sub-vector and the corresponding sample sub-vector is determined. Samples whose distance threshold is less than the corresponding preset distance threshold are used as target similar samples of the image to be processed. The classification results corresponding to the target similar samples are used as the target classification results of the image to be processed.

2. The method according to claim 1, characterized in that, The step of determining the distance threshold between the target feature vector and the corresponding sample vector based on the Euclidean distance of each target and the sample Euclidean distance of the sample subvectors in the sample database includes: Determine the Euclidean distances of the sample subvectors in the second and third models of the sample library, respectively; For sample subvectors in the same model stage, a distance threshold between the target feature vector and each sample subvector is determined based on the target Euclidean distance and the magnitude of the target feature subvector in the current model stage, as well as the sample subvector in the current model stage and the corresponding sample Euclidean distance.

3. The method according to claim 2, characterized in that, The step of using samples whose distance threshold is less than a corresponding preset distance threshold as target similar samples of the image to be processed includes: For each sample, if the cosine similarity value of the current sample in the first model stage is greater than the preset similarity threshold, and the distance threshold of the current sample in the second model stage and the third model stage is less than the corresponding preset distance threshold, then the current sample is determined as a similar sample to be selected. If the number of similar samples to be selected is greater than a preset threshold, then a target similar sample is determined from the similar samples to be selected based on the pre-determined weight values ​​corresponding to each model stage.

4. The method according to claim 3, characterized in that, The step of determining target similar samples from the pool of similar samples to be selected based on pre-determined weight values ​​corresponding to each model stage includes: For each sample to be selected, the similarity value of the sample is determined based on the cosine similarity value, distance threshold and corresponding weight value corresponding to each model stage. The sample with the highest similarity is selected as the target similar sample.

5. An image processing apparatus, characterized in that, include: The feature vector acquisition module is used to process the input image to be processed based on the pre-trained feature extraction model to obtain the feature vectors corresponding to the three model stages in the feature extraction model. The target feature vector acquisition module is used to retain the elements that match the region of interest if the receptive field of the neuron corresponding to each feature vector is consistent with a pre-defined region of interest, thereby obtaining the target feature vector corresponding to each feature vector. The target similarity sample determination module is used to determine the distance threshold between each target feature sub-vector and the sample sub-vector of each sample in the sample library, and to take samples whose distance threshold is less than the corresponding preset distance threshold as target similar samples of the image to be processed. The target classification result determination module is used to take the classification result corresponding to the target similar sample as the target classification result of the image to be processed; The feature vector acquisition module includes: The feature extraction unit is used to input the image to be processed into the feature extraction model, and process it sequentially through the at least three model stages to obtain feature sub-vectors that match each model stage. The target feature sub-vector acquisition module includes: The masking unit is used to perform masking processing on the sub-image corresponding to the current feature sub-vector for each feature sub-vector, and to use the unmasked area as the receptive field of the neuron. The retention unit is used to retain the element corresponding to the unmasked region if the receptive field and the region of interest overlap. The target feature sub-vector acquisition unit is used to repeatedly perform sub-image mask processing on the current feature sub-vector to obtain the target feature sub-vector corresponding to the current feature sub-vector; The target similarity sample determination module includes: The cosine similarity value determination unit is used to sequentially determine the cosine similarity value between the target feature sub-vector in the first model stage and the sample sub-vector of each sample in the sample library in the first model stage. The target Euclidean distance determination unit is used to normalize the target feature vectors in the second and third model stages and determine the target Euclidean distance corresponding to each target feature vector. The distance threshold calculation unit is used to determine the distance threshold between the target feature vector and the corresponding sample vector based on the Euclidean distance of each target and the sample Euclidean distance of the sample subvectors in the sample library.

6. An electronic device, characterized in that, The device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the image processing method as described in any one of claims 1-4.

Images