Image target retrieval method, retrieval device, electronic device, and storage medium

Abstract

The application discloses a retrieval method of an image target, a retrieval device, an electronic device and a storage medium, and the retrieval method comprises the following steps: performing feature extraction on a pretreated image to obtain a multi-scale feature image; performing adaptive weighted fusion on the multi-scale feature image to obtain an enhanced image; and outputting a target feature of an original image based on the enhanced image, so as to retrieve the image target. In the above manner, the detection precision of the image target is improved by performing adaptive weighted fusion on the multi-scale feature image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monitoring, in particular to an image target retrieval method, a retrieval device, an electronic device and a readable storage medium. BACKGROUND

[0002] Generally, target detection of remote sensing images is a technology of extracting class and location information of a target of interest from a given image, which has a wide range of applications in military and civilian fields such as investigation monitoring, weapon guidance, resource exploration, disaster detection, city planning, etc. In recent years, with the continuous development of satellite and remote sensing technology, the time resolution, spatial resolution and image quality of remote sensing images have been greatly improved, and higher requirements have been put forward for the detection accuracy and real-time performance of the target detection model. Therefore, the target detection technology of remote sensing images has always been a research hotspot in the field of remote sensing technology, and has very important theoretical research significance and practical application value.

[0003] With the rapid development of deep learning, target detection technology is also constantly progressing. At present, combining deep learning technology to detect targets in remote sensing images has become the mainstream development trend. At present, the size of the image target in the remote sensing image is large or small, and the number distribution also presents diversity, and directly using feature splicing when performing hierarchical processing on the remote sensing image leads to the loss of image features of different sizes, so that the retrieval of the image target is not accurate, and multiple retrievals are performed, which reduces the retrieval efficiency. SUMMARY

[0004] To solve the above technical problems, the technical solution adopted by the first aspect of the present application is to provide an image target retrieval method, which comprises: performing feature extraction on a preprocessed image to obtain a multi-scale feature image; performing adaptive weighted fusion on the multi-scale feature image to obtain an enhanced image; and outputting target features of the original image based on the enhanced image to retrieve the image target.

[0005] To solve the above technical problems, the technical solution adopted by the second aspect of the present application is to provide a retrieval device, which comprises:

[0006] An extraction module is configured to perform feature extraction on a preprocessed image to obtain a multi-scale feature image;

[0007] An adaptive weighted fusion module is configured to perform adaptive weighted fusion on the multi-scale feature image to obtain an enhanced image;

[0008] An output module is configured to output target features of the original image based on the enhanced image to retrieve the image target.

[0009] To solve the above technical problems, the technical solution adopted by the third aspect of the present application is to provide an electronic device, which comprises a processor and a memory, the memory storing a computer program, and the processor being configured to execute the computer program to implement the retrieval method of the first aspect of the present application.

[0010] To solve the above technical problems, the technical solution adopted by the fourth aspect of the present application is to provide a computer readable storage medium storing a computer program, the computer program being capable of being executed by a processor to implement the retrieval method of the first aspect of the present application.

[0011] The beneficial effects of the present application are as follows: by performing feature extraction on the preprocessed image and then performing adaptive weighted fusion on the obtained multi-scale feature image, on the one hand, the image target can be enhanced, and on the other hand, the feature representation capability of the multi-layer feature image can be excavated through adaptive weighted fusion, and the multi-scale features are comprehensively utilized to improve the detection accuracy of the image target, thereby improving the retrieval efficiency of the image target. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0013] Figure 1 is a schematic diagram of the retrieval method flow framework of the image target;

[0014] Figure 2 is a schematic diagram of the first embodiment of the retrieval method flow of the image target of the present application;

[0015] Figure 3 is Figure 2 is a specific implementation flowchart of the preprocessing before step S11 in the embodiment;

[0016] Figure 4 is Figure 2 is a specific implementation flowchart of step S11 in the embodiment;

[0017] Figure 5 is Figure 4 is a schematic diagram of the image feature extraction flow in the embodiment;

[0018] Figure 6 is Figure 5 is a schematic diagram of the residual block structure adopted in the image feature extraction flow in the embodiment;

[0019] Figure 7 is Figure 2 is a specific implementation flowchart of step S12 in the embodiment;

[0020] Figure 8 is Figure 7 a network structure schematic diagram of an embodiment of the image feature enhancement in the method;

[0021] Figure 9 is Figure 7 a specific implementation flowchart of step S43 in the method;

[0022] Figure 10 is Figure 9 a specific implementation flowchart of step S51 in the method;

[0023] Figure 11 is Figure 7 another specific implementation flowchart of step S43 in the method;

[0024] Figure 12 is Figure 2 a detector module network structure schematic diagram of step S13 in the method;

[0025] Figure 13 is Figure 2 a specific embodiment flowchart of step S13 in the method;

[0026] Figure 14 is a structure schematic block diagram of an embodiment of the retrieval device of the application;

[0027] Figure 15 is a structure schematic block diagram of an embodiment of the electronic device of the application;

[0028] Figure 16 is a circuit schematic block diagram of an embodiment of the computer readable storage medium of the application. DETAILED DESCRIPTION

[0029] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc. in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the application with unnecessary detail.

[0030] It should be understood that the term "comprises" when used in this specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] It should also be understood that the terms used in the present application specification are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, "a", "an", and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise.

[0032] It should be further understood that the term "and / or" used in the present application specification and the appended claims means one or more of the associated listed items as well as all possible combinations of the items and includes these combinations.

[0033] As used in the present specification and the appended claims, the term "if" can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrases "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "once it is determined" or "in response to a determination" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0035] In order to illustrate the technical solutions of the present application, please refer to Figure 1 , Figure 1 is a schematic diagram of the retrieval method flow framework of the image target. The present application mainly performs retrieval for the target detection task of the original image, such as a remote sensing image. As shown in Figure 1 , the main modules of the constructed target detection scheme mainly include five parts of data preprocessing module 11, feature extraction module 12, feature enhancement module 12, detector module 14, and post-processing module 15. After the original image is processed by the five modules, the result output is obtained.

[0036] The following will illustrate the detection method of the image target provided by the present application through specific embodiments. Please refer to Figure 2 , Figure 2 is a schematic diagram of the first embodiment flow of the retrieval method of the image target of the present application. The retrieval method specifically includes the following steps:

[0037] S11: performing feature extraction on the preprocessed image to obtain a multi-scale feature image;

[0038] For a large size original image, such as a remote sensing image, the range involved is usually tens of kilometers, and some image targets on the original image often appear in a concentrated form, such as ship containers in a port, and other image targets appear in a scattered form, such as pedestrians, vehicles, animals, etc.

[0039] In order to extract the image target in the original image, the pre-processed image can be segmented, and the large size original image can be segmented to obtain a small size image, so as to more concentratedly search for the image target in a small range.

[0040] Specifically, the pre-processed image is segmented in size by feature extraction, thereby obtaining a multi-scale feature image, and then the multi-scale feature image is processed layer by layer, useless image fragments are removed, and image fragments with image targets are retained, which is beneficial to improve the implementation of the scheme.

[0041] S12: adaptively fusing the multi-scale feature image to obtain an enhanced image;

[0042] The multi-scale feature image is the image feature information of different sizes, which can use different resolution features to represent the image target, and the purpose of feature fusion stage is to improve the utilization rate of extracted features.

[0043] In the feature fusion stage, the low resolution feature has strong semantic information and can distinguish targets and backgrounds, but has less detailed information; and the high resolution feature has rich fine-grained information, and the corresponding semantic information is weak, so how to fully utilize the semantic information and fine-grained information is the key to improve the feature fusion efficiency.

[0044] Specifically, the multi-scale feature image is enhanced by using an adaptive weighted fusion scheme at a weighted node determined by fine-grained information and semantic information, wherein the adaptive weighted fusion is mainly reflected in that the weighted coefficient corresponding to the weighted node is determined by continuous updating and iteration, so that the feature fusion stage is optimized, the multi-scale feature image adaptive weighted fusion is designed, the representation ability of the feature extraction network is excavated, and the multi-scale features are comprehensively utilized to improve the detection accuracy of the image target.

[0045] S13: based on the enhanced image, outputting the target feature of the original image to retrieve the image target.

[0046] Specifically, YOLOv3 predicts and outputs multiple scale extraction features, different scale features have different receptive fields, and are used for detecting different size targets. For multi-scale detection, if more high resolution feature information can be used, the image detection will be more conducive to small target detection.

[0047] Based on the enhanced image, the fused image features are predicted and output, specifically, the prediction information of the multi-scale feature image output can be obtained, and the target features of the enhanced image are output, thereby the target features of the original image are output, so as to realize the retrieval of the image target.

[0048] Therefore, the original image is preprocessed and feature extracted, and the obtained multi-scale feature images are adaptively weighted and fused, which can enhance the image target on the one hand, and can mine the feature representation ability of the multi-layer feature image through adaptive weighted fusion on the other hand, so as to improve the detection accuracy of the image target by comprehensively utilizing the multi-scale features, thereby improving the retrieval efficiency of the image target.

[0049] Further, the obtained original image is preprocessed to obtain a preprocessed image, please refer to Figure 3 , Figure 3 is Figure 2 a specific implementation flowchart of the preprocessing before step S11, specifically comprising the following steps:

[0050] S21: obtaining an original image;

[0051] Specifically, an image shooting device such as a camera is provided in the retrieval system for shooting images or videos, and a storage module can be provided in the retrieval device for storing the shot image or video file, therefore, the original image can be obtained directly through the image shooting device, or can be obtained from the storage module, of course, the person skilled in the art can also use other ways to obtain the original image, which is not limited here.

[0052] S22: data analysis, data cleaning and data enhancement are performed on the number of pictures of the original image, the labeled target class and the target number, to obtain a preprocessed image;

[0053] At present, the target detection algorithm based on deep learning can be divided into two stages based on region and single stage based on regression. The detection accuracy of two-stage detection algorithm is high, but the real-time performance of target detection is low, and the single-stage algorithm can meet the real-time requirement and has certain detection accuracy.

[0054] Because the original image contains various types of information collected, such as target class label error, target position label error and irrelevant label information, etc., which often has certain influence on the image target.

[0055] In order to eliminate the influence of these label errors, target position label errors and irrelevant label information on the image target, the obtained original image can be processed by data cleaning preprocessing method to obtain a preprocessed image for subsequent operation, which is more conducive to the extraction of the image target.

[0056] For deep retrieval, the effectiveness of the training data has a great influence on the performance of the image target, so building a balanced and complete data set is very important.

[0057] Specifically, first, data analysis is performed on the original image, including the number of pictures, the labeled target category and the number; then, according to the actual business needs, data cleaning is performed to remove dirty data, mainly including target category labeling errors, target position labeling errors and irrelevant labeling information and the like; then, the balance and completeness of the data are evaluated, and in this stage, data enhancement is mainly performed on the original picture, wherein the data enhancement includes geometric enhancement and color enhancement.

[0058] Among them, balance refers to the data type, such as ten thousand vehicles, and only two people, which does not have balance; completeness refers to the data category, such as the need for people, but there is no, which does not have completeness.

[0059] Considering that the range covered by the remote sensing image is wide, the data required for collection is not enough to set the required sample number, is relatively single and is not rich enough, which is easy to cause low adaptability. Since the pixel is not changed, but the size is changed, the data is enhanced, and the complexity of the data is enhanced, and the data coverage is close to the real situation; and the color enhancement is operated through the RGB channel, and the contrast, transparency, denoising and denoising of the image are changed.

[0060] Further, feature extraction is performed on the preprocessed image to obtain a multi-scale feature image, please refer to Figure 4 , Figure 4 is Figure 2 a specific implementation flowchart of step S11, specifically including the following steps:

[0061] S31: inputting the preprocessed image into a preset convolutional neural network module to perform feature extraction to obtain a first feature image;

[0062] Specifically, a convolutional neural network (CNN) model DarkNet-53 can be used to extract features from the input preprocessed image. The preset convolutional neural network module includes an input layer module, a convolution layer (Convolution) module, a down-sampling layer module, an inverse convolution layer module, and an output layer module, which are basic components in the network structure. The convolution layer module can at least perform one of convolution, batch normalization (Batch Normalization), and activation (Leaky Relu) functions, which are basic components in the network structure.

[0063] Specifically, refer to Figure 5 , Figure 5 is Figure 4 the image feature extraction process in FIG. 1, which can be as shown in FIG. 2. First, the original image is redefined to a size of 416x416, and then input to DarkNet-53 for feature extraction to obtain a first feature image. Figure 5

[0064] S32: The first feature image is down-sampled multiple times to obtain a multi-scale feature image, wherein the down-sampling uses a preset residual module.

[0065] Specifically, refer to Figure 6 , Figure 6 is Figure 5 the residual block structure used in the image feature extraction process in FIG. 1. As shown in FIG. 3, the first feature image is sequentially subjected to 5 times of down-sampling operation, and a total of 6 feature maps of different sizes can be obtained. The number of feature channels is doubled after each down-sampling, which aims to retain more effective information in the original image. res represents a residual network block, and N represents the number of residual units. Figure 5

[0066] It is worth noting that, for example, 32 in 416x416x32 in the feature map1 represents that a 32-layer convolution kernel is used, and DBL or res1, res2, res4, and res8 is a fixed module.

[0067] As described above, the image feature of a specified size is not necessarily obtained after each convolution calculation, which may be reflected in the difference in the determinant. Therefore, 0 padding can be used to fill the blank of the image feature to obtain a reasonable determinant so as to continue to extract features subsequently. In addition, N is a number after res1, res2, res4, and res8, such as 1, 2, 4, and 8. Figure 6

[0068] Further, the multi-scale feature images are adaptively weighted and fused to obtain an enhanced image. The multi-scale feature images can be weighted and fused based on the weighting nodes and the weighting coefficients corresponding to the weighting nodes to obtain the enhanced image. For example, the different scale feature images can be weighted and fused according to the corresponding weighting coefficients at the weighting nodes to obtain the enhanced image corresponding to the different scale feature images.

[0069] Specifically, refer to Figure 7 and Figure 8 , Figure 7 is Figure 2 a specific implementation process diagram of step S12 in FIG. 1, Figure 8 is Figure 7 ​​​A schematic diagram of the network structure of a specific embodiment of image feature enhancement, which specifically includes the following steps:

[0070] S41: Obtain the weighted nodes and weighted coefficients, where the weighted nodes are determined by the fine-grained information and semantic information of the feature image;

[0071] like Figure 8 As shown, feature maps 2-6 represent feature images of different resolutions obtained after each downsampling in the feature extraction stage, C1, C2 and C3 are three weighting points set in the feature fusion stage, and W1-W9 represent different weighting coefficients.

[0072] S42: Process a multi-scale feature image into multiple feature images of the same resolution;

[0073] Typically, feature images from different layers are first subjected to convolution and deconvolution operations to achieve the same resolution. Different nodes result in different numbers of convolutions or deconvolutions. For example... Figure 8 As shown, the feature overlay operation here overlays adjacent features. DeConv represents deconvolution, where the image feature sizes are increased from small to large; Conv represents convolution, where the image feature sizes are decreased from large to small.

[0074] Specifically, for node C1, the feature image from feature map 5 does not need to be resized; however, the feature image from feature map 6 needs to be deconvolved to increase its size, generating the same size as the feature image from feature map 5; and the feature image from feature map 4 needs to be convolved to decrease its size, generating the same size as the feature image from feature map 5. Similarly, nodes C2 and C3 undergo similar operations. In this way, for feature images that need to be resized, multiple feature images of the same resolution can be obtained through convolution or deconvolution.

[0075] S43: Based on weighted nodes and weighted coefficients, feature images of the same resolution are weighted and fused, where the weighted coefficients are continuously updated iteratively.

[0076] Specifically, at the weighted nodes, weighted fusion is performed with different weight coefficients W1-W9. This feature fusion method treats feature information at different levels differently, and they have different contributions to the final output of the model.

[0077] Furthermore, the weighting coefficients are obtained through continuous iterative updates; please refer to [link / reference]. Figure 9 , Figure 9 yes Figure 7 A schematic diagram of a specific implementation process for step S43, which includes the following steps:

[0078] S51: Obtain an estimated target value and an actual value of the weighting coefficient;

[0079] The training process of the weighting coefficient iterative model includes four framework steps: initialization of weights, forward propagation, loss calculation, and backward error propagation. Like other parameters in the neural network structure, the weighting coefficients W1-W9 are first assigned an initial value, such as 1.0, which means that the feature images of different layers have the same contribution to the final output result at the beginning of model training.

[0080] By comparing the estimated target value and the actual value of the weighting coefficient, it can be determined whether the weighting coefficient needs to be updated iteratively.

[0081] S52: Compare the estimated target value and the actual value to obtain an error value;

[0082] Specifically, the error value can be obtained by subtracting the estimated target value from the actual value. According to the set loss function formula, the value of the loss function is calculated, which is the error value between the target estimated value and the actual value. This error represents the difference between the current model parameter inference result and the true value.

[0083] S53: Based on the error value, use the gradient descent algorithm to take the partial derivative of the multiple weighting coefficients to obtain multiple parameter update values corresponding to the multiple weighting coefficients respectively;

[0084] Specifically, according to the back propagation algorithm (BP algorithm), the partial derivative of the model parameters is taken to obtain the parameter update value of each parameter W1-W9. For example, +0.2, -0.1, etc. The system then uses the nine update values to add and update W1-W9 to iteratively update the weighting coefficients.

[0085] S54: Use the multiple update values to add and update the multiple weighting coefficients to iteratively update the multiple weighting coefficients obtained in the previous step until the error value is less than a preset value.

[0086] The greater the loss, the more feedback adjustment of the preset initial value is needed. The smaller the loss, the more optimal the preset initial value is.

[0087] The neural network model dynamically updates each weight coefficient according to the parameter update value. The neural network model continuously iteratively updates during the training process, and the weighting coefficients W1-W9 are continuously optimized in the direction of reducing the loss function to achieve the purpose of adaptive learning.

[0088] Furthermore, since the neural network model includes a feature enhancement module 12, the feature enhancement module 12 includes multiple weighting layers corresponding to multiple feature images, such as Figure 8As shown, each weighted layer except the first and last layers weights the feature images of other weighted layers.

[0089] Specifically, each weighted layer except the first and last layers can weight the feature images of the corresponding weighted layers of the adjacent upper and lower layers. For example, at the C3 weighting point of the feature map3 weighted layer, the feature images of the feature map2 and the feature map4 are weighted and fused; at the C2 weighting point of the feature map4 weighted layer, the feature images of the feature map3 and the feature map5 are weighted and fused; at the C1 weighting point of the feature map5 weighted layer, the feature images of the feature map4 and the feature map6 are weighted and fused.

[0090] In addition, the weighted layer can also weight the feature images of other cross-layer weighted layers, for example, at the C2 weighting point of the feature map4 weighted layer, the feature images of the feature map2 and the feature map5 can also be weighted and fused; that is, in the weighted layers that can be weighted and fused, the feature images can be weighted and fused by the adjacent layers, and the feature images can also be weighted and fused by the cross-layer, which is determined according to the requirements, and is not limited here.

[0091] Of course, to ensure at least one weighted layer, the feature enhancement module 12 includes at least 3 weighted layers corresponding to the feature images, that is, there is at least one weighted layer, and in fact, in order to achieve better weighting effect, more than 6 weighted layers corresponding to the feature images can be set, for example, as shown in Figure 8 feature map2 to feature map6.

[0092] Further, the estimated target value and the actual value are obtained, please refer to Figure 10 , Figure 10 is Figure 9 a specific implementation flowchart of step S51 in

[0093] S61: based on the neural network model, the plurality of weighting coefficients are inferred to obtain the estimated target value;

[0094] Specifically, after the given image is input, the output of the neural network is calculated by the forward propagation algorithm according to the designed network structure connection mode, and this output is used as the target estimated value of the model, which is assumed to be y.

[0095] S62: Weighted fusion is performed on the plurality of image features of the same resolution based on the plurality of weighted nodes and the weighted coefficients corresponding to the weighted nodes, to obtain a plurality of fused feature values;

[0096] Specifically, the three weighted coefficients of each weighted node (c1, c2, c3) are respectively used for weighted calculation on the corresponding feature map, and then fused to obtain a plurality of fused feature values, and a fused feature image is obtained as an input feature of the next layer of convolutional neural network.

[0097] S63: A plurality of target features are obtained by respectively performing convolution on the plurality of fused feature values.

[0098] Each node obtains a fused feature value corresponding to the weighted node after weighted fusion, and the fused feature value is still a matrix. Corresponding to three nodes, three target features are input to three fused feature values. A plurality of target features are obtained by respectively performing convolution on the plurality of fused feature values. At this time, the target feature is still a matrix.

[0099] S64: The plurality of target features are input into a loss function for processing to obtain an actual value.

[0100] Specifically, the loss function used is:

[0101]

[0102] 1. Specifically, the parameters are updated by back propagation. Assuming that the model is a regression model, the total error of the first forward propagation is calculated using the mean square error as shown in equation (2) as follows:

[0103]

[0104] Therefore, equation (3) is obtained:

[0105]

[0106] 2. Assuming that the activation function uses a Sigmoid function, equation (4) can be obtained as follows:

[0107]

[0108] Equation (5) is obtained:

[0109] σ'(t) = σ(t) [1-σ(t)] (5)

[0110] 3. Taking the weight coefficient as an example, if you want to know how much affects the whole, you can use the whole error to find the partial derivative of , and the chain rule can be used to find equation (6):

[0111] 4. Update the weight coefficient The learning rate η can be set to 0.1, 0.5, …;

[0112]

[0113] 5. Update the weight coefficient

[0114] 6. When the difference between the weight coefficient W and the intercept term b before and after iteration is less than a certain threshold, stop iteration. At this time, the F(w, b) obtained is the final model.

[0115] Further, the feature images of the same resolution are weighted and fused. Please refer to Figure 11 , Figure 11 is Figure 7 is a specific implementation flowchart of step S43 in

[0116] S71: Determine whether the node of the feature image meets the preset value of the fine-grained information and whether the semantic information of the feature image meets the preset value of the semantic information.

[0117] If yes, go to step S72, that is, determine the node as a weighted node, obtain a plurality of weighting coefficients corresponding to the weighted node, and perform step S73, that is, weighted fusion of the feature images of the same resolution is performed using the plurality of weighting coefficients.

[0118] If not, go to step S74, that is, jump to the next node and continue the judgment as in step S71, which is not described here.

[0119] Further, based on the enhanced image, the target feature of the original image is output to retrieve the image target, which specifically includes:

[0120] Based on the enhanced image and the detection scale, the target class information, target position information and confidence score of the candidate box are output. Specifically, based on the enhanced image, the target class information, target position information and confidence score of the candidate box are output according to the detection scale, wherein the detection scale is at least greater than or equal to 4. The four detection scales used can be 13x13, 26x26, 52x52 and 104x104 feature image detection output branches respectively. Please refer to Figure 12 , Figure 12 is Figure 2 is a detector module network structure diagram of step S13 in

[0121] As Figure 13As shown, the detector module 14 is used to output the prediction of the fused features, i.e., the enhanced image, after the feature fusion stage, including target category information, target position information, and confidence score information.

[0122] y1-y4 represent the prediction information of four different scale feature image outputs, and Channel represents the feature channel dimension of the prediction output, which is calculated as shown in equation (7):

[0123] Channel = n x (t x ,t y ,t w ,t h ,t o ,n class ) (7)

[0124] wherein n represents the number of prior boxes at each scale, t x ,t y ,t w ,t h ,t o are parameters to be learned during training, used to predict the center coordinates, width and height of the bounding box, and the confidence score information, and n class represents the number of predicted target categories.

[0125] Generally, the detection scale is 3, but in the field of remote sensing images, the detection scale is at least 4, which can perform more in-depth search on small targets, can utilize more high-resolution feature information, and thus improves the detection capability of small targets and exhibits good application effect.

[0126] wherein based on the enhanced image, the target features of the original image are outputted to search for image targets, please refer to Figure 13 , Figure 13 is Figure 2 a specific implementation flowchart of step S13, specifically including the following steps:

[0127] S81: judging whether the intersection-over-union of the candidate box corresponding to the confidence score is greater than the intersection-over-union preset value;

[0128] The post-processing module 15 is a process of secondary processing of the results outputted by the detector module of the target detection model, and the purpose is to remove the repeated target detection boxes and optimize the detection output results to obtain the optimal expected output according to the task requirements.

[0129] If yes, go to step S82, i.e. remove the candidate frame; specifically, the operation frequently used in this stage is non-maximum suppression NMS (Non-maximum suppression, NMS), and for each class, first set the IoU (Intersection over Union, IoU) threshold, i.e. the intersection ratio threshold. The candidate frames are sorted according to the class scores, and the candidate frame with the highest class score is selected, and the remaining candidate frames are traversed, and if the IoU with the candidate frame with the highest class score is greater than the IoU threshold, it is removed.

[0130] And continue to select a class score from the unprocessed frame, repeat the above process until all frames are processed. The remaining candidate frame is the detection result.

[0131] If no, go to step S83, i.e. attenuate the candidate frame according to the weight function to obtain the search result.

[0132] The NMS algorithm is too simple and rough in processing the target detection frame, and if an object appears in the overlapping area of another object, i.e. when two target frames are close, the frame with a lower score will be deleted due to the large overlapping area, thereby causing the detection of the object to fail and reducing the average detection rate of the algorithm. Therefore, the Soft-NMS algorithm is also used for post-processing operation on the result.

[0133] In the Soft-NMS algorithm, specifically, B represents the set of initial detection frame results, S represents the confidence score of the detection, N t represents the threshold of the set IoU, and D is empty, indicating no target to be saved. argmax S represents the highest score, and M represents the detection frame with the highest score. DUM represents storage, and B-M represents removal or deletion. i represents a Gaussian weight function, and the mathematical expression is shown in equation (8). The Soft-NMS algorithm multiplies the confidence score of the current detection frame by the weight function, and the weight function attenuates the score of the adjacent detection frame overlapping with the detection frame M with the highest score. The more the detection frame overlaps with the M detection frame, the more serious the score attenuation.

[0134]

[0135] Therefore, the application improves and optimizes the feature fusion stage, and proposes a multi-scale feature image adaptive weighted fusion method. By mining the representation ability of the feature extraction network, the multi-scale features are comprehensively utilized to improve the target detection accuracy. On the other hand, based on the YOLOv3 algorithm, the detection scale is increased to four, that is, the detection output branch of the 104x104 feature image is added, and the detection ability of small targets is improved. Compared with the three detection scales in the prior art, the detection ability of the model is improved.

[0136] In order to illustrate the technical scheme of the application, the application further provides a retrieval device, please refer to Figure 14 , Figure 14 The structure schematic block diagram of the retrieval device of the monitoring device of the application embodiment is shown in the figure. The retrieval device 7 comprises a preprocessing module 71, an extraction module 72, an adaptive weighted fusion module 73 and an output module 74.

[0137] The preprocessing module 71 is used for preprocessing the obtained original image to obtain a preprocessed image.

[0138] The extraction module 72 is used for feature extraction on the preprocessed image to obtain a multi-scale feature image.

[0139] The adaptive weighted fusion module 73 is used for adaptive weighted fusion on the multi-scale feature image to obtain an enhanced image.

[0140] The output module 74 is used for outputting the target feature of the original image based on the enhanced image to retrieve the image target.

[0141] Therefore, the application preprocesses and extracts features from the original image through the preprocessing module 71 and the extraction module 72, respectively. Furthermore, the adaptive weighted fusion module 73 is used for adaptive weighted fusion on the obtained multi-scale feature image. On the one hand, the image target can be enhanced. On the other hand, the feature representation ability of the multi-layer feature image can be mined through adaptive weighted fusion, and the multi-scale features are comprehensively utilized to improve the detection accuracy of the image target, thereby improving the retrieval efficiency of the image target.

[0142] In order to illustrate the technical scheme of the application, the application further provides an electronic device, please refer to Figure 15 , Figure 15 The structure schematic block diagram of the electronic device of the application embodiment is shown in the figure. The electronic device 8 comprises a processor 81 and a memory 82. The memory 82 stores a computer program 821. The processor 81 is used for executing the computer program 821 to realize the method of the first aspect of the application. Here, it is not repeated.

[0143] In addition, the application further provides a computer readable storage medium, please refer to Figure 16 ,Figure 16 is a circuit schematic block diagram of a computer readable storage medium embodiment of the present application, the computer readable storage medium 9 stores a computer program 91, the computer program 91 can be executed by a processor to implement the method of the first aspect of the embodiment of the present application, which will not be repeated here.

[0144] If implemented in the form of a software function unit and sold or used as an independent product, it can also be stored in a device with a storage function. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage device, including a plurality of instructions (program data) for executing all or part of the steps of the method of each embodiment of the present application by a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor. The aforementioned storage device includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various media with the above storage medium, computer, mobile phone, notebook computer, tablet computer, camera and other electronic devices.

[0145] The execution process of the program data in the device with a storage function can refer to the description of the image target retrieval method embodiment of the present application described above, which will not be repeated here.

[0146] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of retrieving an image object, characterized by, The retrieval method comprises: performing feature extraction on the preprocessed image to obtain a multi-scale feature image; performing adaptive weighted fusion on the multi-scale feature image to obtain an enhanced image; based on the enhanced image, outputting target features of the original image to retrieve the image target; wherein the adaptive weighted fusion of the multi-scale feature image to obtain the enhanced image comprises: obtaining a weighted node and a weighting coefficient corresponding to the weighted node; processing the multi-scale feature image into a plurality of feature images of the same resolution; based on the weighted node and the weighting coefficient, performing weighted fusion on the plurality of feature images of the same resolution to obtain the enhanced image, wherein the fine-grained information of the feature image corresponding to the weighted node satisfies a fine-grained information preset value and the semantic information of the feature image satisfies a semantic information preset value, and the weighting coefficient is updated iteratively; the iterative updating of the weighting coefficient comprises: based on a neural network model, inferring a plurality of the weighting coefficients to obtain an estimated target value; based on a plurality of the weighted nodes and the weighting coefficients corresponding to the weighted nodes, performing weighted fusion on the plurality of image features of the same resolution to obtain a plurality of fusion feature values; convolving a plurality of the fusion feature values respectively to obtain a plurality of target features; inputting a plurality of the target features into a loss function for processing to obtain an actual value; comparing the estimated target value with the actual value to obtain an error value; based on the error value, using a gradient descent algorithm to take the partial derivative of a plurality of the weighting coefficients to obtain a plurality of parameter update values corresponding to a plurality of the weighting coefficients respectively; using a plurality of the update values to additively update a plurality of the weighting coefficients to iteratively update a plurality of the weighting coefficients obtained in the previous iteration until the error value is less than a preset value; wherein the neural network model comprises a feature enhancement module, and the feature enhancement module comprises a plurality of weighting layers corresponding to a plurality of the feature images, wherein each of the weighting layers except the first layer and the last layer performs weighted fusion on the feature images of the cross-layer weighting layers.

2. The retrieval method of claim 1, wherein: the weighted fusion of the feature images of the same resolution comprises: determining whether the fine-grained information of the node of the feature image satisfies a fine-grained information preset value and whether the semantic information of the feature image satisfies a semantic information preset value; if yes, determining that the node is the weighted node to obtain a plurality of the weighting coefficients corresponding to the weighted node and performing weighted fusion on the feature images of the same resolution using a plurality of the weighting coefficients.

3. The retrieval method of claim 2, wherein: based on the enhanced image, outputting target features of the original image to retrieve the image target comprises: based on the enhanced image and a detection scale, outputting target class information, target position information, and confidence score of a candidate box, wherein the detection scale is at least greater than or equal to 4.

4. The retrieval method of claim 3, wherein: The method comprises: determining whether the intersection-over-union of the candidate box corresponding to the confidence score is greater than a preset intersection-over-union value; if yes, removing the candidate box; if no, attenuating the candidate box according to a weight function to obtain a retrieval result.

5. The retrieval method of claim 1, wherein the feature extraction of the preprocessed image comprises: inputting the preprocessed image into a preset convolutional neural network module to obtain a first feature image, wherein the preset convolutional neural network module comprises an input layer module, a convolutional layer module, a down-sampling layer module, an inverse convolutional layer module, and an output layer module, and the convolutional layer module performs at least one of convolution, batch normalization, and an activation function; performing multiple times of down-sampling on the first feature image to obtain the multi-scale feature image, wherein the down-sampling uses a preset residual module. The retrieval device comprises:

6. A retrieval device characterized by a feature extraction module configured to extract features of a preprocessed image to obtain a multi-scale feature image; an adaptive weighted fusion module configured to adaptively and weightedly fuse the multi-scale feature image to obtain an enhanced image; an output module configured to output target features of an original image based on the enhanced image to retrieve an image target; wherein the adaptive weighted fusion module is configured to obtain a weighted node and a weighting coefficient corresponding to the weighted node, process the multi-scale feature image into a plurality of feature images of the same resolution, and weightedly fuse the plurality of feature images of the same resolution based on the weighted node and the weighting coefficient to obtain the enhanced image, wherein the fine-grained information of the feature image corresponding to the weighted node satisfies a preset fine-grained information value, and the semantic information of the feature image satisfies a preset semantic information value, and the weighting coefficient is iteratively updated; the iteratively updated weighting coefficient comprises: inferring a plurality of weighting coefficients based on a neural network model to obtain an estimated target value; weightedly fusing a plurality of image features of the same resolution based on a plurality of weighted nodes and weighting coefficients corresponding to the weighted nodes to obtain a plurality of fused feature values; respectively convolving the plurality of fused feature values to obtain a plurality of target features; inputting the plurality of target features into a loss function for processing to obtain an actual value; comparing the estimated target value with the actual value to obtain an error value; using a gradient descent algorithm to derive a plurality of weighting coefficients based on the error value to obtain a plurality of parameter update values corresponding to the plurality of weighting coefficients, respectively; and adding the plurality of update values to the plurality of weighting coefficients for update to iteratively update the plurality of weighting coefficients obtained in the previous time until the error value is less than a preset value; wherein the neural network model comprises a feature enhancement module, and the feature enhancement module comprises a plurality of weighting layers corresponding to the feature images, wherein each of the weighting layers except the first layer and the last layer weightedly fuses feature images of cross-layer weighting layers. The method comprises:

7. An electronic device, comprising: ​ a processor and a memory having stored therein a computer program, the processor being operative to execute the computer program to implement the search method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by the processor, implements the search method of any one of claims 1-5.

Images