Neural network searching method, device and equipment and storage medium

Abstract

Embodiments of the present disclosure provide a neural network search method, device and equipment and a computer readable storage medium. The method provided by the embodiments of the present disclosure is more sensitive to the preservation of multi-scale self-attention in the network itself during the network search process, thereby being able to preserve the comprehensive perception of the image content by the Transformer network structure in the neural network search. The method provided by the embodiments of the present disclosure is able to preserve the comprehensive perception of the image content by the Transformer network structure in the neural network search of the present disclosure by establishing a multi-scale backbone network, generating different scale information at each level in the multi-scale Transformer visual network, and preserving the multi-scale sensitivity in the process of optimizing pruning by using the multi-scale backbone network to obtain a sub-network with redundancy removed, thereby improving the performance of the Transformer neural network determined by the neural network search method of the present disclosure in various visual understanding problems.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a neural network search method, apparatus, device, and storage medium for multi-scale feature extraction. Background Technology

[0002] Currently, in visual understanding tasks, visual backbone networks based on the Transformer self-attention model have been proven to achieve excellent performance in numerous existing works. Compared to convolutional neural networks that perform local convolution and understanding of image content, the advantage of the Transformer structure lies in its self-attention mechanism, which controls global pixel relevance learning. This global self-attention mechanism has enabled visual backbone networks to achieve excellent results in various classic visual tasks, such as visual classification, object detection, and object segmentation.

[0003] Although Transformer-based visual neural networks have demonstrated excellent performance in various visual understanding tasks, current visual networks still exhibit a certain degree of structural redundancy in their design. This redundancy significantly limits the inference speed and efficiency of visual Transformer networks, impacting their performance. For example, reducing the number of image tokens input to a visual Transformer network, even with this information removal, does not significantly affect the final visual inference result output by the network, indicating that the current Transformer network structure contains redundancy.

[0004] Therefore, an efficient neural network search method is needed to determine the optimal Transformer visual neural network and remove redundancy in the Transformer visual neural network. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a multi-scale visual Transformer backbone network and optimizes the network structure based on this backbone network, thereby enabling the optimized neural network to achieve more efficient inference with limited computing resources.

[0006] Embodiments of this disclosure provide a neural network search method, apparatus, device, and computer-readable storage medium for multi-scale feature extraction.

[0007] Embodiments of this disclosure provide a neural network search method for multi-scale feature extraction, comprising: establishing a multi-scale backbone network, the multi-scale backbone network including multi-level feature extraction, wherein in each level of feature extraction, multiple different convolutional networks are used to perform multi-scale feature extraction based on a self-attention mechanism; constructing a search space for the multi-scale backbone network, and determining multiple multi-scale sub-networks based on the multi-scale backbone network and the search space, wherein each of the multiple multi-scale sub-networks is a subset of the multi-scale backbone network, wherein each multi-scale sub-network is different from each other and shares at least a part of the multi-scale backbone network; and searching for a multi-scale sub-network from the multiple multi-scale sub-networks as the neural network for multi-scale feature extraction.

[0008] Embodiments of this disclosure provide a neural network search apparatus for multi-scale feature extraction, comprising: a backbone network generation module configured to establish a multi-scale backbone network, the multi-scale backbone network including multi-level feature extraction, wherein in each level of feature extraction, multiple different convolutional networks are used to perform multi-scale feature extraction based on a self-attention mechanism; a sub-network generation module configured to construct a search space for the multi-scale backbone network, and determine multiple multi-scale sub-networks based on the multi-scale backbone network and the search space, wherein each multi-scale sub-network is a subset of the multi-scale backbone network, wherein each multi-scale sub-network is different from each other and shares at least a part of the multi-scale backbone network; and a sub-network search module configured to search for a multi-scale sub-network from the multiple multi-scale sub-networks as the neural network for multi-scale feature extraction.

[0009] Embodiments of this disclosure provide a neural network search device for multi-scale feature extraction, comprising: one or more processors; and one or more memories, wherein the one or more memories store a computer-executable program, which, when executed by the processor, performs the neural network search method for multi-scale feature extraction as described above.

[0010] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the neural network search method for multi-scale feature extraction as described above.

[0011] Embodiments of this disclosure provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a neural network search method for multi-scale feature extraction according to embodiments of this disclosure.

[0012] Compared to traditional neural network search methods for multi-scale feature extraction, the method provided by the embodiments of this disclosure explores the network's sensitivity to multi-scale self-attention during the network search process, thereby preserving the Transformer network structure's comprehensive perception of image content during neural network search.

[0013] The method provided in the embodiments of this disclosure establishes a multi-scale backbone network, generating different scale information at each level of the multi-scale Transformer visual network. Furthermore, it preserves multi-scale sensitivity during the optimization and pruning process using this multi-scale backbone network to obtain redundant sub-networks. This allows the Transformer network structure to retain a comprehensive perception of image content within the neural network search of this disclosure, thereby improving the performance of the Transformer neural network determined by the neural network search method of this disclosure in various visual understanding problems. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0015] Figure 1 This is a schematic diagram illustrating a multi-scale Transformer network structure according to an embodiment of the present disclosure;

[0016] Figure 2 This is a flowchart illustrating a neural network search method for multi-scale feature extraction according to an embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram illustrating a multi-scale backbone network according to an embodiment of the present disclosure;

[0018] Figure 4 This is a schematic diagram illustrating the extraction of the i-th level feature in a multi-scale backbone network according to an embodiment of the present disclosure;

[0019] Figure 5 This is a schematic diagram illustrating a multi-scale self-attention block in a multi-scale backbone network according to an embodiment of the present disclosure;

[0020] Figure 6 This is a schematic diagram illustrating an example neural network search space according to an embodiment of the present disclosure;

[0021] Figure 7This is a schematic diagram illustrating a neural network search apparatus for multi-scale feature extraction according to an embodiment of the present disclosure;

[0022] Figure 8 A schematic diagram of a neural network search device for multi-scale feature extraction according to an embodiment of the present disclosure is shown;

[0023] Figure 9 A schematic diagram of the architecture of an exemplary computing device according to embodiments of the present disclosure is shown; and

[0024] Figure 10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0026] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0028] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0029] The neural network search method for multi-scale feature extraction disclosed herein can be based on artificial intelligence (AI). Artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—theories, methods, technologies, and application systems. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. For example, the neural network search method for multi-scale feature extraction based on artificial intelligence can acquire multi-scale features of an image through multi-scale feature extraction, similar to how humans identify objects of different sizes in an image with their naked eyes. This allows the neural network search method disclosed herein to maintain the same scale sensitivity in recognizing objects of different sizes in an image. By studying the design principles and implementation methods of various intelligent machines, artificial intelligence enables the neural network search method for multi-scale feature extraction disclosed herein to maintain multi-scale sensitivity while reducing model complexity and computational load.

[0030] The neural network search method for multi-scale feature extraction disclosed herein can be based on a visual Transformer model structure. The visual Transformer model structure originates from natural language processing (NLP), which uses image patches as corpora instead of sentences in NLP, thus enabling self-attention operations and network mapping output. For example, when given a 2D image, it is first uniformly divided into a series of sequential 2D image patches, and then these 2D image patches are mapped into high-dimensional feature vectors; this is called image patch embedding. Due to the global self-attention mechanism, the visual Transformer model structure is effective in capturing global image dependency information. A Transformer model structure typically consists of multiple encoders, each encoder module containing multi-head self-attention operations. A range of studies have focused on Transformer structural design, including aspects such as rapid data training, corpus redesign and selection, and improvements to the self-attention mechanism. These designs all consider the Transformer's own operations and the mechanism for processing image corpora. Transformer model structures can be categorized into single-scale and multi-scale structures. In single-scale Transformer model structures, the feature dimensions and 2D resolution of the outputs from the multiple serial encoders are consistent. While single-scale Transformer visual networks offer the advantage of structural simplicity, extensive research has demonstrated that such designs cannot extract features hierarchically and perform poorly on typical downstream visual tasks, such as object detection and image segmentation. Multi-scale Transformer model structures are not limited to Transformers; they can also be hybrids with Convolutional Neural Networks (CNNs). In designing multi-scale Transformer model structures, scale-sensitive dense visual prediction is a core concern. Many CNN-based designs have evolved from this. These designs utilize channel expansion and resolution decreasing along the depth dimension, resembling a pyramid structure. This type of pyramid structure first divides the serial encoders into multiple layers (e.g., typically four layers). As the number of layers increases, the spatial resolution processed by higher-layer encoders gradually decreases, while the feature dimension processed gradually increases. Notably, in this type of design, the features processed by encoders within the same layer have consistent spatial resolution and feature dimension. A similar structure exists in Transformers. In some recent designs, Transformer utilizes a two-dimensional backbone network to generate different scale information at different levels.

[0031] The neural network search method for multi-scale feature extraction disclosed herein can be based on an attention mechanism. The essence of the attention mechanism originates from the human visual mechanism and belongs to the brain signal processing mechanism unique to human vision, namely human attention. However, the attention mechanism only averages the learned features, which has certain limitations. Multi-head attention is a special variation of the attention mechanism. It uses functions to calculate similarity and, through multiple transformations, comprehensively learns features from different aspects of the model. This is equivalent to multiple individual attentions learning features from different subspaces and then combining these features through a linear transformation. Multi-head attention uses multiple queries to compute and select multiple pieces of information from the input information in parallel, with each attention focusing on a different part of the input information. In the embodiments of this disclosure, a multi-head attention mechanism can be used in the self-attention layer of each encoder to help the Transformer model assign different weights to each feature of the input, extracting more crucial information and improving the accuracy of image tasks.

[0032] The neural network search method for multi-scale feature extraction disclosed herein can also be based on neural network architecture search. Neural network architecture search utilizes various optimization algorithms to automatically learn and search for excellent neural network architectures. In the field of neural network search technology, a major branch is based on a weight-sharing mechanism for searching subnetworks. Specifically, this technique encodes the overall architecture search space into a supernetwork. Searching for the optimal structure typically consists of two steps: the first step is to optimize the weights of the supernetwork to achieve excellent performance in visual tasks; the second step is to search for the best-performing subnetworks among the structures associated with the supernetwork. Since there are many combinations of subnetworks in the supernetwork, random search, evolutionary computation, or reinforcement learning are commonly used to search for the optimal subnetwork. The supernetwork, as the backbone network of the searched neural network, is the core network structure for visual recognition tasks, and its design quality determines the recognition performance. In this disclosure, the backbone network can be designed based on the aforementioned multi-scale Transformer network.

[0033] In summary, the solutions provided by the embodiments of this disclosure involve technologies such as artificial intelligence and visual Transformer web search. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram illustrating a multi-scale Transformer network structure according to an embodiment of the present disclosure.

[0035] In visual understanding tasks, Transformer-based visual backbone networks have been proven to achieve excellent performance in numerous existing works. Compared to convolutional neural networks, which locally convolve and understand image content, the advantage of the Transformer structure lies in its self-attention mechanism, which controls global pixel dependency learning. This global self-attention mechanism has enabled visual backbone networks to achieve excellent results in various classic visual tasks, such as visual classification, object detection, and object segmentation.

[0036] like Figure 1 As shown, the multi-scale Transformer network structure resembles a pyramid structure, where the serial encoder is divided into multiple levels according to its components (e.g., Figure 1 (As shown in the diagram, this represents level four). As the level increases, the spatial resolution processed by higher-level encoders gradually decreases, but the feature dimensionality processed gradually increases. In current multi-scale Transformer network structures, encoders within the same level process features with consistent spatial resolution and feature dimensionality. Therefore, if we only consider encoders within a single level, the features they process are still single-scale (same resolution and same feature dimensionality), and the factors of scale preservation are not deeply considered. Such multi-scale feature extraction is insufficient.

[0037] Despite the superior performance of Transformer-based visual neural networks across various tasks, current visual networks still exhibit structural redundancy in their design. This redundancy significantly limits the inference speed and efficiency of visual Transformer networks, impacting their performance. For example, reducing the image patches input to the visual Transformer, even after removing certain information, has little impact on the final visual inference result, indicating redundancy in the current network structure. Therefore, neural network search can be used to optimize the Transformer network, such as by searching for subnetworks based on weight sharing mechanisms, thereby removing redundant parts from the current Transformer network.

[0038] However, in the process of searching for and optimizing Transformer network structures, existing neural network search methods do not take scale sensitivity as an optimization objective for multi-scale Transformer network structures. This results in the searched network structures simply reducing the complexity of the Transformer model, with a certain loss in performance.

[0039] Therefore, to address the aforementioned issues, the method disclosed herein primarily studies how to extract multi-scale features more effectively and fully in the design of multi-scale Transformer visual backbone networks. Furthermore, during the network search process, it explores how to preserve the network's sensitivity to multi-scale self-attention, so as to preserve multi-scale sensitivity during the process of optimizing pruning using the backbone network to obtain sub-networks.

[0040] Based on this, this disclosure provides a neural network search method for multi-scale feature extraction, which designs a multi-scale visual Transformer backbone network and optimizes the network structure based on the backbone network, so that the optimized neural network can achieve more efficient inference with more limited computing resources.

[0041] Compared to traditional neural network search methods for multi-scale feature extraction, the method provided by the embodiments of this disclosure explores the network's sensitivity to multi-scale self-attention during the network search process, thereby preserving the Transformer network structure's comprehensive perception of image content during neural network search.

[0042] The method provided in the embodiments of this disclosure establishes a multi-scale backbone network, generating different scale information at each level of the multi-scale Transformer visual network. Furthermore, it preserves multi-scale sensitivity during the optimization and pruning process using this multi-scale backbone network to obtain redundant sub-networks. This allows the Transformer network structure to retain a comprehensive perception of image content within the neural network search of this disclosure, thereby improving the performance of the Transformer neural network determined by the neural network search method of this disclosure in various visual understanding problems.

[0043] Figure 2 This is a flowchart illustrating a neural network search method 200 for multi-scale feature extraction according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram illustrating a multi-scale backbone network according to an embodiment of the present disclosure.

[0044] In step 201, a multi-scale backbone network can be established. The multi-scale backbone network can include multi-level feature extraction. In each level of feature extraction, multiple different convolutional networks can be used to perform multi-scale feature extraction based on a self-attention mechanism.

[0045] like Figure 3 As shown, the multi-scale backbone network of this disclosure may include multiple levels of feature extraction processing (e.g., Figure 3The diagram shows a 4-level feature extraction, where the feature dimension extracted at each level can gradually decrease as the number of levels increases. Image features obtained through the multi-scale backbone network of this disclosure (e.g., Figure 3 The image features extracted from the fourth-level feature extraction can be widely applied to various image-based understanding scenarios, such as, but not limited to, image content classification, person recognition, object detection and localization, foreground segmentation and background replacement, and then applied to a series of practical business scenarios such as search, recommendation and advertising.

[0046] Optionally, the multi-scale backbone network of this disclosure can be based on a multi-scale Transformer neural network, and in each level of feature extraction of the multi-scale Transformer neural network, multiple different convolutional networks can be used to perform multi-scale feature extraction based on a self-attention mechanism.

[0047] According to embodiments of this disclosure, each level in the multi-scale backbone network may include one or more serial encoders. Optionally, for example, for such... Figure 3 The second level of the multi-scale backbone network shown may include N² serial encoders. According to embodiments of this disclosure, each level of feature extraction in the multi-level feature extraction may correspond to multi-scale feature extraction via one or more serial encoders included in each level of the multi-scale backbone network. Optionally, each level of feature extraction in the multi-scale backbone network may include one or more feature extractions performed via one or more serial encoders. For example, for a multi-scale backbone network, the second level of feature extraction may include one or more feature extractions performed via one or more serial encoders. Figure 3 The second-level feature extraction in the multi-scale backbone network shown can perform N2 feature extractions corresponding to N2 serial encoders.

[0048] Figure 4 This is a schematic diagram illustrating the extraction of the i-th level feature in a multi-scale backbone network according to an embodiment of the present disclosure. Figure 4 As shown, the i-th level feature extraction in a multi-scale backbone network can be achieved through N i The encoder performs N i Therefore, in the i-th level of this multi-scale backbone network, further feature encoding can be performed based on the feature extraction results of the previous level (i.e., the i-1th level), thereby obtaining more effective feature extraction results than the previous level.

[0049] In an embodiment of this disclosure, each level of the multi-scale backbone network may further include image patch embedding processing for converting the output feature dimension of the previous level into the input feature dimension of the current level.

[0050] Since the Transformer network takes sequences as input, it is not directly applicable to two-dimensional image data. Therefore, applying the Transformer network to visual tasks requires image patch embedding processing. Optionally, before feature encoding in each level of the multi-scale backbone network of this disclosure, image patch embedding processing can be performed on the output features of the previous level. This includes, at the beginning of each level of the multi-scale backbone network, dividing the input image into blocks to form image blocks of the desired size, flattening each image block to convert the input image into a sequence of image blocks, and performing a linear transformation on each flattened image block, that is, converting the high-dimensional feature vector into a low-dimensional feature vector, which serves as the input to the subsequent encoder. In the image patch embedding processing described above, the embedding dimension of each image block (i.e., the dimension of the feature vector input to the subsequent encoder of the feature extraction at this level) can be selected, and the choice of the image patch embedding dimension has a significant impact on the feature extraction performance of the Transformer network.

[0051] Optionally, such as Figure 3 As shown, for an image input to a multi-scale backbone network, it can first be processed by image patch embedding, that is, the image is divided into image patches of the same size. The generated image patches serve as the basic modeling units in the multi-scale backbone network and are used to perform global dependency modeling on these image patches.

[0052] Image patch embedding can transform the output features of the previous level into the input features of the current level, which then serve as the input to the first encoder of the current level. For example... Figure 4 As shown, each encoder can perform layer normalization, multi-scale self-attention, and feature weighting through fully connected layers.

[0053] Figure 5 This is a schematic diagram illustrating a multi-scale self-attention block in a multi-scale backbone network according to an embodiment of the present disclosure.

[0054] Figure 5 The multi-scale self-attention processing employed in each encoder of the multi-scale backbone network of this disclosure is illustrated. According to embodiments of this disclosure, in each serial encoder, local feature extraction can be performed on some features of the input features of the serial encoder, and other features of the input features can be extracted at multiple scales based on the self-attention mechanism using the multiple different convolutional networks.

[0055] Optionally, the input features of the multi-scale self-attention block (MSSA) of this disclosure, after layer normalization, can first be grouped for subsequent local and global feature extraction. Correspondingly, the multi-scale self-attention block of this disclosure can be divided into two branches: local processing (MSSA-L) and global processing (MSSA-G). The local processing branch can extract local features from some of the input features, while the global processing branch can be used to calculate the feature dependencies of the remaining features in the input features at different scales from a global perspective.

[0056] like Figure 5 As shown, the input feature vector of the multi-scale self-attention block (e.g., it can be a query vector q, a key vector k, and a value vector v generated based on the result of normalization processing in previous layers) can be divided into five groups. One group can be input into the local processing branch, while the other four groups can be input into the global processing branch. In the local processing branch MSSA-L, cascaded convolutions with smaller kernels (e.g., 1x1 and / or 3x3 kernels) can be used to extract features from the input features that focus on local features.

[0057] In the global processing branch MSSA-G of the multi-scale self-attention block disclosed herein, different convolutional networks can be used to perform different scale expansions on each group of features in the remaining groups to generate multiple groups of features at different scales. Therefore, by performing multi-scale expansion before the input self-attention layer, the self-attention layer can be used to perform feature dependency calculations at different scales based on features at different scales, thereby maintaining the same recognition sensitivity for objects at different resolutions. Optionally, in the aforementioned self-attention layer, the correlation between all input features can be determined at different scales, and the corresponding input features can be updated based on these correlations.

[0058] Therefore, in the embodiments of this disclosure, the feature vectors obtained through local and global processing branches can be integrated (e.g., concatenated) in the feature integration layer, thereby achieving feature modeling at different scales in a single encoder. The output integrated feature vector can be processed by layer normalization in the fully connected layer of the input encoder to generate the input features of the next encoder or the next level through feature weighting.

[0059] Optionally, each encoder in a Transformer network model can typically include multiple fully connected layers. This is because a single fully connected layer may not be able to aggregate the feature vector data to a sufficient degree for accurate understanding, and more fully connected layers can make the feature data more concentrated, thereby improving feature extraction performance. However, increasing the number of fully connected layers will consume a significant amount of computing power to extract feature values ​​from the data, which will greatly affect the performance of multi-scale Transformer networks.

[0060] As described above, in the establishment of the multi-scale backbone network disclosed herein, the multi-scale content capture of the input image is considered in the design of the multi-scale backbone network, which is key to improving network performance. Through detailed analysis of the redundancy and shortcomings in the structural design of current visual Transformer networks, it can be found that various factors, such as the dimension of image patch embedding in the Transformer network structure, the number of heads in the multi-head self-attention mechanism used in the self-attention layer, and the depth of the visual Transformer network (i.e., the number of fully connected layers), significantly affect the inference performance and efficiency of the Transformer network.

[0061] Optionally, based on the aforementioned influencing factors, the designed multi-scale backbone network can be divided into different scales by designing the range of values ​​for these influencing factors. For example, the multi-scale backbone network can be divided into small networks, regular networks, and large networks based on the number of levels, the dimension of input features at each level, and the proportion of fully connected layers in the multi-scale backbone network. The proportion of fully connected layers corresponds to the number of fully connected layers in each encoder of the multi-scale backbone network.

[0062] Therefore, in neural network search based on the above-mentioned multi-scale backbone network, the search space can be designed based on the above-mentioned influencing factors. The sub-network structure can be optimized according to the determined search space, thereby achieving the beneficial effects of reducing computation, reducing redundancy, and maintaining accurate visual understanding.

[0063] In step 202, a search space for the multi-scale backbone network can be constructed, and multiple multi-scale sub-networks can be determined based on the multi-scale backbone network and the search space. Each of the multiple multi-scale sub-networks is a subset of the multi-scale backbone network, wherein each multi-scale sub-network is different from each other and shares at least a part of the multi-scale backbone network.

[0064] Optionally, this disclosure can employ neural network search for structural optimization of multi-scale backbone networks. The main idea of ​​neural network search is to extract the basic structure constituting the neural network from the search space using intelligent algorithms, determine the remaining structure and parameters using a specific network search strategy, and then construct a neural network to achieve a specific task based on the network connection strategy. By adopting such an automated neural network search method, the costs required for manually building networks from scratch and for developers to continuously try and adjust network parameters can be saved.

[0065] Optionally, in the neural network search process disclosed herein, a large visual Transformer network (i.e., the aforementioned multi-scale backbone network) can be determined first, and based on this large network, various smaller networks can be generated by designing network search variables. Each smaller network can be a subset of this large network, and each smaller network can be different from each other but share some of the structure and parameters of the large network.

[0066] According to embodiments of this disclosure, constructing a search space for the multi-scale backbone network may include: designing network search variables based on the multi-scale backbone network; and constructing a search space for the multi-scale backbone network based on the designed network search variables.

[0067] Optionally, as described above, network search variables and a search space for neural network search can be designed based on various factors that affect the performance of multi-scale backbone networks (e.g., performance in visual understanding tasks and computational performance). For example, network search variables can be designed based on parameters such as the number of levels in the multi-scale backbone network, the dimension of input features at each level, and the proportion of fully connected layers in the multi-scale backbone network. Of course, the above-mentioned influencing factors are only used as examples in this disclosure, and other influencing factors with similar effects can also be considered for designing network search variables for neural network search; the method of this disclosure does not limit this.

[0068] According to embodiments of this disclosure, the constructed search space may include at least one of the following dimensions: the dimension of image patch embedding processing in the multi-scale backbone network; the proportion of fully connected layers in the multi-scale backbone network; the depth of the multi-scale backbone network, corresponding to the number of fully connected layers in the multi-scale backbone network; or the convolution scale in the multi-scale backbone network, corresponding to the plurality of different convolutional networks. It should be understood that the dimensions included in the search space described above are merely examples and not limitations in this disclosure, and the neural network search method of this disclosure may also consider other dimensions capable of achieving similar effects for sub-network search.

[0069] According to embodiments of this disclosure, the search space can be deployed in each serial encoder of the multi-scale backbone network. Referring above to step 201 and... Figure 3-5 The main part of the multi-scale backbone network used for feature extraction is included in the encoder, that is, the function of feature extraction is mainly implemented by the encoder. Therefore, the structural optimization of the multi-scale backbone network can be used to optimize the encoder structure in the multi-scale backbone network. Thus, the search space of the neural network search can be deployed in the serial encoder of the multi-scale backbone network.

[0070] Optionally, in the neural network search framework used in this disclosure, a large search space can be utilized, in which the above four considerations (i.e., dimensions, including the dimension of image patch embedding, the proportion of fully connected layers, the depth of the network, and multi-scale convolution) are adopted, all of which have a significant impact on the performance of the network. Figure 6 This is a schematic diagram illustrating an example neural network search space according to an embodiment of the present disclosure. Figure 6 The document illustrates some specific information about the search space, including the possible values ​​of various multi-scale backbone networks in each dimension of the search space, as well as the parameter constraints and computational complexity constraints for various multi-scale backbone networks.

[0071] like Figure 6 As shown, the search space constructed in this disclosure can be optionally designed for multi-scale backbone networks of various sizes mentioned above. Specifically, this disclosure uses the three types of networks mentioned above (small network, regular network, and large network) as backbone networks of different sizes, and designs the search space for each backbone network based on the four dimensions considered above, and deploys this search space in each serial encoder of the Transformer network. Through this design, the search space of the sub-network can be greatly increased. For example, taking a small network as an example, the search space designed in this disclosure can have 1.1 × 10⁻⁶ under the control of the above four dimensions. 11 Subspace.

[0072] According to embodiments of this disclosure, determining multiple multi-scale sub-networks based on the multi-scale backbone network and the search space may include: performing weight decomposition on the multi-scale backbone network based on the network search variables to determine the structure and parameters shared by the multiple multi-scale sub-networks; and determining the multiple multi-scale sub-networks based on the results of the weight decomposition and the search space.

[0073] Optionally, the multi-scale backbone network can be weighted based on the determined network search variables, including determining the common structure and parameters of each sub-network, while the remaining structure and parameters related to the network search variables are used as the parameters to be optimized for each sub-network.

[0074] According to embodiments of this disclosure, determining the plurality of multi-scale subnetworks based on the results of the weight decomposition and the search space may include: for each of the plurality of multi-scale subnetworks, retaining the determined structure and parameters shared by the plurality of multi-scale subnetworks, and determining other structures and parameters in the multi-scale subnetworks based on the search space, thereby determining the multi-scale subnetwork.

[0075] Optionally, in determining each multi-scale sub-network, the already determined shared structure and parameters can be directly read from the constructed multi-scale backbone network and substituted into each multi-scale sub-network for use. Then, for the remaining parameters to be optimized in the multi-scale sub-network, they can be determined based on the possible structures and parameters in the search space. For example, in the embodiments of this disclosure, each multi-scale sub-network can be determined by searching the dimension of image patch embedding, the proportion of fully connected layers, the depth of the network, and the multi-scale convolutions in MSSA.

[0076] like Figure 6 As shown, for multi-scale backbone networks of different sizes (e.g., small networks, regular networks, and large networks), each dimension in the search space of its neural network search can have different possible values. The array form of the possible values ​​for each dimension in the search space corresponds to the possible values ​​for each level in the multi-level feature extraction of the multi-scale backbone network (where the order from bottom to top in the array corresponds to the order from low to high level in the multi-scale backbone network). For example, for a small network, in its 4th level feature extraction, the possible values ​​for the image patch embedding dimension are (40, 80), meaning that the subnetworks of this small network can choose a feature dimension of 40 or 80 for the image patch embedding dimension in the 4th level feature extraction.

[0077] For example, for a regular network, the proportion of its fully connected layers can be selected from (2, 3, 4) in each level of feature extraction, while its network depth (i.e. the number of fully connected layers) can be selected from (3, 4), (10, 11, 12), (3, 4) and (1, 2) in the first to fourth levels of feature extraction, respectively.

[0078] Optionally, in the scale variation space of multi-scale convolution, the feature extraction at each level of the network at all scales can select all scale convolution kernels based on (0, -W1, -W2, -W3, -W4). Here, 0 indicates retaining all scale convolution kernels, and a negative sign indicates removing a specific scale kernel. W1 represents a 1x1 scale convolution kernel, W2 represents a 3x3 scale convolution kernel, W3 represents a 5x5 scale convolution kernel, and W4 represents a 7x7 scale convolution kernel. Optionally, regarding the scale design of multi-scale convolution, the design for Transformer is currently still in the manual design stage; therefore, a more flexible scale design can be implemented from the perspective of adaptive input information.

[0079] certainly, Figure 6 The values ​​shown in the various dimensions of the search space are used as examples only and not as limitations in this disclosure. The neural network search method of this disclosure can cover all possible value ranges of the above search space.

[0080] As described above, by referring to steps 201 and 202 and Figure 2-6 The description allows us to construct a search space for multi-scale backbone networks and find multi-scale subnetworks with optimal performance within that search space.

[0081] Next, in step 203, a multi-scale subnetwork can be searched from the plurality of multi-scale subnetworks as the neural network for multi-scale feature extraction.

[0082] According to embodiments of this disclosure, searching for a multi-scale subnetwork from the plurality of multi-scale subnetworks includes: training and testing each of the plurality of multi-scale subnetworks using a predetermined dataset; and selecting, based on the results of the training and testing, a multi-scale subnetwork that meets predetermined conditions from the plurality of multi-scale subnetworks as the final neural network for multi-scale feature extraction.

[0083] Optionally, based on the various combinations of possible values ​​for each dimension included in the search space, corresponding multi-scale sub-networks can be pruned from the multi-scale backbone network. Therefore, each multi-scale sub-network can be trained and tested using a predetermined dataset for a specific visual understanding task (e.g., image classification) to evaluate the performance of each sub-network based on its performance for that specific visual understanding task. This allows for the selection of the multi-scale sub-network with the best performance as the final neural network for multi-scale feature extraction.

[0084] According to embodiments of this disclosure, searching for a multi-scale subnetwork from the plurality of multi-scale subnetworks may further include: performing the training and testing on each of the plurality of multi-scale subnetworks while constraining the parameters and computational complexity of the plurality of multi-scale subnetworks.

[0085] Optionally, while training and testing each multi-scale sub-network using a pre-defined dataset for a specific visual understanding task, constraints can also be placed on the number of parameters and computational complexity of the multi-scale sub-network model. For example, such as Figure 6 As shown, the number of parameters and computational power (such as GFLOPs, 1 billion floating-point operations per second) of the network model can be limited, thereby obtaining a smaller network model that is more conducive to edge deployment. Optionally, the number of parameters and computational power of the network model can be limited differently for network structures of different sizes. For example, for the above-mentioned regular network, the number of parameters of its corresponding multi-scale sub-network model can be controlled within the range of 19-23M by adjusting the parameters of its corresponding multi-scale sub-network, and the computational complexity of the multi-scale sub-network model can be controlled within the range of 4.5-5.3 GFLOPs.

[0086] By searching for the multi-scale subnetwork with the best performance among multiple multi-scale subnetworks as a neural network for multi-scale feature extraction, redundancy of existing neural networks can be effectively eliminated. The searched neural network can then perform more efficient inference with more limited computing resources, thereby addressing the high computational efficiency requirements of visual understanding tasks in industrial and commercial applications.

[0087] Since the neural network search method disclosed herein aims to search for the optimal neural network for multi-scale feature extraction, the determined neural network can be widely applied to various practical application scenarios that require feature extraction, including but not limited to image classification, object detection, object semantic segmentation, etc., thereby achieving better visual understanding results compared with existing methods.

[0088] As described above, the neural network search method for multi-scale feature extraction disclosed herein generates different scale information at each level of the multi-scale Transformer visual network by establishing a multi-scale backbone network. Furthermore, during the optimization and pruning process using this multi-scale backbone network to obtain redundant sub-networks, multi-scale sensitivity is preserved. This allows the Transformer network structure to retain a comprehensive perception of image content, thereby improving the performance of the Transformer neural network determined by the neural network search method disclosed herein in various visual understanding problems. Currently, network pruning is a popular research direction, but there is limited work on Transformer network pruning. In the method disclosed herein, the overall computational complexity can be reduced by focusing on the self-attention calculation itself.

[0089] Figure 7 This is a schematic diagram illustrating a neural network search apparatus 700 for multi-scale feature extraction according to an embodiment of the present disclosure.

[0090] According to embodiments of this disclosure, the neural network search device 700 for multi-scale feature extraction may include a backbone network generation module 701, a sub-network generation module 702, and a sub-network search module 703.

[0091] The backbone network generation module 701 can be configured to build a multi-scale backbone network, which may include multi-level feature extraction. In each level of feature extraction, multiple different convolutional networks can be used to perform multi-scale feature extraction based on a self-attention mechanism. Optionally, the backbone network generation module 701 can perform the operations described above with reference to step 201.

[0092] Optionally, the multi-scale backbone network of this disclosure may include multiple levels of feature extraction processing, wherein the feature dimension extracted at each level may gradually decrease as the number of levels increases. Image features obtained through the multi-scale backbone network of this disclosure can be widely applied to various image-based understanding scenarios, such as, but not limited to, image content classification, person recognition, object detection and localization, foreground segmentation and background replacement, and subsequently applied to a range of practical business scenarios such as search, recommendation, and advertising. Optionally, each level of feature extraction in the multi-scale backbone network may include one or more feature extractions performed by one or more serial encoders. Optionally, the multi-scale backbone network of this disclosure may be based on a multi-scale Transformer neural network, and in each level of feature extraction in the multi-scale Transformer neural network, multiple different convolutional networks may be used in each serial encoder to perform multi-scale feature extraction based on a self-attention mechanism.

[0093] Optionally, for the image input to the multi-scale backbone network, image patch embedding can be performed first. That is, the image is divided into image patches of uniform size, and the generated image patches serve as the basic modeling units in the multi-scale backbone network, used for global dependency modeling of these image patches. Through image patch embedding, the output features of the previous level can be transformed into the input features of the current level, serving as the input to the first encoder of the current level. Optionally, in each encoder, layer normalization, multi-scale self-attention, and feature weighting through fully connected layers can be performed.

[0094] Optionally, in each serial encoder, local feature extraction can be performed on some features in the input features of the serial encoder, and other features in the input features can be extracted at multiple scales based on the self-attention mechanism using the multiple different convolutional networks.

[0095] For example, the input features of the multi-scale self-attention block of this disclosure, after layer normalization, can first be grouped for subsequent local and global feature extraction. Correspondingly, the multi-scale self-attention block of this disclosure can be divided into two branches: local processing and global processing. The local processing branch can extract local features from a subset of the input features, while the global processing branch can perform feature dependency calculations at different scales on the remaining features from a global perspective. For example, in the local processing branch, cascaded convolutions with smaller kernels (e.g., 1x1 and / or 3x3 kernels) can be used to extract local features from the input features. In the global processing branch of the multi-scale self-attention block of this disclosure, different convolutional networks can be used to perform different scale expansions on each group of features in the remaining groups to generate multiple groups of features at different scales. Therefore, by performing multi-scale expansion before the input self-attention layer, the self-attention layer can be used to perform feature dependency calculations at different scales based on features at different scales, thereby maintaining the same recognition sensitivity for objects at different resolutions. Optionally, in the self-attention layer described above, the correlations between all input features can be determined at different scales, and the corresponding input features can be updated based on these correlations. Therefore, the feature vectors obtained through local processing branches and global processing branches respectively can be integrated (e.g., concatenated) in the feature integration layer, thereby achieving feature modeling at different scales in a single encoder.

[0096] As described above, by designing a multi-scale backbone network for the Transformer structure in the backbone network generation module 701, the multi-scale content of the input image is captured, thereby achieving more efficient and sufficient multi-scale feature extraction.

[0097] The subnetwork generation module 702 can be configured to construct a search space for the multi-scale backbone network and determine multiple multi-scale subnetworks based on the multi-scale backbone network and the search space. Each of the multiple multi-scale subnetworks is a subset of the multi-scale backbone network, wherein each multi-scale subnetwork is distinct from the others and shares at least a portion of the multi-scale backbone network. Optionally, the subnetwork generation module 702 can perform the operations described above with reference to step 202.

[0098] In the construction of the multi-scale backbone network disclosed herein, the multi-scale content capture of the input image is considered in the design of the multi-scale backbone network, which is crucial for improving network performance. Various factors, such as the dimension of image patch embedding in the Transformer network structure, the number of heads in the multi-head self-attention mechanism used in the self-attention layer, and the depth of the visual Transformer network (i.e., the number of fully connected layers), significantly influence the inference performance and efficiency of the Transformer network. Therefore, in neural network search based on the aforementioned multi-scale backbone network, a search space can be designed based on these influencing factors. This allows for the optimization of sub-network structures based on the determined search space, thereby reducing computational load, minimizing redundancy, and maintaining accurate visual understanding.

[0099] Optionally, this disclosure may employ a neural network search approach for structural optimization of the multi-scale backbone network. For example, in the neural network search process of this disclosure, a large visual Transformer network (i.e., the aforementioned multi-scale backbone network) may be first determined, and based on this large network, various smaller networks may be generated by designing network search variables. Each smaller network may be a subset of this large network, and each smaller network may differ from others but share some of the structure and parameters of the large network.

[0100] According to embodiments of this disclosure, constructing a search space for the multi-scale backbone network may include: designing network search variables based on the multi-scale backbone network; and constructing a search space for the multi-scale backbone network based on the designed network search variables.

[0101] For example, network search variables and a search space can be designed based on various factors that influence the performance of multi-scale backbone networks (e.g., performance in visual understanding tasks and computational performance). For instance, network search variables can be designed based on parameters such as the number of levels in the multi-scale backbone network, the dimension of input features at each level, and the proportion of fully connected layers in the multi-scale backbone network. Therefore, the constructed search space can employ four considerations based on network search variables: the dimension of image patch embeddings, the proportion of fully connected layers, the network depth, and multi-scale convolutions. These considerations all significantly impact network performance.

[0102] Therefore, after determining the network search variables and establishing the search space, multiple multi-scale sub-networks can be determined based on the multi-scale backbone network and the search space.

[0103] Optionally, the multi-scale backbone network can be weighted based on the determined network search variables to identify the common structures and parameters of each sub-network, while the remaining structures and parameters related to the network search variables serve as the parameters to be optimized for each sub-network. Therefore, in determining each multi-scale sub-network, the already determined shared structures and parameters can be directly read from the constructed multi-scale backbone network and substituted into each multi-scale sub-network for use. Then, the remaining parameters to be optimized in the multi-scale sub-networks can be determined based on the possible structures and parameters in the search space, thereby determining all multi-scale sub-networks.

[0104] The subnetwork search module 703 can be configured to search for a multi-scale subnetwork from the plurality of multi-scale subnetworks as the neural network for multi-scale feature extraction. Optionally, the subnetwork search module 703 can perform the operations described above with reference to step 203.

[0105] Optionally, based on the various combinations of possible values ​​for each dimension included in the search space, corresponding multi-scale sub-networks can be pruned from the multi-scale backbone network. Therefore, each multi-scale sub-network can be trained and tested using a predetermined dataset for a specific visual understanding task (e.g., image classification) to evaluate its performance for that specific task, thereby selecting the multi-scale sub-network with the best performance as the final neural network for multi-scale feature extraction. Furthermore, while training and testing each multi-scale sub-network using a predetermined dataset for a specific visual understanding task, constraints can be placed on the number of parameters and computational complexity of the multi-scale sub-network model. For example, the number of parameters and computational power of the network model can be limited to obtain a smaller network model more suitable for on-device deployment.

[0106] As described above, the method disclosed herein establishes an effective multi-scale Transformer network model, and at the same time, it uses a specific neural network search algorithm to perform quantitative optimization on the designed network model, thereby improving its reasoning performance and efficiency in visual understanding tasks.

[0107] By employing the neural network search method disclosed herein, the determined neural network can maintain multi-scale sensitivity while reducing model complexity and computational load. Furthermore, the obtained network model is more suitable for on-device deployment, such as on mobile devices, thereby achieving better results for various visual understanding tasks on mobile devices. Therefore, the technical solution disclosed herein has strong industrialization and commercialization potential.

[0108] According to another aspect of this disclosure, a neural network search device for multi-scale feature extraction is also provided. Figure 8 A schematic diagram of a neural network search device 2000 for multi-scale feature extraction according to an embodiment of the present disclosure is shown.

[0109] like Figure 8 As shown, the neural network search device 2000 for multi-scale feature extraction may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the neural network search method for multi-scale feature extraction as described above.

[0110] The processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0111] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0112] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 9 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 9As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the neural network search method for multi-scale feature extraction provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 9 One or more components in the computing device shown.

[0113] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 10 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0114] like Figure 10 As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, a neural network search method for multi-scale feature extraction according to embodiments of the present disclosure, as described above with reference to the accompanying drawings, can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Link Dynamic Random Access Memory (SLDRAM), and Direct Memory Bus Random Access Memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0115] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a neural network search method for multi-scale feature extraction according to embodiments of this disclosure.

[0116] Embodiments of this disclosure provide a neural network search method, apparatus, device, and computer-readable storage medium for multi-scale feature extraction.

[0117] Compared to traditional neural network search methods for multi-scale feature extraction, the method provided by the embodiments of this disclosure explores the network's sensitivity to multi-scale self-attention during the network search process, thereby preserving the Transformer network structure's comprehensive perception of image content during neural network search.

[0118] The method provided in the embodiments of this disclosure establishes a multi-scale backbone network, generating different scale information at each level of the multi-scale Transformer visual network. Furthermore, it preserves multi-scale sensitivity during the optimization and pruning process using this multi-scale backbone network to obtain redundant sub-networks. This allows the Transformer network structure to retain a comprehensive perception of image content within the neural network search of this disclosure, thereby improving the performance of the Transformer neural network determined by the neural network search method of this disclosure in various visual understanding problems.

[0119] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0121] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A neural network search method for multi-scale feature extraction of images, comprising: A multi-scale backbone network is established, comprising multi-level feature extraction. In each level of feature extraction, multiple different convolutional networks are used to perform multi-scale feature extraction on the input image based on a self-attention mechanism. Each level of the multi-scale backbone network includes one or more serial encoders. In each serial encoder, the input features are grouped, a portion of the grouped features is extracted locally, and another portion of the grouped features is expanded at different scales using the multiple different convolutional networks to generate multiple sets of features at different scales. A self-attention layer is used to calculate feature dependencies at different scales based on the multiple sets of features at different scales, and the results of the local feature extraction and the feature dependency calculation are integrated. A search space is constructed for the multi-scale backbone network, and multiple multi-scale sub-networks are determined based on the multi-scale backbone network and the search space. Each of the multiple multi-scale sub-networks is a subset of the multi-scale backbone network, wherein each multi-scale sub-network is distinct from the others and shares at least a portion of the multi-scale backbone network; and One multi-scale subnetwork is searched from the plurality of multi-scale subnetworks to serve as the neural network for multi-scale feature extraction of the input image.

2. The method as described in claim 1, wherein, Constructing the search space for the multi-scale backbone network includes: Based on the aforementioned multi-scale backbone network, network search variables are designed; and Based on the designed network search variables, a search space is constructed for the multi-scale backbone network; The determination of multiple multi-scale sub-networks based on the multi-scale backbone network and the search space includes: Based on the network search variables, the multi-scale backbone network is weighted to determine the structure and parameters shared by the multiple multi-scale sub-networks; and Based on the results of the weight decomposition and the search space, the plurality of multi-scale sub-networks are determined.

3. The method as described in claim 2, wherein, Based on the results of the weight decomposition and the search space, the plurality of multi-scale sub-networks are determined to include: For each of the plurality of multi-scale subnetworks, the determined structure and parameters shared by the plurality of multi-scale subnetworks are retained, and other structures and parameters in the multi-scale subnetworks are determined based on the search space to determine the multi-scale subnetwork.

4. The method of claim 2, wherein, Each level of feature extraction in the multi-level feature extraction corresponds to multi-scale feature extraction through one or more serial encoders included in each level of the multi-scale backbone network; The search space is deployed in each serial encoder of the multi-scale backbone network.

5. The method of claim 2, wherein, Each level in the multi-scale backbone network also includes image patch embedding processing, which transforms the output feature dimension of the previous level into the input feature dimension of the current level. The constructed search space includes at least one of the following dimensions: The dimension of image patch embedding processing in the multi-scale backbone network; The proportion of fully connected layers in the multi-scale backbone network; The depth of the multi-scale backbone network corresponds to the number of fully connected layers in the multi-scale backbone network; or The convolutional scales in the multi-scale backbone network correspond to the multiple different convolutional networks.

6. The method of claim 1, wherein, Searching for a multi-scale subnetwork from the plurality of multi-scale subnetworks includes: Each of the multiple multi-scale sub-networks is trained and tested using a predetermined dataset; and Based on the training and testing results, one multi-scale subnetwork that meets predetermined conditions is selected from the plurality of multi-scale subnetworks as the final neural network for multi-scale feature extraction.

7. The method of claim 6, wherein, Searching for a multi-scale subnetwork from the plurality of multi-scale subnetworks also includes: The training and testing are performed on each of the multiple multi-scale sub-networks, subject to constraints on the parameters and computational complexity of the multiple multi-scale sub-networks.

8. A neural network search device for multi-scale feature extraction of images, comprising: A backbone network generation module is configured to establish a multi-scale backbone network, which includes multi-level feature extraction. In each level of feature extraction, multiple different convolutional networks are used to perform multi-scale feature extraction on the input image based on a self-attention mechanism. Each level of the multi-scale backbone network includes one or more serial encoders. In each serial encoder, the input features of the serial encoder are grouped, local feature extraction is performed on a portion of the grouped features, and the other portion of the grouped features are expanded at different scales using the multiple different convolutional networks to generate multiple sets of features at different scales. A self-attention layer is used to calculate feature dependencies at different scales based on the multiple sets of features at different scales, and the results of the local feature extraction and the results of the feature dependency calculation are integrated. A subnetwork generation module is configured to construct a search space for the multi-scale backbone network, and determine multiple multi-scale subnetworks based on the multi-scale backbone network and the search space, wherein each multi-scale subnetwork is a subset of the multi-scale backbone network, and each multi-scale subnetwork is distinct from the others but shares at least a portion of the multi-scale backbone network; and The sub-network search module is configured to search for a multi-scale sub-network from the plurality of multi-scale sub-networks as a neural network for multi-scale feature extraction of the input image.

9. The apparatus of claim 8, wherein, Constructing the search space for the multi-scale backbone network includes: Based on the aforementioned multi-scale backbone network, network search variables are designed; and Based on the designed network search variables, a search space is constructed for the multi-scale backbone network; The determination of multiple multi-scale sub-networks based on the multi-scale backbone network and the search space includes: Based on the network search variables, the multi-scale backbone network is weighted to determine the structure and parameters shared by the multiple multi-scale sub-networks; and Based on the results of the weight decomposition and the search space, the plurality of multi-scale sub-networks are determined.

10. The apparatus of claim 9, wherein, Each level in the multi-scale backbone network also includes image patch embedding processing, which transforms the output feature dimension of the previous level into the input feature dimension of the current level. The constructed search space includes at least one of the following dimensions: The dimension of image patch embedding processing in the multi-scale backbone network; The proportion of fully connected layers in the multi-scale backbone network; The depth of the multi-scale backbone network corresponds to the number of fully connected layers in the multi-scale backbone network; or The convolutional scales in the multi-scale backbone network correspond to the multiple different convolutional networks.

11. A neural network search device for multi-scale feature extraction, comprising: One or more processors; as well as One or more memories storing a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-7.

12. A computer program product stored on a computer-readable storage medium and comprising computer instructions that, when executed by a processor, cause a computer device to perform the method of any one of claims 1-7.

13. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images