Multi-scale perception model, target recognition method and related equipment based on converter model

Abstract

The present disclosure provides a multi-scale perception model based on a converter model, comprising: an image feature extraction network, comprising: an encoder formed by stacking multi-level attention modules, for outputting a plurality of image block token (Patch Token) sequences extracted by the multi-level attention modules based on an input image to be identified; a feature fusion network, for fusing the plurality of image block token sequences to obtain a fused target image block token sequence; and at least one convolutional classification head network, for performing target recognition on the target image block token sequence to obtain at least one target object in the image to be identified and its corresponding recognition result. Based on the above-mentioned multi-scale perception model based on a converter model, the present disclosure also provides a target recognition method and related equipment.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a multi-scale perception model based on a converter model, a target recognition method, and related devices. Background Art

[0002] The Visual Transformer (ViT) model applies the Transformer model to image classification. The Transformer uses a self-attention mechanism, which allows the model to obtain information about all other pixels when processing a single pixel, better capturing the global structure of the image.

[0003] With the continuous development of deep learning, a large number of excellent pre-trained models have emerged in recent years. These pre-trained models, trained on large-scale data, can learn universal features and have strong generalization capabilities. Performing downstream image classification tasks based on such pre-trained models often achieves better results than training a small model from scratch. Furthermore, since the number of parameters involved in training is reduced, iteration speed is also faster. However, most of the training of pre-trained models involves fine-tuning the global features output by the model, and therefore, they are often ineffective in recognizing small objects. For small object recognition, training a classification model may be considered, but the cost of labeling the training data is high, and adding categories requires retraining the classification model, which increases the iteration cost. Summary of the Invention

[0004] In light of this, embodiments of the present disclosure provide a multi-scale perception model, object recognition method, and related devices based on a transformer model. These models can achieve multi-scale object recognition in image classification tasks, particularly effective in identifying small objects. Furthermore, this transformer-based multi-scale perception model features a small number of training parameters, fast iteration speed, and low iteration cost.

[0005] The multi-scale perception model based on the converter model described in the embodiments of the present disclosure may include:

[0006] An image feature extraction network comprising: an encoder formed by stacking multi-level attention modules, configured to output a sequence of multiple image patch tokens extracted by the multi-level attention modules based on an input image to be recognized;

[0007] a feature fusion network, configured to fuse the plurality of Patch Token sequences to obtain a fused target Patch Token sequence; and

[0008] At least one convolutional classification head network is used to perform target recognition on the target Patch Token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result.

[0009] In an embodiment of the present disclosure, the image feature extraction network includes:

[0010] An image segmentation module, configured to segment the image to be identified into a plurality of image patches;

[0011] A linear mapping module, configured to perform linear mapping on the plurality of patches respectively to obtain feature vectors (Embedding) corresponding to the plurality of patches; and

[0012] The encoder includes a multi-level stacked attention module, which is used to encode the embeddings corresponding to the multiple patches, and obtain the patch token sequences extracted by each attention module in the multi-level attention module.

[0013] In an embodiment of the present disclosure, the image feature extraction network is implemented by a pre-trained transformer-based image classification model.

[0014] In an embodiment of the present disclosure, the feature fusion network is used to fuse the multiple PatchToken sequences by adding them together or by splicing the feature dimensions together.

[0015] In an embodiment of the present disclosure, the feature fusion network includes:

[0016] A Patch Token sequence selection module, configured to select a candidate Patch Token sequence from the plurality of Patch Token sequences; and

[0017] The fusion module is used to fuse the selected candidate Patch Token sequences to obtain the target Patch Token sequence.

[0018] In an embodiment of the present disclosure, the convolutional classification head network includes: a classification network formed by stacking multiple convolutional modules; and a fully connected layer.

[0019] In an embodiment of the present disclosure, the convolution module is a residual block; or, the convolution module is a dense block.

[0020] Corresponding to the multi-scale perception model based on the transformer model, embodiments of the present disclosure also disclose a target recognition method. This target recognition method may include: extracting multiple Patch Token sequences corresponding to an input image to be recognized; fusing the multiple Patch Token sequences to obtain a fused target Patch Token sequence; and performing target recognition on the target Patch Token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result.

[0021] In an embodiment of the present disclosure, extracting multiple Patch Token sequences corresponding to an input image to be identified includes: dividing the image to be identified into multiple Patches based on an image segmentation module based on a pre-trained Transformer-based image classification model; linearly mapping the multiple Patches respectively based on a linear mapping module based on a pre-trained Transformer-based image classification model to obtain Embeddings corresponding to the multiple Patches; and encoding the Embeddings corresponding to the multiple Patches using a multi-level stacked attention module based on an encoder based on a pre-trained Transformer-based image classification model to obtain Patch Token sequences extracted by each attention module in the multi-level attention module.

[0022] In an embodiment of the present disclosure, fusing the multiple Patch Token sequences includes: fusing the multiple Patch Token sequences by adding them together or fusing the multiple Patch Token sequences by splicing their feature dimensions together.

[0023] In an embodiment of the present disclosure, fusing the multiple Patch Token sequences includes: selecting a candidate Patch Token sequence from the multiple Patch Token sequences; and fusing the selected candidate Patch Token sequences to obtain the target Patch Token sequence.

[0024] In an embodiment of the present disclosure, fusing the selected candidate Patch Token sequences includes fusing the multiple candidate Patch Token sequences by adding them together or fusing the multiple candidate Patch Token sequences by splicing feature dimensions together.

[0025] In an embodiment of the present disclosure, performing target recognition on the target Patch Token sequence includes: performing target recognition on the target Patch Token sequence based on a convolutional classification head network; wherein the convolutional classification head network includes: a classification network formed by stacking multiple convolutional modules; and a fully connected layer.

[0026] In addition, an embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned target recognition method when executing the program.

[0027] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned target recognition method.

[0028] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned target recognition method.

[0029] The convolutional classification head network used in the multi-scale perception model, target recognition method, and related devices based on the transformer model described in the embodiments of the present disclosure employs a convolutional structure, resulting in a strong perception of local image information. The image feature extraction network employed employs a transformer model structure, resulting in a strong global inductive modeling capability. Therefore, the multi-scale perception model, target recognition method, and related devices based on the transformer model provided in the embodiments of the present disclosure can simultaneously possess both global information and information perception capabilities, thereby enabling multi-scale target recognition with good performance for both large and small targets.

[0030] Furthermore, the image feature extraction network can be implemented using a pre-trained transformer-based network, resulting in strong generalization capabilities. Furthermore, since the image feature extraction network does not require training, both the model training cost and the iteration cost are very low in the disclosed embodiments.

[0031] Furthermore, since the above-mentioned image feature extraction network does not participate in training, when deploying the above-mentioned multi-scale perception model based on the converter model, multiple trained convolutional classification head networks can be merged together and share a backbone network to construct a multi-classification model. This construction method can support multiple categories without basically increasing the inference cost. At the same time, the addition of new categories will not affect the existing categories, which greatly reduces the cost of deployment resources and the cost of model iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 The structure of the multi-scale perception model based on the converter model described in some embodiments of the present disclosure is shown.

[0034] Figure 2 The internal structure of the image feature extraction network 110 described in the embodiment of the present disclosure is shown.

[0035] Figure 3 Shows the visualization results of reconstructing the first 10 dimensions of the Patch Token output by the 12 attention modules of a ViT model.

[0036] Figure 4 The internal structure of the feature fusion network 120 described in the embodiment of the present disclosure is shown.

[0037] Figure 5 The internal structure of the convolutional classification head network 130 according to an embodiment of the present disclosure is shown.

[0038] Figure 6 The structure of the multi-classification model described in the embodiment of the present disclosure is shown.

[0039] Figure 7 The implementation process of the target recognition method described in some embodiments of the present disclosure is shown.

[0040] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0043] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0044] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0045] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0046] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0047] As mentioned above, although performing downstream image classification tasks based on pre-trained models has the characteristics of fast iteration speed and better results, performing downstream image classification tasks based on pre-trained models usually does not perform well in recognizing small objects.

[0048] To address the above problems, embodiments of the present disclosure provide a multi-scale perception model based on a converter model. Figure 1The structure of the multi-scale perception model based on the converter model described in some embodiments of the present disclosure is shown. Figure 1 As shown in FIG, the multi-scale perception model based on the converter model includes the following network modules:

[0049] An image feature extraction network 110, comprising an encoder composed of a stack of multi-level attention modules, configured to output a sequence of tokens (Patch Tokens) of multiple image patches extracted by the multi-level attention modules based on an input image to be recognized;

[0050] A feature fusion network 120 is used to fuse the multiple Patch Token sequences to obtain a fused target Patch Token sequence; and

[0051] At least one convolutional classification head network 130 is used to perform target recognition on the target Patch Token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result.

[0052] The structures of the above-mentioned networks are described in detail below with reference to the accompanying drawings and specific examples.

[0053] Figure 2 The internal structure of the image feature extraction network 110 according to the embodiment of the present disclosure is shown. Figure 2 As shown, the image feature extraction network 110 may include:

[0054] An image segmentation module 210 is used to segment the image to be identified into multiple image patches;

[0055] A linear mapping module 220 is configured to perform linear mapping on the plurality of image blocks to obtain feature vectors (Embedding) corresponding to the plurality of image blocks; and

[0056] The encoder 230 includes a multi-level stacked attention module, which is used to encode the feature vectors corresponding to the above-mentioned multiple image blocks, and obtain the above-mentioned multiple Patch Token sequences extracted by each attention module in the above-mentioned multi-level attention module.

[0057] In the embodiments of the present disclosure, the image feature extraction network 110 can be implemented using a pre-trained Transformer-based image classification model, such as a pre-trained ViT model. Therefore, the specific structures and training methods of the image segmentation module 210, linear mapping module 220, and encoder 230 can refer to the ViT model or other pre-trained Transformer-based image classification models and will not be described in detail here.

[0058] In addition, it should be noted that, under normal circumstances, in addition to the length and width of the image, each pixel of an image will also correspond to information of multiple dimensions. For example, for an RGB image, each pixel will correspond to a pixel value of three dimensions. Therefore, in the embodiment of the present disclosure, the above-mentioned multiple Patch Token sequences generally refer to Patch Token sequences containing multiple dimensions. In other words, assuming that the image to be identified is divided into K patches, each patch corresponds to information of N dimensions, and the above-mentioned image feature extraction network 110 includes M-level attention modules, then the above-mentioned image feature extraction network 110 will output M groups of Patch Token sequences, and each group of Patch Token sequences corresponds to the output of an attention module. Among them, each group of Patch Token sequences contains K Patch Tokens, and each Patch Token also corresponds to N dimensions. Further, the number of Patch Tokens contained in each Patch Token sequence is the number of image blocks into which the image to be identified is divided. Among them, for the convenience of display, Figure 2 Only one dimension of data is shown after a set of Patch Token sequences is transformed into two-dimensional features.

[0059] It can be understood that by stacking the patch tokens output by each attention module, the ViT model can simultaneously capture shallow and deep semantic features of the image, allowing the ViT model to better understand both physical objects (e.g., animals, flags, sculptures, etc.) and abstract semantics (e.g., horror, disgust, etc.). However, Transformer-based models suffer from an "over-smoothing" problem. The model uses a self-attention mechanism to capture the relationship between input image patches. This process can cause the model's feature representation to gradually lose its distinctiveness during continuous learning, causing all features to converge or become too close, reducing the gap between features and thus decreasing performance. Experiments have found that using only the patch tokens output by the ViT model's final attention module for classification tasks results in poor recognition of some physical objects (e.g., flags, logos, etc.), which typically rely more on shallow features such as basic visual elements such as color, texture, edges, and brightness. Figure 3 The visualization results of the reconstruction of the first 10 dimensions of the Patch Token output by the 12 attention modules of a ViT model are shown. Figure 3 It can be seen that the Patch Token output by the middle-level attention module contains more intuitive shallow semantic information, such as Figure 3Based on the above research, we can conclude that by fusing the Patch Tokens output by the multi-level attention module, the model can better understand both the entity objects in the image and the abstract semantics of the image. This allows the model to retain both the shallow features and deep semantics of the image during downstream classification training, resulting in better object recognition.

[0060] Specifically, in an embodiment of the present disclosure, the feature fusion network 120 can fuse the multiple Patch Token sequences by adding them together to obtain a fused target Patch Token sequence. That is, the corresponding Patch Tokens of the multiple Patch Token sequences output by the multi-level attention module are directly added together to obtain the added target Patch Token sequence. As an alternative, other fusion methods can also be used for fusion, such as feature fusion by splicing feature dimensions, etc. The embodiment of the present disclosure does not limit the fusion method of multiple Patch Token sequences.

[0061] In other embodiments of the present disclosure, as an alternative to the above-mentioned solution, it is also possible to first select multiple candidate Patch Token sequences from the above-mentioned multiple Patch Token sequences, and then fuse the selected candidate Patch Token sequences. In the process of selecting candidate Patch Token sequences, multiple factors such as the number of attention modules and the level of the attention modules can be considered. For example, considering that the Patch Token sequence output by the attention module at the middle level contains more intuitive shallow semantic information, it is possible to start selecting from the Patch Token sequence output by the attention module at a certain middle level, and select the Patch Token sequence output by the attention module at the last level as the above-mentioned candidate Patch Token sequence. Specifically, according to the number of attention modules, the Patch Token sequence output by the attention modules at the last two-thirds levels can be selected as the above-mentioned candidate Patch Token sequence, and so on. For another example, it is also possible to select multiple candidate Patch Token sequences from the above-mentioned multiple Patch Token sequences according to pre-set selection rules, such as selecting at a certain interval, etc. It should be noted that the embodiments of the present disclosure do not limit the selection method of the candidate Patch Token sequence. In the above situation, Figure 4 FIG. 1 shows the internal structure of the feature fusion network 120 according to an embodiment of the present disclosure. Figure 4 As shown, the feature fusion network 120 may include:

[0062] A Patch Token sequence selection module 410 is configured to select a candidate Patch Token sequence from the plurality of Patch Token sequences; and

[0063] The fusion module 420 is configured to fuse the selected candidate Patch Token sequences to obtain the target Patch Token sequence.

[0064] Specifically, in the embodiments of the present disclosure, the fusion module 420 may fuse the selected target Patch Token sequences by summing them. Alternatively, other fusion methods may be used, such as feature fusion by splicing feature dimensions. The embodiments of the present disclosure do not limit the fusion method used by the fusion module 420.

[0065] Figure 5 FIG. 1 shows the internal structure of the convolution classification head network 130 according to an embodiment of the present disclosure. Figure 5 As shown, the convolutional classification head network 130 may include: a classification network 510 formed by stacking multiple convolution modules 512 and a fully connected layer 520. The classification network 510 formed by stacking multiple convolution modules 512 may be implemented in a variety of ways. For example, the convolution module 512 may be a residual block or a dense block. In other words, the classification network 510 may be implemented as a residual network composed of multiple residual blocks, or as a dense network composed of multiple dense blocks, and so on.

[0066] As previously mentioned, in the embodiments of the present disclosure, the above-mentioned image feature extraction network 110 can be implemented by a pre-trained ViT model, or can also be implemented by other pre-trained Transformer-based image classification models. Therefore, the above-mentioned image feature extraction network 110 does not participate in the update of the model weights during training. The training of the Transformer-based multi-scale perception model described in the embodiments of the present disclosure only involves the parameter training of the above-mentioned convolutional classification head network 130. That is, the parameters of the convolutional classification head network 130 are trained based on the error between the target recognition results of the images in the sample data and the annotation results of the corresponding images.

[0067] In addition, since the above-mentioned image feature extraction network 110 does not participate in training, multiple trained convolutional classification head networks can be merged together and share a backbone network to construct a multi-classification model. This construction method greatly reduces the deployment resource cost. Figure 6 The structure of the multi-classification model described in the embodiment of the present disclosure is shown. Figure 6 As shown, the convolutional classification head network 130 connected to the feature fusion network 120 may include multiple convolutional classification head networks, each of which corresponds to a classification task, thereby forming a multi-classification model to implement multi-classification tasks.

[0068] It can be seen that in the multi-scale perception model based on the converter model provided in the embodiment of the present disclosure, on the one hand, its convolutional classification head network 130 adopts a convolutional structure, and thus has a strong perception capability for local image information; on the other hand, its image feature extraction network 110 adopts a converter model structure, and thus has a strong global inductive modeling capability. Therefore, the multi-scale perception model based on the converter model provided in the embodiment of the present disclosure can simultaneously have the ability to summarize global information and the ability to perceive local information by combining the backbone network of the converter model and the convolutional head network, thereby realizing multi-scale target recognition, and has good recognition performance for targets of larger or smaller scales, and also has a good recognition effect for targets with relatively large scale changes. Moreover, the self-attention mechanism of the converter model can capture information in a global range and better obtain the long-distance dependency between patch tokens. In particular, in the embodiment of the present disclosure, Patch Token is used instead of selecting another output classification token (Class Token) of the encoder for downstream classification tasks. This is because Patch Token still retains the position information of image features to a certain extent, and it is similar to the feature map extracted by the convolutional neural backbone network. It is possible to further select a classification head network with a convolutional structure during classification. By performing operations on features through small-scale convolution kernels, it is possible to more effectively capture local features of the image.

[0069] In addition, the above-mentioned image feature extraction network 110 can be implemented through a pre-trained ViT network or other pre-trained networks based on transformer models, and thus has strong generalization capabilities. Moreover, since the above-mentioned image feature extraction network 110 does not participate in training, in the embodiments of the present disclosure, both the training cost and the iteration cost of the model are very low. Experiments have confirmed that the training of the above-mentioned multi-scale perception model greatly reduces the requirements for the amount of positive example data. Training can be completed with dozens of positive examples, and overfitting problems are not prone to occur. Compared with training a classification model from scratch, the speed of model training can be increased by more than 20 times.

[0070] Based on the above-mentioned multi-scale perception model based on the converter model, an embodiment of the present disclosure further discloses a target recognition method. Figure 7 The implementation process of the target recognition method described in the embodiment of the present disclosure is shown. Figure 7 As shown, the target recognition method may include:

[0071] In step 710, multiple Patch Token sequences corresponding to the input image to be recognized are extracted;

[0072] In step 720, the plurality of Patch Token sequences are fused to obtain a fused target Patch Token sequence; and

[0073] In step 730, target recognition is performed on the target Patch Token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result.

[0074] In an embodiment of the present disclosure, the above-mentioned multiple Patch Token sequences are respectively output by the multi-level stacked attention modules constituting the encoder in the above-mentioned image extraction network.

[0075] In an embodiment of the present disclosure, the above-mentioned step 710 may specifically include: first, dividing the image to be identified into multiple patches; second, linearly mapping the above-mentioned multiple patches respectively to obtain the embeddings corresponding to the above-mentioned multiple patches; finally, encoding the embeddings corresponding to the above-mentioned multiple patches respectively by the above-mentioned multi-level attention module to obtain the above-mentioned multiple Patch Token sequences.

[0076] In practical applications, the above step 710 can be implemented by a pre-trained ViT model or other pre-trained Transformer-based image classification models.

[0077] In some embodiments of the present disclosure, the above step 720 may specifically include: fusing the above multiple Patch Token sequences in an addition manner to obtain a fused target Patch Token sequence.

[0078] In some other embodiments of the present disclosure, the above step 720 may specifically include: fusing the above multiple Patch Token sequences by using a feature dimension splicing method to obtain a fused target Patch Token sequence.

[0079] In some further embodiments of the present disclosure, the above step 720 may specifically include: first selecting multiple candidate Patch Token sequences from the above multiple Patch Token sequences, and then fusing the selected candidate Patch Token sequences to obtain a fused target Patch Token sequence.

[0080] Specifically, in the process of selecting candidate Patch Token sequences, multiple factors such as the number of attention modules and the level at which the attention modules are located can be considered. For example, considering that the Patch Token sequence output by the attention module at the middle level contains more intuitive shallow semantic information, it is possible to start selecting from the Patch Token sequence output by the attention module at a certain middle level, and select the Patch Token sequence output by the attention module at the last level as the above-mentioned candidate Patch Token sequence. For another example, multiple candidate Patch Token sequences can also be selected from the above-mentioned multiple Patch Token sequences according to pre-set selection rules, such as selecting at a certain interval, etc. It should be noted that the embodiments of the present disclosure do not limit the selection method of the candidate Patch Token sequence.

[0081] In an embodiment of the present disclosure, the above step 730 can be implemented by a convolutional classification head network. The above convolutional classification head network may include: a classification network formed by stacking multiple convolutional modules and a fully connected layer. The above classification network formed by stacking multiple convolutional modules can be implemented in a variety of ways, for example, it can be implemented by a residual network composed of multiple ResidualBlocks, or it can be implemented by a dense network composed of multiple Dense Blocks, etc. The above convolutional classification head network can be trained based on the error between the target recognition result of the image in the sample data and the annotation result of the corresponding image.

[0082] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the target recognition method described in any of the above embodiments is implemented.

[0083] Figure 8 1 is a schematic diagram showing the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0084] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0085] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0086] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0087] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0088] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0089] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0090] The electronic device of the above embodiment is used to implement the corresponding target recognition method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0091] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the target recognition method described in any of the above embodiments.

[0092] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0093] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the task processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0094] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0095] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0096] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0097] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A multi-scale perception model based on a transformer model, comprising: An image feature extraction network comprising: an encoder formed by stacking multi-level attention modules, configured to output a sequence of multiple image patch tokens extracted by the multi-level attention modules based on an input image to be recognized; a feature fusion network, configured to fuse the plurality of image block token sequences to obtain a fused target image block token sequence; and At least one convolutional classification head network is used to perform target recognition on the target image block token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result; wherein, The convolution classification head network includes: a classification network formed by stacking multiple convolution modules and a fully connected layer; the convolution module is a residual block; or, the convolution module is a dense block.

2. The multi-scale perception model based on the transformer model according to claim 1, wherein: The image feature extraction network includes: An image segmentation module, configured to segment the image to be identified into a plurality of image blocks; a linear mapping module, configured to perform linear mapping on the plurality of image blocks respectively to obtain feature vectors Embedding corresponding to the plurality of image blocks; and The encoder includes a multi-level stacked attention module, which is used to encode the feature vectors corresponding to the multiple image blocks, and obtain the image block token sequences extracted by each attention module in the multi-level attention module.

3. The multi-scale perception model based on the transformer model according to claim 1 or 2, wherein: The image feature extraction network is implemented through a pre-trained transformer-based image classification model.

4. The multi-scale perception model based on the transformer model according to claim 1, wherein: The feature fusion network is used to fuse the multiple image block token sequences by adding them together or by splicing the multiple image block token sequences with feature dimensions.

5. The multi-scale perception model based on the transformer model according to claim 1, wherein: The feature fusion network includes: an image block token sequence selection module, configured to select a candidate image block token sequence from the plurality of image block token sequences; and The fusion module is used to fuse the selected candidate image block token sequences to obtain the target image block token sequence.

6. A target recognition method, applied to the multi-scale perception model based on the transformer model according to claim 1, comprising: Extracting a plurality of image block token sequences corresponding to an input image to be recognized by an image feature extraction network of the multi-scale perception model; fusing the plurality of image block token sequences using a feature fusion network of the multi-scale perception model to obtain a fused target image block token sequence; as well as At least one convolutional classification head network of the multi-scale perception model performs target recognition on the target image block token sequence to obtain at least one target object in the image to be recognized and its corresponding recognition result.

7. The target recognition method according to claim 6, wherein: The image feature extraction network of the multi-scale perception model extracts a plurality of image block tokens corresponding to the input image to be recognized, and the image block token sequence includes: An image segmentation module based on a pre-trained converter-based image classification model segments the image to be identified into a plurality of image blocks; Performing linear mapping on the plurality of image blocks respectively based on a linear mapping module of a pre-trained converter-based image classification model to obtain feature vectors corresponding to the plurality of image blocks; and An encoder based on a pre-trained converter-based image classification model encodes the feature vectors corresponding to the multiple image blocks using a multi-level stacked attention module to obtain image block token sequences extracted by each attention module in the multi-level attention module.

8. The target recognition method according to claim 6, wherein: The fusing of the multiple image block token sequences by the feature fusion network of the multi-scale perception model includes: fusing the multiple image block token sequences by addition or fusing the multiple image block token sequences by feature dimension splicing.

9. The target recognition method according to claim 6, wherein: Fusing the plurality of image block token sequences by the feature fusion network of the multi-scale perception model includes: selecting a candidate image block token sequence from the plurality of image block token sequences; and The selected candidate image block token sequences are fused to obtain the target image block token sequence.

10. The target recognition method according to claim 9, wherein: Fusing the selected candidate image block token sequences includes: fusing the multiple candidate image block token sequences by adding them together or fusing the multiple candidate image block token sequences by splicing them together in feature dimensions.

11. The target recognition method according to claim 8, wherein: Performing target recognition on the target image block token sequence by at least one convolutional classification head network of the multi-scale perception model includes: performing target recognition on the target image block token sequence based on the convolutional classification head network.

12. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the target recognition method according to any one of claims 6 to 11 is implemented.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the target recognition method according to any one of claims 6 to 11.

14. A computer program product comprising computer program instructions, which, when executed on a computer, enable the computer to execute the target recognition method according to any one of claims 6 to 11.

Images