Image recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an image recognition method and device, electronic equipment and storage medium, belonging to the technical field of computer vision. The method comprises: performing global feature extraction on a target object in an acquired to-be-processed image to obtain initial features, inputting the initial features into a pre-trained image recognition model to obtain encoded features, performing cross fusion processing on the initial features and the encoded features based on feature enhancement to obtain cross fusion features; performing dynamic position query on the encoded features to obtain dynamic query features, and acquiring static query features corresponding to the dynamic query features; decoding the cross fusion features based on the dynamic query features and the static query features to obtain decoded features, and recognizing the target object in the to-be-processed image based on the decoded features to obtain a target recognition result. The present application can improve the recognition quality of small target objects in the to-be-processed image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image recognition requires first detecting objects in the image to be processed, and then performing image recognition on the objects to obtain the recognition result. Currently, image recognition can be performed based on neural network models.

[0003] Typically, the target objects to be identified in an image can be divided into large and small objects. In related technologies, neural network models can easily detect and identify large objects in the image. However, because small objects contain relatively limited information, the features that neural network models can detect in small objects are very vague. These vague features cannot support high-confidence predictions from the neural network model during the recognition process. Therefore, related technologies have poor recognition quality for small objects in the image. Summary of the Invention

[0004] The main objective of this application is to provide an image recognition method, apparatus, electronic device, and storage medium, which aims to improve the recognition quality of small target objects in the image to be processed.

[0005] To achieve the above objectives, a first aspect of this application proposes an image recognition method, the method comprising:

[0006] The image to be processed is acquired, and global feature extraction is performed on the target object in the image to obtain initial features. The initial features are then input into a pre-trained image recognition model.

[0007] The initial features are encoded to obtain encoded features. Feature enhancement is performed on both the initial features and the encoded features. Based on the enhanced initial features and encoded features, cross-fusion processing is performed to obtain cross-fused features.

[0008] Dynamic location lookup is performed on the encoded features to obtain dynamic query features, and static query features corresponding to the dynamic query features are obtained.

[0009] The cross-fusion features are decoded based on dynamic and static query features to obtain decoded features. The target objects in the image to be processed are then identified based on the decoded features to obtain the target recognition results.

[0010] In some embodiments, feature enhancement is performed on the initial features and the encoded features, respectively, including:

[0011] The initial features are activated by performing the first fully connected processing and the first normalization processing on the results, and the activated results are used to extract features to obtain the first extracted features corresponding to the initial features.

[0012] The first product is calculated based on the first extracted features and the encoded features. The initial features are then summed with the first product to obtain the initial features after feature enhancement.

[0013] The results of the second fully connected processing and the second normalization processing of the encoded features are activated, and the activated results are used for feature extraction to obtain the second extracted features corresponding to the encoded features.

[0014] The second product is calculated based on the second extracted features and the initial features. The encoded features are then summed with the second product to obtain the enhanced encoded features.

[0015] In some embodiments, cross-fusion processing is performed based on the enhanced initial features and encoded features to obtain cross-fused features, including:

[0016] The initial features and encoded features after feature enhancement are spliced ​​together to obtain the connection features, and the third extracted features are extracted from the connection features.

[0017] The third extracted features are cross-fused with the initial features and encoded features at different fusion ratios to obtain cross-fused features.

[0018] In some embodiments, the third extracted features are cross-fused with the initial features and encoded features at different fusion ratios, including:

[0019] Based on the preset first weight value, calculate the third product between the encoded features and the third extracted features;

[0020] Based on the preset second weight value, calculate the fourth product between the initial feature and the third extracted feature, where the first weight value is greater than the second weight value;

[0021] The cross-fusion feature is obtained by summing the third and fourth products.

[0022] In some embodiments, dynamic location lookup is performed on the encoded features to obtain dynamic query features, including:

[0023] Linear processing of the encoded features yields multiple corresponding linear processing results, where each linear processing result includes a corresponding linear processing score.

[0024] Sort the results of multiple linear processing based on the order of their linear processing scores.

[0025] Based on the preset target number, the sorted linear processing results are filtered to obtain dynamic query features.

[0026] In some embodiments, the image recognition model is trained through the following steps:

[0027] Acquire sample images, perform global feature extraction on the target objects in the sample images to obtain initial sample features, and input the initial sample features into the image recognition model;

[0028] The initial features of the sample are encoded to obtain the sample encoded features. The initial features and the sample encoded features are then enhanced. Based on the enhanced initial features and the sample encoded features, a cross-fusion process is performed to obtain the sample cross-fusion features.

[0029] Dynamic location lookup is performed on the sample coding features to obtain the sample dynamic query features, and the sample static query features corresponding to the sample dynamic query features are obtained.

[0030] The sample cross-fusion features are decoded based on the sample dynamic query features and sample static query features to obtain sample decoded features. Based on the sample decoded features, the sample target objects in the sample image are identified to obtain the sample target recognition results.

[0031] Based on the target recognition results of the samples, the sample loss value is determined, and the parameters of the image recognition model are adjusted according to the sample loss value to obtain the trained image recognition model.

[0032] In some embodiments, after determining the sample loss value, the method further includes:

[0033] The image recognition model and the backbone model are connected. The backbone model is used to extract global features of the target objects in the sample image to obtain the initial features of the sample.

[0034] Based on the sample loss value, the image recognition model and the backbone model are jointly trained to obtain the trained image recognition model and backbone model.

[0035] In some embodiments, before performing global feature extraction on the target object in the sample image, the method further includes:

[0036] Obtain the first bounding box of the sample image, and determine the height and width of the first bounding box.

[0037] Based on the target object in the sample, determine the second bounding box, and determine the height and width of the second bounding box.

[0038] The sample target objects confined within the second bounding box are classified to obtain the true category information indicating the category to which the sample target objects belong;

[0039] Calculate the first ratio between the height of the second bounding box and the height of the first bounding box, and calculate the second ratio between the width of the second bounding box and the width of the first bounding box;

[0040] Based on the first ratio, the second ratio, and the true category information, the true label of the target object in the sample is determined.

[0041] In some embodiments, the sample target identification result includes sample location result and sample category result, wherein the sample category result includes sample category confidence.

[0042] Based on the target recognition results of the samples, the sample loss value is determined, including:

[0043] Determine the third bounding box corresponding to the sample location result, and determine the height and width of the third bounding box.

[0044] Calculate the third ratio between the height of the third bounding box and the height of the first bounding box, and calculate the fourth ratio between the width of the third bounding box and the width of the first bounding box;

[0045] Based on the third ratio, the fourth ratio, and the sample category results, the predicted label of the target object is determined;

[0046] Perform a first logarithmic operation on the predicted label to obtain the first logarithmic result, and calculate the first sub-value based on the first logarithmic result and the true label;

[0047] The confidence scores of the sample categories are processed by a second logarithm to obtain the second logarithm result. The second sub-value is calculated based on the second logarithm result and the sample category information.

[0048] Based on the first and second sub-values, the sample loss value is obtained.

[0049] To achieve the above objectives, a second aspect of this application provides an image recognition device, the device comprising:

[0050] The input module is used to acquire the image to be processed, perform global feature extraction on the target objects in the image to be processed, obtain initial features, and input the initial features into the pre-trained image recognition model;

[0051] The cross-fusion processing module is used to encode the initial features to obtain encoded features, and to perform feature enhancement on the initial features and encoded features respectively. Based on the enhanced initial features and encoded features, cross-fusion processing is performed to obtain cross-fused features.

[0052] The query module is used to perform dynamic location queries on the encoded features, obtain dynamic query features, and acquire the static query features corresponding to the dynamic query features.

[0053] The target recognition result module is used to decode the cross-fused features based on dynamic query features and static query features to obtain decoded features, and then to identify the target objects in the image to be processed based on the decoded features to obtain the target recognition result.

[0054] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image recognition method of the first aspect described above.

[0055] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image recognition method of the first aspect described above.

[0056] The image recognition method, apparatus, electronic device, and storage medium proposed in this application acquire an image to be processed, perform global feature extraction on target objects in the image to obtain initial features, and input the initial features into a pre-trained image recognition model. Next, the initial features are encoded to obtain encoded features, and feature enhancement is performed on both the initial and encoded features. Based on the enhanced initial and encoded features, cross-fusion processing is performed to obtain cross-fused features, which include enhanced small target object features, making the originally blurry small target object features in the initial features clearer, thus providing a reliable feature foundation for subsequent image recognition. Dynamic location lookup is performed on the encoded features to obtain dynamic query features, and corresponding static query features are obtained. The image recognition model can learn the relationship between the model's predicted results and the actual results based on the dynamic and static query features. The bias information of the bounding box makes the position and category represented by the final target recognition result closer to the real situation, thereby improving the image recognition quality of the image to be processed, especially the recognition quality of small target objects. Finally, the cross-fusion features are decoded based on dynamic query features and static query features to obtain decoded features, and the target objects in the image to be processed are identified based on the decoded features to obtain the target recognition result. Since the cross-fusion processing features reduce the ambiguity of small target object features, and further, based on dynamic query features and static query features, the cross-fusion features are extracted with emphasis. Therefore, the small target object features contained in the encoded features obtained by the encoder are richer, more comprehensive and clearer, that is, the features of the corresponding small target objects represented are more accurate. As a result, the target recognition result obtained by the recognition processing based on the small target object features is of higher quality, improving the detection accuracy of small target objects. Attached Figure Description

[0057] Figure 1 This is an optional image recognition schematic diagram of the image to be processed in related technologies;

[0058] Figure 2 This is a schematic diagram of an optional model structure of the image recognition model provided in this application embodiment;

[0059] Figure 3 This is an optional flowchart of the image recognition method provided in the embodiments of this application;

[0060] Figure 4 This is a schematic diagram of an optional feature processing flow of the image recognition method provided in the embodiments of this application;

[0061] Figure 5This is an optional bounding box diagram of the image recognition method provided in the embodiments of this application;

[0062] Figure 6 This is a schematic diagram of an optional functional module of the image recognition device provided in the embodiments of this application;

[0063] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] First, let's analyze some of the terms used in this application:

[0067] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0068] The ImageNet-1K dataset is a subset of the ImageNet dataset, containing images from 1000 different categories. "1K" indicates that the dataset contains image samples from 1000 categories, covering a wide range of objects, animals, scenes, etc., and is widely used in computer vision research. The ImageNet dataset is a large-scale image database commonly used for training and evaluating computer vision tasks such as image classification, object detection, and image segmentation.

[0069] The Common Objects in Context 2017 (COCO2017) dataset is a well-known public image dataset created by Microsoft Research for tasks such as image understanding, object detection, segmentation, and keypoint detection. This dataset contains over 330,000 images covering more than 80 different object categories. The images are richly annotated, including object bounding boxes, instance segmentation, and keypoint information. One characteristic of the COCO2017 dataset is that the objects in the images are often in complex scenes with rich backgrounds and diverse layouts, requiring models to have stronger generalization capabilities. This dataset is widely used for training and evaluation of tasks such as object detection, instance segmentation, and keypoint detection, and holds an important position in the field of computer vision.

[0070] The WiderPerson dataset is a publicly available dataset focused on human detection, part of the Wider series of datasets. This dataset emphasizes human detection tasks, providing a large number of human images and corresponding bounding box annotations, enabling models to learn the ability to accurately detect humans in various scenes. Images in the WiderPerson dataset typically contain multiple human instances and cover a wide range of shooting scenes and angles, from frontal, side, and back views, as well as large and small targets. This dataset is commonly used to train and evaluate human detection models and has high practical value for tasks requiring human detection in complex scenes.

[0071] Image recognition requires first detecting objects in the image to be processed, and then performing image recognition on the objects to obtain the recognition result. Currently, image recognition can be performed based on neural network models.

[0072] Typically, the target objects to be identified in an image can be divided into large and small objects. In related technologies, neural network models can easily detect and identify large objects in the image. However, because small objects contain relatively limited information, the features that neural network models can detect in small objects are very vague. These vague features cannot support high-confidence predictions from the neural network model during the recognition process. Therefore, related technologies have poor recognition quality for small objects in the image.

[0073] For example, such as Figure 1 As shown, Figure 1 This is an optional image recognition illustration of the image to be processed in related technologies, wherein, Figure 1(1) is an image A to be processed without image recognition processing. It can be found that the image A to be processed includes two large target objects (vehicles) and two small target objects (butterflies). The purpose of image recognition is to obtain accurate recognition results of large and small target objects. Figure 1 (2) is to Figure 1 (1) The image to be processed, A, is input into the neural network model of the related technology to obtain the recognition result A. The recognition result A includes recognition result a, recognition result b, recognition result c, and recognition result d. Among them, recognition result a and recognition result b are both cars, and the recognition results are correct; recognition result c is none, indicating that the related technology mistakenly determined that there was no target object in the area, and the recognition result is incorrect; recognition result d is a lamp, indicating that the related technology mistakenly identified a butterfly as a lamp, and the recognition result is incorrect. In other words, in the related technology, the neural network model can realize the detection and recognition of large target objects, and the recognition results obtained are usually highly accurate; however, the related technology is less effective in detecting and recognizing small target objects. Specifically, it may be unable to detect small target objects, or the accuracy of recognizing small target objects after detection is poor, and it is not able to identify the category of small target objects well.

[0074] Based on this, embodiments of this application provide an image recognition method, apparatus, electronic device, and storage medium, aiming to improve the recognition quality of small target objects in images to be processed by neural network models.

[0075] It should be noted that in this application embodiment, when information related to user identity or characteristics, such as basic user information, is required, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of a user, the user's individual permission or consent will be obtained first. Only after obtaining the user's individual permission or consent will the necessary data for the normal operation of this application embodiment be obtained. For example, when obtaining an image to be processed taken by a user, the consent of the relevant user will be obtained first; or, if the obtained image to be processed involves, but is not limited to, the user's portrait privacy or personal information privacy, the consent of the relevant user will be obtained first; otherwise, the image to be processed in this application embodiment cannot be obtained.

[0076] The image recognition method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the application scenario of the image recognition apparatus in this application is described. The image recognition apparatus includes an image recognition model, which is trained using the image recognition method proposed in this application. Figure 2 As shown, Figure 2This is a schematic diagram of an optional model structure for the image recognition model provided in this application embodiment. The backbone network, encoder, linear layer, cross-fusion module, decoder, and prediction network are all included in the image recognition model, which can be located in an image recognition device. Furthermore, the backbone network typically refers to the basic network structure used for feature extraction in a deep learning model. In other embodiments, the backbone network may not be located in the image recognition device. After acquiring the image to be processed, it is first input into the backbone network to extract initial features. Next, the initial features are input into the image recognition device, where the encoder encodes them to obtain coded features. Then, the coded features are input into the Cross CodingTwice Module (CCTM) to enhance them into cross-fused features. Simultaneously, the coded features are input into a linear layer to fuse the obtained dynamic query features with pre-defined static query features. The fused result is then input into the decoder to obtain decoded features. Finally, the decoded features are input into the prediction network to obtain the target recognition result corresponding to the image to be processed. The target recognition result includes the location and category recognition results of the target object in the image to be processed. The location recognition result represents the coordinates of the target object in the image to be processed, and the category recognition result represents the classification of the target object. Thus, the image to be processed can obtain target recognition results based on the image recognition device, completing the image recognition task.

[0077] The image to be processed includes large and small target objects. In addition to recognizing large target objects with high accuracy, the image recognition device can also detect and recognize small target objects with high quality.

[0078] Next, the image recognition method in the embodiments of this application will be specifically described through the following examples.

[0079] In this application embodiment, the description will focus on the image recognition device, which can be integrated into a computer device, such as a server. Figure 3 As shown, Figure 3 This is an optional flowchart of the image recognition method provided in the embodiments of this application. Figure 3 The method may include, but is not limited to, steps 101 to 104. When the image recognition device executes the image recognition method, the specific process is as follows. It should be noted that this embodiment... Figure 2 The order of steps 101 to 104 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0080] Step 101: Obtain the image to be processed, perform global feature extraction on the target object in the image to be processed to obtain initial features, and input the initial features into the pre-trained image recognition model.

[0081] Step 101 will be described in detail below.

[0082] In some embodiments, the image to be processed may be captured in real time, obtained from a publicly available image database, or synthesized based on an artificial intelligence model. This application does not specifically limit the source of the image to be processed. All obtained images to be processed can be recognized based on the image recognition method proposed in the embodiments of this application.

[0083] Furthermore, the target object in the image to be processed can be one or more, or there may be no target object, depending on the actual image acquired. Target objects include large target objects and small target objects, where small target objects are defined relative to large target objects. Generally, in the image to be processed, a target object whose corresponding bounding box exceeds a size threshold is defined as a large target object, while a target object whose corresponding bounding box is smaller than the size threshold is defined as a small target object. The rectangular region that can encompass all the details of the target object is called the target bounding box. Each target object corresponds to one target bounding box, and both the target bounding box and the size threshold include the height and width of the target bounding box. The size threshold is specifically determined based on the resolution of the image to be processed. For example, if the resolution of the image B to be processed is high, then the size threshold is small; if the resolution of the image B to be processed is low, then the size threshold is correspondingly large. Since the embodiments of this application aim to improve the detection accuracy of small target objects, the specific implementation methods described in the embodiments of this application are for small target objects. In fact, when using the image recognition method provided by the embodiments of this application, the detection accuracy of large target objects can also be improved, but the effect is more significant for small target objects.

[0084] Furthermore, after acquiring the image to be processed, global feature extraction is performed on the target objects in the image to obtain initial features. A backbone network can be used to extract these initial features; this backbone network can be based on a deep multilayer perceptron (MLP) network (hereinafter referred to as an MLP network). Initial features are a higher-level, more abstract representation of the pixel information corresponding to the target objects in the image to be processed. They contain global context and structural information within the image, indicating the overall information features of the image rather than just features of local regions. Initial features provide important clues about the content and semantics of the image. A single target object can have one or more extracted initial features.

[0085] It should be noted that the backbone network can also be a convolutional neural network (CNN), a visual geometry group model (VGG), or a residual network (ResNet), which can extract initial features of a global abstract representation from the image to be processed. The specific network can be set according to the actual situation, and the embodiments of this application do not impose specific limitations.

[0086] Furthermore, after obtaining the initial features, these initial features are input into a pre-trained image recognition model so that the image recognition model can use the initial features for subsequent detection and recognition.

[0087] Step 102: Encode the initial features to obtain encoded features, and perform feature enhancement on both the initial features and the encoded features. Then, perform cross-fusion processing on the enhanced initial features and encoded features to obtain cross-fused features.

[0088] Step 102 is described in detail below.

[0089] In some embodiments, the image recognition model includes a neural network containing an encoder and a decoder. The encoder encodes initial features to obtain encoded features, which are then used for feature enhancement. The enhanced initial and encoded features are then cross-fused to obtain cross-fused features. These cross-fused features include enhanced small object features, making previously blurry small object features in the initial features clearer, thus providing a reliable feature base for subsequent image recognition.

[0090] In some embodiments, feature enhancement is performed on the initial features and the encoded features respectively, including the following steps 201 to 204:

[0091] Step 201: Activate the result of the first fully connected processing and the first normalization processing on the initial features, and extract features from the activated result to obtain the first extracted features corresponding to the initial features.

[0092] Step 202: Calculate the first product based on the first extracted features and the encoded features, and sum the initial features with the first product to obtain the initial features after feature enhancement.

[0093] Steps 201 to 202 are described in detail below.

[0094] In some embodiments, such as Figure 4 As shown, Figure 4 This is a schematic diagram of an optional feature processing flow of the image recognition method provided in this application embodiment. After obtaining the encoded features, both the initial features and the encoded features are input into the cross-fusion processing module. The cross-fusion module is provided with a fully connected layer a, a normalization layer a, and an activation layer a connected in sequence. The initial features are processed by the fully connected layer a to obtain result a1. Result a1 is input into the normalization layer a, and the result a1 is processed by the normalization layer a to obtain result a2. Result a2 is activated by the activation layer a to obtain result a3. Then, result a3 is input into the gating unit, and the result a3 is further filtered and updated by the gating unit to obtain the first extracted feature B'.

[0095] Furthermore, to determine the difference between the first extracted feature and the encoded feature, the difference between the value 1 and the first extracted feature is calculated to obtain (1-B'). Then, the first product between this difference and the encoded feature is calculated. The result is added to the initial feature to obtain the initial feature after feature enhancement, as shown in the following formulas (1) and (2):

[0096] B'=δ(GELU(LN(FC(B)))) (1)

[0097]

[0098] Where δ(*) is the sigmoid function, a commonly used non-linear function typically used to map real numbers to the interval between 0 and 1, and it is usually set in a gated unit; GELU represents the activation layer, LN represents the normalization layer (LayerNorm layer), and FC represents the fully connected layer (Dense layer); B is the initial feature, E is the encoded feature, and B' is the first extracted feature. These are the initial features after feature enhancement.

[0099] Furthermore, feature enhancement of the initial features can improve the abstract level of the initial features, thereby helping to extract more feature information from small target objects in subsequent steps, and thus helping to improve the image recognition model's ability to recognize small target objects.

[0100] Step 203: Activate the result of the second fully connected processing and the second normalization processing of the encoded features, and extract features from the activated result to obtain the second extracted features corresponding to the encoded features.

[0101] Step 204: Calculate the second product based on the second extracted features and the initial features, and sum the encoded features with the second product to obtain the enhanced encoded features.

[0102] Steps 203 to 204 are described in detail below.

[0103] In some embodiments, such as Figure 4 As shown, the cross-fusion module also includes a fully connected layer b, a normalization layer b, and an activation layer b connected in sequence. Based on the fully connected layer b, the encoded features are processed by a second fully connected layer to obtain result b1. Result b1 is then input into the normalization layer b, and based on the normalization layer b, result b1 is processed by a second normalization layer to obtain result b2. Result b2 is activated based on the activation layer b to obtain result b3. Then, result b3 is input into the gating unit, and based on the gating unit, result b3 is further processed by feature filtering and updating to obtain the second extracted feature E'.

[0104] Furthermore, to determine the difference between the second extracted feature and the initial feature, the difference between the value 1 and the second extracted feature is calculated to obtain (1-E'). Then, the second product between this difference and the initial feature is calculated. The result is summed with the encoded feature to obtain the encoded feature after feature enhancement, as shown in the following formulas (3) and (4):

[0105] E'=δ(GELU(LN(FC(E)))) (3)

[0106]

[0107] in, The encoding features are enhanced features. The interpretations of the other variables in formulas (3) and (4) are the same as those in formulas (1) and (2), and will not be repeated here.

[0108] Understandably, both initial features and encoded features retain their own characteristics while selectively combining features different from themselves to obtain more comprehensive and richer information, resulting in enhanced initial features and encoded features.

[0109] Furthermore, to improve the efficiency of image recognition models in processing image features, initial features and encoded features can be enhanced in parallel to obtain enhanced initial features and encoded features simultaneously, facilitating subsequent cross-fusion processing of the enhanced initial features and encoded features. Moreover, feature enhancement of encoded features can improve the abstract level of the encoded features, helping to extract more feature information from small target objects in subsequent processing, thereby improving the recognition ability of small target objects.

[0110] In some embodiments, cross-fusion processing is performed based on the enhanced initial features and the encoded features to obtain cross-fused features, including the following steps 301 to 302:

[0111] Step 301: Concatenate the initial features and encoded features after feature enhancement to obtain the connection features, and extract the third extracted features from the connection features.

[0112] Step 302: The third extracted feature is cross-fused with the initial feature and the encoded feature at different fusion ratios to obtain the cross-fused feature.

[0113] Steps 301 to 302 are described in detail below.

[0114] In some embodiments, after obtaining the initial features and encoded features after feature enhancement, these two types of features are first concatenated to obtain connected features, and then features are extracted from the connected features to obtain third extracted features. For example... Figure 4 As shown, where, Figure 4 The symbol C in the text means feature splicing.

[0115] Furthermore, such as Figure 4 As shown, the connection features are input into a sequentially connected MLP network, a Global Response Normalization (GRN) module, and then back to the MLP network. The MLP network performs complex pattern recognition, classification, and regression tasks on the input data through multi-layered nonlinear transformations and learning. The GRN module enhances the contrast differences between the feature channels. Since the enhanced initial and encoded features have different information representations than the unenhanced initial and encoded features, the concatenated connection features need to undergo further feature extraction processing. Through multi-level feature extraction using different networks, a third extracted feature containing richer and more crucial content is obtained.

[0116] MLP networks, in particular, tend to extract local, static features from the input data, such as edges and textures. Each neuron in an MLP's hidden layer corresponds to a different local region of the input data, and by learning weights, local feature representations of the input data can be extracted. Gated recurrent networks, on the other hand, tend to extract dynamic features related to time or space from the data. They control the flow of information through gating units within the network to effectively capture temporal or spatial dependencies.

[0117] It should be noted that the output of the second MLP network can also be input into the gating unit again to continue feature extraction, further increasing the level of feature extraction and thus obtaining the third extracted feature. In practical applications, the specific networks or modules that perform the feature extraction operation on the connected features, as well as the connection order between these networks or modules, can be set according to the specific situation. This application does not make specific limitations in this regard.

[0118] In some embodiments, the third extracted features are cross-fused with the initial features and encoded features at different fusion ratios, including the following steps 401 to 403:

[0119] Step 401: Based on the preset first weight value, calculate the third product between the encoded features and the third extracted features.

[0120] Step 402: Based on the preset second weight value, calculate the fourth product between the initial feature and the third extracted feature, wherein the first weight value is greater than the second weight value.

[0121] Step 403: Sum the third product and the fourth product to obtain the cross-fusion feature.

[0122] Steps 401 to 403 are described in detail below.

[0123] In some embodiments, the third extracted feature is denoted as B'E', and the third extracted feature is cross-fused with the initial feature and the encoded feature to obtain the cross-fused feature. It can be understood that the cross-fused feature includes the feature corresponding to the enhanced small target object, so that subsequent recognition processing can be performed based on the enhanced small target object feature.

[0124] Furthermore, to better fuse the initial features and the encoded features, this embodiment pre-sets a first weight value corresponding to the encoded features and a second weight value corresponding to the initial features. Based on the first weight value and the second weight value, the cross-fusion features can be calculated, as shown in the following formula (5):

[0125] E cf =2E·B'E'+B·(1-B'E') (5)

[0126] Among them, E cf The cross-fusion feature is represented by the first weight value of 2, the second weight value of 1, 2E·B'E' as the third product, and B·(1-B'E') as the fourth product.

[0127] Furthermore, since the encoder plays a crucial role in information compression and extraction, dimensionality reduction, and redundancy removal, it is necessary to retain and emphasize the encoded features, combining both the initial and encoded features with the third extracted features. Therefore, the first weight value is set to be greater than the second weight value. Next, based on the second weight value and the third extracted features, while retaining the high-level abstract feature representation extracted by the encoder, the encoded features can effectively learn from the initial features and extract the missing detailed information and / or contextual information, thereby enhancing the image recognition model's detection accuracy for small objects and improving the feature quality of small object features.

[0128] Step 103: Perform dynamic position query on the encoded features to obtain dynamic query features, and obtain the static query features corresponding to the dynamic query features.

[0129] Step 103 will be described in detail below.

[0130] Dynamic Positional Query (DPQ) refers to dynamically searching and processing information representing the position and content of a target object based on the encoded features corresponding to different images to be processed. Dynamic query features are used to represent several features related to the position and content obtained after dynamic positional query. Static Content Query (SCQ) features are pre-set features corresponding to the dynamic query features. Based on dynamic and static query features, the image recognition model can learn the bias information of the bounding box between the model's predicted results and the actual results, thereby making the position and category represented by the final target recognition result closer to the real situation, thus improving the image recognition quality of the image to be processed, especially the recognition quality of small target objects.

[0131] In some embodiments, after extracting the encoded features, on the one hand, the encoded features need to be cross-fused based on step 102 to obtain cross-fused features; on the other hand, the encoded features need to be input into a linear layer to perform feature transformation and mapping, mapping the encoded features to a new feature space. The linear layer, through its spatial feature recombination capability, helps the image recognition model better capture the location information of the target object in the image to be processed, and effectively fuses the captured location information based on a weight matrix preset in the linear layer, outputting multiple dynamic query features representing the location and feature content of the target object. This allows for subsequent decoding of the cross-fused features based on the dynamic query features to obtain decoded features for target object recognition.

[0132] In some embodiments, dynamic location lookup is performed on the encoded features to obtain dynamic query features, including the following steps 501 to 503:

[0133] Step 501: Perform linear processing on the encoded features to obtain multiple corresponding linear processing results, wherein the linear processing results include the corresponding linear processing scores.

[0134] Step 502: Sort the multiple linear processing results according to the order of their linear processing scores.

[0135] Step 503: Based on the preset target quantity, filter the sorted linear processing results to obtain dynamic query features.

[0136] Steps 501 to 503 are described in detail below.

[0137] In some embodiments, after extracting the encoded features, on the one hand, the encoded features need to be cross-fused based on step 102 to obtain cross-fused features; on the other hand, the encoded features need to be input into the linear layer of the image recognition model to perform feature transformation and mapping, mapping the encoded features to a new feature space. The linear layer, through its spatial feature recombination capability, helps the image recognition model better capture category information representing the category to which the target object belongs, as well as the location information of the target object in the image to be processed; and based on the weight matrix preset in the linear layer, it effectively fuses the captured location and category information, outputting multiple linear processing results representing the location and category of the target object.

[0138] Furthermore, two linear layers can be deployed in the image recognition model. One linear layer 'a' processes content related to the location of the encoded features, and the other linear layer 'b' processes content related to the category of the encoded features. For example, after linear processing of the encoded features by linear layer 'b', multiple linear processing results are output, each including its corresponding linear processing score. The multiple linear processing scores are sorted from high to low, and a corresponding number of linear processing results 'b' are selected based on a preset target quantity. Based on the feature pixel indexes of these linear processing results 'b', the corresponding processed linear processing result 'a' is determined in linear layer 'a'. The position vector of the linear processing result 'a' is then positionally encoded, and the resulting feature of a specific dimension is used as a dynamic query feature.

[0139] Furthermore, the static query features are matched with the preset number of targets. After the dynamic query features are obtained by filtering the linear processing results, the image recognition model combines the preset static query features with the dynamic query features so that the combined result can be input into the decoder. The decoder obtains the bias information between the real position result and the predicted position result based on the combined result, and adjusts the predicted position result based on the bias information so that the adjusted and updated predicted position result is closer to the real position result, thereby improving the image recognition accuracy.

[0140] It should be noted that the target number can be specifically set according to the actual situation. For example, if the target number is 900, the top 900 will be selected from the multiple linear processing results sorted from high to low as dynamic query features. This application embodiment does not impose specific limitations.

[0141] Furthermore, when the linear processing result includes linear processing scores corresponding to multiple attributes, the results can be sorted according to the linear processing score corresponding to one of the attributes, or multiple linear processing scores can be selected, weighted, and then sorted. This application does not impose any specific restrictions.

[0142] Step 104: Decode the cross-fusion features based on dynamic query features and static query features to obtain decoded features, and identify the target objects in the image to be processed based on the decoded features to obtain the target recognition result.

[0143] Step 104 is described in detail below.

[0144] In some embodiments, after obtaining dynamic query features and static query features that match the dynamic query features, the dynamic query features, static query features, and cross-fusion features are all input into the decoder of the neural network. The decoder then decodes the cross-fusion features based on the dynamic and static query features to obtain decoded features, and performs identification based on the decoded features to obtain the target recognition result. It is understood that the decoded features include enhanced small target object features. Based on the dynamic and static query information indicating the desired location and category, the compressed cross-fusion features can be effectively restored with emphasis, enabling the identification of target objects in the image to be processed based on the obtained decoded features, thus obtaining the target recognition result. Since the cross-fusion processing module effectively transfers detailed information from the initial features to the encoded features to obtain the cross-fusion features, and further performs emphasized feature extraction on the cross-fusion features based on the dynamic and static query features, the encoded features obtained through encoder processing reduce the ambiguity of small target object features, thereby improving the detection accuracy of small target objects when subsequently using these features for identification.

[0145] In some embodiments, the image recognition model is trained through the following steps 601 to 605:

[0146] Step 601: Obtain the sample image, perform global feature extraction on the target object in the sample image to obtain the initial features of the sample, and input the initial features of the sample into the image recognition model.

[0147] Step 602: Encode the initial features of the sample to obtain the encoded features of the sample, and perform feature enhancement on the initial features of the sample and the encoded features of the sample respectively. Based on the enhanced initial features of the sample and the encoded features of the sample, perform cross-fusion processing to obtain the cross-fused features of the sample.

[0148] Step 603: Perform dynamic location query on the sample coding features to obtain the sample dynamic query features, and obtain the sample static query features corresponding to the sample dynamic query features.

[0149] Step 604: Decode the sample cross-fusion features based on the sample dynamic query features and sample static query features to obtain sample decoded features, and identify the sample target objects in the sample image based on the sample decoded features to obtain the sample target recognition result.

[0150] Step 605: Based on the sample target recognition results, determine the sample loss value, and adjust the parameters of the image recognition model according to the sample loss value to obtain the trained image recognition model.

[0151] Steps 601 to 605 are described in detail below.

[0152] In some embodiments, to better utilize the image recognition model, it is typically necessary to pre-train it. First, sample images required for model training are acquired. The method for acquiring these sample images can be the same as that for acquiring the images to be processed, and will not be elaborated here. After acquiring the sample images, global feature extraction is performed to extract the features corresponding to the target objects in the sample images, obtaining initial sample features. Then, the initial sample features are input into the image recognition model, enabling the model to perform encoding, feature enhancement, cross-fusion, dynamic location lookup, decoding, and image recognition processing. The specific methods of these processing steps are similar to those used by the image recognition model to process the images to be processed, and will not be elaborated here. The difference is that, to facilitate model training, before inputting the sample images into the image recognition model, the corresponding real image recognition results are usually generated first. These real image recognition results include the real labels corresponding to each target object, where the real labels represent the location and category information of the target object. The real labels are compared with the sample target recognition results obtained by the image recognition model through a series of processing steps on the sample images to determine the sample loss value.

[0153] Furthermore, the parameters of the image recognition model are adjusted using the sample loss value. This allows the image recognition model to continuously improve its image recognition quality by making the output sample recognition results as close as possible to the true labels corresponding to the sample images during training. Specifically, the parameters of the image recognition model can include encoder parameters such as the learning rate, linear layer parameters such as the weight matrix, cross-fusion processing model parameters such as activation layer parameters, decoder parameters such as the size and number of hidden layers, and prediction network parameters such as normalization parameters. It should be noted that the parameters adjusted based on the sample loss value can be specifically set according to actual conditions, and this application does not impose specific limitations.

[0154] Furthermore, in order to refine the parameter tuning of the image recognition model and enhance its image recognition capabilities, multiple sample loss values ​​can be calculated at different process nodes of the image recognition model in processing sample images. Based on the corresponding sample loss values, the parameters of different parts of the image recognition model can be adjusted. In this way, the parameters can be adjusted according to the different tasks of each module, thereby improving the overall performance of the model. On the other hand, the parameters can be adjusted according to the different training requirements of each module, thereby accelerating the convergence speed and improving training efficiency.

[0155] In some embodiments, after determining the sample loss value, the method further includes steps 701 to 702:

[0156] Step 701: Connect the image recognition model and the backbone model. The backbone model is used to extract global features of the target objects in the sample image to obtain the initial features of the sample.

[0157] Step 702: Based on the sample loss value, jointly train the image recognition model and the backbone model to obtain the trained image recognition model and backbone model.

[0158] Steps 701 to 702 are described in detail below.

[0159] In some embodiments, the backbone model with a backbone network can be connected to the image recognition model. This allows for joint training of the image recognition model and the backbone model using the obtained sample loss values ​​while training the image recognition model. For example, a first sample loss value is calculated based on the linear result of the linear layer output. The parameters of the backbone model and the encoder of the neural network are adjusted according to this first sample loss value. A second sample loss value is calculated based on the final target recognition result. The parameters of the decoder and prediction network of the neural network are then adjusted according to this second sample loss value. Different sample loss values ​​reflect the degree of fit of the image recognition model in each module. Adjusting different modules of the image recognition model using corresponding sample loss values ​​can prevent overfitting in certain modules, thereby improving the generalization ability of the image recognition model. Furthermore, different sample loss values ​​provide a better understanding of the model's performance in each module, helping to interpret the prediction results of the image recognition model and improving its interpretability and understandability.

[0160] It is understandable that by jointly training the image recognition model and the backbone model, the two can influence each other and learn together during the training process. This helps the image recognition model to complement each other during training, thereby improving the generalization ability and robustness of the image recognition model and achieving higher recognition accuracy and efficiency for input images.

[0161] In some embodiments, before performing global feature extraction on the target object in the sample image, the steps 801 to 805 are further included:

[0162] Step 801: Obtain the first bounding box of the sample image, and determine the height and width of the first bounding box.

[0163] Step 802: Determine the second bounding box based on the sample target object, and determine the height and width of the second bounding box.

[0164] Step 803: Classify the sample target objects confined within the second bounding box to obtain the true category information indicating the category to which the sample target objects belong.

[0165] Step 804: Calculate the first ratio between the height of the second bounding box and the height of the first bounding box, and calculate the second ratio between the width of the second bounding box and the width of the first bounding box.

[0166] Step 805: Determine the true label of the target object in the sample based on the first ratio, the second ratio, and the true category information.

[0167] Steps 801 to 805 are described in detail below.

[0168] In some embodiments, to obtain the true label, a first bounding box corresponding to the sample image is first obtained. The first bounding box is used to encompass all content in the sample image. The first bounding box is typically a rectangle composed of four identical boundary lines, where each boundary line defines the boundary position of the sample image. Then, a second target box corresponding to each target object in the sample image is obtained. The second target box is used to encompass all content in the corresponding target object. For better understanding, as... Figure 5 As shown, Figure 5 This is an optional bounding box diagram of the image recognition method provided in the embodiments of this application. The first bounding box is used to encompass all the contents of the sample image A, including all large target objects and small target objects; the second bounding boxes a, b, and c are all second bounding boxes, which are used to encompass their respective target objects.

[0169] For example, after determining the first bounding box, its corresponding first bounding box height and width are determined; after determining the second bounding box, its corresponding second bounding box height and width are determined. Next, the target objects within each second bounding box are first classified and identified to obtain true category information. For example, the target object within second bounding box a is identified as a vehicle, and the target object within second bounding box b is identified as a butterfly.

[0170] Next, the true label that effectively combines the target object category and size is calculated using the following formula (6):

[0171]

[0172] Among them, cs i For the i-th real label, h i w i Let H and W represent the height and width of the i-th second bounding box, respectively, where H is the height of the first bounding box and W is the width of the first bounding box. The first ratio, The second ratio, y i This corresponds to the actual category information.

[0173] Understandably, in related technologies, the model performs position prediction and classification prediction for images separately, that is, it ignores the correlation between position prediction and classification prediction. By setting a true label that combines position and category content, it is possible to effectively enhance the image recognition model's attention to small target objects in the newly defined loss function, thereby improving the recognition accuracy of small target objects.

[0174] In some embodiments, determining the sample loss value based on the sample target recognition result includes the following steps 901 to 906:

[0175] Step 901: Determine the third bounding box corresponding to the sample location result, and determine the height and width of the third bounding box.

[0176] Step 902: Calculate the third ratio between the height of the third bounding box and the height of the first bounding box, and calculate the fourth ratio between the width of the third bounding box and the width of the first bounding box.

[0177] Step 903: Based on the third ratio, the fourth ratio, and the sample category results, determine the predicted label of the target object.

[0178] Step 904: Perform a first logarithmic process on the predicted label to obtain the first logarithmic result, and calculate the first sub-value based on the first logarithmic result and the true label.

[0179] Step 905: Perform a second logarithmic processing on the sample category confidence score to obtain the second logarithmic result, and calculate the second sub-value based on the second logarithmic result and the sample category information.

[0180] Step 906: Based on the first sub-value and the second sub-value, obtain the sample loss value.

[0181] Steps 901 to 906 are described in detail below.

[0182] In some embodiments, the sample target recognition result includes sample location result and sample category result. The sample location result is used to characterize the location of the sample target object in the sample image. The sample category result also includes the corresponding sample category confidence score. The sample category information is used to indicate the category to which the sample target object belongs, and the sample category confidence score is used to characterize the degree of confidence of the sample category result.

[0183] In some embodiments, a custom blending category and object size prediction loss function (Boost Loss) is used to calculate the sample loss value. Specifically, firstly, based on the sample target recognition results, a predicted label is obtained using a method similar to steps 801 to 805. Then, the sample loss value is calculated using the following formula (7):

[0184]

[0185] Among them, L Boost For BoostLoss, α and γ are pre-defined weight hyperparameters, N represents the number of target objects, and cs i For the i-th real label, For the i-th predicted label, p i Let be the confidence score of the sample category corresponding to the i-th predicted label, and log be the logarithm.

[0186] in, For the first logarithmic result, log(1-p) i ) represents the second logarithmic result. As the first subvalue, (1-α)p i γ (1-y i log(1-p) i ) is the second sub-value.

[0187] Understandably, logarithmic processing can increase the sample loss value corresponding to small objects, thereby encouraging image recognition models to pay more attention to small objects, increasing the attention of image recognition models to small objects, and thus improving the recognition quality of small objects.

[0188] Furthermore, to better understand the beneficial effects of the image recognition method provided in this application, a comparison of the image recognition results of this application and related technologies is presented below. Both the image recognition model used in this application and the models in related technologies are pre-trained using the ImageNet-1K dataset. During pre-training, the backbone is used as part of the image recognition model. The detection datasets used during training were the COCO2017 dataset and the WiderPerson dataset. The COCO2017 dataset contains 118,000 training sample images and 5,000 corresponding validation sample images. The evaluation metric for the detection results of the COCO2017 dataset is the average precision (AP) under different intersection-over-union (IoU) thresholds. The WiderPerson dataset is a diverse and complex dense pedestrian detection dataset containing 13,382 images, each image containing an average of 29.87 bounding box annotations. The WiderPerson dataset contains 8,000 sample images and 1,000 validation sample images. The evaluation metrics for the detection results of the WiderPerson dataset are AP, recall, and mean minimum recall (mMR).

[0189] As shown in Table 1, Table 1 presents the target recognition results for the COCO2017 dataset. It can be seen that, compared to other methods, the image recognition method proposed in this application, after training under the same conditions, achieves a significantly higher detection accuracy (AP) for small targets. SThe accuracy improved by 4.4%, the AP index improved by 3.6%, and other indicators also improved significantly, verifying the effectiveness of the embodiments of this application. It has great significance and value for the application of small target object detection.

[0190] Table 1

[0191]

[0192] Among them, Faster-RCNN is an object detection model that uses a Region Proposal Network (RPN) to generate candidate regions, and then uses Faster-RCNN to classify and regress these candidate regions; DETR (Detection Transformer) is an end-to-end object detection model that uses a neural network to encode images, and then uses a decoder to detect and classify objects; DETR-DC5 is an improved DETR model; DeformableDETR is a deformable DETR model that adds deformable convolutions to DETR to handle object deformation; H-Deformable DETR is an improved version of the Deformable DETR model; Conditional DETR is a conditional DETR model that adds conditional information to DETR to handle image recognition under different conditions; Efficient... DETR is an efficient DETR model that uses MobileNetV3 as its backbone network for image recognition. MobileNetV3 is a lightweight convolutional neural network architecture developed by Google, designed for image classification and object detection tasks in resource-constrained environments such as mobile devices. ConditionalDETR is an improved object detection model that, based on the DETR model architecture, introduces a conditional encoder. Encoder) is used for image recognition; SparseDETR is a sparse DETR model that uses a sparse attention mechanism to handle large-scale object detection; DAB-DETR is a dual-attention DETR model that uses a dual-attention mechanism to handle relationships between multiple target objects; DN-DETR is an object detection model that introduces a deformable attention mechanism; CF-DETR is an object detection model that introduces context fusion technology; DiffusionDet is a diffusion model that uses object detection methods, where the diffusion model is a generative model used to model pixel-level image distribution; Focus-DETR is a neural network-based model that introduces a mechanism called "Focus"; DINO (DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection) refers to an end-to-end object detection model based on DETR, used for image classification and other computer vision tasks; DINO* model and Stable-DINO model are both improved versions of the DINO model; ours refers to the image recognition model using the image recognition method proposed in the embodiments of this application.

[0193] Where Year represents the training time of the corresponding model, Backbone refers to the backbone network used, and AP 50 This means only considering prediction boxes with an IoU greater than 50%, AP 75 This means only considering prediction boxes with an IoU greater than 75%, AP S AP refers to the average accuracy of small-sized target objects. M This refers to the average accuracy of a medium-sized target object, while AP... L The average accuracy of large-sized target objects is represented by: Params, which is the number of parameters used to measure the complexity of the model. The more parameters, the more complex the model is usually, and the more computing resources may be needed for training and inference. GFLOPs (Giga Floating Point Operations Per Second) is the number of floating-point operations per second, which is an indicator of the computational complexity and inference speed of the model. The higher the GFLOPs, the greater the computational load and complexity of the model, and the more computing resources may be needed for inference.

[0194] As shown in Table 2, the target recognition results for the WiderPerson dataset are presented. It can be seen that the method of this invention achieves the best detection accuracy across all metrics compared to other methods, verifying the effectiveness of this invention and demonstrating significant meaning and value for small target detection applications and tasks.

[0195] Table 2

[0196]

[0197] Among them, PS-RCNN is a point-set-based object detection model; IterDet-1-iter and IterDet-2-iter are both iterative object detection models, where the numbers "1" and "2" represent the number of iterations; Heetal is a multi-scale feature balancing enhancement network used to address the loss of low-level feature information and imbalance of feature map levels when using Feature Pyramid Network (FPN) for multi-scale detection; Cascade refers to a cascaded object detection model based on a neural network architecture; DINO-ResNet50 is an object detection model that uses the ResNet50 residual network as the backbone network in the DINO model. Furthermore, higher AP and Recall values ​​indicate better image recognition performance, while lower mMR values ​​also indicate better image recognition performance.

[0198] like Figure 6 As shown, Figure 6This is a schematic diagram of an optional functional module of the image recognition device provided in this application embodiment, wherein the image recognition device may include:

[0199] The input module 1001 is used to acquire the image to be processed, perform global feature extraction on the target object in the image to be processed, obtain initial features, and input the initial features into a pre-trained image recognition model.

[0200] The cross-fusion processing module 1002 is used to encode the initial features to obtain coded features, and to perform feature enhancement on the initial features and coded features respectively. Based on the enhanced initial features and coded features, cross-fusion processing is performed to obtain cross-fused features.

[0201] The query module 1003 is used to perform dynamic position query on the encoded features, obtain dynamic query features, and acquire the static query features corresponding to the dynamic query features.

[0202] The target recognition result module 1004 is used to decode the cross-fusion features based on dynamic query features and static query features to obtain decoded features, and to identify the target objects in the image to be processed based on the decoded features to obtain the target recognition result.

[0203] The image recognition method, apparatus, electronic device, and storage medium proposed in this application acquire an image to be processed, perform global feature extraction on target objects in the image to obtain initial features, and input the initial features into a pre-trained image recognition model. Next, the initial features are encoded to obtain encoded features, and feature enhancement is performed on both the initial and encoded features. Based on the enhanced initial and encoded features, cross-fusion processing is performed to obtain cross-fused features, which include enhanced small target object features, making the originally blurry small target object features in the initial features clearer, thus providing a reliable feature foundation for subsequent image recognition. Dynamic location lookup is performed on the encoded features to obtain dynamic query features, and corresponding static query features are obtained. The image recognition model can learn the relationship between the model's predicted results and the actual results based on the dynamic and static query features. The bias information of the bounding box makes the position and category represented by the final target recognition result closer to the real situation, thereby improving the image recognition quality of the image to be processed, especially the recognition quality of small target objects. Finally, the cross-fusion features are decoded based on dynamic query features and static query features to obtain decoded features, and the target objects in the image to be processed are identified based on the decoded features to obtain the target recognition result. Since the cross-fusion processing features reduce the ambiguity of small target object features, and further, based on dynamic query features and static query features, the cross-fusion features are extracted with emphasis. Therefore, the small target object features contained in the encoded features obtained by the encoder are richer, more comprehensive and clearer, that is, the features of the corresponding small target objects represented are more accurate. As a result, the target recognition result obtained by the recognition processing based on the small target object features is of higher quality, improving the detection accuracy of small target objects.

[0204] The specific implementation of this image recognition device is basically the same as the specific implementation of the image recognition method described above, and will not be repeated here.

[0205] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image recognition method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0206] like Figure 7 As shown, Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0207] The processor 1101 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0208] The memory 1102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and called and executed by the processor 1101 using the image recognition method of the embodiments of this application.

[0209] Input / output interface 1103 is used to implement information input and output;

[0210] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0211] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0212] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0213] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image recognition method.

[0214] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0215] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0216] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0217] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0219] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0220] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0222] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0224] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image recognition method, characterized in that, The method includes: The image to be processed is acquired, and global feature extraction is performed on the target object in the image to be processed to obtain initial features. The initial features are then input into a pre-trained image recognition model. The initial feature is encoded to obtain the encoded feature. The result of the first fully connected processing and the first normalization processing of the initial feature is activated, and the activated result is used to extract features to obtain the first extracted feature corresponding to the initial feature. Calculate the first difference between the preset unit value and the first extracted feature, determine the first product based on the first difference and the encoded feature, and sum the initial feature with the first product to obtain the feature-enhanced initial feature; The result of the second fully connected processing and the second normalization processing on the encoded features is activated, and the activated result is used to extract features to obtain the second extracted features corresponding to the encoded features; Calculate the second difference between the preset unit value and the second extracted feature, determine the second product based on the second difference and the initial feature, and sum the encoded feature with the second product to obtain the enhanced encoded feature; The initial feature and the encoded feature after feature enhancement are spliced ​​together to obtain the connection feature, and the third extracted feature is extracted from the connection feature; The third extracted feature is cross-fused with the initial feature and the encoded feature at different fusion ratios to obtain cross-fused features; The encoded features are linearly processed to obtain multiple corresponding linear processing results, wherein the linear processing results include corresponding linear processing scores; Sort the multiple linear processing results according to the order of their linear processing scores; Based on the preset target number, the sorted linear processing results are filtered to obtain dynamic query features, and the static query features corresponding to the dynamic query features are obtained. The cross-fusion features are decoded based on the dynamic query features and the static query features to obtain decoded features, and the target object in the image to be processed is identified based on the decoded features to obtain the target identification result.

2. The method according to claim 1, characterized in that, The process of cross-fusioning the third extracted feature with the initial feature and the encoded feature at different fusion ratios includes: Based on a preset first weight value, calculate the third product between the encoded feature and the third extracted feature; Calculate the third difference between the preset unit value and the third extracted feature, and calculate the fourth product between the initial feature and the third difference based on the preset second weight value, wherein the first weight value is greater than the second weight value; The third product and the fourth product are summed to obtain the cross-fusion feature.

3. The method according to claim 1, characterized in that, The image recognition model is trained through the following steps: Acquire a sample image, perform global feature extraction on the target object in the sample image to obtain initial sample features, and input the initial sample features into the image recognition model; The initial features of the sample are encoded to obtain sample encoded features. Feature enhancement is performed on the initial features of the sample and the sample encoded features respectively. Based on the enhanced initial features of the sample and the sample encoded features, cross-fusion processing is performed to obtain sample cross-fusion features. Dynamic location query is performed on the sample coding features to obtain dynamic query features of the sample, and static query features of the sample corresponding to the dynamic query features of the sample are obtained. Based on the sample dynamic query features and the sample static query features, the sample cross-fusion features are decoded to obtain sample decoding features. Based on the sample decoding features, the sample target objects in the sample image are identified to obtain sample target identification results. Based on the target recognition results of the samples, the sample loss value is determined, and the parameters of the image recognition model are adjusted according to the sample loss value to obtain the trained image recognition model.

4. The method according to claim 3, characterized in that, After determining the sample loss value, the following is also included: The image recognition model and the backbone model are connected, wherein the backbone model is used to perform global feature extraction on the target object in the sample image to obtain the initial features of the sample; Based on the sample loss value, the image recognition model and the backbone model are jointly trained to obtain the trained image recognition model and the backbone model.

5. The method according to claim 4, characterized in that, Before performing global feature extraction on the target objects in the sample image, the method further includes: Obtain the first bounding box of the sample image, and determine the height and width of the first bounding box corresponding to the first bounding box; Based on the sample target object, determine the second bounding box, and determine the second bounding box height and the second bounding box width corresponding to the second bounding box; The sample target object confined within the second bounding box is classified to obtain true category information indicating the category to which the sample target object belongs; Calculate a first ratio between the height of the second bounding box and the height of the first bounding box, and calculate a second ratio between the width of the second bounding box and the width of the first bounding box; Based on the first ratio, the second ratio, and the true category information, the true label of the sample target object is determined.

6. The method according to claim 5, characterized in that, The sample target identification results include sample location results and sample category results, and the sample category results include sample category confidence. The step of determining the sample loss value based on the sample target recognition result includes: Determine the third bounding box corresponding to the sample location result, and determine the height and width of the third bounding box. Calculate a third ratio between the height of the third bounding box and the height of the first bounding box, and calculate a fourth ratio between the width of the third bounding box and the width of the first bounding box; Based on the third ratio, the fourth ratio, and the sample category result, the predicted label of the target object in the sample is determined; The predicted label is subjected to a first logarithmic process to obtain a first logarithmic result, and a first sub-value is calculated based on the first logarithmic result and the true label. The confidence score of the sample category is subjected to a second logarithmic process to obtain a second logarithmic result. A second sub-value is calculated based on the second logarithmic result and the sample category information. Based on the first sub-value and the second sub-value, the sample loss value is obtained.

7. An image recognition device, characterized in that, The device includes: The input module is used to acquire the image to be processed, perform global feature extraction on the target object in the image to be processed to obtain initial features, and input the initial features into a pre-trained image recognition model; The cross-fusion processing module is used to encode the initial feature to obtain the encoded feature, activate the result of the first fully connected processing and the first normalization processing of the initial feature, and extract features from the activated result to obtain the first extracted feature corresponding to the initial feature. Calculate the first difference between the preset unit value and the first extracted feature, determine the first product based on the first difference and the encoded feature, and sum the initial feature with the first product to obtain the feature-enhanced initial feature; The result of the second fully connected processing and the second normalization processing on the encoded features is activated, and the activated result is used to extract features to obtain the second extracted features corresponding to the encoded features; Calculate the second difference between the preset unit value and the second extracted feature, determine the second product based on the second difference and the initial feature, and sum the encoded feature with the second product to obtain the enhanced encoded feature; The initial feature and the encoded feature after feature enhancement are spliced ​​together to obtain the connection feature, and the third extracted feature is extracted from the connection feature; The third extracted feature is cross-fused with the initial feature and the encoded feature at different fusion ratios to obtain cross-fused features; The query module is used to perform linear processing on the encoded features to obtain multiple corresponding linear processing results, wherein the linear processing results include corresponding linear processing scores; Sort the multiple linear processing results according to the order of their linear processing scores; Based on the preset target number, the sorted linear processing results are filtered to obtain dynamic query features, and the static query features corresponding to the dynamic query features are obtained. The target recognition result module is used to decode the cross-fusion features based on the dynamic query features and the static query features to obtain decoded features, and to identify the target object in the image to be processed based on the decoded features to obtain the target recognition result.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image recognition method according to any one of claims 1 to 6.

Images