Image detection method, device, electronic device and storage medium

Abstract

The present invention discloses an image detection method, device, electronic device, and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: determining a first vector sequence based on an image block sequence corresponding to an image to be processed; sequentially performing attention encoding on the first vector sequence based on multiple attention encoding layers to obtain a first feature map; obtaining a first attention distribution map based on the target attention features corresponding to the classification embedding vector; inverting the first attention distribution map and fusing it with the first feature map to obtain a second feature map; and performing target object detection based on the first feature map, the first attention distribution map, and the second feature map to obtain category information and location information of the target object in the image to be processed. The present invention improves the accuracy of the location information in the detection results.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image detection method, device, electronic equipment and storage medium. Background Art

[0002] Object detection can provide the location and category information of the objects of interest in the image, and is a basic prerequisite task for subsequent computer vision tasks.

[0003] Related technologies typically use convolutional neural networks (CNNs) to detect objects in images. This involves first using a CNN to extract a deep semantic feature map from the image. This deep semantic feature map is then globally pooled, using a fully connected layer to integrate local information with class distinctions. This information is then fed into a classification layer for classification, and the classification results are combined to determine the positioning box and obtain location information. Because CNNs only extract local features, related technologies focus solely on the most discriminative features of the target, resulting in poor positioning accuracy and inaccurate location information in the detection results. Summary of the Invention

[0004] In order to solve the problems of the prior art, the embodiments of the present invention provide an image detection method, device, electronic device and storage medium. The technical solution is as follows:

[0005] In one aspect, an image detection method is provided, the method comprising:

[0006] Determining a first vector sequence according to an image block sequence corresponding to the image to be processed; the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence;

[0007] Attention encoding is performed on the first vector sequence in sequence based on multiple attention encoding layers to obtain a first feature map;

[0008] Determine an attention matrix corresponding to each of the attention encoding layers, extract a target attention feature corresponding to the classification embedding vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map;

[0009] Inverting the first attention distribution map to obtain a reverse attention distribution map, and fusing the reverse attention distribution map with the first feature map to obtain a second feature map;

[0010] Target object detection is performed based on the first feature map, the first attention distribution map, and the second feature map to obtain category information and position information of the target object in the image to be processed.

[0011] In another aspect, an image detection device is provided, comprising:

[0012] A first determining module is configured to determine a first vector sequence based on an image block sequence corresponding to an image to be processed; the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence;

[0013] an attention encoding module, configured to sequentially perform attention encoding on the first vector sequence based on multiple attention encoding layers to obtain a first feature map;

[0014] A first attention distribution determination module is configured to determine an attention matrix corresponding to each of the attention encoding layers, extract target attention features corresponding to the classification embedding vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map;

[0015] a negation processing module, configured to perform negation processing on the first attention distribution map to obtain a reverse attention distribution map, and fuse the reverse attention distribution map with the first feature map to obtain a second feature map;

[0016] A target object detection module is used to detect the target object based on the first feature map, the first attention distribution map and the second feature map to obtain the category information and position information of the target object in the image to be processed.

[0017] In an exemplary embodiment, the target object detection module includes:

[0018] a second attention distribution determining module, configured to determine category information of a target object in the image to be processed and generate a second attention distribution map based on the second feature map and the first feature map;

[0019] a target attention distribution determination module, configured to multiply the first attention distribution map by the second attention distribution map pixel by pixel to obtain a target attention distribution map;

[0020] A position information determination module is used to determine the position information of the target object in the image to be processed based on the target attention distribution map.

[0021] In an exemplary embodiment, the second attention distribution determination module includes:

[0022] a first classification module, configured to perform classification based on the second feature map to obtain a first classification result; wherein the first classification result includes a probability value corresponding to each of a plurality of preset categories;

[0023] a re-parameter module, configured to perform weighted summation on a plurality of initialized convolution kernels according to the first classification result to obtain a target convolution kernel;

[0024] a feature extraction module, configured to perform feature extraction on the first feature map according to the target convolution kernel to obtain a third feature map;

[0025] The second determination module is used to determine the category information of the target object in the image to be processed based on the third feature map and generate a second attention distribution map.

[0026] In an exemplary embodiment, the first classification module includes:

[0027] A dimensionality reduction module, configured to perform dimensionality reduction processing on the second feature map to obtain a reduced-dimensionality feature map;

[0028] A first pooling module is used to perform global maximum pooling and global average pooling on the dimensionality reduction feature map to obtain a first pooling feature and a second pooling feature;

[0029] A pooling feature fusion module, configured to fuse the first pooling feature and the second pooling feature to obtain a fused pooling feature;

[0030] The classification submodule is used to classify the fused pooled features to obtain a first classification result.

[0031] In an exemplary embodiment, the second determining module includes:

[0032] A second pooling module is used to perform global average pooling on the third feature map to obtain a third pooling feature;

[0033] A second classification module is configured to perform classification based on the third pooled feature to obtain a second classification result; the second classification result represents category information of the target object in the image to be processed;

[0034] An attention map generation module is used to generate a second attention distribution map based on the third pooled features and the third feature map.

[0035] In an exemplary embodiment, when the inversion processing module fuses the inverse attention distribution map with the first feature map to obtain the second feature map, it is specifically used to multiply the inverse attention distribution map with the first feature map to obtain the second feature map.

[0036] In an exemplary embodiment, the first attention distribution determination module includes:

[0037] A first averaging module is used to determine the average of the attention matrices of the respective attention mechanism modules for each of the attention encoding layers to obtain the attention matrix corresponding to the attention encoding layer;

[0038] An extraction module, configured to extract a target attention feature corresponding to the classification embedding vector from the attention matrix corresponding to the attention encoding layer;

[0039] The second averaging module averages the target attention features corresponding to each of the attention encoding layers to obtain a first attention distribution map.

[0040] In an exemplary embodiment, the first determining module includes:

[0041] An image segmentation module is used to obtain an image to be processed, and segment the image to be processed into a plurality of image blocks to obtain an image block sequence;

[0042] an embedding module, configured to perform vector embedding on image blocks in the image block sequence to obtain an image block embedding vector sequence;

[0043] A first adding module is configured to add an initialized classification embedding vector to the image block embedding vector sequence to obtain an embedding vector sequence;

[0044] The second adding module is used to add a position code to each embedded vector in the embedded vector sequence to obtain the first vector sequence; the position code represents the position information of the corresponding embedded vector in the embedded vector sequence.

[0045] In an exemplary embodiment, the above-mentioned image detection method is implemented based on an image detection model, and the apparatus further includes a training module, wherein the training module includes:

[0046] A sample acquisition module is used to acquire a sample image and a category label corresponding to the sample image; the category label indicates reference category information of a target object in the sample image;

[0047] A first sample feature map determining module is configured to input a sample image block sequence corresponding to the sample image into an attention coding unit of a preset neural network model, and perform attention coding based on multiple attention coding layers in the attention coding unit to obtain a first sample feature map;

[0048] A category prediction module is configured to determine the attention matrix of each of the attention encoding layers based on the local feature mining unit of the preset neural network model, extract the target attention features corresponding to the classification embedding vector in each of the attention matrices, fuse the target attention features to obtain a first sample attention distribution map, invert the first sample attention distribution map to obtain a sample reverse attention distribution map, fuse the sample reverse attention distribution map with the first sample feature map to obtain a second sample feature map; and perform category prediction based on the second sample feature map and the first sample feature map to obtain predicted category information;

[0049] A training submodule is used to determine a loss value based on the difference between the predicted category information corresponding to the sample image and the category label, reversely adjust the model parameters of the preset neural network model based on the loss value, and perform iterative training until the training end condition is met to obtain the image detection model.

[0050] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the above-mentioned image detection method.

[0051] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the image detection method as described above.

[0052] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the above-described image detection method.

[0053] The embodiment of the present invention determines a first vector sequence based on an image block sequence corresponding to the image to be processed, and sequentially performs attention encoding on the first vector sequence based on multiple attention coding layers to obtain a first feature map, thereby performing sufficient correlation analysis on the local areas in the image based on the global attention mechanism to obtain effective global features, and then extracts the target attention features of the corresponding classification character vectors in the attention matrix corresponding to each attention coding layer, fuses each target attention feature to obtain a first attention distribution map, and inverts the first attention distribution to obtain a reverse attention distribution map, and then fuses the reverse attention distribution map with the first feature map to obtain a second feature map, and performs target object detection based on the first feature map, the first attention distribution map and the second feature map. Since the second feature map is obtained based on the fusion of the reverse attention distribution map and the first feature map, it can activate the weak response area in the first feature map that is helpful for classification, thereby greatly improving the positioning accuracy of the target detection during the above-mentioned target object detection, and improving the accuracy of the position information in the detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1a This is an example of a thermal activation map generated when performing object detection based on a convolutional neural network;

[0056] Figure 1b This is an example of a thermodynamic activation diagram obtained using the technical solution of the present invention;

[0057] Figure 2 is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0058] Figure 3 2 is a schematic diagram of the detection results of the image detection method provided by an embodiment of the present invention applied to industrial AI quality inspection;

[0059] Figure 4 1 is a flow chart of an image detection method provided by an embodiment of the present invention;

[0060] Figure 5 is a flow chart of another image detection method provided by an embodiment of the present invention;

[0061] Figure 6 This is a structural block diagram of an image detection device provided by an embodiment of the present invention;

[0062] Figure 7 This is a hardware structure block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] The target detection in the related art uses a convolutional neural network as the backbone network to extract the deep semantic feature map of the image. It then uses global pooling to integrate local information with category discrimination and feeds it to the classification layer for classification. The CAM (Class Activation Mapping) method extracts the areas where the classification layer focuses its attention and forms a thermal activation map. The connected region analysis method is used on this thermal activation map to determine the positioning box and obtain the position information. Since convolution is an operator with the ability to extract local features, the convolutional neural network can only extract the most discriminative features of each category, so that the network in the related art only focuses on some highly discriminative areas of the target object in the image. In other words, only some areas of the thermal activation map show high response.

[0066] like Figure 1a The figure shows an example of a thermal activation map generated when performing target detection based on a convolutional neural network. The target object in the image to be processed is a bird. Since the head area of ​​a bird is more discriminative than other parts and is the key area to distinguish birds from other categories, when performing target detection based on a convolutional neural network, the network only focuses on the head area of ​​the bird, that is, Figure 1aIn the thermal activation map shown in , only the head area shows a high response, which makes the positioning frame determined based on the thermal activation map incomplete, the positioning accuracy is poor, and the position information in the detection result is not accurate enough.

[0067] Based on this, an embodiment of the present invention provides an image detection method, which extracts the global features of the image to be processed based on multiple attention coding layers and further combines the mining of local features to obtain a fully activated thermal activation map, such as Figure 1b The figure shows an example of the thermodynamic activation diagram obtained by using the technical solution of the present invention. It can be seen that compared with Figure 1a Thermodynamic activation diagram of the embodiment of the present invention Figure 1a The areas in the weak response area that are helpful for classification are activated, thereby improving the integrity of the positioning frame, improving the positioning accuracy, and ensuring the accuracy of the position information in the detection results.

[0068] See also Figure 2 , which is a schematic diagram of an implementation environment provided by an embodiment of the present invention, the implementation environment may include a terminal 210 and a server 220, wherein the terminal 210 and the server 220 can communicate through a wired connection or a wireless connection.

[0069] The terminal 210 includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The terminal 210 runs client software with an image detection function, such as an application (Application, referred to as App for short). The application can be an independent application or a subroutine within the application.

[0070] The server 220 may be a server that provides background services for the application in the terminal 210, and specifically may be a server that performs target positioning detection on an image. For example, the server 220 may store a pre-trained image detection model, and the terminal 210 may call the image detection model in the server 220 when performing image detection on the image to be processed to perform image detection using the image detection method of an embodiment of the present invention, and obtain the detection result returned by the server 220, which includes the location information and category information of the target object. It is understandable that the terminal 210 may also download the trained image detection model from the server 220 and store it locally, and directly call the local image detection model when performing image detection on the image to be processed, thereby improving the efficiency of image detection.

[0071] It should be noted that server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] In one exemplary embodiment, terminal 210 and server 220 may both be node devices in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thereby enabling information sharing among multiple node devices. Multiple node devices in a blockchain system may be configured with the same blockchain, which is composed of multiple blocks, with adjacent blocks having an associated relationship. This allows any tampering of data in any block to be detected by the next block, thereby preventing tampering of the blockchain data and ensuring the security and reliability of the blockchain data.

[0073] The image processing method of the embodiment of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0074] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0075] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, which uses cameras and computers to replace the human eye to identify and measure targets, and then further processes images to make them more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0076] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0077] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0078] Taking the application of industrial AI quality inspection scenarios as an example, the images taken by the camera for industrial manufacturing components can be input into the image detection model as the image to be processed, and the image detection method based on the embodiment of the present invention can be used to perform defect detection and output the location information (detection frame) and defect type of the defect in the image. Figure 3 The figure shows a schematic diagram of the detection results of the image detection method according to an embodiment of the present invention applied to industrial AI quality inspection, where the defect type is a dirt type and the box is a positioning box of the defect.

[0079] See also Figure 4 , which is a flow chart of an image detection method provided by an embodiment of the present invention, which can be applied to Figure 2 The electronic device in the embodiment may be a terminal or a server. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but may include more or fewer operation steps based on conventional or non-creative work. The order of steps listed in the embodiment is only one way of executing the steps among many steps, and does not represent the only execution order. When the actual system or product is executed, it can be executed sequentially or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 4 As shown, the method may include:

[0080] S401 : Determine a first vector sequence according to an image block sequence corresponding to an image to be processed.

[0081] The first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence.

[0082] Exemplarily, the above step S401 may include:

[0083] Acquire an image to be processed, and divide the image to be processed into a plurality of image blocks to obtain an image block sequence;

[0084] Performing vector embedding on image blocks in the image block sequence to obtain an image block embedding vector sequence;

[0085] Adding the initialized classification embedding vector to the image block embedding vector sequence to obtain an embedding vector sequence;

[0086] A position code is added to each embedding vector in the embedding vector sequence to obtain the first vector sequence; the position code represents position information of the corresponding embedding vector in the embedding vector sequence.

[0087] Specifically, the image to be processed can be segmented according to a preset resolution, so that the image to be processed can be divided into image blocks of the same size, and then each image block is expanded to form an image block sequence. The vector embedding method can use any embedding tool in the prior art, such as Word2Vec for vector embedding of image blocks. The initialization of the classification embedding vector (usually expressed as [Class] token) can be random, and the output corresponding to the classification embedding vector can be used to implement image classification. Usually, the classification embedding vector is located at the head of the sequence, that is, at position 0. Assuming that the length of the image block sequence is N, that is, there are N image blocks, then the length of the embedding vector sequence is N+1.

[0088] Position encoding, or position embedding, is used to encode the relative position information of each embedded vector. In practical applications, each embedded vector can be added to its corresponding position encoding to add its position information in the sequence to each embedded vector, thereby obtaining a first vector sequence.

[0089] It can be understood that the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block, wherein the classification embedding vector is a vector obtained by adding corresponding position coding to the initialized classification embedding vector, and the image block vector is a vector obtained by adding corresponding position coding to the image block embedding vector.

[0090] S403: Perform attention encoding on the first vector sequence in sequence based on multiple attention encoding layers to obtain a first feature map.

[0091] It should be noted that multiple attention encoding layers are in a cascade relationship, that is, only the input of the first-level attention encoding layer is the first vector sequence, and the input of subsequent attention encoding layers are all the output of the previous-level attention encoding layer. The output of the last-level attention encoding layer can be converted (Reshaped) to obtain the first feature map.

[0092] In practical applications, when the first vector sequence is input into the first-level attention encoding layer, it is usually input in the form of a matrix. For example, if the length of the first vector sequence is N+1 (where N is the number of image blocks and 1 represents an additional classification embedding vector), then the matrix input into the first-level attention encoding layer is (N+1)x D, where D represents the dimension of each vector in the first vector sequence.

[0093] S405, determine the attention matrix corresponding to each of the attention encoding layers, extract the target attention features corresponding to the classification character vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map.

[0094] In the embodiment of the present invention, each attention encoding layer is encoded based on the self-attention mechanism. Each attention encoding layer may include multiple self-attention mechanism modules. Assume that the output of the lth attention encoding layer is X l ∈R (N+1)×D , then the attention matrix A corresponding to the self-attention mechanism module l ∈R S×(N+1)×(N+1) It can be expressed as:

[0095]

[0096] Among them, Q l , K lThey represent the query matrix Queries and key matrix Keys obtained by linear mapping the output of the previous level attention encoding layer, S represents the number of attention heads in the attention encoding layer, that is, the number of self-attention mechanism modules in the attention encoding layer, T is the transpose operator, and Softmax() is the normalization function, which is normalized to between 0 and 1 and the sum is 1.

[0097] For each attention encoding layer, the attention matrix A′ corresponding to each attention encoding layer can be obtained by fusing the attention matrices of its S self-attention mechanism modules l ∈R (N+1)×(N+1) , where the fusion method can be to take the average. Then from each A′ l ∈R (N+1)×(N+1) Extract the target attention feature of the corresponding classification embedding vector, i.e. [Class] token Thus, the target attention features corresponding to each attention encoding layer are obtained Should Characterizes the influence of each image block on image classification.

[0098] In the embodiment of the present invention, each of the multiple attention encoding layers can model attention at a different semantic level, so the target attention features corresponding to the multiple attention encoding layers are fused to obtain a first attention distribution map. The fusion processing can be to average the target attention features corresponding to each attention encoding layer.

[0099] Understandably, the first attention distribution map It characterizes the influence of each image patch on the image classification, that is, it reflects the degree of correlation between the classification embedding vector and all image patches, focusing on global correlation. In practical applications, the first attention distribution map can be represented as a thermal activation map.

[0100] Based on this, in an exemplary embodiment, the above step S405 may include:

[0101] For each of the attention encoding layers, determining the average of the attention matrices of the respective attention mechanism modules to obtain the attention matrix corresponding to the attention encoding layer;

[0102] Extracting the target attention feature corresponding to the classification embedding vector in the attention matrix corresponding to the attention encoding layer;

[0103] The target attention features corresponding to each of the attention encoding layers are averaged to obtain a first attention distribution map.

[0104] In the above embodiment, the first attention distribution map is obtained by averaging the target attention features corresponding to different semantic levels, thereby improving the accuracy of the attention distribution in the first attention distribution map.

[0105] S407: Invert the first attention distribution map to obtain a reverse attention distribution map, and fuse the reverse attention distribution map with the first feature map to obtain a second feature map.

[0106] S409: Detect the target object based on the first feature map, the first attention distribution map, and the second feature map to obtain category information and position information of the target object in the image to be processed.

[0107] It should be noted that, through the first attention distribution map Binarization and finding the maximum connected domain can obtain the rough outline of the target object. Further, by taking the maximum circumscribed rectangle of the outline, the positioning frame of the target object can be obtained. However, due to It focuses on global correlation and ignores the correlation between local regions, that is, the correlation between image blocks. Therefore, the first attention distribution map will have weak local responses. Directly determining the positioning box of the target object based on the first attention distribution map is not conducive to maximizing the positioning accuracy.

[0108] Based on this, after obtaining the first attention distribution map, the embodiment of the present invention performs the following steps on the first attention distribution map: Perform the inversion process to obtain the reverse attention distribution map, which can highlight the first attention distribution map The weak response areas in the image are mainly composed of two parts: one is the complete background area, which has no contribution to image classification; the other is the first attention distribution map. The foreground area that is not paid attention to in the image is helpful for classification, but due to the inherent defects of the target attention features corresponding to the classification embedding vector, this part of the area is not responded to. Normalize to between 0 and 1, and then perform the inversion operation.

[0109] To this end, an embodiment of the present invention fuses the reverse attention distribution map with the first feature map to obtain a second feature map. The fusion process can multiply the reverse attention distribution map with the first feature map, and then combine the first feature map, the first attention distribution map and the second feature map to perform target object detection to obtain the category information and position information of the target object in the image to be processed. Since the second feature map is obtained based on the fusion of the reverse attention distribution map and the first feature map, it can activate the weak response area in the first feature map that is helpful for classification, thereby improving the positioning accuracy of target detection and improving the accuracy of the detection results.

[0110] In an exemplary embodiment, performing target object detection based on a first feature map, a first attention distribution map, and a second feature map may include: determining the category information of the target object in the image to be processed and generating a second attention distribution map based on the second feature map and the first feature map; multiplying the first attention distribution map and the second attention distribution map pixel by pixel to obtain a target attention distribution map; and determining the position information of the target object in the image to be processed based on the target attention distribution map.

[0111] Among them, the second attention distribution map activates the areas in the weak response area that are helpful for classification, so that the second attention distribution map complements the first attention distribution map. The target attention distribution map is obtained by multiplying the first attention distribution map and the second attention distribution map pixel by pixel, and then the connected area analysis of the target attention distribution map can obtain a more complete and accurate positioning frame, thereby greatly improving the accuracy of the position information of the target object in the image to be processed.

[0112] In an exemplary embodiment, performing target object detection based on the first feature map, the first attention distribution map, and the second feature map may include:

[0113] Classification is performed according to the second feature map to obtain a first classification result; the first classification result represents a probability value corresponding to each preset category in a plurality of preset categories;

[0114] Performing weighted summation on multiple initialized convolution kernels according to the first classification result to obtain a target convolution kernel;

[0115] Performing feature extraction on the first feature map according to the target convolution kernel to obtain a third feature map;

[0116] According to the third feature map, category information of the target object in the image to be processed is determined and a second attention distribution map is generated.

[0117] In the above embodiment, the target convolution kernel is obtained by weighted summing of multiple initialized convolution kernels according to the first classification result, thereby realizing re-parameterization of the convolution kernel. By re-applying the re-parameterized convolution kernel to the first feature map, the area in the weak response area that is helpful for classification occupies a higher weight in the final convolution, so that this area can be activated in the subsequent second attention distribution map.

[0118] In an exemplary embodiment, when classifying according to the second feature map to obtain a first classification result, the following steps may be included:

[0119] Performing dimensionality reduction processing on the second feature map to obtain a reduced dimensionality feature map;

[0120] Performing global maximum pooling and global average pooling on the dimensionality reduction feature map to obtain a first pooling feature and a second pooling feature;

[0121] Fusing the first pooling feature and the second pooling feature to obtain a fused pooling feature;

[0122] The fused pooled features are classified to obtain a first classification result.

[0123] In a specific implementation, the dimensionality reduction process may be to process the second feature map through a 1x1 convolution, thereby compressing the second feature map to one dimension. The fusion method of the first pooling feature and the second pooling feature may be to add the two. The Sigmoid activation function may be used when classifying the fused pooling features. In the above embodiment, the features of the weak response area are measured from different dimensions through global average pooling (GAP) and global maximum pooling (GMP), thereby improving the accuracy of the first classification result.

[0124] In an exemplary embodiment, determining the category information of the target object in the image to be processed and generating the second attention distribution map based on the third feature map may include:

[0125] Performing global average pooling on the third feature map to obtain a third pooled feature;

[0126] Classification is performed based on the third pooled feature to obtain a second classification result; the second classification result represents category information of the target object in the image to be processed;

[0127] Based on the third pooled features and the third feature map, a second attention distribution map is generated.

[0128] In a specific implementation, the third feature map includes feature maps of each channel, and the third pooled feature includes features corresponding to each channel. When generating the second attention distribution map, the features of each channel in the third pooled feature are used as weights of the feature map of the corresponding channel, and then the weighted sum of each weight and the feature map of the corresponding channel is calculated, so that the second attention distribution map can be obtained. In practical applications, the second attention distribution map can also be called a category activation map.

[0129] In an exemplary embodiment, the image detection method of an embodiment of the present invention can be implemented based on an image detection model, which includes an attention encoding unit and a local feature mining unit. The attention encoding unit is a Transformer-based neural network, which can include an embedding layer and multiple attention encoding layers. The embedding layer is used to determine a first vector sequence based on an image block sequence corresponding to the image to be processed, and the multiple attention encoding layers are used to sequentially perform attention encoding on the first vector sequence to obtain a first feature map as the output of the attention encoding unit.

[0130] The local feature mining unit (CDM, Cue Digging Module) is used to determine the attention matrix corresponding to each of the attention encoding layers, extract the target attention features of the classification embedding vector corresponding to each attention matrix, fuse the target attention features to obtain a first attention distribution map, invert the first attention distribution map to obtain a reverse attention distribution map, fuse the reverse attention distribution map with the first feature map output by the attention encoding unit to obtain a second feature map, and further perform target object detection based on the first feature map, the first attention distribution map and the second feature map to obtain category information and position information of the target object in the image to be processed.

[0131] In practical applications, transformer-based neural networks can be pre-trained classification networks based on transformers, such as the Vision Transformer (ViT) network. The ViT network uses the original BERT (Bidirectional Encoder Representation from Transformers) transformer structure, which mainly converts images into word-like (i.e., token) forms. It introduces the concept of image blocks, which divides the input image into image blocks of the same size, and then flattens each image block into a one-dimensional image block embedding vector for easy input to the encoder.

[0132] In addition to the aforementioned image block embedding vectors, the ViT network also requires another special input: position encoding. Unlike convolutional neural networks, the transformer requires position encoding to encode the relative position information of words. This is mainly because the self-attention structure in the internal encoder of the transformer is insensitive to the order of the input sequence. In other words, shuffling the order of words in the sequence does not change the output result. If the relative position information of the image blocks is not actively provided to the network, then the network needs to learn to infer the relative position relationship between image blocks through the semantics of the image blocks, which increases the learning cost and reduces the model accuracy.

[0133] To achieve image-level classification, the ViT network adds a special [Class] token, inspired by BERT. Image classification can be achieved by applying a linear classifier to the features output by the [Class] token.

[0134] like Figure 5 The figure shows a flow chart of another image detection method provided by an embodiment of the present invention. Figure 5 The image detection method according to the embodiment of the present invention is described in detail.

[0135] After the image to be processed is divided into N image blocks of the same size, the N image blocks are stretched into an image block sequence, and the N image block sequence is input into the vector embedding layer. In the vector embedding layer, each image block is vector embedded, and position encoding and initialization [Class] token are added to obtain the output of the first vector sequence of length N+1, with X l Represents the vector in the first vector sequence, X l The dimension is D.

[0136] The first vector sequence output by the vector embedding layer is further input into the Transformer-based encoding unit. The output of the encoding unit is transformed (Reshape) to obtain the first feature map X L Among them, the encoding unit includes multiple transformer blocks (i.e., attention encoding layers), and each attention encoding layer encodes based on the multi-head attention mechanism. Figure 5As shown, each attention encoding layer includes a Norm layer, a Multi-Head Attention layer, an Add layer, and an MLP layer. The Norm layer normalizes the hidden layers in the neural network to a standard normal distribution. The Multi-Head Attention layer is a multi-head attention layer, which includes multiple self-attention mechanism modules. Its input is a matrix obtained through three linear mappings: the query matrix Q, the key matrix K, and the value matrix V. For the encoder mechanism of the Multi-Head Attention layer, please refer to the relevant description in the prior art. The Add layer is used for residual connections. The MLP layer is a multi-layer perception network. The specific working principle of the attention encoding layer can be found in the relevant description in the prior art and will not be repeated here.

[0137] The local feature mining unit CDM averages the attention matrices of each attention mechanism module in each attention coding layer to obtain the attention matrix A′ corresponding to the attention coding layer. l ∈R (N+1)×(N+1) , and from this A′ l ∈R (N+1)×(N+1) Extract the attention distribution map corresponding to [Class] token Then the attention distribution maps corresponding to all attention encoding layers are averaged (i.e., the Mean operation is performed) to obtain the first attention distribution map Next, the first attention distribution map After the inverse processing (i.e., reverse operation) and the first feature map X L Multiply, and the multiplication result (i.e., the second feature map) passes through a 1x1 convolution and uses two branches (global maximum pooling layer GMP+fully connected layer FC, global average pooling layer GAP+fully connected layer FC) to characterize the characteristics of the weak response area from different dimensions, and the outputs of the two branches are superimposed (i.e., Add) and classified by the Sigmoid activation function. The classification result is used for the re-parameterization of the convolution kernel (i.e., Weighting). Specifically, m convolution kernels (for example, 6) can be initialized first, and then these m convolution kernels are linearly weighted according to the output result of Sigmoid to generate a new convolution kernel and used for convolution operation. For example, if a certain area is classified as background, the corresponding convolution kernel will be activated, thereby occupying a higher weight in the final convolution.

[0138] The convolution kernel with heavy parameters acts on the first feature map X L Output the third feature map X CDM , the third feature map X CDM After passing through the global average pooling layer again, it is sent to the classifier for classification, so as to obtain the category information of the target object in the output image to be processed.

[0139] In addition, the third feature map X CDM After the global average pooling layer, a second attention distribution map M is generated based on the class activation map method. CDM , and the second attention distribution map M CDM and the first attention distribution map Obtain a final class activation map M by pixel-by-pixel multiplication fuse , the final activation map M fuse The image is sent to the localization module for localization, thereby obtaining the location information of the target object in the output image to be processed. The localization module mainly performs connected domain analysis based on the class activation map to determine the target localization box.

[0140] The embodiment of the present invention uses a feature mining mechanism driven by a reverse attention map to effectively mine areas in weak response areas that are also helpful for classification, thereby compensating for the influence of the attention distribution map corresponding to the [Class] token on positioning accuracy due to ignoring the correlation between image blocks, and ultimately obtaining a more complete and accurate positioning frame, greatly improving the positioning accuracy of objects in the image, and ensuring the accuracy of location information.

[0141] The following describes the training process of the image detection model. Specifically, the training of the above image detection model may include:

[0142] Obtaining a sample image and a category label corresponding to the sample image; the category label indicates reference category information of a target object in the sample image;

[0143] Inputting a sample image block sequence corresponding to the sample image into an attention coding unit of a preset neural network model, and performing attention coding based on multiple attention coding layers in the attention coding unit to obtain a first sample feature map;

[0144] The local feature mining unit based on the preset neural network model determines the attention matrix of each attention encoding layer, extracts the target attention features corresponding to the classification embedding vector in each attention matrix, fuses the target attention features to obtain a first sample attention distribution map, inverts the first sample attention distribution map to obtain a sample reverse attention distribution map, fuses the sample reverse attention distribution map with the first sample feature map to obtain a second sample feature map; and performs category prediction based on the second sample feature map and the first sample feature map to obtain predicted category information;

[0145] The loss value is determined according to the difference between the predicted category information corresponding to the sample image and the category label, and the model parameters of the preset neural network model are reversely adjusted based on the loss value, and iterative training is performed until the training end condition is met to obtain the image detection model.

[0146] Among them, the attention coding unit of the preset neural network model includes an embedding layer and multiple attention coding layers. After receiving the sample image block sequence, the embedding layer performs vector embedding on each sample image block in the sample image block sequence to obtain a sample image block embedding vector sequence, and adds the initialized classification embedding vector to the sample image block embedding vector sequence to obtain a sample embedding vector sequence, and then obtains a sample first vector sequence by adding corresponding position coding to each sample embedding vector in the sample embedding vector sequence. The first vector sequence is sequentially attention-encoded by multiple attention coding layers, and finally a first sample feature map output by the attention coding unit is obtained.

[0147] The sample image block sequence may be obtained by evenly dividing the sample image and then expanding and arranging the sample image into a sequence, and the sample image blocks obtained by evenly dividing the sample image may have the same size.

[0148] The specific determination process of the sample reverse attention distribution map and the second sample feature map by the local feature mining unit of the preset neural network model can refer to the relevant contents of the aforementioned steps S405 to S407 in the embodiment of the present invention, which will not be repeated here. In addition, when performing category prediction based on the second sample feature map and the first sample feature map, the probability values ​​corresponding to each preset category can be obtained by first classifying the second sample feature map, and then performing weighted summation on the multiple initial convolution kernels in the local feature mining unit based on the probability values ​​corresponding to each preset category to obtain the target convolution kernel, and then performing feature extraction on the first sample feature map based on the target convolution kernel to obtain the third sample feature map, performing global average pooling on the third sample feature map, and performing category prediction based on the features after global average pooling, thereby obtaining predicted category information.

[0149] The loss during training can be calculated using a cross-entropy loss function, which is based on the difference between the predicted class information and the class label for the sample image. Training can end when the loss reaches a preset loss threshold or when the number of iterations reaches a preset iteration threshold.

[0150] The training process of the above-mentioned image detection model only requires labeling the true category of the target object in the sample image, and no longer requires providing precise target location information, which can greatly reduce the complexity of labeling. In addition, the model can effectively utilize the massive amount of images obtained from the Internet for model training to achieve target positioning of large-scale / long-tail categories.

[0151] Corresponding to the image detection methods provided in the above-mentioned embodiments, an embodiment of the present invention also provides an image detection device. Since the image detection device provided in the embodiment of the present invention corresponds to the image detection methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned image detection methods are also applicable to the image detection device provided in this embodiment and will not be described in detail in this embodiment.

[0152] See also Figure 6 , which is a schematic diagram of the structure of an image detection device provided by an embodiment of the present invention. The image detection device 600 has the function of implementing the image detection method in the above method embodiment. The function can be implemented by hardware or by hardware executing corresponding software. Figure 6 As shown, the image detection device 600 may include:

[0153] A first determining module 610 is configured to determine a first vector sequence based on an image block sequence corresponding to an image to be processed; the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence;

[0154] an attention encoding module 620, configured to sequentially perform attention encoding on the first vector sequence based on multiple attention encoding layers to obtain a first feature map;

[0155] A first attention distribution determination module 630 is configured to determine an attention matrix corresponding to each of the attention encoding layers, extract target attention features corresponding to the classification embedding vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map;

[0156] a negation processing module 640 configured to perform negation processing on the first attention distribution map to obtain a reversed attention distribution map, and fuse the reversed attention distribution map with the first feature map to obtain a second feature map;

[0157] The target object detection module 650 is used to perform target object detection based on the first feature map, the first attention distribution map and the second feature map to obtain the category information and position information of the target object in the image to be processed.

[0158] In an exemplary embodiment, the target object detection module 650 includes:

[0159] a second attention distribution determining module, configured to determine category information of a target object in the image to be processed and generate a second attention distribution map based on the second feature map and the first feature map;

[0160] a target attention distribution determination module, configured to multiply the first attention distribution map by the second attention distribution map pixel by pixel to obtain a target attention distribution map;

[0161] A position information determination module is used to determine the position information of the target object in the image to be processed based on the target attention distribution map.

[0162] In an exemplary embodiment, the second attention distribution determination module 650 includes:

[0163] a first classification module, configured to perform classification based on the second feature map to obtain a first classification result; wherein the first classification result includes a probability value corresponding to each of a plurality of preset categories;

[0164] a re-parameter module, configured to perform weighted summation on a plurality of initialized convolution kernels according to the first classification result to obtain a target convolution kernel;

[0165] A feature extraction module, configured to perform feature extraction on the first feature map according to the target convolution kernel to obtain a third feature map;

[0166] The second determination module is used to determine the category information of the target object in the image to be processed based on the third feature map and generate a second attention distribution map.

[0167] In an exemplary embodiment, the first classification module includes:

[0168] A dimensionality reduction module, configured to perform dimensionality reduction processing on the second feature map to obtain a reduced-dimensionality feature map;

[0169] A first pooling module is used to perform global maximum pooling and global average pooling on the dimensionality reduction feature map to obtain a first pooling feature and a second pooling feature;

[0170] A pooling feature fusion module, configured to fuse the first pooling feature and the second pooling feature to obtain a fused pooling feature;

[0171] The classification submodule is used to classify the fused pooled features to obtain a first classification result.

[0172] In an exemplary embodiment, the second determining module includes:

[0173] A second pooling module is used to perform global average pooling on the third feature map to obtain a third pooling feature;

[0174] A second classification module is configured to perform classification based on the third pooled feature to obtain a second classification result; the second classification result represents category information of the target object in the image to be processed;

[0175] An attention map generation module is used to generate a second attention distribution map based on the third pooled features and the third feature map.

[0176] In an exemplary embodiment, when the inversion processing module 640 fuses the inverse attention distribution map with the first feature map to obtain the second feature map, it is specifically used to multiply the inverse attention distribution map with the first feature map to obtain the second feature map.

[0177] In an exemplary embodiment, the first attention distribution determination module 630 includes:

[0178] A first averaging module is used to determine the average of the attention matrices of the respective attention mechanism modules for each of the attention encoding layers to obtain the attention matrix corresponding to the attention encoding layer;

[0179] An extraction module, configured to extract a target attention feature corresponding to the classification embedding vector from the attention matrix corresponding to the attention encoding layer;

[0180] The second averaging module averages the target attention features corresponding to each of the attention encoding layers to obtain a first attention distribution map.

[0181] In an exemplary embodiment, the first determining module 610 includes:

[0182] An image segmentation module is used to obtain an image to be processed, and segment the image to be processed into a plurality of image blocks to obtain an image block sequence;

[0183] an embedding module, configured to perform vector embedding on image blocks in the image block sequence to obtain an image block embedding vector sequence;

[0184] A first adding module is configured to add an initialized classification embedding vector to the image block embedding vector sequence to obtain an embedding vector sequence;

[0185] The second adding module is used to add a position code to each embedded vector in the embedded vector sequence to obtain the first vector sequence; the position code represents the position information of the corresponding embedded vector in the embedded vector sequence.

[0186] In an exemplary embodiment, the image detection method of the embodiment of the present invention can be implemented based on an image detection model. Accordingly, the image detection device of the embodiment of the present invention further includes a training module, which includes:

[0187] A sample acquisition module is used to acquire a sample image and a category label corresponding to the sample image; the category label indicates reference category information of a target object in the sample image;

[0188] A first sample feature map determining module is configured to input a sample image block sequence corresponding to the sample image into an attention coding unit of a preset neural network model, and perform attention coding based on multiple attention coding layers in the attention coding unit to obtain a first sample feature map;

[0189] A category prediction module is configured to determine the attention matrix of each of the attention encoding layers based on the local feature mining unit of the preset neural network model, extract the target attention features corresponding to the classification embedding vector in each of the attention matrices, fuse the target attention features to obtain a first sample attention distribution map, invert the first sample attention distribution map to obtain a sample reverse attention distribution map, fuse the sample reverse attention distribution map with the first sample feature map to obtain a second sample feature map; and perform category prediction based on the second sample feature map and the first sample feature map to obtain predicted category information;

[0190] A training submodule is used to determine a loss value based on the difference between the predicted category information corresponding to the sample image and the category label, reversely adjust the model parameters of the preset neural network model based on the loss value, and perform iterative training until the training end condition is met to obtain the image detection model.

[0191] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0192] An embodiment of the present invention provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image detection method provided in the above-mentioned method embodiment.

[0193] The memory can be used to store software programs and modules. The processor executes various functional applications and image detection by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0194] The method embodiments provided in the embodiments of the present invention can be executed in a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 7 This is a hardware structure diagram of a server running an image detection method provided by an embodiment of the present invention, such as Figure 7 As shown, the server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 710 (the processor 710 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 730 for storing data, and one or more storage media 720 (such as one or more mass storage devices) for storing application programs 723 or data 722. Among them, the memory 730 and the storage medium 720 can be temporary storage or permanent storage. The program stored in the storage medium 720 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 710 can be configured to communicate with the storage medium 720 to execute a series of instruction operations in the storage medium 720 on the server 700. The server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input and output interfaces 740, and / or one or more operating systems 721, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0195] The input / output interface 740 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 700. In one embodiment, the input / output interface 740 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 740 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0196] It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown.

[0197] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing an image detection method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image detection method provided by the above method embodiment.

[0198] Embodiments of the present invention further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the above-described image detection method.

[0199] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0200] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0201] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0202] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image detection method, characterized in that: The method comprises: Determining a first vector sequence according to an image block sequence corresponding to the image to be processed; the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence; Attention encoding is performed on the first vector sequence in sequence based on multiple attention encoding layers to obtain a first feature map; Determine an attention matrix corresponding to each of the attention encoding layers, extract a target attention feature corresponding to the classification embedding vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map; Inverting the first attention distribution map to obtain a reverse attention distribution map, and fusing the reverse attention distribution map with the first feature map to obtain a second feature map; Target object detection is performed based on the first feature map, the first attention distribution map, and the second feature map to obtain category information and position information of the target object in the image to be processed.

2. The image detection method according to claim 1, wherein: The detecting the target object according to the first feature map, the first attention distribution map, and the second feature map to obtain category information and position information of the target object in the image to be processed includes: Determining category information of a target object in the image to be processed and generating a second attention distribution map based on the second feature map and the first feature map; Multiplying the first attention distribution map and the second attention distribution map pixel by pixel to obtain a target attention distribution map; Determine the position information of the target object in the image to be processed according to the target attention distribution map.

3. The image detection method according to claim 2, wherein: The determining, based on the second feature map and the first feature map, category information of the target object in the image to be processed and generating a second attention distribution map includes: Classification is performed according to the second feature map to obtain a first classification result; the first classification result includes a probability value corresponding to each preset category in a plurality of preset categories; Performing weighted summation on multiple initialized convolution kernels according to the first classification result to obtain a target convolution kernel; Performing feature extraction on the first feature map according to the target convolution kernel to obtain a third feature map; According to the third feature map, category information of the target object in the image to be processed is determined and a second attention distribution map is generated.

4. The image detection method according to claim 3, wherein: The classifying according to the second feature map to obtain a first classification result includes: Performing dimensionality reduction processing on the second feature map to obtain a reduced-dimensionality feature map; Performing global maximum pooling and global average pooling on the dimensionality reduction feature map to obtain a first pooling feature and a second pooling feature; Fusing the first pooling feature and the second pooling feature to obtain a fused pooling feature; The fused pooled features are classified to obtain a first classification result.

5. The image detection method according to claim 3, wherein: The step of determining category information of the target object in the image to be processed and generating a second attention distribution map based on the third feature map includes: Performing global average pooling on the third feature map to obtain a third pooled feature; Classification is performed based on the third pooled feature to obtain a second classification result; the second classification result represents category information of the target object in the image to be processed; Based on the third pooled features and the third feature map, a second attention distribution map is generated.

6. The image detection method according to claim 1, wherein: The fusing the reverse attention distribution map with the first feature map to obtain a second feature map includes: Multiplying the inverse attention distribution map by the first feature map to obtain a second feature map.

7. The image detection method according to claim 1, wherein: The determining of the attention matrix corresponding to each of the attention encoding layers, extracting the target attention features corresponding to the classification embedding vector in each of the attention matrices, and fusing the target attention features to obtain a first attention distribution map includes: For each of the attention encoding layers, determining the average of the attention matrices of the respective attention mechanism modules to obtain the attention matrix corresponding to the attention encoding layer; Extracting the target attention feature corresponding to the classification embedding vector in the attention matrix corresponding to the attention encoding layer; The target attention features corresponding to each of the attention encoding layers are averaged to obtain a first attention distribution map.

8. The image detection method according to claim 1, wherein: The determining of the first vector sequence according to the image block sequence corresponding to the image to be processed includes: Acquire an image to be processed, and divide the image to be processed into a plurality of image blocks to obtain an image block sequence; Performing vector embedding on image blocks in the image block sequence to obtain an image block embedding vector sequence; Adding the initialized classification embedding vector to the image block embedding vector sequence to obtain an embedding vector sequence; A position code is added to each embedding vector in the embedding vector sequence to obtain the first vector sequence; the position code represents position information of the corresponding embedding vector in the embedding vector sequence.

9. The image detection method according to any one of claims 1 to 8, characterized in that: The image detection method is implemented based on an image detection model, and the training of the image detection model includes: Acquire a sample image and a category label corresponding to the sample image; the category label indicates reference category information of a target object in the sample image; Inputting a sample image block sequence corresponding to the sample image into an attention coding unit of a preset neural network model, and performing attention coding based on multiple attention coding layers in the attention coding unit to obtain a first sample feature map; The local feature mining unit based on the preset neural network model determines the attention matrix of each attention encoding layer, extracts the target attention features corresponding to the classification embedding vector in each attention matrix, fuses the target attention features to obtain a first sample attention distribution map, inverts the first sample attention distribution map to obtain a sample reverse attention distribution map, fuses the sample reverse attention distribution map with the first sample feature map to obtain a second sample feature map; and performs category prediction based on the second sample feature map and the first sample feature map to obtain predicted category information; The loss value is determined according to the difference between the predicted category information corresponding to the sample image and the category label, and the model parameters of the preset neural network model are reversely adjusted based on the loss value, and iterative training is performed until the training end condition is met to obtain the image detection model.

10. An image detection device, characterized in that: The device comprises: A first determining module is configured to determine a first vector sequence based on an image block sequence corresponding to an image to be processed; the first vector sequence includes a classification embedding vector and an image block vector corresponding to each image block in the image block sequence; an attention encoding module, configured to sequentially perform attention encoding on the first vector sequence based on multiple attention encoding layers to obtain a first feature map; A first attention distribution determination module is configured to determine an attention matrix corresponding to each of the attention encoding layers, extract target attention features corresponding to the classification embedding vector in each of the attention matrices, and fuse the target attention features to obtain a first attention distribution map; a negation processing module, configured to perform negation processing on the first attention distribution map to obtain a reverse attention distribution map, and fuse the reverse attention distribution map with the first feature map to obtain a second feature map; A target object detection module is used to detect the target object based on the first feature map, the first attention distribution map and the second feature map to obtain the category information and position information of the target object in the image to be processed.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the image detection method according to any one of claims 1 to 9.

Images