Target classification method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention provides a target classification method and device, electronic equipment, a storage medium and a product, and relates to the technical field of image processing. In the method, the position of a target in a to-be-recognized image is acquired; performing cascaded multiple times of feature extraction on the to-be-identified image to obtain a first feature map; mapping the position to the first feature map according to a size relationship between the first feature map and the to-be-identified image to obtain a local feature region in the first feature map; and performing classification based on the local feature region to obtain the category of the target. The accuracy of target classification can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target classification method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] To reduce the false detection rate of target classification, the industry typically adopts a two-stage processing flow of "detection + classification". In this process, the electronic device first determines the location of the target in the original image, and then crops out a local image region containing the target from the original image based on the determined location; finally, features are extracted from the cropped local image region, and then category recognition is performed based on the extracted features to obtain the category of the target in the local image region.

[0003] However, the cropped local image region contains less information, especially for small targets, where the local image region may only contain a few pixels. This results in a small amount of extracted feature information, leading to low accuracy in target classification. Summary of the Invention

[0004] The purpose of this invention is to provide a target classification method, apparatus, electronic device, storage medium, and product to improve the accuracy of target classification. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide a target classification method, the method comprising:

[0006] Obtain the location of the target in the image to be identified;

[0007] The image to be identified is subjected to multiple cascaded feature extractions to obtain a first feature map;

[0008] Based on the size relationship between the first feature map and the image to be identified, the position is mapped to the first feature map to obtain a local feature region in the first feature map;

[0009] The target category is obtained by classifying based on the local feature regions.

[0010] Optionally, mapping the position to the first feature map according to the size relationship between the first feature map and the image to be identified, to obtain a local feature region in the first feature map, includes:

[0011] The first feature map is subjected to subpixel-level interpolation to obtain a subpixel feature map;

[0012] Based on the size relationship between the first feature map and the image to be identified, the position is mapped to the sub-pixel feature map to obtain a local feature region.

[0013] Optionally, the classification based on the local feature regions to obtain the category of the target includes:

[0014] The local feature region is enlarged according to a set size to obtain a second feature map;

[0015] The target category is obtained by classifying based on the second feature map.

[0016] Optionally, the step of performing cascaded feature extraction on the image to be identified to obtain a first feature map includes:

[0017] The image to be identified is input into the first feature extraction layer of the first classification model, so that the first x cascaded feature extraction layers in the first classification model are cascaded to extract features, resulting in the first feature map output by the xth feature extraction layer. The first classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model also includes a classification layer.

[0018] The classification based on the second feature map to obtain the category of the target includes:

[0019] The second feature map is input into the (x+1)th feature extraction layer in the first classification model for feature extraction, resulting in the third feature map output by the last feature extraction layer.

[0020] The third feature map is input into the classification layer for category identification to obtain the category of the target.

[0021] Optionally, the step of performing cascaded feature extraction on the image to be identified to obtain a first feature map includes:

[0022] The image to be identified is input into the first feature extraction layer of the second classification model, so that the first x cascaded feature extraction layers in the second classification model are cascaded to extract features, resulting in the first feature map output by the xth feature extraction layer. The second classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model also includes an adaptive pooling layer.

[0023] The classification based on the local feature regions to obtain the category of the target includes:

[0024] The local feature region is input into the adaptive pooling layer for category recognition to obtain the category of the target.

[0025] Secondly, embodiments of the present invention provide a target classification device, the device comprising:

[0026] The first acquisition module is used to acquire the location of the target in the image to be recognized;

[0027] The extraction module is used to perform cascaded feature extraction on the image to be identified multiple times to obtain a first feature map;

[0028] The mapping module is used to map the position to the first feature map according to the size relationship between the first feature map and the image to be identified, so as to obtain a local feature region in the first feature map;

[0029] The classification module is used to classify the target based on the local feature regions to obtain the category of the target.

[0030] Optionally, the mapping module is specifically used to perform sub-pixel level interpolation processing on the first feature map to obtain a sub-pixel feature map; and to map the position to the sub-pixel feature map according to the size relationship between the first feature map and the image to be identified to obtain a local feature region.

[0031] And / or,

[0032] The classification module includes: a magnification unit for magnifying the local feature region according to a set size to obtain a second feature map; and a classification unit for classifying the target based on the second feature map to obtain the category of the target.

[0033] or,

[0034] The extraction module is specifically used to input the image to be identified into the first feature extraction layer of the first classification model, so that the first x cascaded feature extraction layers in the first classification model are cascaded for feature extraction, resulting in a first feature map output by the xth feature extraction layer. The first classification model is trained based on sample images and the true categories of sample targets in the sample images; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model also includes a classification layer; the classification unit is specifically used to input the second feature map into the (x+1)th feature extraction layer of the first classification model for feature extraction, resulting in a third feature map output by the last feature extraction layer; the third feature map is then input into the classification layer for category recognition to obtain the category of the target.

[0035] or,

[0036] The extraction module is specifically used to input the image to be identified into the first feature extraction layer of the second classification model, so that the first x cascaded feature extraction layers in the second classification model are cascaded to extract features, resulting in a first feature map output by the xth feature extraction layer. The second classification model is trained based on the sample image and the true category of the target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model also includes an adaptive pooling layer; the classification module is specifically used to input the local feature region into the adaptive pooling layer for category recognition, thereby obtaining the category of the target.

[0037] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0038] Memory, used to store computer programs;

[0039] The processor, when executing the program stored in memory, implements the above-mentioned target classification method.

[0040] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned target classification method.

[0041] This invention also provides a computer program product, including a computer program that implements the above-described target classification method when executed by a processor.

[0042] Beneficial effects of the embodiments of the present invention:

[0043] In this embodiment of the invention, the position of the target in the image to be identified is obtained, and multiple cascaded feature extractions are performed on the image to be identified to obtain a first feature map. Through multiple cascaded feature extractions, features can be extracted layer by layer from shallow to deep within the entire image to be identified. Shallow feature extraction can obtain the detailed features of the target itself in the image to be identified, while deep feature extraction can obtain the global context features of the entire image. Therefore, the first feature map can contain both the target's own features and the features related to the target and its surrounding background. Then, according to the size relationship between the first feature map and the image to be identified, the position of the target is mapped onto the first feature map to obtain a local feature region in the first feature map. That is, the local feature region originates from the first feature map, and the position of the local feature region corresponds to the position of the target in the image to be identified. Thus, the local feature region can contain both the target's own features at the corresponding position and the features related to the target and its background. Therefore, classification based on the local feature region can be performed based on the target's own features and the features related to the target and its background. Especially for small targets with few visual features, this solution can classify them based on the characteristics of the small target itself in the local feature region, as well as the correlation features between the small target and the background. It can effectively distinguish small targets from interference objects based on more feature information, minimize the possibility of misclassification of target categories, and improve the accuracy of target classification.

[0044] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0046] Figure 1 This is a schematic diagram of the first flowchart of the target classification method provided in the embodiments of the present invention;

[0047] Figure 2 This is a schematic diagram of a second process for the target classification method provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the third process of the target classification method provided in the embodiments of the present invention;

[0049] Figure 4 This is a schematic diagram of the first structure of the classification model provided in an embodiment of the present invention;

[0050] Figure 5This is a schematic diagram of the fourth process of the target classification method provided in the embodiments of the present invention;

[0051] Figure 6 This is a schematic diagram of the second structure of the classification model provided in an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the fifth process of the target classification method provided in the embodiments of the present invention;

[0053] Figure 8 A schematic diagram illustrating the principle of a target classification method provided in an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of the structure of a target classification device provided in an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0057] To better understand the technical problems to be solved by the embodiments of the present invention, the prior art will be introduced first.

[0058] In applications of computer vision in security monitoring, remote sensing image analysis, autonomous driving, and industrial inspection, the core challenge of implementing object detection and classification technologies lies in small object detection. Small objects in an image are those that occupy less than a preset threshold, such as 3% or 5%. For example, an image could be a distant photograph of a forest, where a small object could be a flame; or an image could be a photograph of a gas station scene, where a small object could be a cigarette butt.

[0059] As described in the background art, the electronic device determines the location of the target in the original image, and then crops out the local image region where the target is located from the original image based on the determined location, that is, crops out the candidate region of interest (ROI) where the target exists; finally, feature extraction is performed on the cropped local image region. Since feature extraction is only performed on the local image region, the extracted features can only reflect the information of the local image region. The local image region is the area where the small target is located. It is small in size and contains fewer pixels. Therefore, the information that the extracted features can reflect is also less. Thus, the accuracy of target classification through local image regions is low.

[0060] For example, electronic devices can utilize object detection models built on deep convolutional neural networks to perform object classification tasks. When extracting features from local image regions, deep convolutional neural networks can perform multiple convolutional transformations layer by layer on the local image regions to extract features and obtain deep features with strong semantic expressive power. However, local image regions have limited information, and after multiple convolutional transformations, the detailed features of the target in the local image region will be weakened or even completely lost. In scenarios with complex backgrounds or interference objects that resemble the target, classification based on extracted deep features is prone to misclassifying background noise as the target, leading to a high false positive rate. The object detection model can be a You Only Look Once (YOLO) model, a Faster Region-based Convolutional Neural Networks (Faster R-CNN) model, or a Single Shot MultiBox Detector (SSD) model.

[0061] For example, in smoking behavior detection scenarios, because cigarette butts are extremely small and their appearance is not obvious, thin, elongated objects such as pens and sticks are easily misidentified as cigarette butts. Similarly, in fireworks detection scenarios, lights, reflective surfaces, or bright areas are easily misidentified as fireworks.

[0062] It is clear that improving the accuracy of target classification is an urgent problem to be solved.

[0063] To improve the accuracy of target classification, embodiments of the present invention provide a target classification method, apparatus, electronic device, storage medium, and product. The target classification method provided by these embodiments can be applied to an electronic device that can acquire an image to be identified and classify the targets within that image. In specific applications, the electronic device can be a server, a personal computer, or a camera, etc.

[0064] One embodiment of the present invention provides a target classification method that may include the following steps: obtaining the location of the target in the image to be identified; performing cascaded multiple feature extractions on the image to be identified to obtain a first feature map; mapping the location to the first feature map according to the size relationship between the first feature map and the image to be identified to obtain a local feature region in the first feature map; and classifying based on the local feature region to obtain the category of the target. In this embodiment, the location of the target in the image to be identified is obtained, and cascaded multiple feature extractions are performed on the image to be identified to obtain a first feature map. Through cascaded multiple feature extractions, feature extraction can be performed layer by layer from shallow to deep within the entire image to be identified. Shallow feature extraction can obtain the detailed features of the target itself in the image to be identified, while deep feature extraction can obtain the global context features of the entire image to be identified. Therefore, the first feature map can contain both the target's own features and the correlation features between the target and the surrounding background. Then, based on the size relationship between the first feature map and the image to be identified, the target's position is mapped onto the first feature map, resulting in a local feature region within the first feature map. In other words, the local feature region originates from the first feature map, and its position corresponds to the target's position in the image to be identified. Therefore, the local feature region can contain both the target's own features at the corresponding location and the correlation features between the target and the background. Thus, classification based on the local feature region allows for target classification based on both the target's own features and the correlation features between the target and the background. Especially for small targets with limited visual features, this method can classify them based on the small target's own features within the local feature region, as well as the correlation features between the small target and the background. This allows for effective differentiation of small targets from interference based on more feature information, minimizing misclassification and improving the accuracy of target classification.

[0065] The following describes a target classification method provided by an embodiment of the present invention with reference to the accompanying drawings. See also... Figure 1 The target classification method provided in this embodiment of the invention may include steps S101-S104.

[0066] S101, Obtain the location of the target in the image to be identified.

[0067] The image to be identified can be an original image containing the target to be classified. The original image may contain both the target to be classified and background areas, as well as distracting objects that resemble the target in appearance. The location of the target in the acquired image to be identified may contain either the target itself or distracting objects. Subsequent processing is required to identify whether the objects present at the target's location are indeed targets, and to determine the specific target category.

[0068] The target can be an object that needs to be classified in the scene where the image to be identified is captured, and the target can be a small object in the scene, for example, the target occupies only one percent of the image to be identified. For example, the image to be identified can be an image taken in a no-smoking area such as a gas station, and the target can be a cigarette butt. The image to be identified can be an image taken in a fire prevention area such as a forest, and the target can be a flame.

[0069] The classification of targets in the image to be identified is mainly divided into two cases: binary classification and multi-target classification. Both cases are suitable for the target classification method provided in the embodiments of this invention.

[0070] Binary classification involves making a binary judgment of whether a target in an acquired image is a predefined specific target. For example, in no-smoking scenarios like gas stations, binary classification only needs to determine whether the object at the acquired location is a "cigarette butt" or a "non-cigarette butt," without distinguishing the specific category of non-cigarette butt distractors, such as pens or sticks, which would be classified as non-cigarette butts. Similarly, in forest fire prevention scenarios, it only needs to determine whether the object at the acquired location is a "flame" or a "non-flame," without distinguishing the specific type of non-flame distractors, such as lights, which would be classified as non-flame.

[0071] Multi-target classification involves determining the specific category of each target at its location. This requires not only distinguishing between real targets and distracting objects, but also clearly classifying different types of real targets to determine the true category of each target. For example, in a forest fire prevention scenario, multi-target classification needs to determine whether the object at the target's location is "flame," "smoke," or "distracting object," thus achieving accurate differentiation between different types of targets.

[0072] Electronic devices can acquire the location of a target in an image to be identified in various ways. In one implementation, the electronic device can acquire the image to be identified and then input the image into an object detection model to obtain the location of the target in the image; wherein, the object detection model is trained based on sample images and the true location of the target in the sample images.

[0073] In another implementation, the electronic device can receive the target location transmitted by an external device. The external device performs target detection on the image to be identified in advance, determines the location of the target in the image, and then sends the location information to the electronic device. After receiving the information, the electronic device can complete the acquisition of the target's location. The external device can be a dedicated image detection terminal, a high-definition shooting device (integrating a simple detection algorithm), etc. Its detection logic can be consistent with the above object detection model, or a lightweight detection algorithm can be used to adapt to the needs of rapid front-end acquisition and detection. The electronic device does not need to perform additional detection calculations, thus saving the computing resources of the electronic device.

[0074] The location of the target in the acquired image to be recognized can be the coordinates of the candidate bounding box of the target in the image to be recognized. For example, the location of the target can be [x1, y1, x2, y2], where (x1, y1) and (x2, y2) represent the coordinates of the two vertices on the diagonal of the candidate bounding box containing the target, respectively. The dimension of the image to be recognized is... , This can represent the image to be recognized, where H, W, and C represent the height, width, and number of channels of the image to be recognized, respectively.

[0075] S102, perform multiple cascaded feature extractions on the image to be recognized to obtain the first feature map.

[0076] Cascaded feature extraction for images refers to the process of performing feature extraction operations multiple times in a fixed order, where the input for each feature extraction is the output of the previous one. This process is completed through a layered, progressive approach of input and output. Specifically, an electronic device can first extract features from the image to be recognized, obtaining the first extracted features; then extract features from the first extracted features, obtaining the second extracted features; then extract features from the second extracted features, and so on. Each feature extraction targets the features obtained in the previous extraction, thus achieving cascaded feature extraction of the image to be recognized, resulting in the first feature map.

[0077] The number of cascaded feature extractions is preset. This preset number of cascaded feature extractions ensures that the resulting first feature map includes both local detail features of the target and global contextual information of the image to be identified. Specifically, the preset number of cascaded feature extractions can extract local detail features such as edges and textures of the target in the image to be identified; simultaneously, this preset number of feature extractions can extract the correlation features between the target and its surrounding background while preserving these local detail features.

[0078] Multiple cascaded feature extractions of the image to be recognized can be performed in various ways. In one implementation, the electronic device can input the image to be recognized into a backbone network, which contains multiple cascaded feature extraction layers. Features are extracted from the image through these backbone layers to obtain a first feature map. The backbone network can be a deep convolutional neural network, and each feature extraction layer can be a convolutional layer. For example, the backbone network can be a Residual Network (ResNet) or a Practical Pyramid Hierarchical Ghost Network V2 (HgNetv2).

[0079] In another implementation, the electronic device can be implemented using multiple independent and cascaded feature extraction modules. Each feature extraction module is an independent functional module with feature extraction logic, capable of extracting features from the input information according to a preset algorithm. Specifically, the output feature map of the preceding feature extraction module is directly used as the input of the following feature extraction module. After the electronic device inputs the image to be recognized into the first feature extraction module, each subsequent cascaded feature extraction module sequentially extracts features from the feature map output by the preceding module.

[0080] S103, according to the size relationship between the first feature map and the image to be identified, the position is mapped to the first feature map to obtain the local feature region in the first feature map.

[0081] During the cascaded feature extraction process on the image to be recognized, the size of the image is scaled. Specifically, upsampling during feature extraction enlarges the size, while downsampling shrinks it, thus establishing a clear size relationship between the first feature map and the image to be recognized.

[0082] To determine the size relationship between the first feature map and the image to be recognized, the electronic device can obtain the step size of each downsampling and each upsampling step in the feature extraction process from the first feature map to the image to be recognized. Then, it calculates the reciprocal of each downsampling step and the product of each upsampling step to obtain the size ratio. For example, in the feature extraction process from the first feature map to the image to be recognized, there are two downsampling steps, each with a step size of 2, and one upsampling step with a step size of 2. Therefore, 1 / 2 × 1 / 2 × 2 can be used to calculate the size ratio between the first feature map and the image to be recognized as 1 / 2.

[0083] In another implementation, the electronic device can calculate the ratio of the width of the first feature map to the width of the image to be recognized, and calculate the ratio of the height of the first feature map to the height of the image to be recognized, to obtain the size relationship between the first feature map and the image to be recognized.

[0084] Electronic devices can determine the location in the first feature map based on the relationship between position and size. For example, the electronic device can first calculate the size ratio between the first feature map and the image to be recognized, and then calculate the product of this size ratio and the location of the target in the image to be recognized, thus obtaining the location of the local feature region in the first feature map. Specifically, the size ratio between the first feature map and the image to be recognized is... , If the location of the target in the image to be identified is indicated, then the location of the local feature region in the first feature map can be... For example, the size ratio of the first feature map to the image to be identified is 1 / 4; (1,1) and (4,4) are the diagonal vertex coordinates of the candidate box containing the target in the image to be identified, representing the position of the target in the image to be identified. Then the position of the local feature region in the first feature map can be (0.25, 0.25) and (1,1).

[0085] In the first feature map, each coordinate corresponds to a feature value. All coordinates in the first feature map are integers. For example, the feature value at position (1,1) in the first feature map is 1, and the feature value at position (1,2) is 0. However, the first feature map only contains feature values ​​at the positions with integer coordinates. There are no feature values ​​corresponding to positions with non-integer coordinates. For example, there is no feature value at position (1.25,1.25) in the first feature map.

[0086] Electronic devices can determine the feature value of each position in a local feature region in various ways. In one implementation, for each coordinate within the local feature region, if the coordinate value in any direction is a floating-point value (e.g., the horizontal coordinate value, the vertical coordinate value, or both horizontal and vertical coordinate values ​​are floating-point values), the coordinate can be determined to be a non-integer coordinate. The distance between the non-integer coordinate and the adjacent integer coordinate is calculated, and the feature value corresponding to the integer coordinate with the smallest distance is selected as the feature value of the non-integer coordinate. For example, if the non-integer coordinate is (1.25, 1.25), and the nearest integer coordinate is (1, 1), then the feature value corresponding to the coordinate (1, 1) is selected as the feature value corresponding to the coordinate (1.25, 1.25). In this implementation, for each non-integer coordinate, the nearest integer coordinate is determined, and then the feature value corresponding to the determined integer coordinate is used as the feature value of the non-integer coordinate, thus determining the feature value of each position in the local feature region. For each coordinate within a local feature region, if the coordinate values ​​in both the horizontal and vertical directions are integer values, then the coordinate is an integer coordinate and has a corresponding feature value, so there is no need to calculate the feature value corresponding to the coordinate.

[0087] In another implementation, the electronic device can determine the feature value at each location in a local feature region based on sub-pixel level interpolation. This implementation will be described in subsequent embodiments.

[0088] S104, classify based on local feature regions to obtain the category of the target.

[0089] The feature values ​​at each location in the local feature region can represent the feature information of the target, as well as the correlation features between the target and the background. Electronic devices can process the local feature region based on the classification layer of the classification model, whereby the classification layer of the classification model can classify the target based on the feature values ​​in the local feature region.

[0090] In one implementation, the classification layer of the classification model can perform pooling and normalization operations on local feature regions, converting the feature information contained in the local feature regions into probability values ​​corresponding to each category. The probability value is positively correlated with the probability that the target corresponding to the local feature region belongs to the corresponding category. Subsequently, the electronic device determines the final category of the target based on the probability values ​​of each category output by the classification layer. For example, the electronic device can select the category with the highest probability value as the category of the target.

[0091] In another implementation, the electronic device can magnify local feature regions and then classify them based on the magnified feature maps. This implementation will be described in detail in subsequent embodiments and will not be elaborated on here.

[0092] There can be at least one target in the image to be identified. In one implementation, there are multiple locations of targets in the acquired image to be identified, and there are also multiple local feature regions. Each local feature region corresponds to the location of a target. When classifying based on local feature regions, the electronic device can classify based on each local feature region to determine the category of the target corresponding to that local feature region, thereby determining the category of each target in the image to be identified.

[0093] In another implementation, there is only one location of the target in the acquired image to be identified, and there is only one local feature region. This local feature region corresponds to the location of the target. When classifying based on the local feature region, the electronic device can classify based on the local feature region to determine the category of the target and obtain the category of the target in the image to be identified.

[0094] In this scheme, the first feature map is obtained by cascading multiple feature extractions from the global image to be identified. The local feature regions within it are not isolated local information, but rather implicitly contain global image features and contextual relationships. They include both the target's own features and fully preserve the relationship between the target and its surrounding background. For small targets with few visual features, this scheme relies on local feature regions containing global relational information for discrimination. It can combine the relationship between the target and the background to assist in classification, avoiding isolating the image region where the target is located from the original image to be identified, severing the semantic connection between the target and its surrounding background, avoiding the loss of contextual information, and maintaining the semantic relationship between the target and the global environment of the background. This effectively distinguishes small targets from interference objects, minimizes the possibility of misclassification of target categories, and improves the accuracy of target classification.

[0095] Since feature extraction is performed on the complete image to be identified, each location in the feature map contains contextual information within its receptive field. Electronic devices can fully utilize the semantics of the target's surrounding environment (such as fingers around a cigarette butt or smoke around a flame) to assist in discrimination, enhancing classification accuracy through background semantics. Furthermore, it breaks through the "cropping before classification" processing flow in existing technologies, employing full-image features from the first feature map and a classification mode guided by the coordinates of the location in the image to be identified to the region in the feature map, thus achieving a non-cropping classification paradigm.

[0096] Furthermore, for cases where the image to be identified contains multiple targets, the embodiments of the present invention can eliminate the need to crop the image region. This avoids the repeated cropping of each target region from the image to be identified and scaling the cropped region to the input size of the classification model, as required by existing technologies. Especially for cases where a large image contains multiple targets, this invention avoids repeatedly performing cropping, scaling, and other processing, reducing the inference time required for target classification, minimizing computational redundancy, and making it more suitable for real-time applications.

[0097] In one possible embodiment, in a target classification method provided by an embodiment of the present invention, the step of mapping the position to the first feature map according to the size relationship between the first feature map and the image to be identified, to obtain a local feature region in the first feature map, includes: performing sub-pixel level interpolation processing on the first feature map to obtain a sub-pixel feature map; and mapping the position to the sub-pixel feature map according to the size relationship between the first feature map and the image to be identified, to obtain a local feature region. In this implementation, the local feature region obtained through subpixel-level interpolation can completely and accurately cover the entire feature space corresponding to the target in the subpixel feature map. This avoids the quantization error introduced by the rounding operation of directly assigning the feature values ​​of integer coordinates to floating-point coordinates before ROI pooling in traditional techniques. It avoids quantizing continuous floating-point spatial positions into discrete integer coordinate positions, thereby avoiding positional offset of floating-point coordinates relative to integer coordinates during the quantization process. It avoids region offset due to coordinate rounding and does not miss key feature parts of the target, thus truly restoring the spatial features of the target. Especially for small targets, all positions within this local feature region have corresponding feature values. Whether the coordinates are integer or floating-point, they can be directly obtained from the subpixel feature map, improving the accuracy of the target features represented by the feature values ​​in the local feature region.

[0098] See Figure 2 Step S103 may include steps S1031-S1032.

[0099] S1031, perform sub-pixel level interpolation on the first feature map to obtain a sub-pixel feature map.

[0100] Subpixel level refers to coordinates relative to the original pixel-level integer coordinates in the first feature map. The feature values ​​of the first feature map are only stored at positions where both the horizontal and vertical coordinates are integers. These integer coordinates constitute the basic grid of the feature map. Subpixel level, on the other hand, points to all non-integer coordinates between these grids, i.e., floating-point coordinates, which are the subdivisions between integer coordinates. For example, a non-integer coordinate could be a position with a horizontal coordinate of 1.25 and a vertical coordinate of 1.25, or a position with a horizontal coordinate of 2.5 and a vertical coordinate of 2.5. Both of these fall within the subpixel level coordinate range. Simply put, subpixel level is a finer subdivision of the original integer pixel coordinates of the feature map, expanding the coordinate space of the feature map from a discrete set of integers to a continuous set of real numbers.

[0101] Interpolation is a numerical calculation method that derives unknown data from known data. In the feature map processing scenario of this step, it is the process of deriving the feature values ​​of unknown sub-pixel level coordinates in the feature map based on the known integer coordinates in the first feature map and through specific operation rules. Through this processing, the first feature map, which originally only had discrete integer coordinates with feature values, can be transformed into a continuous feature space where all coordinate positions (whether integer or non-integer coordinates) have corresponding feature values.

[0102] Specifically, the electronic device can first identify the sub-pixel level coordinate positions of all the sub-pixel level coordinates that need to be supplemented with feature values ​​in the first feature image. For example, there are three sub-pixel level coordinates evenly distributed between every two adjacent integer coordinate values. For instance, there are three sub-pixel level coordinates (1, 1), (1, 1.25), (1, 1.5), and (1, 1.75) between (1, 1) and (1, 2) in the first feature image. Then, for each sub-pixel level coordinate, the feature values ​​of multiple known coordinates surrounding it are combined and weighted according to a preset calculation rule to finally obtain the feature value corresponding to the sub-pixel level coordinate.

[0103] The entire calculation process covers all sub-pixel level coordinate positions of the first feature map. After completing the supplementation of all sub-pixel level coordinate feature values, the first feature map, which originally only contained integer coordinate values, is transformed into a sub-pixel feature map with corresponding feature values ​​for all coordinate positions. This sub-pixel feature map retains all the original feature information of the first feature map, while adding feature values ​​for all sub-pixel level coordinates, forming a continuous feature coordinate space and preserving the refined structural features of small targets.

[0104] In this scheme, there are several ways to implement the above-mentioned sub-pixel level interpolation processing. In one implementation, the electronic device can use bilinear interpolation to first determine the four adjacent integer coordinates around each sub-pixel level coordinate; based on the distance between the sub-pixel level coordinate and the four determined coordinates, a weight is assigned to the feature value of each coordinate; and the feature value of the sub-pixel level coordinate is obtained by weighted summation of the feature values ​​of the four coordinates; wherein, the weight of the feature value of each coordinate is inversely correlated with the distance to be utilized for that coordinate, and the distance to be utilized for that coordinate is the distance between that coordinate and the sub-pixel level coordinate.

[0105] In another implementation, the electronic device can use linear interpolation to first determine the coordinates of two adjacent integer coordinate values ​​in the horizontal direction for each sub-pixel level coordinate. Based on the distance between the sub-pixel level coordinate and the two determined coordinates, a weight is assigned to each determined coordinate. The feature value of the sub-pixel level coordinate is obtained by weighted summation of the feature values ​​of the two coordinates. The weight of the feature value of each coordinate is inversely correlated with the distance to be utilized for that coordinate, and the distance to be utilized for that coordinate is the distance between that coordinate and the sub-pixel level coordinate.

[0106] S1032, according to the size relationship between the first feature map and the image to be identified, the position is mapped to the sub-pixel feature map to obtain the local feature region.

[0107] The electronic device can first obtain the size relationship between the aforementioned first feature map and the image to be recognized. Then, it can obtain the position of the target in the image to be recognized, which represents the target's location within the image. The electronic device will map each coordinate value in the position according to the size relationship to obtain the corresponding mapped coordinates in the sub-pixel feature map. This conversion result will retain the floating-point form completely without any rounding or quantization. Since the sub-pixel feature map is obtained by sub-pixel level interpolation of the first feature map, the two are completely consistent in spatial coordinate system. Furthermore, the sub-pixel feature map has already supplemented the feature values ​​of all floating-point coordinates. Therefore, the mapped coordinates can be adapted to the coordinate space of the sub-pixel feature map. The electronic device will directly match the floating-point mapped coordinates to the sub-pixel feature map to obtain a local feature region containing the feature values ​​of the sub-pixel level coordinates.

[0108] In this implementation, the local feature region can completely and accurately cover the entire feature space corresponding to the target in the sub-pixel feature map. This avoids the quantization error introduced by the rounding operation of assigning the feature value of the integer coordinate to the floating coordinate before the pooling of the region of interest (ROI) in traditional techniques. It will not cause the region to shift due to coordinate rounding, nor will it miss the key feature parts of the target. At the same time, all positions in this local feature region have corresponding feature values. Whether it is an integer coordinate or a floating coordinate, it can be directly obtained from the sub-pixel feature map, which lays the foundation for accurate classification.

[0109] The process of obtaining local feature regions based on sub-pixel level interpolation described above can be represented by the following formula: f_roi=RoIAlign(F_i, ,output_size=h×w). Where f_roi represents the local feature region; RoIAlign() represents the region of interest alignment process, i.e., ROI alignment, where the region of interest alignment process includes: determining the local feature region based on sub-pixel level interpolation, and magnifying the local feature region; F_i represents the first feature map; This indicates the position of the local feature region in the first feature map; output_size=h×w indicates that the size of the output local feature region is h in width and w in height.

[0110] In this embodiment of the invention, physical cropping of the original image to be identified and forced scaling of the cropped image are avoided. Subpixel-level alignment is achieved in the feature space by ROI alignment, preserving the original discrimination information to the greatest extent and achieving high-fidelity feature expression.

[0111] In one possible embodiment, a target classification method provided by this invention, wherein classifying the target based on the local feature region to obtain the target category includes: enlarging the local feature region by a set size to obtain a second feature map; and classifying the target based on the second feature map to obtain the target category. In this embodiment, by enlarging the local feature region to obtain the second feature map, the feature distribution, details, and spatial correlation of the local feature region are preserved, while increasing the amount of information in the feature map used for classification, thereby improving the accuracy of target classification.

[0112] See Figure 3 Step S104 includes steps S1041-S1042.

[0113] S1041, the local feature region is enlarged according to the set size to obtain the second feature map.

[0114] Electronic devices can acquire a set size. In one implementation, if the second feature map needs to be classified using a classification layer of a classification model, the set size can be determined based on the input size of the subsequent classification layer. For example, the set size can be the input size of the classification layer, so that the enlarged second feature map can fit the input size of the classification layer and can be directly input into the classification layer for category determination. In another implementation, the set size can be a pre-set size larger than the size of the local feature region, without considering whether a classification layer is input or its input size. By expanding the size of the local feature region with a preset size, the second feature map has more feature information than the local feature region.

[0115] Electronic devices can magnify a local feature region to a set size. For each new location resulting from this magnification, an interpolation algorithm is used to calculate the weighted sum of the feature values ​​of nearby locations to obtain the feature value of that new location, thus generating a second feature map. The interpolation algorithm can be a bilinear interpolation algorithm or a linear interpolation algorithm, etc.

[0116] In one implementation, the local feature region can be obtained through sub-pixel level interpolation. For each new position, the electronic device can determine the coordinates closest to that new position. The determined coordinates can be floating-point coordinates or integer coordinates. Then, the weighted sum of the feature values ​​of the determined coordinates is calculated to obtain the feature value of the new position.

[0117] This process is based on the feature values ​​of all locations within a local feature region, including feature values ​​of both integer and floating-point coordinates. Through interpolation, sub-pixel feature values ​​corresponding to the magnification ratio are added between the original feature points. This allows the magnified second feature map to fully retain the feature distribution, detailed information, and spatial correlation of the original local feature region within a set size. It will not miss the key features of the target, nor will it introduce new feature deviations due to the magnification operation.

[0118] The entire magnification process is completed in the continuous feature space of the sub-pixel feature map. Relying on the complete feature value distribution formed by the previous sub-pixel interpolation processing, each position in the magnified second feature map has a precise corresponding feature value, and the relative spatial position of the feature value is consistent with the original local feature region. While obtaining the second feature map containing feature information, the accuracy of the second feature map is improved, providing a feature map with size adaptation and accurate feature information for subsequent classification operations.

[0119] In another implementation, the local feature region can be obtained by directly mapping the first feature map without subpixel-level interpolation. For each new position, the electronic device can determine the coordinates closest to that new position. The determined coordinates can all be integer coordinate values. Then, the weighted sum of the feature values ​​of the determined coordinates is calculated to obtain the feature value of the new position.

[0120] S1042, classify based on the second feature map to obtain the category of the target.

[0121] The second feature map obtained by magnifying the local feature region contains detailed feature information of the target and the correlation features between the target and the background. The electronic device can process the second feature map based on the classification layer of the classification model, whereby the classification layer of the classification model can classify the target based on the feature values ​​in the second feature map.

[0122] In one implementation, the electronic device can input the second feature map into the classification layer of the classification model. The classification layer can perform pooling and normalization operations on the second feature map, converting the feature information contained in the second feature map into probability values ​​corresponding to each category. The probability value is positively correlated with the probability that the target corresponding to the second feature map belongs to the corresponding category. Subsequently, the electronic device determines the final category of the target based on the probability values ​​of each category output by the classification layer. For example, the electronic device can select the category with the highest probability value as the category of the target.

[0123] In another implementation, the electronic device can further extract features from the second feature map to obtain a new feature map, and then perform classification based on the new feature map. This implementation will be described in detail in subsequent embodiments, and will not be elaborated on here.

[0124] In this embodiment of the invention, a second feature map is obtained by magnifying sub-pixel-level local feature regions. This preserves the feature distribution, details, and spatial correlation of the local feature regions while reducing noise introduced by magnification. It minimizes the smoothing or even distortion of target edges and texture details, thus avoiding resolution and feature representation loss and improving the accuracy of target classification. Furthermore, features of local feature regions are extracted and reconstructed using ROI alignment on the intermediate layer feature map, maintaining feature scale consistency and supporting subsequent deep semantic modeling, achieving precise feature-level alignment and reconstruction.

[0125] In one possible embodiment, in a target classification method provided by this invention, the step of performing cascaded multiple feature extractions on the image to be identified to obtain a first feature map includes: inputting the image to be identified into the first feature extraction layer of a first classification model, such that the first x cascaded feature extraction layers in the first classification model are cascaded for feature extraction, to obtain a first feature map output by the xth feature extraction layer, wherein the first classification model is trained based on sample images and the true categories of sample targets in the sample images; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model further includes a classification layer; the step of classifying based on the second feature map to obtain the category of the target includes: inputting the second feature map into the (x+1)th feature extraction layer of the first classification model for feature extraction, to obtain a third feature map output by the last feature extraction layer; inputting the third feature map into the classification layer for category recognition to obtain the category of the target. In this embodiment of the invention, the entire process of feature processing and classification is completed by relying on the original cascaded feature extraction layer and classification layer of the first classification model without making any changes to the model structure. While accurately classifying based on the feature map output by the intermediate feature extraction layer of the model, there is no need to retrain the model. The pre-trained model parameters can be reused directly, reducing the complexity of operation and development costs.

[0126] The first classification model is trained based on sample images and the true categories of the target objects within those images. During training, sample images are input into the model to obtain the predicted target category output by the model. The difference between the predicted category and the true category of the target object is then compared to obtain a loss value. The network parameters of the first classification model are then adjusted based on this loss value. The first classification model can be a convolutional neural network or a deep residual network, etc., and each feature extraction layer can be a downsampling layer or an upsampling layer, etc. This embodiment of the invention does not specifically limit the specific features; it is merely illustrative.

[0127] The first classification model comprises cascaded feature extraction layers and classification layers. An electronic device can input the image to be recognized into the first feature extraction layer of the first classification model to obtain the first feature map output by the x-th feature extraction layer in the first classification model. Here, x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model. For example, x could be half the total number of feature extraction layers in the first classification model. See also... Figure 4 The first classification model contains 5 cascaded feature extraction layers, where x is 3, x is greater than 1 and less than 5, the first feature extraction layer is feature extraction layer a, and the xth feature extraction layer is feature extraction layer c.

[0128] See Figure 5 Step S102 includes step S1021, and step S1042 includes steps S1042a-S1042b.

[0129] S1021, the image to be identified is input into the first feature extraction layer in the first classification model, so that the first x feature extraction layers arranged in the first classification model are cascaded together to extract features, and the first feature map output by the xth feature extraction layer is obtained.

[0130] The electronic device can process the image to be recognized through the feature extraction layer of the first classification model to obtain a first feature map. The first classification model can be a model used to classify targets in the image, and it is pre-trained in the manner described above.

[0131] The first classification model can contain multiple cascaded feature extraction layers, and a first feature map can be obtained through these layers. To ensure that the extracted feature map contains the global context information of the entire image, the first classification model contains at least two feature extraction layers. Furthermore, to avoid losing shallow structural details of the target in the image while performing deeper semantic feature extraction, the electronic device can acquire the feature map output by the intermediate feature extraction layers of the first classification model. That is, the electronic device can input the image to be recognized into the first feature extraction layer, extract features through the first x cascaded feature extraction layers (x > 1 and x < the maximum number of feature extraction layers in the first classification model), and obtain the feature map output by the intermediate feature extraction layers. For example, the number of intermediate feature extraction layers can be half the total number of feature extraction layers in the first classification model.

[0132] For example, the relationship between the image to be recognized and the feature map output by the feature extraction layer can be expressed by the following formula: ,in, This indicates that the first classification model has four cascaded feature extraction layers. Indicates the first Feature maps output by each feature extraction layer Indicates passing through the first The feature extraction layer for the first Feature extraction is performed on the feature maps output by each feature extraction layer. This represents the image to be identified.

[0133] For example, such as Figure 4 As shown, the first classification model includes cascaded feature extraction layers a, b, c, d, and e. The first classification model contains five cascaded feature extraction layers: feature extraction layer a is the first feature extraction layer, feature extraction layer c is the x-th feature extraction layer, feature extraction layer d is the third feature extraction layer, and x is 3, where x is greater than 1 and less than 5. The electronic device can input the image to be recognized into feature extraction layer a to obtain the feature map output by feature extraction layer a; then input the feature map output by feature extraction layer a into feature extraction layer b to obtain the feature map output by feature extraction layer b; then input the feature map output by feature extraction layer b into feature extraction layer c to obtain the feature map output by feature extraction layer c. Since feature extraction layer c is the x-th feature extraction layer, the feature map output by feature extraction layer c can be used as the first feature map.

[0134] S1042a, the second feature map is input into the (x+1)th feature extraction layer in the first classification model for feature extraction, and the third feature map output by the last feature extraction layer is obtained.

[0135] The local feature region is a part of the first feature map. The size of the local feature region is smaller than that of the first feature map. If the subsequent structure of the first classification model is used for classification, the size of the local feature region can be synchronized to the input size of the subsequent structure of the first classification model. In other words, the local feature region is enlarged to the size of the first feature map to obtain the second feature map. That is, the size is set to the size of the first feature map.

[0136] For example, such as Figure 4 As shown, feature extraction layer d is the (x+1)th feature extraction layer. The electronic device can magnify the local feature region in the first feature map output by feature extraction layer c to obtain the second feature map, and then input the second feature map into feature extraction layer d to obtain the feature map output by feature extraction layer d; then input the feature map output by feature extraction layer d into the last feature extraction layer e in the first classification model to obtain the third feature map output by feature extraction layer e.

[0137] S1042b inputs the third feature map into the classification layer for category recognition to obtain the category of the target.

[0138] Electronic devices can input a third feature map into the classification layer of a first classification model for category recognition to obtain the target's category. The classification layer can perform pooling and normalization on the input feature map to obtain the probability value of each target belonging to each category, and then use the category with the highest probability value as the target's category. The classification layer can perform global average pooling and fully connected classification on the third feature map. For example,... Figure 4 As shown, the third feature map output by the feature extraction layer e can be input into the classification layer for category recognition.

[0139] In this embodiment of the invention, since the second feature map has the same input size as the (x+1)th feature extraction layer, features can be directly extracted from the second feature map through the (x+1)th feature extraction layer. This allows for further processing based on the subsequent structure of the first classification model, obtaining deeper global semantic information and providing deeper semantic features to improve classification accuracy. Simultaneously, relying on the existing cascaded feature extraction and classification layers of the first classification model, the entire process of feature processing and classification is completed. A processing flow targeting the feature map is inserted during the first classification model's processing. While accurately classifying based on the feature map output from the intermediate feature extraction layers, the model remains unchanged. Following a deep feature interaction classification method, combined with the classification model's own network structure and additional inserted feature processing, the pre-trained model parameters can be directly reused, reducing operational complexity and development costs.

[0140] In one possible embodiment, the target classification method provided by this invention includes the following steps: performing cascaded feature extraction on the image to be identified to obtain a first feature map. This includes: inputting the image to be identified into the first feature extraction layer of a second classification model, such that the first x cascaded feature extraction layers in the second classification model are cascaded to extract features, resulting in the first feature map output by the xth feature extraction layer. The second classification model is trained based on sample images and the true categories of sample targets in the sample images; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model further includes an adaptive pooling layer. The step of classifying the target based on the local feature region to obtain its category includes: inputting the local feature region into the adaptive pooling layer for category recognition to obtain the target's category. In this embodiment, the adaptive pooling layer can automatically adapt to any size of the local feature region, eliminating the need for preprocessing operations such as magnification of the local feature region, saving intermediate processing steps, achieving lightweight and fast feature processing, reducing computational overhead, and improving classification inference efficiency.

[0141] The second classification model is trained based on sample images and the true categories of the target objects within those images. It includes cascaded feature extraction layers and classification layers. The electronic device can input the image to be recognized into the first feature extraction layer of the second classification model to obtain the first feature map output by the x-th feature extraction layer, where x represents a natural number greater than 1 and is less than the maximum number of feature extraction layers in the second classification model. The training method of the second classification model used in this embodiment is similar to that of the first classification model mentioned in the previous embodiments, and the feature extraction layers in the second classification model can also be similar to those in the first classification model. However, the first classification model does not include an adaptive pooling layer. See also... Figure 6 The second classification model contains multiple cascaded feature extraction layers and an adaptive pooling layer.

[0142] It should be noted that the first classification model and the second classification model used in the target classification method provided in the embodiments of the present invention are not limited. In addition to the feature extraction layer, classification layer and adaptive pooling layer mentioned above, other structures used in the first classification model and the second classification model are optional, and the shape of the feature extraction layer and the shape of the classifier are also optional.

[0143] See Figure 7 Step S102 includes step S1022, and step S104 includes step S1043.

[0144] S1022, the image to be identified is input into the first feature extraction layer in the second classification model, so that the first x feature extraction layers in the second classification model are cascaded together to extract features, and the first feature map output by the xth feature extraction layer is obtained.

[0145] The second classification model includes an adaptive pooling layer. The electronic device can calculate the pooling window size and stride of the adaptive pooling layer based on the size of the feature map input to it. Then, it can process the input feature map according to the calculated pooling window size and stride to achieve target classification. In other words, the adaptive pooling layer can adapt to the size of the input feature map for classification. In contrast, the classification layer in the first classification model can only process feature maps of a fixed size.

[0146] In this embodiment, the size of the local feature region and the input size of the adaptive pooling layer satisfy the following relationship:

[0147] ,in, Represents a local feature region. This indicates the region of interest alignment process, which includes: determining local feature regions based on sub-pixel level interpolation, and magnifying the local feature regions. Represents the first feature map; This indicates the location of the local feature region in the first feature map; output_size= This indicates the size of the output local feature region and its corresponding position in the first feature map. The actual dimensions are consistent, and no additional size changes have been made.

[0148] For example, such as Figure 6 As shown, the classification model contains cascaded feature extraction layers a, b, c, d, and e. The classification model comprises five cascaded feature extraction layers: feature extraction layer a is the first feature extraction layer, feature extraction layer c is the x-th feature extraction layer, and feature extraction layer d is the third feature extraction layer. x is 3, greater than 1 and less than 5. The electronic device can input the image to be recognized into feature extraction layer a to obtain the feature map output by feature extraction layer a; then input the feature map output by feature extraction layer a into feature extraction layer b to obtain the feature map output by feature extraction layer b; then input the feature map output by feature extraction layer b into feature extraction layer c to obtain the feature map output by feature extraction layer c. Since feature extraction layer c is the x-th feature extraction layer, the feature map output by feature extraction layer c can be used as the first feature map.

[0149] S1043, input the local feature region into the adaptive pooling layer for category recognition to obtain the category of the target.

[0150] Electronic devices can directly input local feature regions from the first feature map into an adaptive pooling layer. The adaptive pooling layer automatically adapts to the size of the local feature regions, performing pooling and linear classification to obtain the probability value of each target belonging to each category. Then, the category with the highest probability value is taken as the category of that target. For example, such as... Figure 6 As shown, the electronic device does not input the local feature regions in the first feature map into the feature extraction layer d again, but directly inputs the local feature regions in the first feature map into the adaptive pooling layer.

[0151] In this embodiment of the invention, the original structure of the second classification model is not modified, and the pre-trained model parameters are directly reused without additional training and debugging, making the operation simpler. The adaptive pooling layer can automatically adapt to any size of the local feature region without the need for preprocessing operations such as magnification of the local feature region, eliminating intermediate processing steps and achieving lightweight and fast feature processing, which can reduce computational overhead and improve classification inference efficiency.

[0152] Furthermore, since this embodiment of the invention does not modify the original structure of the second classification model, it can be easily embedded into existing object detection frameworks. It is a lightweight and fast classification method that utilizes features extracted by the feature extraction layer in existing object detection frameworks to improve the accuracy of object classification while enhancing the applicability of the solution, possessing end-to-end integration potential. Moreover, both the previous embodiment and this embodiment provide a deep feature interaction method and a lightweight and fast classification method, allowing users to choose the appropriate method based on their actual application scenarios. This multi-strategy implementation mechanism balances high accuracy requirements with engineering deployment efficiency.

[0153] See Figure 8 Target classification methods can include the following steps:

[0154] Step 1: Input the complete image and target bounding box.

[0155] Electronic devices can acquire the location of the target in the image to be identified and the image to be identified, and then input the image to be identified into the first classification model.

[0156] Step 2: Extract features from the backbone network.

[0157] Electronic devices can extract features from images to be recognized through multiple cascaded feature extraction layers in the first classification model.

[0158] Step 3: Locate the target in the intermediate feature layer.

[0159] The electronic device can select the first feature map output by the middle feature extraction layer from the feature maps output by multiple cascaded feature extraction layers. The middle feature extraction layer can be any feature extraction layer other than the first and last ones; that is, the middle feature extraction layer is the x-th feature extraction layer, where x represents a natural number greater than 1 and is less than the maximum number of feature extraction layers in the first classification model. Then, according to the size relationship between the first feature map and the image to be recognized, the position of the target in the image to be recognized is mapped to the first feature map, obtaining the local feature region in the first feature map.

[0160] Step 4: Execute the region of interest alignment feature.

[0161] In the process of acquiring local feature regions, the electronic device can perform subpixel-level interpolation on the first feature map to obtain a subpixel feature map; according to the size relationship between the first feature map and the image to be recognized, the position is mapped to the subpixel feature map to obtain the local feature region. Then, the local feature region is enlarged according to a set size to obtain the second feature map.

[0162] Step 5: Input the data into the subsequent network for classification.

[0163] The electronic device can input the second feature map into the (x+1)th feature extraction layer in the first classification model for feature extraction, and obtain the third feature map output by the last feature extraction layer; input the third feature map into the classification layer for category recognition to obtain the category of the target.

[0164] In this embodiment of the invention, since feature extraction is performed on the complete image to be identified, each position in the first feature map contains contextual information within its receptive field. This allows for full utilization of the semantics of the target's surrounding environment (such as fingers around a cigarette butt or smoke around a flame) to assist in the discrimination. Furthermore, it avoids physical cropping and forced scaling operations at the image level. By achieving sub-pixel alignment in the feature space, the accuracy of feature values ​​in local feature regions is improved.

[0165] In experiments such as smoking detection and fire detection, the embodiments of the present invention outperform existing technologies in terms of performance evaluation metrics for classification tasks. For example, the area under the curve (AUROC) of the embodiments of the present invention is closer to 1, which can more effectively distinguish between targets and interference objects. Especially in scenarios with out-of-distribution (OOD) objects such as pens and lights, it is easier to detect the difference between cigarette butts and out-of-distribution objects, thereby more accurately identifying cigarette butt targets and significantly reducing false alarm rates.

[0166] See Figure 9 This invention provides a target classification device, the device comprising:

[0167] The first acquisition module 910 is used to acquire the position of the target in the image to be recognized;

[0168] The extraction module 920 is used to perform cascaded feature extraction on the image to be identified multiple times to obtain a first feature map;

[0169] The mapping module 930 is used to map the position to the first feature map according to the size relationship between the first feature map and the image to be identified, so as to obtain a local feature region in the first feature map;

[0170] The classification module 940 is used to classify the target based on the local feature region to obtain the category of the target.

[0171] Optionally, the mapping module is specifically used to perform sub-pixel level interpolation processing on the first feature map to obtain a sub-pixel feature map; and to map the position to the sub-pixel feature map according to the size relationship between the first feature map and the image to be identified to obtain a local feature region.

[0172] Optionally, the classification module includes: a magnification unit for magnifying the local feature region according to a set size to obtain a second feature map; and a classification unit for classifying the target based on the second feature map to obtain the category of the target.

[0173] Optionally, the extraction module is specifically used to input the image to be identified into the first feature extraction layer of the first classification model, so that the first x cascaded feature extraction layers in the first classification model are cascaded for feature extraction to obtain the first feature map output by the xth feature extraction layer. The first classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model also includes a classification layer; the classification unit is specifically used to input the second feature map into the (x+1)th feature extraction layer of the first classification model for feature extraction to obtain the third feature map output by the last feature extraction layer; and to input the third feature map into the classification layer for category recognition to obtain the category of the target.

[0174] Optionally, the extraction module is specifically used to input the image to be identified into the first feature extraction layer in the second classification model, so that the first x feature extraction layers arranged in a cascaded manner in the second classification model are cascaded to extract features, thereby obtaining the first feature map output by the xth feature extraction layer. The second classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model also includes an adaptive pooling layer; the classification module is specifically used to input the local feature region into the adaptive pooling layer for category recognition, thereby obtaining the category of the target.

[0175] In this embodiment of the invention, the position of the target in the image to be identified is obtained, and multiple cascaded feature extractions are performed on the image to be identified to obtain a first feature map. Through multiple cascaded feature extractions, features can be extracted layer by layer from shallow to deep within the entire image to be identified. Shallow feature extraction can obtain the detailed features of the target itself in the image to be identified, while deep feature extraction can obtain the global context features of the entire image. Therefore, the first feature map can contain both the target's own features and the features related to the target and its surrounding background. Then, according to the size relationship between the first feature map and the image to be identified, the position of the target is mapped onto the first feature map to obtain a local feature region in the first feature map. That is, the local feature region originates from the first feature map, and the position of the local feature region corresponds to the position of the target in the image to be identified. Thus, the local feature region can contain both the target's own features at the corresponding position and the features related to the target and its background. Therefore, classification based on the local feature region can be performed based on the target's own features and the features related to the target and its background. Especially for small targets with few visual features, this solution can classify them based on the characteristics of the small target itself in the local feature region, as well as the correlation features between the small target and the background. It can effectively distinguish small targets from interference objects based on more feature information, minimize the possibility of misclassification of target categories, and improve the accuracy of target classification.

[0176] This invention also provides an electronic device, such as... Figure 10 As shown, the system includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004. The processor 1001, communication interface 1002, and memory 1003 communicate with each other via the communication bus 1004. The memory 1003 stores computer programs; the processor 1001 executes the programs stored in the memory 1003 to implement the aforementioned target classification method.

[0177] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices. The memory can include Random Access Memory (RAM), or Non-Volatile Memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0178] The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. In another embodiment provided by the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-mentioned target classification methods.

[0179] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the target classification methods described above.

[0180] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0181] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0182] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, electronic devices, storage media, and products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0183] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A target classification method, characterized in that, The method includes: Obtain the location of the target in the image to be identified; The image to be identified is subjected to multiple cascaded feature extractions to obtain a first feature map; Based on the size relationship between the first feature map and the image to be identified, the position is mapped to the first feature map to obtain a local feature region in the first feature map; The target category is obtained by classifying based on the local feature regions.

2. The method according to claim 1, characterized in that, The step of mapping the position to the first feature map according to the size relationship between the first feature map and the image to be identified, to obtain a local feature region in the first feature map, includes: The first feature map is subjected to subpixel-level interpolation to obtain a subpixel feature map. Based on the size relationship between the first feature map and the image to be identified, the position is mapped to the sub-pixel feature map to obtain a local feature region.

3. The method according to claim 1 or 2, characterized in that, The classification based on the local feature regions to obtain the category of the target includes: The local feature region is enlarged according to a set size to obtain a second feature map; The target category is obtained by classifying based on the second feature map.

4. The method according to claim 3, characterized in that, The step of performing cascaded feature extraction on the image to be identified to obtain a first feature map includes: The image to be identified is input into the first feature extraction layer of the first classification model, so that the first x cascaded feature extraction layers in the first classification model are cascaded to extract features, resulting in the first feature map output by the xth feature extraction layer. The first classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model also includes a classification layer. The classification based on the second feature map to obtain the category of the target includes: The second feature map is input into the (x+1)th feature extraction layer in the first classification model for feature extraction, resulting in the third feature map output by the last feature extraction layer. The third feature map is input into the classification layer for category identification to obtain the category of the target.

5. The method according to claim 1 or 2, characterized in that, The step of performing cascaded feature extraction on the image to be identified to obtain a first feature map includes: The image to be identified is input into the first feature extraction layer of the second classification model, so that the first x cascaded feature extraction layers in the second classification model are cascaded to extract features, resulting in the first feature map output by the xth feature extraction layer. The second classification model is trained based on the sample image and the true category of the sample target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model also includes an adaptive pooling layer. The classification based on the local feature regions to obtain the category of the target includes: The local feature region is input into the adaptive pooling layer for category recognition to obtain the category of the target.

6. A target classification device, characterized in that, The device includes: The first acquisition module is used to acquire the location of the target in the image to be recognized; The extraction module is used to perform cascaded feature extraction on the image to be identified multiple times to obtain a first feature map; The mapping module is used to map the position to the first feature map according to the size relationship between the first feature map and the image to be identified, so as to obtain a local feature region in the first feature map; The classification module is used to classify the target based on the local feature regions to obtain the category of the target.

7. The apparatus according to claim 6, characterized in that, The mapping module is specifically used to perform subpixel-level interpolation on the first feature map to obtain a subpixel feature map; and to map the position to the subpixel feature map according to the size relationship between the first feature map and the image to be identified to obtain a local feature region. And / or, The classification module includes: a magnification unit for magnifying the local feature region according to a set size to obtain a second feature map; and a classification unit for classifying the target based on the second feature map to obtain the category of the target. or, The extraction module is specifically used to input the image to be identified into the first feature extraction layer of the first classification model, so that the first x cascaded feature extraction layers in the first classification model are cascaded to extract features, resulting in a first feature map output by the xth feature extraction layer. The first classification model is trained based on sample images and the true categories of sample targets in the sample images; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the first classification model; the classification model also includes a classification layer; the classification unit is specifically used to input the second feature map into the (x+1)th feature extraction layer of the first classification model for feature extraction, resulting in a third feature map output by the last feature extraction layer; the third feature map is then input into the classification layer for category recognition to obtain the category of the target. or, The extraction module is specifically used to input the image to be identified into the first feature extraction layer of the second classification model, so that the first x cascaded feature extraction layers in the second classification model are cascaded to extract features, resulting in a first feature map output by the xth feature extraction layer. The second classification model is trained based on the sample image and the true category of the target in the sample image; x represents a natural number greater than 1, and x is less than the maximum number of feature extraction layers in the second classification model; the second classification model also includes an adaptive pooling layer; the classification module is specifically used to input the local feature region into the adaptive pooling layer for category recognition, thereby obtaining the category of the target.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images