Method, electronic device and storage medium for classification and localization of image objects

Abstract

The present disclosure relates to a method, an electronic device and a storage medium for image target classification and positioning. The method comprises: acquiring an image to be processed; performing feature extraction processing, feature fusion processing and prediction processing on the image to obtain a feature map with a region frame and a category of each target; performing mask discrimination processing on the feature map to obtain a mask of each target in the image; and performing spatial positioning processing on the feature map to obtain an inference result of a space map formed by each target, and correcting the region frame position and the category of each target. The method of the present disclosure can improve the accuracy and speed of image target classification and positioning operations.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to methods, electronic devices, and storage media for classifying and locating targets in images. Background Technology

[0002] Object detection, which aims to identify all objects of interest in an image and determine their category and location, is one of the core problems in computer vision. Due to the different appearances, shapes, and poses of various objects, coupled with interference from factors such as lighting and occlusion during imaging, object detection has always been one of the most challenging problems in computer vision. Summary of the Invention

[0003] This disclosure provides methods for classifying and locating targets in images, as well as corresponding electronic devices, computing devices, and non-transitory machine-readable storage media for performing these methods.

[0004] According to a first aspect of this disclosure, a method for classifying and locating targets in an image is provided, comprising: acquiring an image to be processed; performing feature extraction, feature fusion, and prediction processing on the image to obtain a feature map with region boxes and categories of each target; performing mask discrimination processing on the feature map to obtain masks of each target in the image; performing spatial localization processing on the feature map to obtain inference results of spatial maps formed by each target, and correcting the region box positions and categories of each target.

[0005] Optionally, the feature extraction process is a multi-scale feature extraction, including: sequentially performing a first phantom processing and channel attention processing on the intermediate feature map of at least one stage of input; and performing a first concatenation processing on the intermediate feature map and the intermediate feature map after the first phantom processing and the channel attention processing.

[0006] Optionally, the feature map is subjected to mask discrimination processing to obtain the mask of each target in the image, including: firstly, edge pixel extraction is performed on the feature map, and then the extracted edge pixels are predicted and judged to obtain the category of the target in the image; the edge pixel extraction includes: performing edge feature enhancement processing on the feature map to obtain edge feature pixels, selecting any k pixels around the edge feature pixels for independent prediction, and upsampling the edge feature pixels determined after independent prediction by transpose convolution and bilinear interpolation to obtain finer edge pixel curves, thereby extracting the edge pixels, where k is a natural number; the prediction and judgment includes: performing pixel block correction processing on the feature map of each target region using shape and size, and comparing the result of multiplying each pixel value of the extracted edge pixels with the corresponding weights with the threshold values ​​of various label pixels according to the RGB model to determine the category of the edge pixels, thereby obtaining the category of the target.

[0007] Optionally, the feature extraction process further includes: performing two channel-wise convolution processes on the input initial image to obtain a first intermediate feature map; performing convolution, first phantom processing, channel attention processing, and convolution processing on the first intermediate feature map to obtain a second intermediate feature map, and performing a convolution process on the first intermediate feature map to obtain a third intermediate feature map; and concatenating the second intermediate feature map and the third intermediate feature map.

[0008] Optionally, the two channel-wise convolution processes reduce the size of the image in the height and width dimensions and / or increase the number of channels in the image.

[0009] Optionally, the first phantom processing includes: performing convolution processing and channel-by-channel convolution processing sequentially on the intermediate feature map; and concatenating the intermediate feature map after the convolution processing with the intermediate feature map after the convolution processing and channel-by-channel convolution processing sequentially.

[0010] Optionally, the first phantom processing further includes, after the stitching process, performing convolution processing and channel-by-channel convolution processing sequentially on the intermediate feature map after the stitching process; and stitching the intermediate feature map after the convolution process with the intermediate feature map after the channel-by-channel convolution process, wherein the stitched intermediate feature map is used as the input to the channel attention processing.

[0011] Optionally, the channel attention processing includes: sequentially performing global pooling, convolution, and sigmoid function processing on the intermediate feature map; and performing element-wise multiplication of the weight values ​​of each channel obtained after the sigmoid function processing with the feature maps of the corresponding channels of the intermediate feature map.

[0012] Optionally, the feature fusion processing includes feature pyramid network processing, which includes: performing upsampling processing on the smallest-size feature map obtained by the feature extraction processing; concatenating the upsampled feature map with the second smallest-size feature map to obtain a fused feature map; performing a second phantom processing on the fused feature map; wherein the second phantom processing includes: performing two convolution processing and one channel-wise convolution processing on the fused feature map in sequence to obtain a first fused feature map; performing another convolution processing on the fused feature map after the aforementioned first convolution processing to obtain a second fused feature map; and concatenating the first fused feature map with the second fused feature map.

[0013] Optionally, the feature fusion processing further includes path aggregation network processing after the feature pyramid network processing, which includes: downsampling the largest-size fused feature map obtained by the feature pyramid network processing; concatenating the largest-size fused feature map after the downsampling processing with the second-largest-size fused feature map; and performing the second phantom processing on the fused feature map after the concatenation processing.

[0014] Optionally, the spatial localization process includes: learning a sparse adjacency matrix from the visual features of each target in the feature map, wherein each target in the sparse adjacency matrix uses its top t most relevant targets as its adjacency nodes, where t is an integer greater than 1; calculating the spatial information between each target and its adjacent targets based on a Gaussian kernel with learnable parameters to obtain the corresponding Gaussian kernel weights; and performing graph convolution processing on the visual embedding obtained by soft mapping of each target's category, the sparse adjacency matrix, and the Gaussian kernel weights to obtain the inference result of the spatial graph formed by each target, and correcting the region box position and category of each target.

[0015] According to a second aspect of this disclosure, an electronic device is provided, comprising: a camera for capturing images; a processor; a memory storing executable code thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above on the images captured by the camera, thereby obtaining the location and category of a target contained in the images; and a display / warning component for displaying and / or warning to a user accordingly based on the obtained location and category of the target.

[0016] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above. Attached Figure Description

[0018] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0019] Figure 1 A schematic flowchart of a method for classifying and locating image targets according to at least one embodiment of the present disclosure is shown.

[0020] Figure 2 A schematic flowchart of a feature extraction process according to at least one embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic structural diagram of a neural network model portion for one stage of feature extraction processing according to at least one embodiment of the present disclosure is shown as an example.

[0022] Figure 4 A schematic structural diagram of a phantom block in a feature extraction network according to at least one embodiment of the present disclosure is shown as an example.

[0023] Figure 5 A schematic structural diagram of a channel attention block in a feature extraction network according to at least one embodiment of the present disclosure is shown as an example.

[0024] Figure 6 A schematic flowchart of a feature fusion process according to at least one embodiment of the present disclosure is shown.

[0025] Figure 7 A schematic structural diagram of a phantom block in a feature fusion network according to at least one embodiment of the present disclosure is shown as an example.

[0026] Figure 8 A schematic flowchart of edge pixel extraction processing in mask discrimination processing according to at least one embodiment of the present disclosure is shown.

[0027] Figure 9 A schematic flowchart of the prediction judgment process in the mask discrimination process according to at least one embodiment of the present disclosure is shown.

[0028] Figure 10 A schematic flowchart of a spatial positioning process according to at least one embodiment of the present disclosure is shown.

[0029] Figure 11 A schematic diagram of the structure of a computing device according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0030] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0031] Those skilled in the art should understand that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence, and do not have any additional limiting effect.

[0032] Currently, several methods have been proposed for classifying and locating targets in images, aiming to improve the accuracy and speed of target detection, especially in scenarios requiring target detection in images captured in complex environments or with occlusion. The goal is to achieve better classification and localization of image targets. For example, drivers often encounter complex road conditions such as nighttime, fog, rain, road debris, or obstructed views by vehicles. Every year, tens of thousands of traffic accidents occur due to these complex road conditions, resulting in huge economic and personal losses. In other words, complex road conditions pose significant safety hazards to drivers. Therefore, it is beneficial to provide drivers with electronic devices that can accurately identify various targets on the road and provide corresponding displays or warnings even in complex road conditions, assisting drivers in driving more safely and effectively reducing the risk of traffic accidents. Such electronic devices can be in-vehicle devices built into the vehicle or installed by the driver, or portable electronic devices carried by the driver, such as mobile phones or XR glasses. In this scenario, both the accuracy and speed of image target detection are highly demanding.

[0033] In view of this, this application proposes a novel scheme for classifying and locating targets in images, which can improve the accuracy and speed of detection. In particular, the scheme disclosed herein can better determine the category and location of targets in images captured in complex environments or under conditions of occlusion. The scheme disclosed herein will be described in detail below with reference to the accompanying drawings. Those skilled in the art should understand that the application scenarios of the scheme disclosed herein are not limited to the specific scenarios or complex environment scenarios mentioned above and below, but can be various scenarios.

[0034] Figure 1 A schematic flowchart of a method for classifying and locating image targets according to at least one embodiment of the present disclosure is shown.

[0035] like Figure 1 As shown, in step S110, the image to be processed is acquired.

[0036] In some implementations, images can be captured by various electronic devices with cameras (e.g., in-vehicle devices, mobile phones, XR glasses, etc.) and used as images to be processed.

[0037] Then, as Figure 1 As shown, in step S120, the acquired image is subjected to feature extraction, feature fusion and prediction processing to obtain a feature map with region boxes and categories of each target.

[0038] In some implementations, the feature extraction process can be multi-scale feature extraction, that is, extracting feature maps of multiple scales of the image in multiple stages. Then, the feature fusion process can fuse the extracted feature maps of multiple sizes to obtain a fused feature map. Then, the prediction process can predict the location (e.g., represented by bounding boxes) and category of the targets contained in the image based on the obtained fused feature map, and obtain and output a feature map with bounding boxes and categories of all possible targets contained in the image.

[0039] In some implementations, the feature extraction process as multi-scale feature extraction may include: sequentially performing a first phantom processing and channel attention processing on an intermediate feature map of at least one input stage; and performing a first concatenation process on the intermediate feature map and the intermediate feature map after the first phantom processing and the channel attention processing. It should be understood that the term "intermediate" in the term "intermediate feature map" herein is not intended to limit the feature map in any way, but is only used to indicate the feature map in a certain processing step.

[0040] In this field, ghost processing typically refers to the following operation: first, a regular convolution is performed on the input feature map to generate an intrinsic feature map with fewer channels (i.e., fewer channels than the input). Then, a simple linear operation (e.g., channel-wise convolution or shift operation) is performed on this intrinsic feature map to augment it and fill in the missing channels. Finally, this intrinsic feature map is concatenated with the linearly processed intrinsic feature map. In this paper, ghost processing also includes some variants, as detailed later. For example, in some cases, the input feature map may be subjected to two regular convolutions to obtain a first intrinsic feature map, which is then linearly processed. Furthermore, the input feature map may be subjected to two more regular convolutions to obtain a second intrinsic feature map. The second intrinsic feature map is then concatenated with the linearly processed first intrinsic feature map, where the first regular convolution used to obtain these two intrinsic feature maps can be shared.

[0041] In this field, channel attention processing is used to implement the channel attention mechanism. It obtains the weights of each channel based on the input image features and weights the feature maps of each channel accordingly. This allows for the enhancement or suppression of different channels based on their importance, thus focusing more on important channels and suppressing less important ones. Channel attention processing includes squeeze-and-excitation (SE) processing, which typically involves the following steps: First, the input feature map is squeezed, i.e., the two-dimensional feature map of each channel is compressed into a real number. Then, an excitation operation is performed, i.e., the weights of each channel are learned from the compressed feature map to explicitly represent the importance of each channel. Finally, the obtained weight values ​​of each channel are multiplied by the corresponding input channel's feature map to obtain the feature map after channel attention processing.

[0042] In this field, concatenation typically refers to the operation of concatenating two or more input feature maps along the channel dimension. That is, the channels become the sum of the channels of all the feature maps to be concatenated, while the individual feature maps within each channel remain unchanged. This requires the feature maps to be concatenated to have the same scale in all dimensions except for the channels, although their number of channels can differ.

[0043] In some embodiments, the feature extraction process may further include: performing two channel-wise convolutional processes on the input initial image (e.g., the image obtained in step S110) to obtain a first intermediate feature map; performing convolution, first phantom processing, channel attention processing, and convolution processing on the first intermediate feature map to obtain a second intermediate feature map; and performing a single convolution processing on the first intermediate feature map to obtain a third intermediate feature map; and concatenating the second and third intermediate feature maps. In some embodiments, the two channel-wise convolutional processes reduce the size of the image in the height and width dimensions and / or increase the number of channels in the image.

[0044] In some embodiments, the aforementioned first phantom processing may include: sequentially performing convolution processing and channel-wise convolution processing on the intermediate feature map; and concatenating the intermediate feature map after the convolution processing with the intermediate feature map after the sequential convolution processing and channel-wise convolution processing. In some embodiments, the aforementioned first phantom processing may further include, after the concatenation processing, sequentially performing convolution processing and channel-wise convolution processing on the intermediate feature map after the concatenation processing; and concatenating the intermediate feature map after the convolution processing with the intermediate feature map after the channel-wise convolution processing, wherein the concatenated intermediate feature map serves as the input to the channel attention processing.

[0045] In some implementations, the aforementioned channel attention processing includes: sequentially performing global pooling, convolution, and sigmoid function processing on the intermediate feature map; and performing element-wise multiplication of the weight values ​​of each channel obtained after the sigmoid function processing with the feature maps of the corresponding channels of the intermediate feature map.

[0046] For example, Figure 2 An exemplary schematic flowchart of feature extraction processing as a multi-scale feature extraction is shown.

[0047] like Figure 2 As shown, the image to be processed obtained in step S110 can be input, and the input image can be processed continuously in five stages. After processing in stages three through five (S3-S5), feature maps P3-P5 are output respectively. The feature map output after the processing of the previous stage is also an intermediate feature map input to the next stage. Furthermore, at least in stages three through five (S3-S5), an operation is performed to reduce the size of the image's feature maps (e.g., reduce it to half the input size), therefore the extracted feature maps P3-P5 have progressively decreasing sizes.

[0048] Those skilled in the art will understand that intermediate feature maps obtained during feature extraction processing of an image (also referred to as extracted feature maps if output) and intermediate fused feature maps / fusion feature maps obtained during the fusion processing described later have three dimensions: height, width, and channel. In this document, unless otherwise specified, the size / scale of intermediate feature maps / feature maps / intermediate fused feature maps / fusion feature maps described later generally refers to the size / scale in the height and width dimensions. Furthermore, Figure 2 The number of processing stages, the number of stages for outputting feature maps, and the selection of stages for outputting feature maps are merely exemplary and not restrictive. Those skilled in the art can set an appropriate number of stages and at which stages to output feature maps as needed.

[0049] In particular, Figure 2 The improved extraction process according to this disclosure is illustrated in detail using the second stage S2 as an example. Those skilled in the art will understand that, although not shown, the processing in other stages may be similar to or different from the second stage S2. For example, in the initial first stage S1, only depthwise (DW) convolution processing may be performed on the input image, while subsequent stages employ operations similar to the second stage S2.

[0050] like Figure 2 As shown, the extraction process in the second stage S2 includes the sequential execution of phantom processing S210, channel attention processing S220, and splicing processing S230.

[0051] Specifically, the intermediate feature map input to the second stage S2 is subjected to phantom processing S210 and channel attention processing S220 in sequence, and then the input intermediate feature map is concatenated with the intermediate feature map output after channel attention processing S220 in process S230.

[0052] The channel attention processing S220 described above focuses on channels with higher weights, making the extracted image features more effective. Simultaneously, the phantom processing S210 described above compensates for the features lost during the weighted fusion of the channel attention processing S220, improving accuracy and reducing computational load. Furthermore, the stitching processing S230 described above ensures that the output feature map retains a significant amount of input information after one stage of processing, thereby improving the accuracy and other performance aspects of feature extraction. Therefore, the improved feature extraction processing according to this disclosure, as described above, combines speed and accuracy, enabling faster and better image feature extraction.

[0053] In some embodiments, a neural network model can be used to implement the improved feature extraction processing according to this disclosure described above. The neural network model portion used for feature extraction processing (also referred to as the feature extraction network) is generally referred to as the backbone of the entire object detection model.

[0054] Figure 3 An exemplary illustration shows a stage in a feature extraction process according to at least one embodiment of the present disclosure (e.g. Figure 2 A schematic diagram of the neural network model part of the second stage (S2) in the model.

[0055] like Figure 3 As shown, this feature extraction stage (e.g., corresponding to...) Figure 2 The second stage (S2) includes a channel-wise (DW) convolutional layer 310 and an extraction module 320.

[0056] In some embodiments, the DW convolutional layer 310 may have a 3*3 kernel size and a stride of 2, thereby reducing the height and width of the input intermediate feature map by half, achieving downsampling processing.

[0057] In addition, such as Figure 3 As shown, the extraction module 320 includes a phantom block 302, a channel attention block 303, and a splicing layer 305, which are respectively used to implement the above-mentioned... Figure 2 The phantom processing S210, channel attention processing S220, and splicing processing S230 are shown.

[0058] As mentioned earlier, the channel attention block 303 focuses on channels with higher weights, making the extracted image features more effective. Simultaneously, the phantom block 302 compensates for features lost during the weighted fusion of the channel attention block 303, improving model accuracy and reducing model parameters and computational cost. Furthermore, the stitching layer 305 preserves residual connections, effectively preventing gradient vanishing and allowing for a deeper model with stronger representational capabilities, thereby improving model accuracy and other performance aspects. Therefore, the improved feature extraction network according to this disclosure, as described above, achieves both speed and accuracy, enabling faster and better image feature extraction.

[0059] In some embodiments, the phantom block 302 and the channel attention block 303 may respectively have the following characteristics: Figure 4 and Figure 5 The schematic structures are shown respectively.

[0060] like Figure 4As shown, the phantom block 402 comprises two parts enclosed by dashed boxes, each part being used to perform one phantom process. That is, the phantom block 402 is actually used to perform two consecutive phantom processes on the input intermediate feature map. It should be understood that this disclosure is not limited to this, but rather one or more phantom processes can be performed as needed, and other operations can be performed between more than one consecutive phantom process.

[0061] The portion of the phantom block 402 used to perform one phantom processing step may include a convolutional layer 403, a DW convolutional layer 404, and a stitching layer 405 connected in sequence. The stitching layer 405 is used to stitch the output of the convolutional layer 403 with the output of the DW convolutional layer 404. The portion used to perform the next phantom processing step may have a similar structure, namely, it may include a convolutional layer 406, a DW convolutional layer 407, and a stitching layer 408 connected in sequence. The stitching layer 408 is used to stitch the output of the convolutional layer 406 with the output of the DW convolutional layer 407.

[0062] like Figure 5 As shown, the channel attention block 503 includes a global pooling layer 504, a convolutional layer 505, a sigmoid layer 506, and an element-wise multiplication layer 507 connected in sequence. The global pooling layer 504, the convolutional layer 505, and the sigmoid layer 506 are used to obtain the weight values ​​of each channel based on the intermediate feature map of the input, while the element-wise multiplication layer 507 is used to multiply the weight values ​​of each channel with the feature map of the corresponding input channel element-wise.

[0063] In this field, global pooling typically refers to transforming the feature map of each channel from a two-dimensional matrix into a numerical value. Therefore, the entire feature map after global pooling is converted into a one-dimensional vector, where each value represents the feature map of a channel. Global pooling typically includes global average pooling and global max pooling. Global average pooling calculates and outputs the mean of all points in the feature map of each channel, while global max pooling takes the maximum value among all points in the feature map of each channel and outputs it. Therefore, in some embodiments, the global pooling layer 504 can be used to perform global average pooling or global max pooling, or both.

[0064] In this field, Sigmoid layers are typically used to perform Sigmoid function calculations on the input, thereby restricting the output value to between 0 and 1.

[0065] In this example, only one convolutional layer (i.e., convolutional layer 505) is used to learn the weight values ​​of each channel. Therefore, compared to the SE model in the prior art which uses two fully connected layers, Figure 5 The channel attention block 503 can reduce the model's parameters and computational load, thereby improving the model's speed.

[0066] Additionally, returning Figure 3 In some embodiments, in addition to the aforementioned phantom block 302, channel attention block 303, and stitching layer 305, the extraction module 320 may further include a convolutional layer 301 connected before the phantom block 302, a convolutional layer 306 connected between the input of the extraction module 320 and the stitching layer 305, a convolutional layer 304 connected between the channel attention block 303 and the stitching layer 305, and a convolutional layer 307 connected after the stitching layer 305. These convolutional layers can increase the depth of the model and improve its accuracy.

[0067] Furthermore, although not shown in this disclosure, those skilled in the art will understand that a non-linear processing / layer, namely an activation processing / layer (e.g., using activation functions such as SiLU or ReLU), is usually followed by a convolutional processing or convolutional layer. Moreover, when implementing this using a neural network model, batch normalization (BN) layers can often be added between at least some convolutional layers and their respective activation layers as needed, thereby facilitating model training. Note that the aforementioned convolutional processing / layers include not only conventional convolutional processing / layers but also other convolutional processing / layers, such as channel-wise convolutional processing / layers.

[0068] As mentioned earlier, in some implementations, after extracting multiple feature maps of different scales using feature extraction processing, feature fusion processing can be performed next. In feature fusion processing, at least two of these multiple feature maps can be fused to obtain one or more fused feature maps.

[0069] In this field, fusion processing typically refers to the following operation: after upsampling and / or downsampling two or more feature maps of different scales to make them into feature maps of the same size, information merging operations such as addition or concatenation are performed to fuse the information of these feature maps of different scales together. In some cases, various convolutional processes can be selectively performed before and after upsampling / downsampling operations, or before and after addition or concatenation operations, to facilitate and / or enhance the fusion processing.

[0070] In some embodiments, when fusing multiple feature maps of different scales, a top-down (e.g., Feature Pyramid Network (FPN)) and / or bottom-up (e.g., Path Aggregation Network (PAN)) fusion approach can be used. Top-down fusion refers to fusing feature maps from the smallest size to those with progressively larger sizes, while bottom-up fusion refers to fusing feature maps from the largest size to those with progressively smaller sizes.

[0071] In this field, top-down FPN processing typically refers to the following operations: starting with the top-level (i.e., smallest size) feature map, it is upsampled and fused with the feature map of the next layer. The resulting fused feature map is then upsampled again and fused with the feature map of the next layer below, and so on, until the bottom-level feature map is completely fused. The top-level feature map (or its variant after convolution, etc.) and the fused feature map obtained from each fusion are also output for subsequent processing (e.g., subsequent bottom-up PAN processing or prediction processing). Bottom-up PAN processing typically refers to the following operations: starting with the bottom-level (i.e., largest size) feature map (or fused feature map), it is downsampled and fused with the feature map of the previous layer. The resulting fused feature map is then downsampled again and fused with the feature map of the next layer above, and so on, until the top-level feature map is completely fused. The bottom-level feature map (or its variant after convolution, etc.) and the fused feature map obtained from each fusion are also output for subsequent processing (e.g., subsequent prediction processing).

[0072] In some embodiments, the aforementioned feature fusion processing may include feature pyramid network processing, which includes: performing upsampling processing on the smallest-sized feature map obtained from the feature extraction processing; concatenating the upsampled feature map with the second-smallest-sized feature map to obtain a fused feature map; performing a second phantom processing on the fused feature map; wherein the second phantom processing includes: performing two convolution processing and one channel-wise convolution processing on the fused feature map sequentially to obtain a first fused feature map; performing another convolution processing on the fused feature map after the first convolution processing to obtain a second fused feature map; and concatenating the first fused feature map with the second fused feature map. In some embodiments, the aforementioned feature fusion processing may further include path aggregation network processing after the feature pyramid network processing, which includes: performing downsampling processing on the largest-sized fused feature map obtained from the feature pyramid network processing; concatenating the downsampled largest-sized fused feature map with the second-largest-sized fused feature map; and performing the second phantom processing on the concatenated fused feature map.

[0073] For example, Figure 6 A schematic flowchart of a feature fusion process according to at least one embodiment of the present disclosure is shown.

[0074] like Figure 6 As shown, the feature fusion processing 600 includes FPN processing 610 and PAN processing 620. FPN processing 610 performs fusion processing on the feature maps P3-P5 of multiple scales extracted by feature extraction processing, starting from the smallest feature map P5 and gradually moving towards the feature maps with progressively larger sizes. PAN processing 620 performs fusion processing on the fused feature maps M3-M4 of multiple scales obtained by FPN processing 610 in each fusion stage and the smallest feature map P5' after convolution processing S611, starting from the largest fused feature map and gradually moving towards the fused feature maps or feature maps with progressively smaller sizes.

[0075] Specifically, in some embodiments, the FPN processing 610 is in such... Figure 6 The first fusion stage S11 may include: upsampling processing S612, and splicing processing S613, which splices the upsampled feature map together with the feature map P4 to be fused in the current fusion stage.

[0076] In addition, to improve the fusion effect, this disclosure also proposes to add additional processing after the splicing processing S613 in the first fusion stage S11, namely, phantom processing S614, convolution processing S615 and convolution processing S616.

[0077] like Figure 6As shown, the minimum size feature map P5' after convolution processing S611 and the fused feature maps M3-M4 obtained in the two fusion stages of FPN processing 610 are both input into the next level of processing, namely the bottom-up PAN processing 620.

[0078] In some embodiments, PAN processing 620 is as follows: Figure 6 The first fusion stage S21 may include: downsampling processing S621, and concatenation processing S622, which concatenates the downsampled fusion feature map M3 with the fusion feature map M4 to be fused in the current fusion stage. Furthermore, to improve the fusion effect, this disclosure also proposes adding additional processing after the concatenation processing S622 in the first fusion stage S21, namely, phantom processing S623 and convolution processing S624.

[0079] Although not shown, it will be understood by those skilled in the art that Figure 6 The processing of other fusion stages can be similar to or different from the first fusion stage S11 / S21. For example, similar operations to the first fusion stage S11 / S21 can be used in each fusion stage.

[0080] The upsampling process S612 described above can be implemented in various ways, such as nearest neighbor interpolation, bilinear interpolation, unpooling, or transposed convolution. Alternatively, the downsampling process S621 described above can also be implemented in various ways, such as using channel-wise convolution with a stride of 2.

[0081] In some embodiments, a neural network model can be used to implement the improved feature fusion processing according to this disclosure described above. The portion of the neural network model used for feature fusion processing (also referred to as the feature fusion network) is often referred to as the neck of the entire object detection model. Thus, the feature fusion network can have a... Figure 6 The same structure can be used, and convolutional layers, upsampling layers, downsampling layers, splicing layers, and phantom blocks can be used to implement the same structure. Figure 6 The diagram shows convolution processing, upsampling processing, downsampling processing, splicing processing, and phantom processing.

[0082] In some embodiments, Figure 6 The phantom processing in S613 and S623 can both utilize, for example... Figure 7 The phantom block 700 shown is used to achieve this.

[0083] like Figure 7As shown, the phantom block 700 includes a convolutional layer 701, a convolutional layer 702, and a DW convolutional layer 703 connected in sequence in one branch to be spliced ​​before the splicing layer 704, and a convolutional layer 701 (shared with the previous branch) and a convolutional layer 705 connected in sequence in another branch to be spliced. Thus, the splicing layer 704 is used to splice the output of the convolutional layer 705 with the output of the DW convolutional layer 703.

[0084] Using the aforementioned phantom blocks makes the fusion process more lightweight and requires fewer parameters, thus allowing for the acquisition of fused feature maps with fewer parameters and faster computation speed.

[0085] As previously described, in some embodiments, after obtaining multiple fused feature maps using feature fusion processing, a prediction process can be performed to predict the location and category of an object contained in the image based on these fused feature maps. In some embodiments, a neural network model can be used to implement the prediction process described above. The neural network model portion used for prediction processing (also referred to as the prediction network) is typically referred to as the head of the entire object detection model.

[0086] This head can employ various methods to perform prediction processing. For example, a two-stage structure can be used to obtain the target's bounding box and category. The first stage includes a Region Proposal Network (RPN), which performs feature mapping on candidate regions on the feature map to form Regions of Interest (RoIs), proposing candidate target bounding boxes. Then, the second stage uses the RoIAlign layer to extract features from each candidate bounding box and performs the same classification and bounding box regression operations as FastR-CNN, thereby obtaining the accurate target location and category.

[0087] In some implementations, the detection heads are decoupled, allowing them to output separately, adapting to more feature sizes and improving detection robustness. For example, after feature extraction and fusion, anchors are assigned using a PRN layer, migrating them from the original image to the feature map. Because the feature map is much smaller than the original image, the computational cost is reduced by an order of magnitude, significantly decreasing classification and regression time. The RPN network maps candidate regions to form Regions of Interest (ROIs) on the feature map, then accurately locates the target region based on the predicted offset, and simultaneously provides a probability score indicating whether the ROI's proposals belong to a certain class in the dataset.

[0088] Back Figure 1After step S120, step S130 is performed, in which the previously obtained feature map is subjected to mask discrimination processing to obtain the mask of each target in the image.

[0089] In some implementations, the mask discrimination process may include: first extracting edge pixels from the feature map, and then predicting and judging the extracted edge pixels to obtain the category of the target in the image; thereby improving the accuracy of target classification and localization in some complex situations (such as occlusion or blurring).

[0090] In some implementations, such as Figure 8 As shown, the aforementioned edge pixel extraction may include:

[0091] Step S810: Perform edge feature enhancement processing on the feature map to obtain edge feature pixels;

[0092] Step S820: Select any k pixels around the edge feature pixel and predict them independently, where k is a natural number;

[0093] Step S830: Upsample the edge feature pixels determined after independent prediction by transpose convolution and bilinear interpolation to obtain finer edge pixel curves, thereby extracting the edge pixels.

[0094] The aforementioned edge pixel extraction process can utilize edge feature enhancement to increase the receptive field, thereby retaining more edge information. In some implementations, edge feature enhancement may include multi-scale receptive field extraction and context attention. Multi-scale receptive field extraction captures the correlation information between receptive fields of different scales, while context attention aggregates multi-scale receptive field information from previous layers, achieving scale diversity of features with different receptive fields and thus enabling effective information interaction between receptive fields of different sizes in different layers. Furthermore, in some implementations, contour coefficients can be used to select any k pixels different from surrounding pixels on the edge feature pixels for independent prediction. In some implementations, the obtained edge pixel curve can be used to predict the precise contour of the target, forming a segmentation between the background and the target. Therefore, the aforementioned edge pixel extraction process can achieve individual pixel mask segmentation, effectively solving the problem of calculating each feature pixel one by one, greatly saving segmentation time and memory, and ensuring rapid discrimination.

[0095] Even after extracting the target edges (i.e., target contours) as described above, it's still unavoidable that the same edge pixel of the target might be predicted as multiple different label pixels. Furthermore, if the target is occluded, target pixels may be missing, leading to shape variations and recognition errors. Therefore, after the edge pixel extraction process described above, a prediction and judgment process can be performed to predict the category of the pixels on the obtained contour, thereby obtaining accurate masks for each category of the target.

[0096] In some implementations, such as Figure 9 As shown, the aforementioned prediction and judgment processing may include:

[0097] Step S910: Perform pixel block correction processing on the feature maps of each target region using shape and size;

[0098] Step S920: Multiply the extracted edge pixel values ​​with their corresponding weights and compare the result with the threshold values ​​of various label pixels according to the RGB model to determine the category of the edge pixel, thereby obtaining the category of the target.

[0099] In some implementations, the pixel value of a feature point is multiplied by its corresponding weight to calculate the size of a certain type of pixel value. Then, a threshold comparison is performed based on the RGB model and the label pixel to determine the pixel category of the feature point. Pixel categories with smaller weights are removed, and what remains is the category of the label pixel corresponding to that point.

[0100] The masks obtained from this process can be used to correct the bounding boxes and categories of the targets obtained in step S120, thereby outputting more accurate target locations and categories; or, they can be output to other processing as needed for processes such as instance segmentation. In some embodiments, the bounding boxes, categories, and masks of the targets in the obtained image can be output individually or in arbitrary combinations to subsequent processing stages (e.g., spatial localization processing).

[0101] The mask discrimination process described above can improve the accuracy of target classification and positioning, especially in complex environments or situations with obstructions, such as at night, in heavy fog, in rain, or when vehicles obstruct the view. In these conditions, the mask discrimination process can better identify targets.

[0102] Back Figure 1After step S130, the process proceeds to step S140, where spatial localization processing is performed on the obtained feature map to obtain the inference results of the spatial map formed by each target, and the position and category of the region box of each target are corrected. The scheme disclosed in this invention can further improve the target detection accuracy in complex situations such as occlusion by adding spatial localization processing, because it can capture and correct the spatial position of occluded or covered targets, thereby achieving accurate localization of occluded or covered targets.

[0103] In some implementations, such as Figure 10 As shown, the aforementioned spatial positioning process may include:

[0104] Step S1010: Learn a sparse adjacency matrix from the visual features of each target in the feature map. In the sparse adjacency matrix, each target uses its top t most relevant targets as its adjacency nodes, where t is an integer greater than 1.

[0105] Step S1020: Calculate the spatial information between each target and its neighboring targets based on the Gaussian kernel with learnable parameters, and obtain the corresponding Gaussian kernel weights;

[0106] Step S1030: Perform graph convolution processing on the visual embedding obtained by soft mapping of each target's category, the sparse adjacency matrix, and the Gaussian kernel weights to obtain the inference result of the spatial graph formed by each target, and correct the region box position and category of each target.

[0107] For example, in some implementations, spatial localization processing can be performed on different target features in space. First, a sparse adjacency matrix is ​​learned from visual features, which preserves the original related connections. Then, different target weights are collected and mapped to Gaussian encoded regions. Pairwise spatial information (distance, angle) between regions is adaptively processed by gradient flow for analytical localization through Gaussian kernel function and spatial graph convolution. Finally, the visual features of different regions are positionally corrected. The output of spatial graph inference is combined with the original region features to improve both classification and localization.

[0108] For example, in some implementations, weight features are first collected, and then an interpretable sparse adjacency matrix is ​​learned from visual features. Pairwise spatial information (distance, angle, etc.) between different regions is calculated based on the sparse adjacency matrix and a Gaussian kernel. The Gaussian kernel is then input to determine the convolution mode of the graph, thereby obtaining the spatial relationships between different targets.

[0109] In some implementations, the spatial location information of the target is calculated as follows:

[0110]

[0111]

[0112] Where d is the relative distance between the two targets; Δd is the positional deviation of the target region box; Δθ is the angular offset between the two target nodes; and v represents the target node.

[0113] In some implementations, the sparse adjacency matrix described above is calculated as follows:

[0114]

[0115] Where N r e represents the total number of target nodes i,j This represents the edge between nodes i and j, indicating the relationship between these two nodes;

[0116] For each target region i, only the top t nodes with the strongest correlation are retained as the neighbor nodes of the region.

[0117] In some implementations, the Gaussian kernel encoding (mapping) described above is calculated as follows:

[0118] First, the spatial correspondence of the target region is represented using polar coordinates as follows:

[0119] v(v1, v2) = V(d, θ)

[0120] For the target node ν1, the coordinates of the target node ν2 can be returned through the mapping relationship ν(ν1,ν2);

[0121] Secondly, the encoding operator is defined by learning a k-order Gaussian kernel with the mean and variance, as shown in the following formula:

[0122]

[0123]

[0124] Among them, f k (i) represents the weight matrix mapping of the target; Nei(i) represents the neighboring nodes; G k σ is a k-th order Gaussian kernel; μ is the mean; σ is the variance.

[0125] In some implementations, the spatial graph convolution operation described above is as follows:

[0126]

[0127] Among them, Z h This indicates the information of the corresponding spatial target node in the region suggested by D. The fundamental product representing the mapping between spatial location information and the weight matrix.

[0128] Furthermore, in some embodiments, the above-described method for classifying and locating image targets according to this disclosure (e.g.) Figure 1 Each of the processes shown can be implemented using a neural network model.

[0129] Furthermore, in some embodiments, the method and neural network model for classifying and locating image targets according to this disclosure can be applied to various scenarios with relevant target detection needs, and are particularly suitable for scenarios requiring target detection in images captured in complex environments or with occlusion. As mentioned above, for example, the solution disclosed in this disclosure can be applied to electronic devices that identify various targets on the road and display or provide warnings accordingly, in order to assist drivers in driving more safely and effectively reduce the risk of traffic accidents. Such electronic devices can be, for example, in-vehicle devices that are built into the vehicle or installed by the driver, or mobile electronic devices carried by the driver, such as mobile phones, XR glasses, etc.

[0130] In some embodiments, such as XR glasses, an electronic device applying the present disclosure is described.

[0131] XR glasses may include a camera for capturing images, a processor, a memory, and a display / warning component. The memory stores executable code that, when executed by the processor, causes the processor to perform the aforementioned method for classifying and locating targets in the images captured by the camera, thereby obtaining the location and category of targets contained in the image. The display / warning component is used to provide appropriate displays and / or warnings to the user based on the obtained target location and category. For example, the memory may store code implementing the neural network model according to this disclosure. The display / warning component may include a display of the XR glasses that displays predictions and suggestions related to the targets.

[0132] In addition, in some embodiments, information obtained from cameras, gyroscopes, and photosensors in the XR glasses can be combined for driving monitoring and alarms. For example, the distance to obstacles ahead can be calculated based on the information obtained from these sensors, and an alarm can be issued to the user through various means (such as display, vibration, voice, etc.) when the distance is less than a safe distance.

[0133] Therefore, XR glasses can be used to automatically identify and detect road targets, provide relevant prompts and predictions, and prevent traffic and other problems that may occur due to complex road conditions. Moreover, due to the use of the image target classification and localization method disclosed herein, the detection accuracy is higher and the response is faster.

[0134] Figure 11A schematic diagram of a computing device according to an embodiment of the present disclosure is shown, which can be used to implement the above-described method for classifying and locating image targets. Note that the following description of the memory 1110 and processor 1120 also applies to the memory and processor in the aforementioned electronic devices, such as XR glasses.

[0135] See Figure 11 The computing device 1100 includes a memory 1110 and a processor 1120.

[0136] Processor 1120 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1120 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 1120 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0137] Memory 1110 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1120 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1110 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 1110 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0138] The memory 1110 stores executable code, which, when processed by the processor 1120, enables the processor 1120 to execute the aforementioned method for classifying and locating image targets.

[0139] Furthermore, the method according to this disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of this disclosure.

[0140] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to this disclosure.

[0141] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0143] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for classifying and locating targets in an image, comprising: Obtain the image to be processed; The image is processed by feature extraction, feature fusion, and prediction to obtain feature maps with bounding boxes and categories for each target. The feature map is subjected to mask discrimination processing to obtain the mask of each target in the image. The obtained mask of each target is used to correct the region box and category of each target. Spatial localization processing is performed on the feature map to obtain the inference results of the spatial map formed by each target, and the position and category of the region box of each target are corrected; The process of performing mask discrimination on the feature map to obtain the mask of each target in the image includes: firstly, extracting edge pixels from the feature map, and then predicting and judging the extracted edge pixels to obtain the category of the target in the image. The edge pixel extraction includes: performing edge feature enhancement processing on the feature map to obtain edge feature pixels, selecting any k pixels around the edge feature pixels for independent prediction, and upsampling the edge feature pixels determined after independent prediction by transpose convolution and bilinear interpolation to obtain finer edge pixel curves, thereby extracting the edge pixels, where k is a natural number; The prediction and judgment include: performing pixel block correction processing on the feature maps of each target region using shape and size; multiplying the pixel values ​​of each extracted edge pixel with the corresponding weights; comparing the result with the threshold values ​​of various label pixels according to the RGB model; determining the category of the edge pixel; and thus obtaining the category of the target.

2. The method according to claim 1, wherein, The feature extraction process is a multi-scale feature extraction, including: The intermediate feature maps of the input at least one stage are sequentially subjected to first phantom processing and channel attention processing; and The intermediate feature map is then joined with the intermediate feature map after the first phantom processing and the channel attention processing.

3. The method of claim 2, wherein, The feature extraction process also includes: The initial input image is subjected to two channel-wise convolution processes to obtain the first intermediate feature map; The first intermediate feature map is sequentially processed by convolution, first phantom processing, channel attention processing, and convolution to obtain the second intermediate feature map. The first intermediate feature map is then processed by convolution once to obtain the third intermediate feature map. The second and third intermediate feature maps are concatenated.

4. The method of claim 3, wherein, The two channel-wise convolution processes reduce the size of the image in the height and width dimensions and / or increase the number of channels in the image.

5. The method of claim 2, wherein, The first phantom processing includes: The intermediate feature maps are subjected to convolution processing and channel-wise convolution processing in sequence; and The intermediate feature map after the convolution process is then concatenated with the intermediate feature map after the convolution process and the channel-by-channel convolution process.

6. The method according to claim 5, wherein, The first phantom processing further includes, after the splicing process: The intermediate feature maps after the aforementioned stitching process are sequentially subjected to convolution and channel-by-channel convolution; and The intermediate feature map after the convolution process is concatenated with the intermediate feature map after the channel-by-channel convolution process, and the concatenated intermediate feature map is used as the input for the channel attention process.

7. The method according to claim 2, wherein, The channel attention processing includes: The intermediate feature maps are then subjected to global pooling, convolution, and sigmoid function processing in sequence; and The weight values ​​of each channel obtained after processing by the Sigmoid function are multiplied element-wise with the feature maps of the corresponding channels of the intermediate feature map.

8. The method according to claim 1, wherein, The feature fusion process includes feature pyramid network processing, which includes: Upsampling is performed on the minimum size feature map obtained from the feature extraction process; The feature map after the upsampling process is concatenated with the second smallest size feature map to obtain a fused feature map. Perform a second phantom processing on the fused feature map; The second phantom processing includes: The fused feature map is subjected to two convolutional processes and one channel-wise convolutional process in sequence to obtain the first fused feature map; Perform another convolution process on the fused feature map obtained after the first convolution process of the two convolution processes mentioned above to obtain a second fused feature map; and The first fused feature map and the second fused feature map are then concatenated.

9. The method according to claim 8, wherein, The feature fusion process, following the feature pyramid network processing, also includes path aggregation network processing, which includes: The maximum-size fused feature map obtained by the feature pyramid network is downsampled. The largest-size fused feature map after the downsampling process is then stitched together with the second-largest-size fused feature map. The second phantom processing is performed on the fused feature map after the splicing process.

10. The method according to claim 1, wherein, The spatial positioning process includes: A sparse adjacency matrix is ​​learned from the visual features of each target in the feature map, where each target uses its top t most relevant targets as its adjacency nodes, and t is an integer greater than 1. Spatial information between each target and its neighboring targets is calculated using a Gaussian kernel with learnable parameters, resulting in corresponding Gaussian kernel weights; and The visual embedding obtained by soft mapping of each target's category, the sparse adjacency matrix, and the Gaussian kernel weights are subjected to graph convolution processing to obtain the inference result of the spatial graph formed by each target, and the region box position and category of each target are corrected.

11. An electronic device, comprising: A camera is used to capture images; processor; A memory storing executable code, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-10 on an image captured by the camera, thereby obtaining the location and category of a target contained in the image; as well as Display / alert component, used to display and / or alert users accordingly based on the location and category of the obtained target.

12. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-10.

13. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-10.

Images