Image recognition method and related device

Abstract

Embodiments of the present application disclose an image recognition method and related equipment, which can include an image recognition device, an electronic device, a computer program product and a computer readable storage medium; in the embodiments of the present application, at least one road image is acquired, at least one target element region of an image element is recognized in the road image, multi-dimensional feature extraction is performed on the target element region to obtain element image features of at least one dimension, then, based on the element image features, a component region of at least one element component is detected in the target element region, feature extraction is performed on the component region to obtain component features, then, the component features are fused with the element image features to obtain element features of the image element, then, according to the element features, the image element is classified to obtain a target element category of the image element; the scheme can improve the accuracy of image recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more specifically to an image recognition method and related equipment, which may include image recognition devices, electronic devices, computer program products, and computer-readable storage media. Background Technology

[0002] In recent years, with the rapid development of internet technology, the collection and processing of map road data has become increasingly convenient. When updating map road data, a large number of complex image elements are often generated (such as road signs, road markers, or useful physical point information like traffic cameras). To improve the accuracy of map road data updates, image recognition of these image elements is necessary. Current image recognition methods often extract high-level semantic features from the image features and classify them to obtain the image recognition result.

[0003] In the process of researching and practicing current technologies, the inventors of this application have found that because some image elements are too complex and the interference between different types of image elements is large, using a single classification network to extract high-level semantic features for classification often results in false detections or recognition errors, thus leading to a low accuracy rate in image recognition. Summary of the Invention

[0004] This application provides an image recognition method and related equipment. The related equipment may include an image recognition device, an electronic device, a computer program product, and a computer-readable storage medium, which can improve the accuracy of image recognition.

[0005] An image recognition method, comprising:

[0006] Acquire at least one road image and identify at least one target feature region of an image feature in the road image, the image feature indicating target physical point information in the road image;

[0007] Multi-dimensional feature extraction is performed on the target feature region to obtain feature image features of at least one dimension;

[0008] Based on the feature image features, at least one component region of a feature component is detected in the target feature region, and feature extraction is performed on the component region to obtain component features;

[0009] The component features are fused with the element image features to obtain the element features of the image elements;

[0010] Based on the feature characteristics, the image features are classified to obtain the target feature category of the image features.

[0011] Accordingly, embodiments of this application provide an image recognition device, including:

[0012] The acquisition unit is configured to acquire at least one road image and identify a target feature region of at least one image feature in the road image, wherein the image feature indicates target physical point information in the road image;

[0013] An extraction unit is used to extract multi-dimensional features from the target feature region to obtain feature image features of at least one dimension.

[0014] The detection unit is configured to detect at least one component region of a component in the target feature region based on the feature image features, and to extract features from the component region to obtain component features.

[0015] A fusion unit is used to fuse the component features with the element image features to obtain the element features of the image elements;

[0016] A classification unit is used to classify the image elements according to the feature characteristics to obtain the target feature category of the image elements.

[0017] In some embodiments, the fusion unit may be specifically used to perform attention weighting on the feature image features based on the component features to obtain the spatial attention features of the image features; extract text features from the road image and fuse the text features with the spatial attention features to obtain the feature features of the image features.

[0018] In some embodiments, the fusion unit may be specifically used to extract the underlying image features corresponding to each feature position in the image features; determine the attention weights corresponding to the underlying image features based on the component features, and weight the underlying image features according to the attention weights; and fuse the weighted underlying image features to obtain the spatial attention features of the image elements.

[0019] In some embodiments, the fusion unit may be specifically used to sort the image features based on the feature size of the image features; according to the sorting result, select the image features with the largest feature size from the image features to obtain the target image features; and extract the image features corresponding to each feature position from the target image features to obtain the bottom layer image features.

[0020] In some embodiments, the fusion unit may be specifically used to perform dimensional transformation on the spatial attention features to obtain target spatial attention features of a preset dimension; perform dimensional transformation on the text features to obtain target text features of the preset dimension; and concatenate the target spatial attention features and the target text features to obtain the feature features of the image elements.

[0021] In some embodiments, the fusion unit may be specifically used to identify at least one text content in the road image and determine the content location of the text content in the road image; based on the content location, to extract features from the text content to obtain the text features of the road image.

[0022] In some embodiments, the detection unit may be specifically used to extract at least one region feature of a feature component from the feature image features, and determine the region location information of the feature component based on the region feature; identify the initial component region of the feature component in the target feature region according to the region location information; and fuse the initial component region to obtain the component region of the feature component.

[0023] In some embodiments, the detection unit may be specifically used to extract the position of the center point of the feature component and the boundary distance between the center point and the region boundary from the region location information; and based on the boundary distance and the position of the center point, to identify the initial component region of the feature component in the target feature region.

[0024] Optionally, in some embodiments, the detection unit may be specifically used to extract the position of the center point of the feature component and the endpoint distance between the center point and at least one endpoint from the regional location information; based on the endpoint distance and the position of the center point, identify at least one endpoint of the feature component in the target feature region; and detect the initial component region of the feature component in the target feature region according to the position of the endpoint.

[0025] In some embodiments, the extraction unit may be specifically used to adjust the size of the target feature region and use an image recognition model to extract multi-dimensional features from the adjusted target feature region to obtain original feature image features of multiple dimensions; filter out the target dimension original feature image features from the original feature image features to obtain initial semantic features; and fuse the original feature image features based on the initial semantic features to obtain feature image features of at least one dimension.

[0026] In some embodiments, the extraction unit may be specifically used to perform multi-dimensional feature extraction on the initial semantic features to obtain at least one semantic feature; to fuse the original element image features at least once to obtain at least one fused element image feature; and to use the semantic features, the initial semantic features, and the fused element image features as the element image features of the image elements.

[0027] In some embodiments, the extraction unit may be specifically used to sort the original feature image features based on the feature size of the original feature image features; according to the sorting result, select the current feature image features that need to be fused with the initial semantic features from the original feature image features; and fuse the initial semantic features with the current feature image features to obtain at least one fused feature image feature.

[0028] In some embodiments, the extraction unit may be specifically used to adjust the size of the initial semantic feature to obtain the adjusted semantic feature; fuse the adjusted semantic feature with the current feature image to obtain the fused feature image, and use the fused feature image as the initial semantic feature; return to the step of selecting the current feature image that needs to be fused with the initial semantic feature from the original feature image based on the sorting result, until all the original feature image features are fused, to obtain at least one fused feature image.

[0029] In some embodiments, the image recognition device may further include a training unit, which is specifically configured to acquire at least one image element sample, and use a preset image recognition model to perform multi-dimensional feature extraction on the image element sample to obtain multi-dimensional sample image features; extract sample component features of at least one element component from the sample image features, and extract sample text features from the image element sample; fuse the sample image features, sample component features, and sample text features, and predict the category of the image element sample based on the fused sample element features to obtain a predicted element category; determine the classification loss of the image element sample based on the predicted element category, and converge the preset image recognition model according to the classification loss to obtain the image recognition model.

[0030] In some embodiments, the acquisition unit may be specifically used to extract features from the road image to obtain original image features, and filter out noise features from the original image features to obtain image features; based on the image features, identify at least one candidate feature region in the road image; and select at least one target feature region of image features from the candidate feature regions.

[0031] In some embodiments, the acquisition unit may be specifically used to identify the position of each feature point in the image features in the road image, and obtain the feature point position; based on the feature point position, segment the image region of at least one scale in the road image according to at least one preset aspect ratio, and use the image region as a candidate feature region.

[0032] Furthermore, this application also provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the image recognition method provided in this application.

[0033] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image recognition methods provided in embodiments of this application.

[0034] Furthermore, this application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image recognition method provided in this application.

[0035] This embodiment of the application acquires at least one road image and identifies at least one target element region of an image element in the road image. Then, it performs multi-dimensional feature extraction on the target element region to obtain at least one dimension of element image features. Based on the element image features, it detects at least one component region of an element within the target element region and extracts features from this component region to obtain component features. These component features are then fused with the element image features to obtain the element features of the image element. Finally, based on the element features, the image element is classified to obtain the target element category of the image element. Since this scheme can extract at least one dimension of element image features from the target element region, detect at least one component region of an element within the target element region, and extract component features from the component region, it utilizes the rich component features within the element component to guide the classification of the image element. This effectively solves the problem of complex and variable information within image elements and significant interference between categories, thus improving the accuracy of image recognition. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of a scene illustrating the image recognition method provided in an embodiment of this application;

[0038] Figure 2 This is a schematic flowchart of the image recognition method provided in the embodiments of this application;

[0039] Figure 3 This is a schematic diagram of a traffic sign provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the large square sign provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the small square sign provided in the embodiments of this application;

[0042] Figure 6 These are schematic diagrams of other small signs provided in the embodiments of this application;

[0043] Figure 7 This is a schematic diagram of candidate bounding boxes segmented from a road image provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the target feature region for detecting image features provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the convolutional neural network provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the target element region provided in the embodiments of this application;

[0047] Figure 11 This is a schematic diagram of a densely packed signboard provided in an embodiment of this application;

[0048] Figure 12 This is a schematic diagram of the component area optimization detection provided in an embodiment of this application;

[0049] Figure 13 This is a schematic diagram of a component for detecting features using a local feature extraction network, provided in an embodiment of this application.

[0050] Figure 14 This is a schematic diagram illustrating the fusion of component features and element image features provided in an embodiment of this application;

[0051] Figure 15 This is a schematic diagram of the overall process for classifying traffic signs provided in the embodiments of this application;

[0052] Figure 16 This is another schematic flowchart of the image recognition method provided in the embodiments of this application;

[0053] Figure 17 This is a schematic diagram of the structure of the image recognition device provided in the embodiments of this application;

[0054] Figure 18 This is another structural schematic diagram of the image recognition device provided in the embodiments of this application;

[0055] Figure 19 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This application provides an image recognition method and related equipment. The related equipment may include an image recognition device, an electronic device, a computer program product, and a computer-readable storage medium. The image recognition device may be integrated into an electronic device, which may be a server or a terminal, etc.

[0058] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0059] For example, see Figure 1Taking the integration of an image recognition device into an electronic device as an example, after acquiring at least one road image and identifying the target element region of at least one image element in the road image, the electronic device performs multi-dimensional feature extraction on the target element region to obtain at least one dimension of element image features. Then, based on the element image features, it detects at least one component region of an element component in the target element region and extracts features from the component region to obtain component features. Then, it fuses the component features with the element image features to obtain the element features of the image element. Finally, based on the element features, it classifies the image element to obtain the target element category of the image element, thereby improving the accuracy of image recognition.

[0060] It is understood that, in the specific embodiments of this application, road images and other related data of the object are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0061] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0062] This embodiment will be described from the perspective of an image recognition device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices capable of image recognition.

[0063] An image recognition method, comprising:

[0064] At least one road image is acquired, and at least one target element region of an image element is identified in the road image. The image element indicates the target physical point information in the road image. Multi-dimensional feature extraction is performed on the target element region to obtain at least one dimension of element image features. Based on the element image features, at least one component region of an element component is detected in the target element region, and feature extraction is performed on the component region to obtain component features. The component features are fused with the element image features to obtain the element features of the image element. Based on the element features, the image element is classified to obtain the target element category of the image element.

[0065] like Figure 2 As shown, the specific process of this image recognition method is as follows:

[0066] 101. Acquire at least one road image and identify the target feature region of at least one image feature in the road image.

[0067] Among them, road images can be understood as images containing roads or various traffic facilities / elements on roads.

[0068] Image elements refer to target physical point information in road images, which can also be understood as useful physical point information in map data images. Examples include various traffic signs (such as speed limit signs and various directional signs), traffic cameras, or other physical point information that can indicate road facilities or regulations. Taking traffic signs as an example, traffic signs can be understood as signs that indicate traffic information, such as… Figure 3 As shown, this can include road name signs, vehicle information signs, direction names, road name-current signs, U-turn points, one-way streets, underpasses, service areas, bus lanes, pedestrian crossings, section speed cameras, emergency lanes, bicycle lanes, etc. Figure 3 It can be observed that each category of signs contains complex content, such as arrows, logos, or text. Traffic signs contain numerous signs with complex and diverse internal elements. At the same time, there are also many signs of the same type with simple internal modalities but varied forms, or signs of different categories but with high similarity; for example, large square signs have a wide variety of internal elements and complex forms, such as... Figure 4 As shown; small rectangular signs have relatively simple content, but there are large differences within the same category, such as... Figure 5 As shown; other smaller cards have relatively simple content, but the differences between categories are small, such as Figure 6 As shown, etc.

[0069] The target feature region can be understood as the image region in the road image that contains image features.

[0070] There are several ways to obtain at least one road image, including the following:

[0071] For example, an image of the road ahead can be acquired by an in-vehicle image acquisition device or component, thereby obtaining at least one road image; or an image acquisition device can acquire at least one image of the target road, thereby obtaining a road image; or an image acquisition vehicle can be received to send at least one road image; or an image database can be obtained from a network or map database; or an image database can be received to send at least one road image uploaded by a terminal or client; or an image recognition request can be received, which carries the storage address of at least one road image, and at least one road image can be obtained based on the storage address, and so on.

[0072] After acquiring at least one road image, a target feature region of at least one image element can be identified within the road image. There are several ways to identify the target feature region of at least one image element in a road image. For example, feature extraction can be performed on the road image to obtain its image features. Based on these features, feature points of at least one image element can be identified in the road image. Based on these feature points, at least one candidate feature region of the image element can be identified in the road image, and the target feature region can be selected from among the candidate feature regions.

[0073] There are several ways to extract features from road images. For example, you can extract features from the road image to obtain the original image features, and then filter out noise features from the original image features to obtain the final image features.

[0074] There are various ways to extract features from road images. For example, convolutional layers can be used to extract the edges, textures, or other basic features of the road image to obtain the original image features.

[0075] After extracting the original image features from the road image, noise features can be filtered out from the original image features to obtain the image features. There are several ways to filter out noise features from the original image features. For example, noise features can be filtered out from the original image features to obtain filtered image features, and then nonlinear mapping can be performed on the filtered image features to obtain the image features of the road image.

[0076] There are several ways to filter out noise features from the original image features. For example, a batch normalization (BN) layer can be used to normalize the original image features according to a normal distribution, thereby filtering out the noise features in the original image features and obtaining the filtered image features.

[0077] After filtering out noise features from the original image features, nonlinear mapping can be applied to the filtered image features to obtain the image features of the road image. There are various ways to apply nonlinear mapping to the filtered image features; for example, an activation layer (ReLU) can be used to apply nonlinear mapping to the filtered image features to obtain the image features of the road image, and so on.

[0078] After feature extraction from a road image, at least one candidate feature region can be identified based on the extracted image features. There are several ways to identify at least one candidate feature region in a road image. For example, the position of each feature point in the image features can be identified to obtain the feature point position. Based on the feature point position, the image region of at least one scale can be segmented in the road image according to at least one preset aspect ratio, and the image region can be used as the candidate feature region.

[0079] Taking three preset aspect ratios {1:1, 2:1, 1:2}, with scales of 1 feature point, 2 feature points, and 3 feature points respectively, as examples, nine candidate boxes can be segmented in a road image, such as... Figure 7 As shown, these 9 candidate boxes can be used as candidate feature regions.

[0080] After identifying at least one candidate feature region in a road image, a target feature region containing at least one image feature can be selected from the candidate feature region. There are several ways to select a target feature region containing at least one image feature from the candidate feature region. For example, image feature detection (i.e., binary classification) can be performed in the candidate feature region. Based on the detection results, regions containing image features can be selected from the candidate feature region to obtain at least one feature region. The position of the feature region can then be adjusted to obtain the target feature region of the image feature.

[0081] The overall process for identifying the target feature region of at least one image feature in a road image can be as follows: Figure 8 As shown, deep learning and convolutional neural networks can be used to extract features from road images. Based on the extracted image features, image elements appearing in the road images can be detected, and the target feature region (boundary) of each image element can be determined. The network structure of the convolutional neural network can be as follows: Figure 9As shown, the system mainly includes a feature extraction network (Conv layers) and a candidate box selection network (Region Proposal Network). The feature extraction network consists of convolutional layers, batch normalization (BN) layers, and activation layers (ReLU). The convolutional layers are responsible for extracting basic features such as edge texture. The normalization layers normalize the features extracted by the convolutional layers according to a normal distribution, filtering out noisy features and making the model's training converge faster. The activation layers are responsible for performing non-linear mapping on the features extracted by the convolutional layers, enhancing the model's generalization ability. The candidate box selection network, based on the extracted image features, uses each feature point as a center point, selects at least one preset aspect ratio (e.g., 1:1, 2:1, 1:2, or any other arbitrary ratio), and uses the feature points as anchors to select at least one scale (e.g., 1 feature point, 2 feature points, or any other arbitrary number of feature points, etc.) to select at least one candidate box in the road image as a candidate box for feature detection, and detects the target detection box (target feature region) of the image feature within these candidate boxes.

[0082] 102. Perform multi-dimensional feature extraction on the target feature region to obtain feature image features of at least one dimension.

[0083] In this context, element image features can be understood as the image features of image elements. There are various methods for multi-dimensional feature extraction of target element regions, including the following:

[0084] For example, the size of the target feature region can be adjusted, and an image recognition model can be used to extract multi-dimensional features from the adjusted target feature region to obtain original feature image features of multiple dimensions. The target dimension of the original feature image features can be selected from the original feature image features to obtain initial semantic features. Based on the initial semantic features, the original feature image features can be fused to obtain feature image features of at least one dimension.

[0085] There are several ways to adjust the size of the target feature area. For example, the size of the target feature area can be adjusted to a preset size to obtain the adjusted target feature area. The preset size can be a pre-defined area size, such as 300*300 pixels, or any other preset size, etc.

[0086] After adjusting the size of the target feature region, an image recognition model can be used to extract multi-dimensional features from the adjusted target feature region, thereby obtaining multi-dimensional original feature image features. There are several ways to use an image recognition model to extract multi-dimensional features from the adjusted target feature region. For example, an image feature extraction network of an image recognition model can be used to extract multi-dimensional features from the target feature region, thereby obtaining multi-dimensional original feature image features.

[0087] The network structure of image feature extraction networks can be varied, such as ResNet (residual network), CNN (convolutional neural network), or other networks that can perform multi-dimensional feature extraction, etc.

[0088] After extracting multi-dimensional features from the adjusted target feature region using an image recognition model, the target dimension original feature can be selected from the original feature image features to obtain the initial semantic features. There are several ways to select the target dimension original feature from the original feature image features. For example, the feature with the smallest feature size can be selected to obtain the initial semantic features corresponding to the target dimension; alternatively, the feature with the highest dimension can be selected to obtain the initial semantic features corresponding to the target dimension, and so on.

[0089] After selecting initial semantic features from the original feature image features, the original feature image features can be fused based on these initial semantic features to obtain feature image features of at least one dimension. There are various ways to fuse the original feature image features based on the initial semantic features. For example, multi-dimensional feature extraction can be performed on the initial semantic features to obtain at least one semantic feature; the original feature image features can be fused at least once to obtain at least one fused feature image; and the semantic features, initial semantic features, and fused feature image features can be used as the feature image features of the image elements.

[0090] There are several ways to fuse the original feature image features at least once. For example, the original feature image features can be sorted based on their feature size. Based on the sorting results, the current feature image features that need to be fused with the initial semantic features can be selected from the original feature image features. The initial semantic features can be fused with the current feature image features to obtain at least one fused feature image feature.

[0091] There are several ways to select the current element image features that need to be fused with the initial semantic features from the original element image features based on the sorting results. For example, if the sorting method is from smallest to largest, the next element image feature after the initial semantic features can be selected from the original element image features as the current element image features that need to be fused with the initial semantic features.

[0092] After selecting the current feature from the original feature image features, the initial semantic features can be fused with the current feature image features to obtain at least one fused feature image. There are several ways to fuse the initial semantic features with the current feature image features. For example, the size of the initial semantic features can be adjusted to obtain adjusted semantic features. These adjusted semantic features are then fused with the current feature image features to obtain fused feature image features. These fused feature image features are then used as the initial semantic features. The process then returns to the original feature image features, selecting the current feature image features to be fused with the initial semantic features based on the sorting results. This process continues until all original feature image features have been fused, resulting in at least one fused feature image.

[0093] There are several ways to adjust the size of the initial semantic features. For example, the initial semantic features can be upsampled to obtain the adjusted semantic features, which can have the same size as the current feature image features.

[0094] After adjusting the size of the initial semantic features, the adjusted semantic features can be fused with the current feature image features to obtain the fused feature image features. There are several ways to fuse the adjusted semantic features with the current feature image features. For example, the adjusted semantic features can be directly concatenated with the current feature image features to obtain the fused feature image features. Alternatively, fusion weights can be obtained, and based on these weights, the adjusted semantic features and the current feature image features can be weighted separately. The weighted adjusted semantic features can then be concatenated with the weighted current feature image features to obtain the fused feature image features, and so on.

[0095] After fusing the adjusted semantic features with the current feature image features, the fused feature image features can be used as the initial semantic features. Then, the process returns to the step of selecting the current feature image features that need to be fused with the initial semantic features from the original feature image features based on the sorting results, until all the original feature image features have been fused, resulting in at least one fused feature image feature.

[0096] One approach to fusing the original feature images at least once is to use an FPN (Feature Fusion Network) for multi-scale feature generation and fusion. This primarily involves fusing high-level refined semantic features with low-level global features, thereby improving the network perception capability of the image recognition model. Taking the generated original feature images C3, C4, and C5, with feature sizes ordered from largest to smallest as C3 / C4 / C5, as an example, C5 can be selected as the initial semantic feature P5. P5 is then upsampled, and C4 is selected as the current feature image to be fused with P5. P5 and C4 are then fused to obtain P4, which is used as the initial semantic feature. Similarly, C3 is selected from the original feature images as the current feature image to be fused with P4, and P4 is fused with C3 to obtain P3. P3 and P4 can then be used as the fused feature images. It should be noted that P3 can be the best fused feature image. In subsequent feature fusion, P3 can be used as the low-level image feature.

[0097] After fusing the original feature image features at least once, the semantic features, the initial semantic features, and the fused feature image features can be used as the feature image features of the image elements.

[0098] Optionally, in some embodiments, the image recognition device can be set up by maintenance personnel or trained automatically by the image recognition model device. That is, before the step "using the image recognition model to extract multi-dimensional features from the adjusted target element region to obtain multi-dimensional original element image features", the image recognition method includes:

[0099] At least one image element sample is acquired, and a pre-defined image recognition model is used to extract multi-dimensional features from the image element sample to obtain multi-dimensional sample image features. At least one element component feature is extracted from the sample image features, and sample text features are extracted from the image element sample. The sample image features, sample component features, and sample text features are fused, and based on the fused sample element features, the category of the image element sample is predicted to obtain the predicted element category. Based on the predicted element category, the pre-defined image recognition model is converged to obtain the image recognition model.

[0100] There are several ways to obtain at least one image element sample. For example, you can obtain at least one road image sample, segment at least one element region from the road image sample, and thus obtain the image element sample. Alternatively, you can directly obtain the image element sample, and so on.

[0101] After acquiring at least one image element sample, a pre-defined image recognition model can be used to extract multi-dimensional features from the image element sample, resulting in multi-dimensional sample image features. The method of using a pre-defined image recognition model to extract multi-dimensional features from image sample elements is similar to the method of using an image recognition model to extract multi-dimensional features from the adjusted target element region, as detailed above, and will not be repeated here.

[0102] After extracting the sample image features, at least one element component feature can be extracted from the sample image features. Sample text features are also extracted from the image element samples. The sample image features, sample component features, and sample text features are then fused. Based on the fused sample element features, the category of the image element sample is predicted, thus obtaining the predicted element category. The predicted image element samples during model training can be similar to the method used to classify image elements in road images, as detailed above.

[0103] After predicting the category of image feature samples based on the fused sample feature features, the classification loss of image feature samples can be determined based on the predicted feature category, and the preset image recognition model can be converged according to the classification loss to obtain the image recognition model.

[0104] The classification loss can be understood as the loss incurred after classifying image feature samples using a pre-defined image recognition model. There are several ways to determine the classification loss of image feature samples based on the predicted feature category. For example, one can obtain the labeled feature categories of the image feature samples and compare them with the predicted feature categories to obtain the classification loss of the image feature samples.

[0105] After determining the classification loss for image feature samples, the preset image recognition model can be converged based on the classification loss to obtain the image recognition model. There are several ways to converge the preset image recognition model based on the classification loss. For example, the network parameters of the preset image recognition model can be updated using the gradient descent algorithm based on the classification loss to obtain an updated image recognition model. This updated model is then used as the preset image recognition model, and the process of obtaining at least one image feature sample is repeated until the preset image recognition model converges, thus obtaining the trained image recognition model. Alternatively, other parameter update algorithms can be used to update the network parameters of the preset image recognition model based on the classification loss to obtain an updated image recognition model. This updated model is then used as the preset image recognition model, and the process of obtaining at least one image feature sample is repeated until the preset image recognition model converges, thus obtaining the trained image recognition model, and so on.

[0106] 103. Based on the features of the feature image, detect at least one component region of the feature component in the target feature region, and extract features from the component region to obtain the component features.

[0107] In this context, a feature component can be understood as a part within an image feature. Feature components can be of various types, such as frames containing patterns, logos, or text within an image feature. Taking the target feature area as an example... Figure 10 As shown, the area within the solid line box represents the feature components. These components can include arrows, road text, and other signage elements, etc. Based on feature image features, there are several ways to detect the component region of at least one feature component within a target feature region, as follows:

[0108] For example, at least one feature region can be extracted from the feature image features, and the feature region location information can be determined based on the feature region. According to the feature region location information, the initial component region of the feature component can be identified in the target feature region, and the initial component region can be fused to obtain the component region of the feature component.

[0109] The regional location information can be understood as the predicted location information of the region where the indicator component is located. This information may include the location of the center point of the component region, the boundary distance from the center point to the boundary of the component region, the endpoint distance from the center point to at least one endpoint of the component region, or other location information that can indicate the component region. Based on the regional location information, there are several ways to identify the initial component region of the indicator component within the target feature region. For example, the location of the center point of the indicator component and the boundary distance between the center point and the region boundary can be extracted from the regional location information. Based on the boundary distance and the center point location, the initial component region of the indicator component can be identified within the target feature region. Alternatively, the location of the center point of the indicator component and the endpoint distance between the center point and at least one endpoint can be extracted from the regional location information. Based on the endpoint distance and the center point location, at least one endpoint of the indicator component can be identified within the target feature region. Based on the endpoint location, the initial component region of the indicator component can be detected within the target feature region, and so on.

[0110] Taking traffic signs as an example, the signs may contain a large amount of outlier component information, such as... Figure 10 As shown. For this type of feature component, the center point of the feature component and the boundary distance from the center point to the region boundary can be predicted. Based on the position of the center point and the boundary distance, the initial component region of the feature component can be identified in the target feature region.

[0111] Among them, traffic signs are still based on image elements, such as... Figure 11As shown, the dense arrangement of components on traffic signs can lead to semantic information overlap. To address this, the distance between the detection center point and the region boundary can be optimized to the distance between the detection center point and the four endpoints, such as... Figure 12 As shown. In this case, the position of the center point and the distances between the center point and the four endpoints can be extracted from the region boundary information, thereby identifying the endpoint positions of the feature components in the target feature region, and further detecting the initial component region of the feature components in the target feature region. It should be noted that detecting the distances between the center point and the four endpoints can yield quadrilaterals of arbitrary shapes and angles, thus solving the problem of semantic information overlap.

[0112] After identifying the initial component regions of a feature component within the target feature region, these initial component regions can be merged to obtain the component regions of the feature component. There are several ways to merge the initial component regions. For example, duplicate initial component regions can be removed to obtain the component regions of the feature component. Alternatively, pairs of initial component regions with overlapping areas can be detected within the initial component regions, and the initial component regions in the pairs can be merged based on the size of the overlapping areas to obtain the component regions of the feature component, and so on.

[0113] Specifically, a local feature extraction network of an image recognition model can detect at least one component region within a target feature region. Taking an example where the extracted original feature image features include three dimensions, the detection process of identifying at least one component region within the target feature region using a local feature extraction network can be described as follows: Figure 13 As shown, the target feature region is scaled to the same size (300*300 pixels) and used as input to the local feature extraction network. The ResNet convolutional neural network is used to extract features from the input image to obtain feature maps (original feature image features). The FPN network is used to generate and fuse multi-scale features, fusing high-level refined semantic features with low-level global features to obtain image features (P3-P7). Then, the component detection sub-network (head) detects component boxes such as patterns, logos, and text in the target feature region, thereby obtaining refined semantic features of the target feature region.

[0114] After detecting at least one component region within the target feature region based on feature image characteristics, feature extraction can be performed on the component region to obtain component features. There are various ways to extract features from the component region. For example, features can be extracted from the image corresponding to the component region to obtain component features, or features can be extracted from the components within the component region to obtain component features, and so on.

[0115] 104. Fuse component features with element image features to obtain the element features of the image elements.

[0116] Among them, feature features can be understood as feature information that indicates the feature category of image features.

[0117] There are several ways to fuse component features with feature image features, including the following:

[0118] For example, attention weighting can be applied to element image features based on component features to obtain spatial attention features of image elements. Text features can be extracted from road images, and the text features can be fused with spatial attention features to obtain element features of image elements. Specifically, this can be done as follows:

[0119] S1. Based on component features, attention weighting is applied to the feature image to obtain the spatial attention features of the image features.

[0120] Spatial attention features can be understood as features obtained by performing spatial attention modeling on the feature image. There are several ways to apply attention weighting to feature image features, including the following:

[0121] For example, the underlying image features corresponding to each feature location can be extracted from the image features. Based on the component features, the attention weights corresponding to the underlying image features can be determined. The underlying image features can be weighted according to the attention weights. The weighted underlying image features can be fused to obtain the spatial attention features of the image elements.

[0122] In this context, feature location can be understood as the position of each pixel in the target feature region, or as the position of each feature point in the image features. There are several ways to extract the underlying image features corresponding to each feature location from the image features. For example, the image features can be sorted based on their feature size, and the image features with the largest feature size can be selected from the sorted features to obtain the target image features. Then, the image features corresponding to each feature location can be extracted from the target image features to obtain the underlying image features.

[0123] After extracting the underlying image features corresponding to each feature location from the image features, the attention weights corresponding to the underlying image features can be determined based on the component features. There are several ways to determine the attention weights corresponding to the underlying image features. For example, one method is to filter for target component features corresponding to the feature location within the component features. When a target component feature exists, it is used as the attention weight for that feature location. When a target component feature does not exist, it means that there is no component at that feature location. In this case, the feature corresponding to that feature location can be extracted from the component's regional features, thus obtaining the attention weight for that feature location. Therefore, the attention weight here can be understood as the feature value output by the local feature extraction network of the recognition model at that feature location.

[0124] After determining the attention weights corresponding to the underlying image features, the underlying image features can be weighted according to the attention weights to obtain the weighted underlying image features, as shown in formula (1). Specifically, it can be as follows:

[0125]

[0126] Wherein, s(x i (x, q) represents the weighted low-level image feature at the i-th feature position, where i is the i-th feature position, and x... i Let q be the underlying image feature at the i-th feature location, and q be the attention weight corresponding to that feature location (in regions where there are feature components, this attention weight can be the component feature).

[0127] After weighting the underlying image features according to the attention weights, the weighted underlying image features can be fused to obtain the spatial attention features of the image elements.

[0128] It should be noted that in the process of spatial attention modeling, the component features are passed to the classification branch (the element classification network of the image recognition model) through attention weighting, which can robustly guide the learning of the classification branch and improve the accuracy of image recognition.

[0129] S2. Extract text features from the road image and fuse the text features with spatial attention features to obtain the feature features of the image elements.

[0130] There are several ways to extract text features from road images, including the following:

[0131] For example, at least one text content can be identified in a road image, and the content location of the text content in the road image can be determined. Based on the content location, feature extraction is performed on the text content to obtain the text features of the road image.

[0132] There are several ways to identify at least one text content in a road image. For example, OCR (Optical Character Recognition) can be used to identify at least one text content in a road image, or a text recognition model can be used to extract text features from the road image and determine at least one text content in the road image based on the text features, and so on.

[0133] After identifying at least one piece of text in a road image, the location of the text content within the road image can be determined. Based on this location, features can be extracted from the text content to obtain the text features of the road image. There are several ways to extract features from text content based on its location. For example, features can be extracted from the text content to obtain initial text features, the location of the text content can be encoded, and the encoded features can be fused with the initial text features to obtain the text features of the road image. Alternatively, regional features of the image region corresponding to the content location can be extracted from the road image, features can be extracted from the text content, and the extracted initial text features can be fused with the regional features to obtain the text features of the road image. Or, features can be extracted directly from the text content to obtain the text features of the road image, and so on.

[0134] After extracting text features from a road image, these text features can be fused with spatial attention features to obtain the feature features of the image element. There are several ways to fuse text features with spatial attention features. For example, the spatial attention features can be dimensionalized to obtain target spatial attention features of a preset dimension, and the text features can be dimensionalized to obtain target text features of a preset dimension. The target spatial attention features and target text features can then be concatenated to obtain the feature features of the image element.

[0135] There are several ways to perform dimensional transformation on spatial attention features. For example, spatial attention features can be transformed using convolutional layers, pooling layers, and fully connected layers to obtain target spatial attention features of a preset dimension.

[0136] The preset dimension can be understood as a pre-defined feature dimension, such as one-dimensional (1*n) or other dimensions, etc.

[0137] The method for dimensional transformation of text features is similar to that for spatial attention features, as detailed above, and will not be repeated here. It should be noted that the transformed target spatial attention features and target text features have the same dimensionality.

[0138] After dimensional transformation of the spatial attention features and text features, the transformed target spatial attention features and target text features can be concatenated to obtain the feature features of the image elements. There are several ways to concatenate the target spatial attention features and target text features. For example, the target spatial attention features and target text features can be concatenated directly to obtain the feature features of the image elements, as shown in formula (2). Specifically, it can be as follows:

[0139]

[0140] Wherein, s(x i ,q,c) are the feature features of the image elements. Let FC(c) be the target text feature, i be the position of the i-th feature, and x be the target spatial attention feature. i Let q be the underlying image feature at the i-th feature location, q be the attention weight corresponding to that feature location, and c be the text feature.

[0141] The process of fusing component features with element image features can be divided into two parts, mainly including spatial attention modeling and multimodal modeling, which can be like... Figure 14 As shown, the specific details are as follows:

[0142] (1) Spatial attention modeling: The image features to be fused are the underlying image features, and the corresponding part features can be the feature values ​​of the part detection branch at the corresponding position. The feature values ​​here can also be understood as attention weights. The part features are passed to the classification branch through multiplication, thereby robustly guiding the learning of the classification branch and improving the accuracy of image recognition.

[0143] (2) Multimodal modeling: Text features (c) can be the text line feature values ​​extracted from the component region. The text features and spatial attention features are transformed in dimension through convolutional layers, pooling layers and fully connected layers to obtain features of the same dimension. Finally, multimodal fusion is performed to improve the accuracy of image feature recognition.

[0144] 105. Based on the feature characteristics, classify the image features to obtain the target feature category of the image features.

[0145] For example, image features can be classified using an image recognition model's classification network based on feature characteristics, thereby obtaining the target feature category of the image features.

[0146] The network structure of a classification network can be varied, such as fully connected layers (FC layers) or other multi-classification networks, etc.

[0147] Taking traffic signs as an example, the overall process of traffic sign classification in the image recognition method of this solution can be described as follows: Figure 15 As shown, after detecting the target element region of the traffic sign in the road image, the component detection module can extract local features to obtain refined component feature information (component features) of the traffic sign image, implicitly improving the classification ability of the image recognition model. Then, the feature fusion module is used to fuse the element image features (image features of the traffic sign), text features, and refined component feature information (component features), explicitly improving the recognition ability of the image recognition model. Finally, the classification result of the traffic sign is output through a fully connected layer, thereby improving the accuracy of image recognition.

[0148] As can be seen from the above, in this embodiment of the application, after acquiring at least one road image and identifying at least one target element region of an image element in the road image, multi-dimensional feature extraction is performed on the target element region to obtain at least one dimension of element image features. Then, based on the element image features, at least one component region of an element component is detected in the target element region, and feature extraction is performed on the component region to obtain component features. Then, the component features are fused with the element image features to obtain the element features of the image element. Then, the image element is classified according to the element features to obtain the target element category of the image element. Since this scheme can extract at least one dimension of element image features in the target element region, detect at least one component region of an element component in the target element region, and extract component features in the component region, thereby using the rich component features within the element component to guide the classification of the image element, it effectively solves the problem of complex and variable information within the image element and large interference between categories. Therefore, it can improve the accuracy of image recognition.

[0149] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0150] In this embodiment, the image recognition device will be specifically integrated into an electronic device, with the electronic device serving as a server, the image element being a traffic sign, the element region being the sign region, the element feature being the sign feature, and the element image feature being the sign image feature, as an example for explanation.

[0151] (I) Training of the Pre-set Image Recognition Model

[0152] The preset image recognition model may include a local feature extraction network and a classification network (FC layer).

[0153] (1) The server obtains at least one traffic sign sample and uses a preset image recognition model to extract multi-dimensional features from the traffic sign sample to obtain sample image features in multiple dimensions.

[0154] For example, the server can obtain at least one road image sample, segment at least one sign area from the road image sample to obtain a traffic sign sample, or it can directly obtain a traffic sign sample, and so on.

[0155] The server uses a pre-defined image recognition model to extract multi-dimensional features from traffic sign samples, resulting in multi-dimensional sample image features.

[0156] (2) The server extracts at least one element component feature from the sample image features and extracts sample text features from the traffic sign sample.

[0157] (3) The server fuses the sample image features, sample component features and sample text features, and predicts the category of the traffic sign sample based on the fused sample element features to obtain the predicted element category.

[0158] (4) The server converges the preset image recognition model based on the predicted element category to obtain the image recognition model.

[0159] For example, the server can obtain the labeled feature categories of traffic sign samples and compare them with the predicted feature categories to obtain the classification loss of the traffic sign samples.

[0160] The server can update the network parameters of a preset image recognition model using a gradient descent algorithm based on the classification loss, obtaining an updated image recognition model. This updated model is then used as the preset image recognition model, and the server returns to execute the step of acquiring at least one traffic sign sample until the preset image recognition model converges, thus obtaining the trained image recognition model. Alternatively, the server can update the network parameters of the preset image recognition model using other parameter update algorithms based on the classification loss, obtaining an updated image recognition model. This updated model is then used as the preset image recognition model, and the server returns to execute the step of acquiring at least one traffic sign sample until the preset image recognition model converges, thus obtaining the trained image recognition model, and so on.

[0161] (II) Processing road images using image recognition models

[0162] like Figure 16 As shown, an image recognition method has the following specific process:

[0163] 201. The server obtains at least one road image.

[0164] For example, a server can acquire images of the road ahead using an in-vehicle image acquisition device or component, thereby obtaining at least one road image; or it can acquire at least one image of the target road using an image acquisition device, thereby obtaining a road image; or it can receive at least one road image sent by an image acquisition vehicle; or it can obtain at least one road image from a network or map database; or it can receive at least one road image uploaded by a terminal or client; or it can receive an image recognition request carrying the storage address of at least one road image, and obtain at least one road image based on that storage address, and so on.

[0165] 202. The server identifies at least one target sign area for a traffic sign in the road image.

[0166] For example, a server can use convolutional layers to extract the edges, textures, or other basic features of a road image, thereby obtaining the original image features.

[0167] The server can use a normalization layer to normalize the original image features according to a normal distribution, thereby filtering out noise features in the original image features and obtaining filtered image features. An activation layer can then be used to perform a non-linear mapping on the filtered image features to obtain the image features of the road image, and so on.

[0168] The server can identify the location of each feature point in the road image and obtain the feature point location. Based on the feature point location, it can segment the image region of at least one scale in the road image with three preset aspect ratios of {1:1, 2:1, 1:2} and scales of 1 feature point, 2 feature points, and 3 feature points, respectively, and use the image region as a candidate sign region.

[0169] The server can perform traffic sign detection (i.e., binary classification) in the candidate sign area. Based on the detection results, it can filter out the areas containing traffic signs in the candidate sign area to obtain at least one sign area. The server can then adjust the position of the sign area to obtain the target sign area for the traffic sign.

[0170] 203. The server performs multi-dimensional feature extraction on the target signage area to obtain signage image features in at least one dimension.

[0171] For example, the server can adjust the size of the target sign area to 300*300 pixels to obtain the adjusted target sign area.

[0172] The server can use the ResNet image recognition model to extract multi-dimensional features from the target sign area, thereby obtaining the original sign image features in multiple dimensions.

[0173] The server can filter out the original signage image features with the smallest feature size from the original image features to obtain the initial semantic features corresponding to the target dimension. Alternatively, it can filter out the original signage image features with the highest dimension from the original signage image features to obtain the initial semantic features corresponding to the target dimension, and so on.

[0174] The server can perform multi-dimensional feature extraction on the initial semantic features to obtain at least one semantic feature. Based on the feature size of the original element image, the original sign image features are sorted. Taking sorting in ascending order as an example, the next-ranked original sign image feature from the original sign image features can be selected as the current sign image feature to be fused with the initial semantic feature.

[0175] The server upsamples the initial semantic features to obtain adjusted semantic features, which can be the same size as the current sign image features.

[0176] The server can directly concatenate the adjusted semantic features with the current sign image features to obtain the fused sign image features. Alternatively, it can obtain the fusion weights and, based on these weights, weight the adjusted semantic features and the current sign image features separately, then concatenate the weighted adjusted semantic features with the weighted current sign image features to obtain the fused sign image features, and so on.

[0177] The server uses the fused signage image features as the initial semantic features and returns to execute the step of selecting the current signage image features that need to be fused with the initial semantic features from the original signage image features according to the sorting results, until all the original signage image features are fused, thus obtaining at least one fused signage image feature.

[0178] The server can use semantic features, initial semantic features, and fused sign image features as the sign image features of traffic signs.

[0179] 204. Based on the features of the sign image, the server detects at least one component region of an element in the target sign area.

[0180] For example, a server can extract the regional features of at least one element component from the features of a sign image using a local feature extraction network of an image recognition model.

[0181] The server can extract the location of the center point of the feature component and the boundary distance between the center point and the region boundary from the region location information. Based on the boundary distance and the center point location, it can identify the initial component region of the feature component in the target signage region. Alternatively, it can extract the location of the center point of the feature component and the endpoint distance between the center point and at least one endpoint from the region location information. Based on the endpoint distance and the center point location, it can identify at least one endpoint of the feature component in the target signage region. Based on the endpoint location, it can detect the initial component region of the feature component in the target signage region, and so on.

[0182] The server can deduplicate the initial component regions to obtain the component regions of the feature components. Alternatively, it can detect pairs of initial component regions that have overlapping areas in the initial component regions, and merge the initial component regions in the pairs of initial component regions according to the size of the overlapping areas to obtain the component regions of the feature components, and so on.

[0183] 205. The server extracts features from the component region to obtain component features.

[0184] For example, the server can extract features from the image corresponding to the component region to obtain component features, or it can extract features from the components within the component region to obtain component features, and so on.

[0185] 206. The server performs attention weighting on the sign image features based on component features to obtain the spatial attention features of traffic signs.

[0186] For example, the server can sort image features based on their feature size, select the image features with the largest feature size from the sorted features to obtain the target image features, and extract the image features corresponding to each feature position from the target image features to obtain the underlying image features.

[0187] The server filters the target component features corresponding to the feature locations from the component features. When a target component feature exists, it is used as the attention weight for that feature location. When no target component feature exists, it means that there is no component at that feature location. In this case, the feature corresponding to that feature location can be extracted from the regional features of the component, thus obtaining the attention weight for that feature location. Therefore, the attention weight here can be understood as the feature value output by the local feature extraction network of the recognition model at that feature location.

[0188] The server weights the underlying image features according to the attention weights to obtain the weighted underlying image features, as shown in formula (1). The weighted underlying image features are then fused to obtain the spatial attention features of the traffic sign.

[0189] 207. The server extracts text features from the road image and fuses the text features with spatial attention features to obtain the sign features of the traffic sign.

[0190] For example, the server can use OCR (Optical Character Recognition) technology to identify at least one text content in a road image, or it can use a text recognition model to extract text features from the road image and determine at least one text content in the road image based on the text features, and so on.

[0191] The server determines the location of text content within a road image. Features are extracted from the text content to obtain initial text features. The content location is then encoded, and the encoded features are fused with the initial text features to obtain the text features of the road image. Alternatively, region features corresponding to the content location can be extracted from the road image, and features can be extracted from the text content. The extracted initial text features are then fused with the region features to obtain the text features of the road image. Alternatively, features can be extracted directly from the text content to obtain the text features of the road image, and so on.

[0192] The server can perform dimensionality transformation on spatial attention features using convolutional layers, pooling layers, and fully connected layers to obtain one-dimensional target spatial attention features. Similarly, the server performs dimensionality transformation on text features to obtain one-dimensional target text features.

[0193] The server can directly concatenate the target space attention features and the target text features to obtain the sign features of the traffic sign, as shown in formula (2).

[0194] 208. The server classifies traffic signs according to their characteristics to obtain the target sign category.

[0195] For example, the server can classify traffic signs using a fully connected layer based on the sign characteristics, thereby obtaining the target sign category.

[0196] As can be seen from the above, in this embodiment of the application, after acquiring at least one road image and identifying at least one target sign area of ​​a traffic sign in the road image, the server performs multi-dimensional feature extraction on the target sign area to obtain at least one dimension of sign image features. Then, based on the sign image features, at least one component area of ​​an element is detected in the target sign area, and features are extracted from the component area to obtain component features. Then, the component features are fused with the sign image features to obtain the sign features of the traffic sign. Then, the traffic sign is classified according to the sign features to obtain the target sign category of the traffic sign. Since this scheme can extract at least one dimension of sign image features in the target sign area, detect at least one component area of ​​a sign component in the target sign area, and extract component features from the component area, thereby using the rich component features within the sign component to guide the classification of traffic signs, it effectively solves the problem of complex and variable information within traffic signs and large interference between categories. Therefore, it can improve the accuracy of image recognition.

[0197] To better implement the above methods, this application also provides an image recognition device that can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0198] For example, such as Figure 17 As shown, the image recognition device may include an acquisition unit 301, an extraction unit 302, a detection unit 303, a fusion unit 304, and a classification unit 305, as follows:

[0199] (1) Obtain unit 301;

[0200] The acquisition unit 301 is used to acquire at least one road image and identify at least one target element region of an image element in the road image, wherein the image element indicates target physical point information in the road image.

[0201] For example, the acquisition unit 301 can be used to acquire at least one road image, extract features from the road image to obtain image features of the road image, identify feature points of at least one image element in the road image based on the image features, identify at least one candidate element region of the image element in the road image based on the feature points, and select the target element region from the candidate element region.

[0202] (2) Extraction unit 302;

[0203] Extraction unit 302 is used to extract multi-dimensional features from the target feature region to obtain feature image features of at least one dimension.

[0204] For example, extraction unit 302 can be used to adjust the size of the target feature region and use an image recognition model to extract multi-dimensional features from the adjusted target feature region to obtain original feature image features of multiple dimensions. The target dimension of the original feature image features are selected from the original feature image features to obtain initial semantic features. Based on the initial semantic features, the original feature image features are fused to obtain feature image features of at least one dimension.

[0205] (3) Detection unit 303;

[0206] The detection unit 303 is used to detect at least one component region of a component in the target component region based on the feature of the feature image, and to extract features from the component region to obtain component features.

[0207] For example, the detection unit 303 can be used to extract the regional features of at least one element component from the feature image features, and based on the regional features, determine the regional location information of the element component, identify the initial component region of the element component in the target feature region according to the regional location information, and fuse the initial component regions to obtain the component region of the element component.

[0208] (4) Fusion unit 304;

[0209] The fusion unit 304 is used to fuse component features with feature image features to obtain feature features of image features.

[0210] For example, the fusion unit 304 can be used to extract the underlying image features corresponding to each feature position in the image features, determine the attention weights corresponding to the underlying image features based on the component features, and weight the underlying image features according to the attention weights. The weighted underlying image features are then fused to obtain the spatial attention features of the image elements. Text features are extracted from the road image, and the spatial attention features are dimensionally transformed to obtain the target spatial attention features of the preset dimensions. The text features are dimensionally transformed to obtain the target text features of the preset dimensions. The target spatial attention features and the target text features are then concatenated to obtain the element features of the image elements.

[0211] (5) Classification unit 305;

[0212] The classification unit 305 is used to classify image elements according to their features to obtain the target element category of the image elements.

[0213] For example, classification unit 305 can be used to classify image elements according to the feature characteristics using a classification network of an image recognition model, thereby obtaining the target feature category of the image elements.

[0214] Optionally, in some embodiments, the image recognition device may further include a training unit 306, such as... Figure 18 As shown, the specific details are as follows:

[0215] Training unit 306 is used to train a preset image recognition model to obtain an image recognition model.

[0216] For example, training unit 306 can be used to acquire at least one image element sample and use a preset image recognition model to extract multi-dimensional features from the image element sample to obtain multi-dimensional sample image features. It extracts sample component features of at least one element from the sample image features and sample text features from the image element sample. The sample image features, sample component features, and sample text features are then fused. Based on the fused sample element features, the category of the image element sample is predicted to obtain the predicted element category. Based on the predicted element category, the preset image recognition model is converged to obtain the image recognition model.

[0217] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0218] As can be seen from the above, in this embodiment of the application, after the acquisition unit 301 acquires at least one road image and identifies at least one target element region of an image element in the road image, the extraction unit 302 performs multi-dimensional feature extraction on the target element region to obtain at least one dimension of element image features. Then, the detection unit 303 detects at least one component region of an element component in the target element region based on the element image features, and extracts features from the component region to obtain component features. Then, the fusion unit 304 fuses the component features with the element image features to obtain the element features of the image element. Then, the classification unit 305 classifies the image element according to the element features to obtain the target element category of the image element. Since this scheme can extract at least one dimension of element image features in the target element region, detect at least one component region of an element component in the target element region, and extract component features from the component region, thereby using the rich component features within the element component to guide the classification of the image element, it effectively solves the problem of complex and variable information within the image element and large interference between categories. Therefore, it can improve the accuracy of image recognition.

[0219] This application also provides an electronic device, such as... Figure 19 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0220] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 19 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0221] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0222] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0223] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0224] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0225] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0226] At least one road image is acquired, and at least one target element region of an image element is identified in the road image. The image element indicates the target physical point information in the road image. Multi-dimensional feature extraction is performed on the target element region to obtain at least one dimension of element image features. Based on the element image features, at least one component region of an element component is detected in the target element region, and feature extraction is performed on the component region to obtain component features. The component features are fused with the element image features to obtain the element features of the image element. Based on the element features, the image element is classified to obtain the target element category of the image element.

[0227] For example, an electronic device can acquire at least one road image, extract features from the road image to obtain its image features, identify feature points of at least one image element in the road image based on these features, identify at least one candidate element region of the image element based on these feature points, and select a target element region from the candidate element regions. The size of the target element region is adjusted, and an image recognition model is used to extract multi-dimensional features from the adjusted target element region to obtain original element image features of multiple dimensions. The target dimension of the original element image features is selected from the original element image features to obtain initial semantic features. Based on the initial semantic features, the original element image features are fused to obtain element image features of at least one dimension. At least one element component's region features are extracted from the element image features, and based on these region features, the region location information of the element component is determined. Based on the region location information, the initial component region of the element component is identified in the target element region, and the initial component regions are fused to obtain the component region of the element component. The process involves extracting the underlying image features corresponding to each feature location from the image features. Based on component features, attention weights are determined for each underlying image feature. These attention weights are then weighted, and the weighted underlying image features are fused to obtain the spatial attention features of the image elements. Text features are extracted from the road image. The spatial attention features undergo dimensionality transformation to obtain target spatial attention features of a preset dimension. Similarly, the text features undergo dimensionality transformation to obtain target text features of a preset dimension. The target spatial attention features and target text features are then concatenated to obtain the element features of the image elements. Based on these element features, a classification network of an image recognition model is used to classify the image elements, thereby obtaining the target element category of the image elements.

[0228] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0229] As can be seen from the above, in this embodiment of the application, after acquiring at least one road image and identifying at least one target element region of an image element in the road image, multi-dimensional feature extraction is performed on the target element region to obtain at least one dimension of element image features. Then, based on the element image features, at least one component region of an element component is detected in the target element region, and feature extraction is performed on the component region to obtain component features. Then, the component features are fused with the element image features to obtain the element features of the image element. Then, the image element is classified according to the element features to obtain the target element category of the image element. Since this scheme can extract at least one dimension of element image features in the target element region, detect at least one component region of an element component in the target element region, and extract component features in the component region, thereby using the rich component features within the element component to guide the classification of the image element, it effectively solves the problem of complex and variable information within the image element and large interference between categories. Therefore, it can improve the accuracy of image recognition.

[0230] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0231] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image recognition methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0232] At least one road image is acquired, and at least one target element region of an image element is identified in the road image. The image element indicates the target physical point information in the road image. Multi-dimensional feature extraction is performed on the target element region to obtain at least one dimension of element image features. Based on the element image features, at least one component region of an element component is detected in the target element region, and feature extraction is performed on the component region to obtain component features. The component features are fused with the element image features to obtain the element features of the image element. Based on the element features, the image element is classified to obtain the target element category of the image element.

[0233] For example, acquire at least one road image, extract features from the road image to obtain its image features, identify feature points of at least one image element in the road image based on these features, identify at least one candidate element region of the image element based on these feature points, and select a target element region from the candidate element regions. Adjust the size of the target element region, and use an image recognition model to extract multi-dimensional features from the adjusted target element region to obtain original element image features of multiple dimensions. Select the target dimension original element image features from the original element image features to obtain initial semantic features. Based on the initial semantic features, fuse the original element image features to obtain element image features of at least one dimension. Extract the region features of at least one element component from the element image features, determine the region location information of the element component based on the region features, identify the initial component region of the element component in the target element region based on the region location information, and fuse the initial component regions to obtain the component region of the element component. The process involves extracting the underlying image features corresponding to each feature location from the image features. Based on component features, attention weights are determined for each underlying image feature. These attention weights are then weighted, and the weighted underlying image features are fused to obtain the spatial attention features of the image elements. Text features are extracted from the road image. The spatial attention features undergo dimensionality transformation to obtain target spatial attention features of a preset dimension. Similarly, the text features undergo dimensionality transformation to obtain target text features of a preset dimension. The target spatial attention features and target text features are then concatenated to obtain the element features of the image elements. Based on these element features, a classification network of an image recognition model is used to classify the image elements, thereby obtaining the target element category of the image elements.

[0234] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0235] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0236] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image recognition methods provided in the embodiments of this application, the beneficial effects that any of the image recognition methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0237] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the image recognition, image feature classification, or traffic sign classification methods described above.

[0238] The foregoing has provided a detailed description of an image recognition method and related equipment provided in the embodiments of this application. The related equipment may include image recognition devices, electronic devices, computer program products, and computer-readable storage media. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image recognition method, characterized in that, include: Acquire at least one road image and identify at least one target feature region of an image feature in the road image, the image feature indicating target physical point information in the road image; Multi-dimensional feature extraction is performed on the target feature region to obtain feature image features of at least one dimension; Based on the feature image features, at least one component region of a feature component is detected in the target feature region, and feature extraction is performed on the component region to obtain component features; The component features are fused with the element image features to obtain the element features of the image elements; Based on the feature characteristics, the image features are classified to obtain the target feature category of the image features.

2. The image recognition method according to claim 1, characterized in that, The process of fusing the component features with the element image features to obtain the element features of the image features includes: Based on the component features, attention weighting is applied to the element image features to obtain the spatial attention features of the image elements; Text features are extracted from the road image, and the text features are fused with the spatial attention features to obtain the feature features of the image elements.

3. The image recognition method according to claim 2, characterized in that, The step of performing attention weighting on the element image features based on the component features to obtain the spatial attention features of the image features includes: Extract the underlying image features corresponding to each feature location from the image features; Based on component features, the attention weights corresponding to the underlying image features are determined, and the underlying image features are weighted according to the attention weights. The weighted underlying image features are fused to obtain the spatial attention features of the image elements.

4. The image recognition method according to claim 3, characterized in that, The step of extracting the underlying image features corresponding to each pixel position from the image features includes: The image features are sorted based on their feature dimensions. Based on the sorting results, the image features with the largest feature size are selected from the image features to obtain the target image features; The image features corresponding to each feature position are extracted from the target image features to obtain the bottom image features.

5. The image recognition method according to claim 2, characterized in that, The process of fusing the text features with the spatial attention features to obtain the feature features of the image elements includes: The spatial attention features are dimensionally transformed to obtain target spatial attention features of a preset dimension; The text features are transformed to obtain the target text features of the preset dimensions; The target spatial attention features and the target text features are concatenated to obtain the feature features of the image elements.

6. The image recognition method according to claim 2, characterized in that, The extraction of text features from the road image includes: Identify at least one text content in the road image and determine the content location of the text content in the road image; Based on the content location, feature extraction is performed on the text content to obtain the text features of the road image.

7. The image recognition method according to claim 1, characterized in that, The step of detecting at least one component region of a feature component in the target feature region based on the feature image features includes: At least one feature region is extracted from the feature image features, and the region location information of the feature region is determined based on the feature region. Based on the location information of the region, the initial component region of the element component is identified in the target element region; The initial component regions are merged to obtain the component regions of the element components.

8. The image recognition method according to claim 7, characterized in that, The step of identifying the initial component region of the feature component in the target feature region based on the region location information includes: Extract the location of the center point of the element component and the boundary distance between the center point and the region boundary from the region location information. Based on the boundary distance and the position of the center point, the initial component region of the feature component is identified in the target feature region.

9. The image recognition method according to claim 7, characterized in that, The step of identifying the initial component region of the feature component in the target feature region based on the region location information includes: Extract the location of the center point of the element component and the endpoint distance between the center point and at least one endpoint from the regional location information; Based on the endpoint distance and the position of the center point, at least one endpoint of the feature component is identified in the target feature region; Based on the position of the endpoint, the initial component region of the feature component is detected in the target feature region.

10. The image recognition method according to claim 1, characterized in that, The step of extracting multi-dimensional features from the target feature region to obtain feature image features of at least one dimension includes: The size of the target feature region is adjusted, and an image recognition model is used to extract multi-dimensional features from the adjusted target feature region to obtain multi-dimensional original feature image features. The target dimension of the original feature image features are selected from the original feature image features to obtain the initial semantic features; Based on the initial semantic features, the original feature image features are fused to obtain feature image features of at least one dimension.

11. The image recognition method according to claim 10, characterized in that, The step of fusing the original feature image features based on the initial semantic features to obtain feature image features of at least one dimension includes: Multi-dimensional feature extraction is performed on the initial semantic features to obtain at least one semantic feature; The original feature image features are fused at least once to obtain at least one fused feature image feature. The semantic features, the initial semantic features, and the fused element image features are used as the element image features of the image elements.

12. The image recognition method according to claim 11, characterized in that the step of fusing the original element image features at least once to obtain at least one fused element image feature includes: Based on the feature size of the original feature image, the original feature image features are sorted. Based on the sorting results, select the current element image features that need to be fused with the initial semantic features from the original element image features; The initial semantic features are fused with the current feature image features to obtain at least one fused feature image feature.

13. The image recognition method according to claim 12, characterized in that, The step of fusing the initial semantic features with the current feature image to obtain at least one fused feature image includes: The size of the initial semantic features is adjusted to obtain the adjusted semantic features; The adjusted semantic features are fused with the current element image features to obtain fused element image features, and the fused element image features are used as the initial semantic features; Return to the step of selecting current feature images that need to be fused with the initial semantic features from the original feature image features based on the sorting result, until all the original feature image features have been fused, to obtain at least one fused feature image feature.

14. The image recognition method according to claim 10 is characterized in that, Before using an image recognition model to extract multi-dimensional features from the adjusted target feature region to obtain multi-dimensional original feature image features, the process also includes: At least one image element sample is obtained, and a preset image recognition model is used to extract multi-dimensional features from the image element sample to obtain sample image features in multiple dimensions. Extract at least one element component feature from the sample image features, and extract sample text features from the image element samples; The sample image features, sample component features, and sample text features are fused together, and the category of the image element sample is predicted based on the fused sample element features to obtain the predicted element category; Based on the predicted feature category, the classification loss of the image feature sample is determined, and the preset image recognition model is converged according to the classification loss to obtain the image recognition model.

15. The image recognition method according to claim 1 is characterized in that, The process of identifying a target feature region containing at least one image feature in the road image includes: Feature extraction is performed on the road image to obtain the original image features, and noise features are filtered out from the original image features to obtain the image features; Based on the image features, at least one candidate feature region is identified in the road image; The target feature region is selected from the candidate feature region to identify at least one image feature.

16. The image recognition method according to claim 15, characterized in that, The step of identifying at least one candidate feature region in the road image based on the image features includes: The location of each feature point in the image features is identified in the road image to obtain the feature point location; Based on the location of the feature points, an image region of at least one scale is segmented in the road image according to at least one preset aspect ratio, and the image region is used as a candidate feature region.

17. An image recognition device, characterized in that, include: The acquisition unit is configured to acquire at least one road image and identify a target feature region of at least one image feature in the road image, wherein the image feature indicates target physical point information in the road image; An extraction unit is used to extract multi-dimensional features from the target feature region to obtain feature image features of at least one dimension. The detection unit is configured to detect at least one component region of a component in the target feature region based on the feature image features, and to extract features from the component region to obtain component features. A fusion unit is used to fuse the component features with the element image features to obtain the element features of the image elements; A classification unit is used to classify the image elements according to the feature characteristics to obtain the target feature category of the image elements.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program within the memory to perform the steps of the image recognition method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the image recognition method according to any one of claims 1 to 16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image recognition method according to any one of claims 1 to 16.

Images