Image recognition method, device, equipment and storage medium

Abstract

The present application discloses an image recognition method, apparatus, device, and storage medium. The method comprises: obtaining an image to be recognized, performing feature extraction processing on the image to be recognized, obtaining a target area of a map element to be recognized in the image to be recognized; generating multiple sub-areas based on the target area, performing feature extraction on each sub-area respectively, obtaining image features of each sub-area; fusing the image features of all sub-areas to obtain fused features, and determining the recognition result of the map element to be recognized based on the fused features. This technical solution can extract the image features of each sub-area in the image to be recognized at a finer granularity, enhance the saliency of the target area by fusing the image features of all sub-areas, combine more comprehensive features to determine the recognition result of the map element to be recognized, and improve the accuracy of map element recognition.

Description

Technical Field

[0001] The present invention generally relates to the field of image recognition technology, and in particular to an image recognition method, device, equipment and storage medium. Background Art

[0002] With the continuous development of computer technology and image processing technology, image recognition, as the foundation of practical technologies such as stereo vision, motion analysis, and data fusion, has been widely applied in various fields, such as autonomous driving, vehicle navigation, map and terrain registration, natural resource analysis, environmental monitoring, and physiological pathology research. In the application process of image recognition, in order to facilitate the use of image elements to update map data or perform unmanned driving, it is necessary to identify the category of image elements from the collected road images.

[0003] At present, related technologies can use a large amount of labeled data for training to obtain a classification network model, and then use the classification network model to extract semantic features and classify images to obtain image recognition results.

[0004] However, in the process of collecting map road data, due to the poor quality of collected images, the large number of map elements in the annotated data and their uneven distribution, image recognition through the classification network model may result in false detection and recognition type errors, resulting in low accuracy of image type recognition. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide an image recognition method, apparatus, device, and storage medium that can extract image features of each sub-region in the image to be recognized in a more fine-grained manner, thereby improving the accuracy of the recognition results of the map elements to be recognized. The technical solution is as follows:

[0006] According to one aspect of the present application, there is provided an image recognition method, the method comprising:

[0007] Acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified;

[0008] generating a plurality of sub-regions based on the target region, performing feature extraction on each of the sub-regions to obtain image features of each sub-region;

[0009] The image features of all the sub-regions are fused to obtain fused features, and the recognition result of the map element to be recognized is determined based on the fused features.

[0010] In one embodiment, generating a plurality of sub-regions based on the target region includes:

[0011] Performing background addition processing on the target area to obtain a candidate window; the candidate window includes the target area;

[0012] The candidate window is divided n times according to n division sizes to obtain a plurality of sub-regions of different sizes, where n is a positive integer and n≥1.

[0013] In one embodiment, performing background addition processing on the target area to obtain a candidate window includes:

[0014] The target area is resized to obtain the candidate window.

[0015] In one embodiment, the expanding the target area includes:

[0016] The target area is expanded according to a preset ratio with the map element to be identified as the center.

[0017] In one embodiment, fusing the image features of all the sub-regions to obtain fused features includes:

[0018] Performing vectorization processing on the image features of each of the sub-regions to obtain a feature vector corresponding to the sub-region;

[0019] Determine a direction value corresponding to each eigenvalue in the eigenvector corresponding to each sub-region, wherein the direction value is used to indicate whether the sub-region includes a target region;

[0020] A fusion feature is obtained according to the feature value corresponding to each of the sub-regions and the direction value corresponding to the sub-region.

[0021] In one embodiment, determining the recognition result of the map element to be recognized based on the fusion feature includes:

[0022] The fusion features are input into a trained detection network, and the recognition results of the traffic elements to be identified are determined based on the output of the detection network; the detection model is trained based on historical images and map element recognition results of historical images, and the historical images are marked with traffic element areas.

[0023] In one embodiment, the training process of the detection network includes:

[0024] generating a plurality of sample sub-regions based on the historical image, and inputting each of the sample sub-regions into a feature extraction network to be trained for feature extraction processing to obtain image features of each sample sub-region;

[0025] The image features of all sample sub-regions are input into the fusion network to be trained for fusion processing to obtain the sample fusion features;

[0026] Inputting the sample fusion features into the detection network to be trained to obtain the prediction results of the map elements;

[0027] Calculating a loss function based on the prediction results of the map elements and the map element recognition results of the historical images;

[0028] According to the minimization of the loss function, an iterative algorithm is used to iteratively adjust the parameters of the feature extraction network to be trained, the fusion network to be trained, and the detection network to be trained to obtain a detection network.

[0029] In one embodiment, performing feature extraction processing on the image to be identified to obtain a target area of the map element to be identified in the image to be identified includes:

[0030] Performing feature extraction processing on the image to be identified to obtain a feature map of the image to be identified;

[0031] For each feature point in the feature map, determining a candidate box corresponding to each feature point; the candidate box contains the feature point;

[0032] Detecting map elements for the image features in each candidate frame to obtain a confidence level corresponding to the candidate frame; the confidence level is used to represent the probability that the image in the candidate frame is a map element;

[0033] The area corresponding to the candidate box whose confidence meets the preset rules is determined as the target area of the map element to be identified in the image to be identified.

[0034] In one embodiment, for each feature point in the feature map, determining a candidate box corresponding to each feature point includes:

[0035] For each feature point in the feature map, with the feature point as the center, candidate boxes corresponding to each feature point are generated according to different sizes and scaling ratios.

[0036] According to another aspect of the present application, an image recognition device is provided, the device comprising:

[0037] an acquisition module, configured to acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified;

[0038] A feature extraction module is used to generate multiple sub-regions based on the target region, perform feature extraction on each sub-region, and obtain image features of each sub-region;

[0039] The element recognition module is used to fuse the image features of all the sub-areas to obtain fusion features, and determine the recognition result of the map element to be recognized based on the fusion features.

[0040] According to another aspect of the present application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image recognition method when executing the program.

[0041] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is used to implement the above-mentioned image recognition method.

[0042] According to another aspect of the present application, a computer program product is provided, which includes instructions, and when the instructions are executed, the image recognition method as described above is implemented.

[0043] The image recognition method, apparatus, device, and storage medium provided in the embodiments of the present application obtain an image to be recognized and perform feature extraction on the image to be recognized to obtain a target area of the map element to be recognized in the image to be recognized. Based on the target area, multiple sub-areas are generated. Feature extraction is performed on each sub-area to obtain image features of each sub-area. The image features of all sub-areas are then fused to obtain fused features. The recognition result of the map element to be recognized is determined based on the fused features. Compared with the prior art, the technical solution of the present application, on the one hand, generates multiple sub-areas after identifying the target area, thereby extracting image features of each sub-area in the image to be recognized at a finer granularity, so that the map element in the image can be recognized based on more detailed features, which can effectively improve the recognition accuracy of the map element. On the other hand, by fusing the image features of all sub-areas, the saliency of the target area can be enhanced, the interference of the background area in the image to be recognized can be reduced, and the recognition result of the map element to be recognized can be determined by combining more comprehensive features. This can also significantly improve the recognition accuracy of the method provided by the present application compared with the prior art.

[0044] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0046] Figure 1A system architecture diagram of an image recognition application system provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the image recognition process provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a process for generating multiple sub-regions based on a target region according to an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a structure for generating multiple sub-regions based on a target region provided in an embodiment of the present application;

[0051] Figure 6 A flowchart of a method for training a detection network according to an embodiment of the present application is provided;

[0052] Figure 7 A schematic diagram of some map elements obtained according to an embodiment of the present application;

[0053] Figure 8 A schematic structural diagram of obtaining a recognition result of a map element to be recognized provided in another embodiment of the present application;

[0054] Figure 9 A schematic diagram of the structure of generating a target area provided in an embodiment of the present application;

[0055] Figure 10 A schematic diagram of the structure of obtaining a candidate frame provided in an embodiment of the present application;

[0056] Figure 11 A schematic diagram of a structure for extracting features from an image to be identified provided in an embodiment of the present application;

[0057] Figure 12 A flowchart of a method for identifying map elements to be identified provided in another embodiment of the present application;

[0058] Figure 13 A flowchart of a method for identifying map elements to be identified provided in an embodiment of the present application;

[0059] Figure 14 A schematic diagram of the structure of a method for identifying map elements to be identified provided in an embodiment of the present application;

[0060] Figure 15 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0061] Figure 16A schematic structural diagram of an image recognition device provided in another embodiment of the present application;

[0062] Figure 17 A schematic diagram of the structure of a computer device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only the portions relevant to the invention are shown in the accompanying drawings.

[0064] It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments. For ease of understanding, some technical terms involved in the embodiments of this application are explained below:

[0065] (1) Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0066] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software primarily encompasses computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] (2) Machine Learning (ML): It is a multidisciplinary interdisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0068] (3) Convolutional Neural Network (CNN): A feedforward neural network with a deep structure that includes convolutional computations, and is one of the representative algorithms for deep learning. Convolutional neural networks have the ability to learn representations and can perform translation-invariant classification of input information based on their hierarchical structure.

[0069] (4) Classification network: This refers to a neural network that recognizes objects contained in an image. The input of the classification network is image data, and the output is the category of the object contained in the image. The object can be, for example, a traffic element.

[0070] (5) Map elements: These are useful physical point information in electronic map images, which can be traffic elements, including traffic signs or control facilities on the road, such as speed limit signs, electronic eyes, traffic restriction signs, traffic lights, etc.

[0071] (6) Feature similarity: It is a measure used to evaluate the similarity between different spatial features. Feature similarity can be measured by distance, angle, etc.

[0072] (7) Spatial potential field: used to describe the properties of two features or objects that attract or repel each other. For example, when two features attract each other, there is an attractive force between the two features; when two features repel each other, there is a repulsive force between the two features.

[0073] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable social security, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0074] The solutions provided in the embodiments of this application involve technologies such as artificial intelligence neural networks, which are specifically illustrated through the following embodiments.

[0075] Currently, related technologies can utilize large amounts of labeled data to train a classification network model, which is then used to extract semantic features and classify collected road images to obtain image recognition results. However, during the process of collecting map road data, the labeled data may suffer from problems such as poor image quality, numerous map elements with uneven coverage and distribution. For example, rockfall signs only appear on a few sections of mountain roads, resulting in a small number of samples, while speed limit signs are more common on urban roads and other roads, resulting in a large number of samples. The different sample sizes of different types of map elements can lead to an imbalance in the labeled data samples, resulting in low recognition accuracy of the classification network model after sample learning. Image recognition using this classification network model can result in false detections and incorrect recognition types. Furthermore, road images may be collected while a vehicle is in motion, or due to low resolution of the image acquisition device or poor lighting conditions, resulting in poor quality of the collected road images and low accuracy in identifying map element types.

[0076] Based on the above-mentioned defects, the present application provides an image recognition method, apparatus, device and storage medium. Compared with the existing technology, by generating multiple sub-regions based on the target area, the image features of each sub-region in the image to be identified can be extracted in a more fine-grained manner, and by fusing the image features of all sub-regions, the significance of the target area can be enhanced, and the interference of the background area in the image to be identified can be reduced. Furthermore, more comprehensive features are combined to determine the recognition results of the map elements to be identified, thereby improving the accuracy of the recognition results of the map elements to be identified.

[0077] Figure 1 This is an implementation environment architecture diagram of an image recognition method provided in an embodiment of the present application. Figure 1 As shown, the implementation environment architecture includes: a terminal 10 and a server 20.

[0078] In the field of image recognition, the process of identifying map elements in an image to be identified can be performed either on the terminal 10 or on the server 20. For example, the image to be identified can be captured by the terminal 10, and image recognition can be performed locally on the terminal 10 to obtain a recognition result of the map element to be identified. Alternatively, the image to be identified can be sent to the server 20, so that the server 20 acquires the image to be identified, performs image recognition based on the image to be identified, obtains a recognition result of the map element to be identified, and then sends the recognition result of the map element to be identified to the terminal 10, thereby realizing the recognition result of the type of the map element to be identified in the image to be identified.

[0079] The image recognition solution provided in the embodiments of this application can be applied to common scenarios such as autonomous driving, vehicle navigation, map data collection, and road data collection. In these application scenarios, it is usually necessary to capture road scene images and then analyze them to obtain information such as the recognition results of map elements. Subsequent operations based on this information can be performed, such as map updates, travel route planning, and controlling autonomous vehicle driving.

[0080] In addition, an operating system can be run on the terminal 10, and the operating system can include but is not limited to Android system, IOS system, Linux system, Unix, Windows system, etc. It can also include a user interface (UI) layer, which can provide external display of the image to be identified and the recognition results of the map elements to be identified through the UI layer. In addition, the image to be identified required for image recognition can be sent to the server 20 based on the application programming interface (API).

[0081] Optionally, the terminal 10 may be a terminal device in various AI application scenarios. For example, the terminal 10 may be a laptop computer, a tablet computer, a desktop computer, a vehicle-mounted terminal, a mobile device, etc. The mobile device may be, for example, a smartphone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, or other terminal types, which are not specifically limited in the present embodiment.

[0082] The server 20 may be a single server, or a server cluster or distributed system composed of several servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0083] The terminal 10 and the server 20 establish a communication connection via a wired or wireless network. Optionally, the wireless or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but may be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network, or any combination of a virtual private network.

[0084] For ease of understanding and explanation, the following Figures 2 to 17 The image recognition method, apparatus, device and storage medium provided in the embodiments of the present application are described in detail.

[0085] Figure 2 The figure shows a flow chart of the image recognition method according to an embodiment of the present application. The method can be executed by a computer device, which can be the above-mentioned Figure 1 The server 20 or the terminal 10 in the system shown, or the computer device can also be a combination of the terminal 10 and the server 20. Figure 2 As shown, the method includes:

[0086] S101 , obtaining an image to be recognized, performing feature extraction processing on the image to be recognized, and obtaining a target area of a map element to be recognized in the image to be recognized.

[0087] The above-mentioned image to be identified may be an image that requires image recognition, and may include map elements to be identified, and may also include background information. The map elements to be identified are map elements contained in the image to be identified. Among them, map elements are the basic content that constitutes the map, and are used to identify some warning situations that exist in the geographical environment. For example, map elements may include traffic elements, human elements, etc. Among them, traffic elements may include traffic signs or control facilities on the road. Human elements may include information such as human form, appearance, and clothing. In addition, background information refers to image information in the image to be identified other than the map elements to be identified, such as vehicles, roads, poles, buildings, sky, ground, trees, etc.

[0088] It should be noted that traffic signs are road facilities that use words, numbers, or symbols to convey guidance, restrictions, warnings, or instructions. Examples include speed limit signs, traffic signs, and traffic lights. Control facilities, for example, include electronic eye systems. For example, traffic signs may include arrows indicating directions such as go straight, turn left, turn right, turn left and go straight, turn right and go straight, U-turn, and continuous downhill. They may also include bus images indicating bus lanes, bicycle images indicating non-motorized vehicle lanes, car images indicating motorized vehicle lanes, and staircase images indicating overpasses and underpasses.

[0089] In the embodiment of the present application, when acquiring the image to be identified, the image acquisition device may be called to capture the image of the road to obtain the image to be identified, or the image may be acquired through the cloud, or the image may be acquired through a database or blockchain, or the image may be imported from an external device to obtain the image to be identified.

[0090] In one possible implementation, the image acquisition device may be a camera or a still camera, or a radar device such as a laser radar or a millimeter-wave radar. The camera may be a monocular camera, a binocular camera, a depth camera, a three-dimensional camera, or the like. Optionally, during the process of acquiring images through the camera, the camera may be controlled to start a video mode, scan the target object in the camera's field of view in real time, and shoot at a specified frame rate to obtain a road video, which is then processed to generate an image to be identified. During the process of acquiring images through a radar device, a detection signal may be transmitted to the target object in real time through the radar device, and then an echo signal reflected by the target object may be received. Based on the difference between the detection signal and the echo signal, the characteristic data of the target object may be determined, and based on the characteristic data, the image to be identified may be determined.

[0091] It should be noted that the above-mentioned image to be identified may be in the format of an image sequence, a three-dimensional point cloud image, or a video image format.

[0092] In one embodiment, after acquiring an image to be identified, a computer device may perform feature extraction on the image using preset feature extraction rules to obtain a target region within the image containing the map element to be identified. It should be noted that the target region refers to an image containing only the map element to be identified. The target region may be a rectangular region, a circular region, a triangular region, or the like.

[0093] Optionally, the above-mentioned feature extraction rules refer to feature extraction strategies pre-set for the image to be identified based on the actual application scenario, which can be a trained regional prediction model or a general feature extraction algorithm. As an implementable method, the image to be identified can be subjected to feature extraction processing by the regional prediction model to obtain the target area of the map element to be identified in the image to be identified. Among them, the regional prediction model is a network structure model with map element extraction capabilities learned by training sample data. The regional prediction model takes the image to be identified as input and outputs the target area of the map element to be identified in the image to be identified. It has the ability to perform image recognition on the image to be identified and is a neural network model that can predict the target area of the map element to be identified in the image to be identified. The regional prediction model can include a multi-layer network structure, in which the network structures of different layers perform different processing on the input data and transmit their output results to the next network layer until they are processed by the last network layer to obtain the target area of the map element to be identified in the image to be identified.

[0094] As another possible implementation method, in the process of extracting features of the map elements to be identified from the image to be identified by using a feature extraction algorithm, for example, a Scale-Invariant Feature Transform (SIFT) algorithm, a Speeded Up Robust Features (SURF) algorithm, or an ORB feature detection (Oriented FAST and Rotated BRIEF, ORB) algorithm may be used to obtain a target area of the map elements to be identified in the image to be identified.

[0095] As another possible implementation, a pre-established template image database can be consulted to compare the image features of the image to be identified with those in the template image database. The portion of the image to be identified that matches the template image features in the template image database is then identified as the target region of the map element to be identified in the image to be identified. The template image database can be flexibly configured based on the image feature information of map elements in actual application scenarios, and is constructed by aggregating and organizing map elements with different feature types, morphologies, and structures.

[0096] It should be noted that the above-mentioned various implementation methods of extracting features from the image to be identified and obtaining the target area of the map element to be identified in the image to be identified are merely examples and are not limited in the embodiments of the present application.

[0097] In this embodiment, by performing feature extraction processing on the image to be identified, the target area of the map element to be identified in the image to be identified can be accurately obtained, thereby obtaining image features of a finer-grained sub-area for the correct target area, and further making the identification result of the map element to be identified more accurate.

[0098] S102 : generating a plurality of sub-regions based on the target region, performing feature extraction on each sub-region respectively, and obtaining image features of each sub-region.

[0099] Specifically, since the image to be recognized includes not only the map elements to be recognized but also background information other than the map elements to be recognized, in order to improve the accuracy of the recognition of the map elements to be recognized, the background information in the image to be recognized can be fully utilized to enhance the saliency of the target area. Therefore, after determining the target area, the corresponding background information is added to the target area, and the image area formed after the background information is added is divided into multiple sub-areas. The multiple sub-areas obtained are different when the division is performed according to different division strategies.

[0100] It should be noted that, in the process of dividing the image area formed after adding background information, the area formed after adding background information can be divided evenly according to different division sizes or division ratios, or the division process can be irregular. The number of division sizes or division ratios can be multiple. For example, the area formed after adding background information can be divided using two division sizes to obtain two sub-areas; the area formed after adding background information can be divided using three division sizes to obtain three sub-areas; or the area formed after adding background information can be divided using four division sizes to obtain four sub-areas.

[0101] The above-mentioned division size can be any size and the division ratio can be any ratio. It is understood that the larger the division size is when dividing the area formed after adding the background information, the fewer the number of sub-areas obtained; and the smaller the division size is when dividing the area formed after adding the background information, the more the number of sub-areas obtained.

[0102] In one embodiment, after generating multiple sub-regions based on the target region, feature extraction can be performed on each sub-region to obtain image features of each sub-region. Optionally, feature extraction processing can be performed using a feature extraction strategy to obtain image features of each sub-region. Optionally, the image features can be expressed in matrix or vector form.

[0103] Optionally, the above-mentioned feature extraction strategy refers to a strategy for feature extraction that is pre-set according to the actual application scenario, which can be a feature extraction network obtained after training, or a general feature extraction algorithm, etc. As an implementable method, the feature extraction network can be used to extract features of each sub-region to obtain the image features of each sub-region. Among them, the feature extraction network is a network structure model with feature extraction capabilities learned by training sample data. The input of the feature extraction network is each sub-region among multiple sub-regions, and the output is the image features of each sub-region. It has the ability to perform image recognition on each sub-region and is a neural network model that can predict the image features of each sub-region. The feature extraction network can include a multi-layer network structure, and the network structures of different layers perform different processing on the input data, and transmit their output results to the next network layer until they are processed by the last network layer to obtain the image features of each sub-region.

[0104] As another possible implementation method, feature extraction is performed on each sub-region through a feature extraction algorithm to obtain image features of each sub-region. The feature extraction algorithm may be, for example, a Scale-Invariant Feature Transform (SIFT) algorithm, a Speeded Up Robust Features (SURF) algorithm, or an Oriented FAST and Rotated BRIEF (ORB) feature detection.

[0105] It can be understood that by performing feature extraction on each sub-region respectively, different image features are obtained for each sub-region.

[0106] In this embodiment, multiple sub-regions are generated based on the target region, and features are extracted for each sub-region respectively, so that image features of each sub-region can be extracted in a finer granularity, further improving the accuracy of identifying map elements.

[0107] S103: Perform fusion processing on the image features of all sub-regions to obtain fusion features, and determine the recognition result of the map element to be recognized based on the fusion features.

[0108] Specifically, the image features of all the sub-regions may include image feature vectors or image feature matrices. For example, after acquiring the image features of all the sub-regions, the image features of all the sub-regions may be fused. When the image features of all the sub-regions are represented by vectors, information fusion may be performed by combining vectors; when the image features of all the sub-regions are represented by matrices, information fusion may be performed by concatenating matrices, thereby obtaining corresponding fused features.

[0109] After obtaining the fused features, the fused features can be input into a trained detection network to determine the recognition result of the map element to be identified, or a classification algorithm can be used to determine the recognition result of the map element to be identified.

[0110] It should be noted that the above-mentioned detection network is a model structure that has the ability to recognize map elements through sample learning. The input of the detection network is the fusion feature, and the output is the recognition result of the map element. It is a neural network model that can predict the recognition result of the map element to be identified.

[0111] As an implementable method, the detection network may include a fully connected layer and an activation function. After obtaining the fused features, the fused features may be processed by the fully connected layer to obtain fully connected vector features, and the fully connected vector features may be processed using the activation function to obtain the recognition results of the map elements to be identified. The recognition results include multiple map element types and may also include multiple element attributes under the element type.

[0112] Another possible implementation approach is to cluster the fused features using a clustering algorithm based on prior knowledge of the corresponding domain to obtain clustering results. The recognition result for each clustered result in the fused features is then determined using artificial prior knowledge of feature features, thereby obtaining the recognition result for the map feature to be identified. The clustering algorithm can be a clustering function, such as Mean, Pool, or LSTM.

[0113] As another possible implementation, a pre-established feature feature database of known feature types can be queried, and the feature features of the unknown type of map feature to be identified, which is fused with features, can be compared with the feature features in the feature feature database of known feature types. Feature types with identical feature features can be identified as the recognition results of the map feature to be identified. The feature feature database can be constructed by aggregating and categorizing feature data of different map feature types, feature forms, and structural characteristics.

[0114] The recognition result of the map element to be identified is used to identify the map element to be identified, so that the information, characteristics, etc. of the map element to be identified can be quickly obtained through the recognition result of the map element to be identified. For example, the recognition result of the map element to be identified may include the element type of the map element to be identified, or may include multiple element attributes of the map element to be identified under the element type. Exemplarily, the element type may be a route sign, a traffic light, a road sign, a road condition sign, etc. The element attributes corresponding to the route sign may be, for example, going straight, turning left, turning right, turning left and going straight, turning right and going straight, U-turn, etc. The functions of the map elements corresponding to different element attributes are different. For example, a straight sign has the function of indicating that the vehicle can go straight; a left turn sign has the function of indicating that the vehicle can turn left; a left turn and straight sign has the function of indicating that the vehicle can go straight or turn left; and a U-turn sign has the function of indicating that the vehicle can make a U-turn.

[0115] See Figure 3 As shown, when the image to be identified 3-1 is obtained, feature extraction processing is performed on the image to be identified 3-1 to obtain a target area 3-2 of the map element to be identified in the image to be identified, and then multiple sub-areas 3-3 are generated based on the target area 3-2, and the image features of all sub-areas 3-3 are fused to obtain a fused feature 3-4, and the recognition result 3-5 of the map element to be identified is determined based on the fused feature 3-4.

[0116] Compared to the prior art, the image recognition method provided in the embodiments of the present application, on the one hand, generates multiple sub-regions based on the target region, thereby being able to extract the image features of each sub-region in the image to be identified in a more fine-grained manner; on the other hand, by fusing the image features of all sub-regions, it effectively integrates the background information in the image to be identified, which can enhance the saliency of the target region. Furthermore, it combines more comprehensive image features to determine the recognition results of the map elements to be identified, more accurately obtaining the recognition results of the map elements to be identified, and greatly improving the accuracy of the map elements to be identified in the image to be identified. It can also be applied to map update systems to accurately predict the map elements to be identified in the image to be identified, greatly improving the quality and efficiency of map element identification, and providing strong support for the update and analysis of map data.

[0117] In another embodiment of the present application, background information may be added to the target area to obtain multiple sub-areas. Figure 4 Provides a specific implementation method for generating multiple sub-regions based on the target region. Figure 4 As shown, specifically including:

[0118] S201 , performing background addition processing on the target area to obtain a candidate window; the candidate window includes the target area.

[0119] It should be noted that the above-mentioned background refers to the rest of the image in the image to be recognized except for the map elements to be recognized, for example, it may include trees, sky, roads, poles, vehicles, etc. in the image to be recognized.

[0120] In this embodiment, as an optional implementation method, in the process of adding background to the target area, background information can be first selected in the image to be identified, and the area where the background information is mapped in the image to be identified is used as the background area. Then, the background area is combined with the target area to obtain a candidate window. Optionally, in the process of combining the background area with the target area, the background area and the target area can be framed by a regular graphic window, or by an irregular image window. In this step, the combination method is not specifically limited, as long as the background area and the target area can be combined to obtain a candidate window, where the regular graphic can be, for example, a rectangle, triangle, diamond, circle, etc.

[0121] Wherein, a plurality of background information may be selected in the image to be identified, and each background information is mapped to a different background area in the image to be identified. Different candidate windows are formed by combining different background areas with target areas.

[0122] Furthermore, the target region can be resized to obtain a candidate window. The size of the target region in the image to be identified can be determined first, and then the target region can be resized based on the size to obtain a candidate window. The candidate window contains the target region. Optionally, the candidate window can be any shape, such as a rectangle, a triangle, or a circle.

[0123] When the target area is expanded based on its size, the target area may be expanded using preset software. The user may customize relevant parameters according to actual needs. For example, the parameters may be user-defined function options "expand" and "size." The preset software is then run to expand the target area of the image to be recognized based on the relevant parameters, thereby obtaining a candidate window. The preset software may be image processing software.

[0124] When the target area is expanded, the target area may be expanded with the map element to be identified as the center according to a preset ratio. The preset ratio is customized according to actual needs. For example, the preset ratio of the target area to the candidate window can be any value such as 1:2, 1:3, 1:4, or 2:3.

[0125] For example, when it is determined that the size of the target area in the image to be identified is W×H, assuming that the preset ratio determined according to the size is 1:2, the target area is size-expanded to obtain a candidate window, then the size of the obtained candidate window is 2W×2H, and the candidate window contains the background information in the image to be identified.

[0126] S202 : Divide the candidate window n times according to n division sizes to obtain a plurality of sub-regions of different sizes, where n is a positive integer and n≥1.

[0127] It should be noted that each of the above division sizes has a corresponding sub-region, and different division sizes result in different numbers and sizes of sub-regions.

[0128] Specifically, after obtaining the candidate window, the candidate window can be divided and processed, and the candidate window can be divided n times according to n division sizes, where n is a positive integer and n≥1. Each time the candidate window is divided, it can be a uniform division process, in which case the size of each sub-region obtained after the division process is the same; it can also be an uneven division process, in which case the size of each sub-region obtained after the division process is different.

[0129] For example, see Figure 5As shown, when n is 3, feature extraction is performed on the image to be identified 5-1. After determining the target area of the map element to be identified in the image to be identified, the target area is resized to obtain a candidate window 5-2. The candidate window is then divided three times according to three division sizes, resulting in three sub-regions 5-3 of different sizes. The three different division sizes can be, for example, the length and width dimensions of each sub-region are 4×4, 8×8, and 16×16, respectively. Different division sizes result in different sizes of the corresponding sub-regions. Different sub-regions can be used to extract features with different receptive fields in the image to be identified.

[0130] In this embodiment, background processing is performed on the target area to obtain a candidate window. By introducing background information, the recognition area is expanded. Then, the candidate window is divided to obtain multiple sub-areas of different sizes, so that the features of each sub-area can be focused on in a finer granularity to improve the accuracy of image recognition.

[0131] In another embodiment of the present application, a method for obtaining fused features based on the image features of each sub-region is also provided. Specifically, after determining the image features of each sub-region, the image features of each sub-region can be vectorized to obtain a feature vector corresponding to the sub-region; then, the direction value corresponding to each eigenvalue in the feature vector corresponding to each sub-region is determined. The direction value is used to indicate whether the sub-region includes the target region, and a fused feature is obtained based on the eigenvalue corresponding to each sub-region and the direction value corresponding to the sub-region.

[0132] It should be noted that a feature vector refers to the result of expressing other forms of data in a digital form. The image features of each subregion can be vectorized to obtain a feature vector corresponding to the subregion. This feature vector can include the feature vector of the map element to be identified or the pure background feature vector. Different feature vectors correspond to feature values and direction values of different dimensions. The feature value is a numerical value used to represent the attraction or repulsion between the information in that dimension and the target category, while the direction value indicates whether the subregion contains the target area. When the subregion does not contain the target area, that is, contains the background area, the corresponding direction value is "-"; when the subregion contains the target area, the corresponding direction value is "+". A feature value with a direction value of "+" indicates that the information in that dimension has an attraction with the target type of the map element, and the magnitude of the attraction is the numerical value of the dimension; a feature value with a direction value of "-" indicates that the information in that dimension has a repulsion with the target type of the map element, and the magnitude of the repulsion is the numerical value of the dimension.

[0133] It should be noted that the above-mentioned attraction and repulsion can be understood as the degree of influence of the image features of the sub-region on the target type of the map element to be identified, that is, if the information of a certain dimension has an attraction with the target type of the map element and the greater the magnitude of the attraction, the greater the influence of the image features of the sub-region on the target type of the map element to be identified; if the information of a certain dimension has a repulsion with the target type of the map element and the greater the magnitude of the repulsion, the smaller the influence of the image features of the sub-region on the target type of the map element to be identified.

[0134] Exemplarily, the image features of the sub-region are expressed as the mathematical form "[0,10,2,0]", or "[0,-10,2,0]". In this case, "[0,10,2,0]" and "[0,-10,2,0]" are the feature vectors corresponding to the sub-region. For example, the eigenvalue of "-10" in the eigenvector "[0,-10,2,0]" is "10", and the direction value is "-", which means that the information of the second dimension has a repulsive force with the target type of the map element, and the amplitude of the repulsion is 10. For another example, the eigenvalue of "2" in the eigenvector "[0,-10,2,0]" is "2", and the direction value is "+", which means that the information of the third dimension has an attractive force with the target type of the map element, and the amplitude of the attraction is 2.

[0135] It is understood that the specific form of the feature vector is not limited in this embodiment, as long as the image features of each sub-region can be mathematically represented and include feature values and direction values. For example, the image features of the sub-region can be converted into a high-dimensional sparse vector or a low-dimensional dense vector.

[0136] In this embodiment, after determining the direction value and eigenvalue corresponding to each eigenvalue in the eigenvector corresponding to each subregion, vector convolution and weighting can be performed based on the eigenvalue and directional value corresponding to each subregion to obtain a fused feature. The fused feature integrates the image features of each subregion.

[0137] The feature vectors of the target area and the feature vectors of the background area in each sub-area can be fused separately to obtain fused features. It should be noted that during the fusion process of the feature vectors containing the target area, if the direction of a certain dimension is repulsive, its direction value is set to 0, that is, it does not participate in the feature fusion of attraction. Similarly, during the fusion process of the feature vectors containing the background area, if the direction of a certain dimension is attractive, its direction value is set to 0, that is, it does not participate in the feature fusion of repulsion.

[0138] In this embodiment, the image features of all sub-regions are fused to obtain fused features, which effectively integrates the background information in the image to be identified, can enhance the saliency of the target area, and combines more comprehensive image features to determine the recognition results of the map elements to be identified, which facilitates the subsequent prediction of the element type of the map elements to be identified.

[0139] Another embodiment of the present application provides an implementation method for determining the recognition result of the map element to be identified based on the fused features after obtaining them. The fused features can be input into a trained detection network, and the recognition result of the map element to be identified is determined based on the output of the detection network. The detection model is trained based on historical images and the map element recognition results of the historical images, where the historical images are annotated with traffic element areas.

[0140] It should be noted that the above-mentioned detection network is a neural network model that takes fused features as input and outputs the recognition results of the map elements to be identified, and has the ability to identify the feature types of the map elements to be identified and predict the recognition results. The detection network is responsible for establishing the relationship between the fused features and the target feature types, and its model parameters are already in an optimal state. Among them, the detection network may include but is not limited to convolutional layers, fully connected layers and activation functions. The convolutional layers and fully connected layers may include one layer, or may also include multiple layers. The convolutional layer is used to extract features from the fused features, and the fully connected layer is mainly used to classify the fused features. The fused features can be processed by the convolutional layer to obtain convolutional features, and then the convolutional features can be processed by the fully connected layer to obtain a fully connected vector, and then the fully connected vector can be processed by the activation function to obtain the output result of the detection network. The output result includes the feature type of the map element to be identified, or may include multiple feature attributes of the map element to be identified under the feature type.

[0141] The activation function may be a Sigmoid function, a Tanh function, or a ReLU function. By processing the fully connected vector through the activation function, the result can be mapped to a value between 0 and 1.

[0142] In one possible implementation, the detection network's processing of fused features specifically includes: During the processing of the fused features, the fused features may be operated on using a multi-classification function to output the map feature type. The fused features may also be operated on using a multi-class binary classification function to output the fused feature attributes. Optionally, the multi-classification function may be a softmax function, and the multi-class binary classification function may be multiple sigmoid functions, where one sigmoid function can implement a binary classification prediction. The multi-classification function is used to incorporate nonlinear factors, as linear models lack the expressive power to transform continuous real-valued inputs into outputs between 0 and 1.

[0143] For example, the fused features are input into the detection network, and the prediction results of the detection network can include any one of the map element types such as "route sign," "traffic light," and "road condition sign." The prediction results can also include map element attributes. For example, the feature attributes corresponding to the map element type "route sign" can include "straight ahead," "left turn," "right turn," "left turn and straight ahead," and "U-turn."

[0144] Here, we use three classifications as an example to introduce the output of the multi-classification function. For example, the multi-classification function can predict the feature types of "route sign," "traffic light," and "road condition sign," respectively. The output of the above detection network can be represented by a vector, such as a 3*1-dimensional vector, where each element corresponds to a feature type, and each element value in the vector represents the probability that the map feature to be identified is of the corresponding label type. Assuming that the output vector of the multi-classification function is [0.61, 0.31, 0.08], the probability that the map feature to be identified is "route sign" is 0.61, the probability that the map feature to be identified is "traffic light" is 0.31, and the probability that the map feature to be identified is "road condition sign" is 0.08. The element value with the highest probability can be selected as the prediction result for the map feature to be identified, that is, "route sign" is the identification result for the map feature to be identified.

[0145] Taking a ternary binary classification function as an example, we will explain the output of a multivariate binary classification function. For example, the cell attributes that a multivariate binary classification function can predict are represented by "straight," "left turn," "right turn," and "left turn and straight." Its output can be represented by a vector, such as a 4x1-dimensional vector. Each element in this vector corresponds to a feature attribute, and each element value in the vector represents the probability that the map feature to be identified has the corresponding feature attribute. Assuming the output vector of the ternary binary classification function is [0.51, 0.15, 0.22, 0.62], this indicates that the probability that the map feature to be identified is "straight" is 0.51, the probability that the map feature to be identified is "left turn" is 0.15, the probability that the map feature to be identified is "right turn" is 0.22, and the probability that the map feature to be identified is "left turn and straight" is 0.62. Assuming a preset threshold of 0.5, the element value with a probability greater than the preset threshold is considered the prediction result for the map feature to be identified, that is, "straight" and "right turn" are considered the identification results of the multivariate binary classification function.

[0146] In the embodiment of the present application, by inputting the fusion features into the detection network for predictive processing, the accuracy of the recognition results of the map elements to be identified is greatly improved, the recognition results can be obtained more accurately, and higher-precision map element recognition can be achieved.

[0147] In another embodiment of the present application, a specific implementation method for the training process of the detection network is also provided. Figure 6 As shown, specifically including:

[0148] S301 , generating a plurality of sample sub-regions based on historical images, and inputting each sample sub-region into a feature extraction network to be trained for feature extraction processing, thereby obtaining image features of each sample sub-region.

[0149] The above-mentioned historical images may be multiple or one, wherein each historical image may include at least one map element, for example, the historical image may include a route sign, a traffic light, or a road condition sign. The historical image is an image of a known map element recognition result. The map element recognition result can be found in Figure 7 As shown, Figure 7 As shown, Figure 7 The schematic diagram of some elements of the map elements provided in the embodiment of the present application may include various warning signs, prohibition signs and information signs.

[0150] Specifically, after acquiring historical images, corresponding background information can be added to the historical images, and the resulting image region can be partitioned to obtain multiple sample subregions. Each sample subregion can be randomly divided into a training set and a validation set according to a certain ratio. The training set is used to train the initial detection network to obtain a trained detection network, and the validation set is used to verify the trained detection network to verify its performance. Each sample subregion in the training set is then input into the feature extraction network to be trained for feature extraction, obtaining image features for each sample subregion.

[0151] S302: Input the image features of all sample sub-regions into the fusion network to be trained for fusion processing to obtain sample fusion features.

[0152] S303: Input the sample fusion features into the detection network to be trained to obtain the prediction results of the map elements.

[0153] S304: Calculate a loss function based on the prediction results of the map elements and the map element recognition results of the historical image.

[0154] S305 , according to the minimization of the loss function, an iterative algorithm is used to iteratively adjust the parameters of the feature extraction network to be trained, the fusion network to be trained, and the detection network to be trained to obtain a detection network.

[0155] After determining the image features of all sample sub-regions, the image features of all sample sub-regions can be input into the fusion network to be trained for fusion processing, which can be done by vector splicing or superposition to obtain sample fusion features. The sample fusion features are then input into the detection network to be trained, which can include a fully connected layer and an activation function. The obtained sample fusion features are input into the fully connected layer to obtain a sample fully connected vector, and the activation function is used to process the sample fully connected vector to obtain the corresponding output result. The feature extraction network, fusion network, and detection network to be constructed are trained using the training set to obtain the feature extraction network, fusion network, and detection network to be verified.

[0156] In the process of training the detection network, the computer equipment uses the feature extraction network, fusion network and detection network to be verified in the verification set, and optimizes the feature extraction network, fusion network and detection network to be verified according to the minimization of the loss function to obtain the feature extraction network, fusion network and detection network. According to the difference between the detection network to be verified and the annotation result input in the verification set, the parameters in the feature extraction network, fusion network and detection network to be constructed are updated to achieve the purpose of training the feature extraction network, fusion network and detection network, wherein the above-mentioned annotation result can be a map element recognition result obtained by manually annotating historical images.

[0157] Optionally, the updating of parameters in the feature extraction network, fusion network, and detection network to be verified may be updating matrix parameters such as weight matrices and bias matrices in the feature extraction network, fusion network, and detection network to be constructed. The weight matrices and bias matrices include, but are not limited to, matrix parameters in the convolutional layers, feedforward network layers, and fully connected layers in the feature extraction network, fusion network, and detection network to be verified.

[0158] In an embodiment of the present application, a loss function can be used to calculate the loss value of the result and label result obtained in the detection network to be verified by inputting the verification set, so as to update the parameters in the feature extraction network, fusion network and detection network to be verified. Optionally, the loss function can use a cross entropy loss function, a normalized cross entropy loss function, or a Focalloss.

[0159] Among them, when the parameters in the feature extraction network, fusion network and detection network to be verified are updated by the loss function, it can be determined according to the loss function that the feature extraction network, fusion network and detection network to be verified have not converged, and then the parameters in the model are adjusted to make the feature extraction network, fusion network and detection network to be verified converge, thereby obtaining the feature extraction network, fusion network and detection network. The convergence of the feature extraction network, fusion network and detection network to be verified can mean that the difference between the output result of the feature extraction network, fusion network and detection network to be verified on the verification set and the labeled result of the training data is less than a preset threshold, or the rate of change of the difference between the output result and the labeled result of the training data approaches a certain lower value. When the calculated loss function is small, or the difference between the loss function output from the previous round of iteration approaches 0, it is considered that the feature extraction network, fusion network and detection network to be verified have converged.

[0160] Further, after training the feature extraction network, fusion network and detection network, please refer to Figure 8 As shown, after obtaining the image 8-1 to be identified, feature extraction processing can be performed on the image 8-1 to be identified to obtain the target area 8-2 of the map element to be identified in the image to be identified, and then multiple sub-areas 8-3 are generated based on the target area 8-2. For example, there are three sub-areas 8-3, namely the first sub-area, the second sub-area and the third sub-area. Then, the image features of the three sub-areas 8-3 are processed through the feature extraction network to obtain the image features of the first sub-area, the image features of the second sub-area and the image features 8-4 of the third sub-area respectively. The image features of the first sub-area, the image features of the second sub-area and the image features 8-4 of the third sub-area are fused through the fusion network to obtain the fused features 8-5, and the fused features 8-5 are input into the detection network to obtain the recognition result 8-6 of the map element to be identified.

[0161] In another embodiment of the present application, a specific implementation method for performing feature extraction processing on an image to be identified to obtain a target area of a map element to be identified in the image to be identified is also provided. Feature extraction processing can be performed on the image to be identified to obtain a feature map of the image to be identified; then, for each feature point in the feature map, a candidate box corresponding to each feature point is determined; the candidate box contains the feature point; and map element detection is performed on the image features in each candidate box to obtain a confidence level corresponding to the candidate box; the confidence level is used to characterize the probability that the image within the candidate box is a map element; and the area corresponding to the candidate box whose confidence level meets a preset rule is determined as the target area of the map element to be identified in the image to be identified.

[0162] See Figure 9 As shown, the region prediction network can be used to extract image features from the image to be identified 9-1 to obtain the target region 9-2 of the map element to be identified in the image to be identified. Optionally, the region prediction network can be a convolutional neural network (CNN) model or an artificial neural network (ANN) model.

[0163] Specifically, the structure of the region prediction network may include a feature prediction network, a candidate box generation network, and a feature detection network. Among them, the feature prediction network may include a convolution layer, a normalization layer (Batch Normalization, BN), and an activation layer. The image to be identified can be passed through a convolution layer to extract image features such as edges and textures of the image to be identified to obtain image features of the image to be identified. The normalization layer in the feature prediction network then normalizes the image features extracted by the convolution layer according to a normal distribution to filter out noise features in the image features to obtain filtered image features, and the filtered image features are nonlinearly mapped through the activation layer to enhance the generalization ability of the feature extraction model to obtain a feature map. Among them, the feature map includes multiple feature points.

[0164] After obtaining the feature map, for each feature point in the feature map, a candidate box corresponding to each feature point is determined according to the candidate box generation rules. Optionally, the candidate box can be any shape, such as a square, rectangle, circle, diamond, or triangle. The number of candidate boxes can be a predetermined random number, such as 8, 9, or 10.

[0165] As an implementation method, the computer device can generate candidate boxes corresponding to each feature point in the feature map, centered on the feature point and at different sizes and scaling ratios. Alternatively, the computer device can generate candidate boxes corresponding to each feature point at a specific size combined with a specific aspect ratio. The number of candidate boxes corresponding to each feature point can be random, the different sizes can be randomly determined, such as 1 feature point, 2 feature points, or 3 feature points, and the different aspect ratios can be randomly determined, such as 1:1, 2:1, or 1:3.

[0166] For example, Figure 10 A schematic diagram of determining the correspondence between each feature point and a candidate frame provided in an embodiment of the present application. For each feature point 10-1, the computer device can set the size of the candidate frame to 1 feature point, 2 feature points, or 3 feature points, respectively. At the same time, three sets of aspect ratios are set at these sizes, for example, 1:1, 2:1, and 1:2. Three corresponding candidate frames are then generated for each size, resulting in nine candidate frames.

[0167] The computer device detects the map elements of the image features in each candidate box through the feature detection network, thereby obtaining the confidence corresponding to each candidate box, and also obtaining the detection result of each candidate box. The detection result is used to characterize whether the candidate box contains map elements, and the confidence is used to characterize the probability that the image in the candidate box is a map element. The area corresponding to the candidate box whose confidence meets the preset rules is then determined as the target area of the map element to be identified in the image to be identified. It should be noted that the above-mentioned compliance with the preset rules can be the area corresponding to the candidate box with the highest confidence, or the area corresponding to the candidate box with a confidence higher than a preset threshold as the target area of the map element to be identified in the image to be identified. The preset threshold is customized according to the needs of the actual application scenario.

[0168] It can be understood that in the process of detecting map elements separately by using the feature detection model to detect the image features in each candidate box, the image features of each candidate box can be extracted first, and then the image features can be classified. The image features of each candidate box can be processed by the fully connected layer to obtain a fully connected vector, and the fully connected vector can be processed using an activation function to obtain a prediction result of the candidate box. The prediction result includes multiple confidence levels, wherein the confidence levels corresponding to each candidate box type are different. Then, the candidate box whose confidence level meets the preset rules is determined. The candidate box corresponding to the highest confidence level can be selected, and the area corresponding to the candidate box is determined as the target area of the map element to be identified in the image to be identified.

[0169] For example, Figure 11This is a schematic diagram of the process of detecting the target area of the image to be identified provided by the embodiment of the present application. Figure 11 As shown, the computer device can first input the image 11-1 to be identified into the feature prediction network to obtain the feature Figure 11-2 Then the feature Figure 11-2 Processing is performed through the candidate box generation network. For each feature point in the feature map, a candidate box 11-3 corresponding to each feature point is generated. Then, the image features in each candidate box are classified and detected as map elements through the feature detection network to obtain the confidence corresponding to each candidate box to determine whether the image features in the candidate box contain map elements. From the confidences corresponding to multiple candidate boxes, the area corresponding to the candidate box whose confidence meets the preset rules is selected, and the area is determined as the target area of the map element to be identified in the image to be identified.

[0170] It should be noted that the feature prediction network is capable of extracting features from the image to be recognized, the candidate box generation network is capable of generating candidate boxes for each feature point, and the feature detection network is capable of detecting the target area of map features. These feature prediction network, candidate box generation network, and feature detection network can be trained using a preset training algorithm, i.e., their model parameters are already optimized.

[0171] Furthermore, this embodiment provides a specific training process for the above-mentioned feature prediction network, candidate box generation network, and feature detection network. The training samples can be input into the above-mentioned feature prediction model, candidate box generation network, and feature detection network, and the loss between the true label and the output of the training sample can be determined according to a pre-set loss function. The feature prediction model, candidate box generation network, and feature detection network are iteratively trained based on the obtained loss.

[0172] In one possible implementation, the training process of the feature prediction model, candidate box generation network, and feature detection network in the aforementioned region prediction network specifically includes: randomly dividing the initial historical image into a training set and a validation set according to a certain ratio, and then using the training set and validation set to construct the feature prediction network, candidate box generation network, and feature detection network according to the training learning algorithm. The training set is used to train the initial feature prediction network, candidate box generation network, and feature detection network to obtain the trained feature prediction network, candidate box generation network, and feature detection network, and the validation set is used to verify the trained feature prediction network, candidate box generation network, and feature detection network to verify the performance of the feature prediction network, candidate box generation network, and feature detection network.

[0173] In the process of training the feature prediction network, candidate box generation network, and feature detection network, the computing device uses the feature prediction network, candidate box generation network, and feature detection network to be verified in the verification set, and optimizes the feature prediction network, candidate box generation network, and feature detection network to be verified according to the minimization of the loss function to obtain the feature prediction network, candidate box generation network, and feature detection network. According to the difference between the results obtained from the feature prediction network, candidate box generation network, and feature detection network to be verified inputted into the verification set and the historical area detection results, the parameters in the feature prediction network, candidate box generation network, and feature detection network to be constructed are updated to achieve the purpose of training the feature prediction network, candidate box generation network, and feature detection network. The above-mentioned historical area detection results can be the results obtained by manually annotating the initial historical images.

[0174] In an embodiment of the present application, when training a feature prediction network, a candidate box generation network, and an element detection network, a loss function can be used to calculate the loss value of the results obtained in the verification set input feature prediction network, candidate box generation network, and element detection network to be verified and the historical area detection results, thereby updating the parameters in the feature prediction network, candidate box generation network, and element detection network to be verified. Optionally, the loss function can use a cross entropy loss function, a normalized cross entropy loss function, or a Focalloss, etc.

[0175] In this embodiment, features are extracted from the map elements to be identified, and then for each feature point in the feature map, a candidate box corresponding to each feature point is determined, and map elements are detected for the image features in each candidate box to obtain the confidence level corresponding to the candidate box. Then, the area corresponding to the candidate box whose confidence level meets the preset rules is determined as the target area of the map element to be identified in the image to be identified. This can facilitate the accurate generation of multiple sub-areas, further improving the accuracy of the recognition results of the map elements to be identified.

[0176] In order to better understand the embodiments of the present application, the complete flowchart method of the image recognition method proposed in the present application is further explained below.

[0177] Figure 12 A flow chart of the image recognition method provided in the embodiment of the present application is shown as follows: Figure 12 As shown, the method may include the following steps:

[0178] S401 , obtaining an image to be recognized, performing feature extraction processing on the image to be recognized, and obtaining a target area of a map element to be recognized in the image to be recognized.

[0179] Specifically, see Figure 13As shown, image acquisition can be performed by an image acquisition device or a vehicle-mounted camera, specifically, taking a photo of the road ahead to obtain an image to be recognized, which includes not only the map element to be recognized but also background information. After obtaining the recognized image, the image to be recognized can be subjected to feature extraction processing by a region prediction network to detect map elements. Specifically, the image to be recognized is input into the feature prediction network in the region prediction network, and sequentially passes through the convolution layer, normalization layer, and activation layer to obtain a feature map. The feature map is then processed by a candidate box generation network. For each feature point in the feature map, a candidate box corresponding to each feature point is generated. For example, 9 candidate boxes are generated for each feature point. Then, the image features in each candidate box are classified and detected as map elements by the feature detection network to obtain the confidence corresponding to each candidate box to determine whether the image features in the candidate box contain map elements. From the confidences corresponding to the 9 candidate boxes, the area corresponding to the candidate box with the highest confidence is selected, and the area is determined as the target area of the map element to be recognized in the image to be recognized.

[0180] S402: Perform background addition processing on the target area to obtain a candidate window; the candidate window includes the target area.

[0181] S403 : Divide the candidate window n times according to n division sizes to obtain multiple sub-regions of different sizes, where n is a positive integer and n≥1.

[0182] After determining the target area, background information can be added to more accurately identify the map feature category. Specifically, this can be done by configuring multi-level window information, which means that multiple sub-areas are obtained according to different division sizes.

[0183] For example, the size of the target area can be determined first. For example, when the size of the target area is determined to be W×H, the target area is then expanded in size according to a preset ratio with the map element to be identified as the center. Assuming that the preset ratio is 1:2, the size of the candidate window after expansion is 2W×2H, and the candidate window contains the background information in the image to be identified.

[0184] In this embodiment, after the candidate window is determined, the candidate window can be divided and processed, and the candidate window can be divided n times according to n division sizes, where n is a positive integer and n≥1. For example, when n is 3, the three division sizes can be the length × width sizes of each sub-region: 4×4, 8×8, 16×16, respectively. When the candidate window is divided three times according to the three division sizes, three sub-regions corresponding to different sizes are obtained, and the three sub-regions corresponding to different sizes are used to extract features of different receptive fields, respectively.

[0185] S404: Perform vectorization processing on the image features of each sub-region to obtain a feature vector corresponding to the sub-region.

[0186] S405 : Determine the direction value corresponding to each eigenvalue in the eigenvector corresponding to each sub-region, where the direction value is used to indicate whether the sub-region includes the target region.

[0187] S406: Obtain fusion features according to the feature values corresponding to each sub-region and the direction values corresponding to the sub-region.

[0188] Specifically, after determining each sub-region, multi-window feature extraction and multi-window feature fusion and recognition can be performed, that is, the image features of each sub-region can share a feature extraction model. After obtaining three sub-regions with length × width of 4×4, 8×8, and 16×16 respectively, the three sub-regions can be input into the feature extraction model respectively to obtain the image features corresponding to each sub-region. The image features corresponding to each sub-region can be represented by feature vectors. For example, the image features of 1*Z dimension are extracted for each sub-region, and the feature vector is used. Indicates that Z is the dimension of the feature, where if the sub-region has a target area to be detected, it is represented as If it is a pure background area, it is represented as The sub-region features containing the target enhance the recognition of map elements, while the background region features inhibit the recognition of map elements, thereby increasing the discriminative ability of element recognition. The three sets of feature vectors corresponding to the three sub-regions of 4×4, 8×8, and 16×16 are obtained respectively. The three sets of feature vectors are Z-dimensional feature vectors, including the feature vector of the target to be identified and the pure background feature vector. Among them, for the feature vector containing the target to be identified, the value of the i-th dimension represents the attraction between the information of this dimension and the target category, and the magnitude of the attraction is the value of the i-th dimension; for the pure background feature vector, the value of the i-th dimension represents the repulsion between the information of this dimension and the non-target category, and the magnitude of the repulsion is the value of the i-th dimension. Figure 14 As shown, when performing multi-window feature extraction, the marked image features represent the hit map elements, and the remaining image features are background information, which can be, for example, information such as poles, trees, sky, and ground.

[0189] After obtaining the three sets of feature vectors corresponding to the three sub-regions, the three sets of feature vectors can be subjected to multi-feature fusion processing through the fusion network, such as splicing or combining the vectors, to obtain the fusion feature. The fusion feature is The eigenvector F of Z , the feature vectors of each sub-region can be fused by the following formula:

[0190]

[0191] Among them, each set of eigenvectors includes eigenvalues and direction values, [K1, K2, ..., K m ] represents the image feature vector The eigenvalue corresponding to each dimension in a m Represents the direction value corresponding to the mth dimension in the feature vector. When a subregion does not contain the target area, that is, contains the background area, the corresponding direction value is "-"; when a subregion contains the target area, the corresponding direction value is "+". i∈n represents the number of subregions, and Z is the dimension of the feature. A feature value with a direction value of "+" indicates that the information in that dimension has an attractive force with the target type of the map feature, and the magnitude of the attraction is the numerical value of the dimension. A feature value with a direction value of "-" indicates that the information in that dimension has a repulsive force with the target type of the map feature, and the magnitude of the repulsion is the numerical value of the dimension.

[0192] It should be noted that, in the process of feature fusion, the feature vectors of the target area and the feature vectors of the background area in each sub-area can be fused separately to obtain fused features. In the process of fusion of the feature vectors of the target area, if the direction of a certain dimension i is repulsive, its direction value a is i Set to 0, that is, it does not participate in the feature fusion of attraction. Similarly, in the process of fusion of feature vectors containing background areas, if the direction of a certain dimension i is attractive, its direction value a i Set to 0, that is, not participating in the feature fusion of repulsive force, thus obtaining the fusion feature F Z .

[0193] In this embodiment, the target area is first scaled up, thereby increasing the recognition area by introducing background information. By extracting the features of the background information, it can play a strong supervisory role in the feature category recognition of the map element to be identified. Secondly, the area to be identified is divided into multiple sub-areas of different sizes, which can focus more on the finer-grained features of each sub-area. Finally, the image features of different sub-areas are fused. By enhancing the saliency of the target area and reducing the interference of the background area, it is possible to combine more comprehensive image features to determine the recognition result of the map element to be identified, facilitating the subsequent prediction of the feature type of the map element to be identified.

[0194] S407: Input the fused features into the trained detection network, and determine the recognition result of the map element to be recognized based on the output of the detection network.

[0195] After determining the post-fusion features, the fusion features are input into the trained detection network for accurate feature identification. The fusion features can be passed through the convolution layer to obtain the convolution features, and then the convolution features are input into the fully connected layer for processing to obtain the fully connected vector, and the fully connected vector is processed through the activation function to obtain the output result of the detection network. The output result includes the feature type of the map feature to be identified, or may include multiple feature attributes of the map feature to be identified under the feature type.

[0196] In this embodiment, multiple sub-regions are generated based on the target region, so that the image features of each sub-region in the image to be identified can be extracted in a finer granularity, and the background information is effectively integrated by fusing the image features of all sub-regions. This can more accurately and comprehensively utilize the background information in the image to be identified, enhance the saliency of the target region, and thus more accurately obtain the identification results of the map elements to be identified, thereby greatly improving the accuracy of element recognition of the map elements to be identified, and avoiding the problems of incomplete category coverage and recognition errors caused by directly using classification networks for image recognition in related technologies.

[0197] In addition, this embodiment can implement this solution using two models: one is an existing coarse-grained model, and the other is the fine-grained detection model provided in this application. The coarse-grained model is used to extract features from the image to be identified, obtaining a first category of the map element to be identified in the image to be identified, as well as the target area of the map element to be identified in the image to be identified. The first category is a coarse-grained category, such as "traffic light" or "traffic sign." The fine-grained model includes the feature extraction network, fusion network, and detection network described above. The feature extraction network is used to extract features from multiple sub-regions to obtain image features for each sub-region; the fusion network is used to fuse the image features of each sub-region to obtain a fused feature; and the detection network is used to classify the fused feature to obtain a recognition result for the map element to be identified, which can be a second category. The second category is a fine-grained category, such as "left turn," "right turn," "U-turn," or "left turn and go straight" on a traffic sign.

[0198] In this embodiment, by using a fine-grained model to process each sub-region, the image features of each sub-region in the image to be identified can be extracted in a finer granularity, thereby improving the accuracy of the recognition result of the map element to be identified.

[0199] It should be noted that although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0200] on the other hand, Figure 15 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application. The device can be a device in a terminal device or a server, such as Figure 15 As shown, the apparatus 700 includes:

[0201] An acquisition module 710 is configured to acquire an image to be identified, perform feature extraction on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified;

[0202] A feature extraction module 720 is configured to generate multiple sub-regions based on the target region, and perform feature extraction on each sub-region to obtain image features of each sub-region;

[0203] The element recognition module 730 is used to perform fusion processing on the image features of all sub-regions to obtain fusion features, and determine the recognition results of the map elements to be recognized based on the fusion features.

[0204] In some embodiments, see Figure 16 As shown, the feature extraction module 720 includes:

[0205] A background adding unit 721 is used to perform background adding processing on the target area to obtain a candidate window; the candidate window includes the target area;

[0206] The division unit 722 is configured to divide the candidate window n times according to n division sizes to obtain a plurality of sub-regions of different sizes, where n is a positive integer and n≥1.

[0207] In some embodiments, the background adding unit 721 is specifically configured to:

[0208] The target area is resized to obtain a candidate window.

[0209] In some embodiments, the background adding unit 721 is further configured to:

[0210] With the map element to be identified as the center, the target area is expanded according to the preset ratio.

[0211] In some embodiments, the element identification module 730 is specifically configured to:

[0212] Perform vectorization processing on the image features of each sub-region to obtain the feature vector corresponding to the sub-region;

[0213] Determine the direction value corresponding to each eigenvalue in the eigenvector corresponding to each sub-region, where the direction value is used to indicate whether the sub-region includes the target region;

[0214] The fusion feature is obtained according to the feature value corresponding to each sub-region and the direction value corresponding to the sub-region.

[0215] In some embodiments, the element identification module 730 is specifically configured to:

[0216] The fused features are input into the trained detection network, and the recognition results of the map elements to be identified are determined based on the output of the detection network; the detection model is trained based on historical images and the map element recognition results of historical images, and the historical images are marked with traffic element areas.

[0217] In some embodiments, the training process of the detection network includes:

[0218] Generate multiple sample sub-regions based on historical images, input each sample sub-region into the feature extraction network to be trained for feature extraction processing, and obtain image features of each sample sub-region;

[0219] The image features of all sample sub-regions are input into the fusion network to be trained for fusion processing to obtain the sample fusion features;

[0220] Input the sample fusion features into the detection network to be trained to obtain the prediction results of map elements;

[0221] Calculate the loss function based on the prediction results of map elements and the map element recognition results of historical images;

[0222] According to the minimization of the loss function, an iterative algorithm is used to iteratively adjust the parameters of the feature extraction network to be trained, the fusion network to be trained, and the detection network to be trained to obtain the detection network.

[0223] In some embodiments, the acquisition module 710 is specifically configured to:

[0224] Perform feature extraction processing on the image to be identified to obtain a feature map of the image to be identified;

[0225] For each feature point in the feature map, determine the candidate box corresponding to each feature point; the candidate box contains the feature point;

[0226] Detect map elements for the image features in each candidate frame and obtain the confidence level corresponding to the candidate frame; the confidence level is used to represent the probability that the image in the candidate frame is a map element;

[0227] The area corresponding to the candidate box whose confidence meets the preset rules is determined as the target area of the map element to be identified in the image to be identified.

[0228] In some embodiments, the acquisition module 710 is further configured to:

[0229] For each feature point in the feature map, with the feature point as the center, generate candidate boxes corresponding to each feature point according to different sizes and scaling ratios.

[0230] It can be understood that the functions of the various functional modules of the image recognition device of this embodiment can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment and will not be repeated here.

[0231] In summary, the image recognition device provided in the embodiment of the present application obtains the image to be recognized through the acquisition module, and performs feature extraction processing on the image to be recognized through the feature extraction module to obtain the target area of the map element to be recognized in the image to be recognized, generates multiple sub-areas based on the target area, performs feature extraction on each sub-area respectively, obtains the image features of each sub-area, and then fuses the image features of all sub-areas through the element recognition module to obtain the fused features, and determines the recognition result of the map element to be recognized based on the fused features. Compared with the prior art, the technical solution in the present application, on the one hand, generates multiple sub-areas based on the target area, thereby being able to extract the image features of each sub-area in the image to be recognized in a more fine-grained manner; on the other hand, by fusing the image features of all sub-areas, it is possible to enhance the saliency of the target area and reduce the interference of the background area in the image to be recognized, thereby combining more comprehensive features to determine the recognition result of the map element to be recognized, and also makes the accuracy of the recognition result of the map element to be recognized determined by the method provided by the present application significantly improved compared with the prior art.

[0232] On the other hand, the device provided in an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image recognition method as described above is implemented.

[0233] Reference below Figure 17 , Figure 17 This is a schematic diagram of the structure of the computer system of the terminal device of an embodiment of the present application.

[0234] like Figure 17As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage unit 303 to a random access memory (RAM) 303. Various programs and data required for the operation of the system 300 are also stored in the RAM 303. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0235] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, and the like; an output section 307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 308 including a hard disk; and a communication section 309 including a network interface card such as a LAN card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read therefrom can be installed into the storage section 308 as needed.

[0236] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 303, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the system of the present application are executed.

[0237] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0238] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0239] The units or modules involved in the embodiments described in the present application can be implemented by software or by hardware. The described units or modules can also be set in a processor. For example, they can be described as: a processor comprising: an acquisition module, a feature extraction module and an element recognition module. Among them, the names of these units or modules do not constitute a limitation on the units or modules themselves under certain circumstances. For example, the acquisition module can also be described as "used to acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain the target area of the map element to be identified in the image to be identified in the image to be identified."

[0240] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently and not be incorporated into the electronic device. The computer-readable storage medium stores one or more programs, and when the aforementioned programs are used by one or more processors to execute the image recognition method described in the present application:

[0241] Acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified;

[0242] generating a plurality of sub-regions based on the target region, performing feature extraction on each of the sub-regions to obtain image features of each sub-region;

[0243] The image features of all the sub-regions are fused to obtain fused features, and the recognition result of the map element to be recognized is determined based on the fused features.

[0244] In summary, the image recognition method, apparatus, device and storage medium provided in the embodiments of the present application obtain an image to be recognized and perform feature extraction processing on the image to be recognized to obtain a target area of the map element to be recognized in the image to be recognized, generate multiple sub-areas based on the target area, perform feature extraction on each sub-area respectively, obtain the image features of each sub-area, then fuse the image features of all sub-areas to obtain fused features, and determine the recognition result of the map element to be recognized based on the fused features. Compared with the prior art, the technical solution in the present application, on the one hand, generates multiple sub-areas based on the target area, thereby being able to extract the image features of each sub-area in the image to be recognized in a more fine-grained manner; on the other hand, by fusing the image features of all sub-areas, it is possible to enhance the saliency of the target area and reduce the interference of the background area in the image to be recognized, thereby combining more comprehensive features to determine the recognition result of the map element to be recognized, and also making the accuracy of the recognition result of the map element to be recognized determined by the method provided by the present application significantly improved compared with the prior art.

[0245] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in this application.

Claims

1. An image recognition method, characterized in that: include: Acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified; After determining the target area, adding corresponding background information to the target area, dividing the image area formed after adding the background information to obtain a plurality of sub-areas, and performing feature extraction on each of the sub-areas to obtain image features of each sub-area; Vectorization is performed on the image features of each sub-region to obtain a feature vector corresponding to the sub-region. Different feature vectors correspond to feature values and direction values of different dimensions. The feature value is used to characterize the numerical value of the attraction or repulsion between the information of the dimension and the target category. The direction value is used to characterize whether the sub-region includes the target area. When the sub-region does not contain the target area, that is, contains the background area, the corresponding direction value is "-"; when the sub-region contains the target area, the corresponding direction value is "+". The feature value with a direction value of "+" indicates that the information of the dimension has an attraction with the target type of the map element, and the magnitude of the attraction is the numerical value of the dimension; the feature value with a direction value of "-" indicates that the information of the dimension has a repulsion with the target type of the map element, and the magnitude of the repulsion is the numerical value of the dimension. The feature vectors of the target area and the feature vectors of the background area in each sub-area are fused to obtain fused features. In the process of fusion of the feature vectors of the target area, if the direction of a certain dimension is repulsive, it will not be included in the feature fusion of attractive force. In the process of fusion of the feature vectors of the background area, if the direction of a certain dimension is attractive, it will not be included in the feature fusion of repulsive force. The recognition result of the map element to be recognized is determined based on the fusion feature.

2. The method according to claim 1, characterized in that After determining the target area, corresponding background information is added to the target area, and the image area formed after adding the background information is divided to obtain multiple sub-areas, including: Performing background addition processing on the target area to obtain a candidate window; the candidate window includes the target area; The candidate window is divided n times according to n division sizes to obtain a plurality of sub-regions of different sizes, where n is a positive integer and n≥1.

3. The method according to claim 2, characterized in that Performing background addition processing on the target area to obtain a candidate window includes: The target area is resized to obtain the candidate window.

4. The method according to claim 3, characterized in that The expanding the target area includes: The target area is expanded according to a preset ratio with the map element to be identified as the center.

5. The method according to claim 1, wherein Determining a recognition result of the map element to be recognized based on the fusion feature includes: The fusion features are input into a trained detection network, and the recognition result of the map element to be identified is determined based on the output of the detection network; the detection network is trained based on historical images and map element recognition results of historical images, and the historical images are marked with traffic element areas.

6. The method according to claim 5, characterized in that The training process of the detection network includes: Generate multiple sample sub-regions based on the historical image, input each sample sub-region into the feature extraction network to be trained for feature extraction processing, and obtain image features of each sample sub-region; The image features of all sample sub-regions are input into the fusion network to be trained for fusion processing to obtain the sample fusion features; Inputting the sample fusion features into the detection network to be trained to obtain the prediction results of the map elements; Calculating a loss function based on the prediction results of the map elements and the map element recognition results of the historical images; According to the minimization of the loss function, an iterative algorithm is used to iteratively adjust the parameters of the feature extraction network to be trained, the fusion network to be trained, and the detection network to be trained to obtain the detection network.

7. The method according to claim 1, characterized in that Performing feature extraction processing on the image to be identified to obtain a target area of the map element to be identified in the image to be identified, including: Performing feature extraction processing on the image to be identified to obtain a feature map of the image to be identified; For each feature point in the feature map, determining a candidate box corresponding to each feature point; the candidate box contains the feature point; Detecting map elements for the image features in each candidate frame to obtain a confidence level corresponding to the candidate frame; the confidence level is used to represent the probability that the image in the candidate frame is a map element; The area corresponding to the candidate box whose confidence meets the preset rules is determined as the target area of the map element to be identified in the image to be identified.

8. The method according to claim 7, characterized in that For each feature point in the feature map, determining a candidate box corresponding to each feature point includes: For each feature point in the feature map, with the feature point as the center, candidate boxes corresponding to each feature point are generated according to different sizes and scaling ratios.

9. An image recognition device, characterized in that: The device comprises: an acquisition module, configured to acquire an image to be identified, perform feature extraction processing on the image to be identified, and obtain a target area of a map element to be identified in the image to be identified; a feature extraction module configured to, after determining the target area, add corresponding background information to the target area, divide the image area formed after adding the background information into a plurality of sub-areas, and extract features from each of the sub-areas to obtain image features of each sub-area; The feature recognition module is used to perform vectorization processing on the image features of each sub-region to obtain the feature vector corresponding to the sub-region. Different feature vectors correspond to feature values and direction values of different dimensions. The feature value is used to characterize the numerical value of the attraction or repulsion between the information of the dimension and the target category. The direction value is used to characterize whether the sub-region includes the target area. When the sub-region does not contain the target area, that is, it contains the background area, the corresponding direction value is "-"; when the sub-region contains the target area, the corresponding direction value is "+". The feature value with a direction value of "+" indicates that the information of the dimension has an attraction with the target type of the map element, and the magnitude of the attraction is the numerical value of the dimension; the feature value with a direction value of "-" indicates that the information of the dimension has a repulsion with the target type of the map element, and the magnitude of the repulsion is the numerical value of the dimension. The feature vectors containing the target area and the feature vectors containing the background area in each sub-area are fused separately to obtain fused features. In the process of fusing the feature vectors containing the target area, if the direction of a certain dimension is repulsive, it will not participate in the feature fusion of attractive force. In the process of fusing the feature vectors containing the background area, if the direction of a certain dimension is attractive, it will not participate in the feature fusion of repulsive force. The recognition result of the map element to be identified is determined based on the fused features.

10. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the image recognition method according to any one of claims 1 to 8 when executing the program.

11. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is used to implement the image recognition method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed, the image recognition method according to any one of claims 1 to 8 is implemented.

Images