Point of interest information processing method, device, electronic device and storage medium

Abstract

The present application discloses a method for processing point of interest information, which relates to the field of image processing, and specifically to intelligent transportation technology. The specific implementation scheme includes: obtaining an image to be processed; extracting at least one candidate point of interest POI signboard image from the image to be processed. And for each candidate POI signboard image in the above-mentioned at least one candidate POI signboard image, the following operations are performed: extracting the first type of features and the second type of features of the candidate POI signboard image; based on the above-mentioned first type of features and second type of features, classifying the candidate POI signboard image to determine the type to which the candidate POI signboard image belongs; and, when it is determined that the candidate POI signboard image belongs to a predetermined type, determining that the candidate POI signboard image is a POI signboard image to be identified. The present application also discloses a point of interest information processing device, an electronic device, and a storage medium.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically to intelligent transportation technology. More specifically, the present application provides a method, device, equipment and storage medium for processing point of interest information. Background Art

[0002] In the prior art, the processing method for the image of the signboard of the point of interest is to detect it first and then recognize it. However, due to the accuracy of the detection algorithm, the detection result of the image of the signboard of the point of interest may be wrong. For example, the image of the billboard in the street view image is mistakenly detected as the image of the signboard of the point of interest. The above detection error will lead to the subsequent invalid recognition process for the image that does not contain the real signboard information of the point of interest, resulting in low recognition accuracy and efficiency of the image of the signboard of the point of interest. Summary of the invention

[0003] Provided are a method, device, equipment and storage medium for processing point of interest information.

[0004] According to a first aspect, a method for processing point of interest information is provided, characterized in that it includes: obtaining an image to be processed; extracting at least one candidate point of interest POI signboard image from the image to be processed. And for each candidate POI signboard image in the at least one candidate POI signboard image, the following operations are performed: extracting first-category features and second-category features of the candidate POI signboard image; classifying the candidate POI signboard image based on the first-category features and the second-category features to determine the type to which the candidate POI signboard image belongs; and when it is determined that the candidate POI signboard image belongs to a predetermined type, determining the candidate POI signboard image as a POI signboard image to be identified.

[0005] According to a second aspect, a device for processing information of a point of interest is provided, comprising: an acquisition module, an image extraction module, a feature extraction module, a classification module and a determination module. The acquisition module is used to acquire an image to be processed. The image extraction module is used to extract at least one candidate point of interest (POI) signboard image from the image to be processed. The feature extraction module is used to extract the first type of features and the second type of features of each candidate POI signboard image in the at least one candidate POI signboard image. The classification module is used to classify the candidate POI signboard image based on the first type of features and the second type of features of the candidate POI signboard image to determine the type to which the candidate POI signboard image belongs. The determination module is used to determine that the candidate POI signboard image is a POI signboard image to be identified when it is determined that the candidate POI signboard image belongs to a predetermined type.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for processing point of interest information provided by the present application.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method for processing point of interest information provided in the present application.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a computer, implements the method for processing point of interest information provided in the present application.

[0009] After extracting the candidate POI sign images from the image to be processed, the POI information processing method of the embodiment of the present application further performs classification processing on each candidate POI sign image to further screen out the candidate POI sign images that meet expectations (i.e., belong to the predetermined type) as the POI sign images to be identified. This can avoid invalid recognition in the later stage due to early detection errors, reduce the time and computing resources occupied by invalid images in the subsequent recognition process, and improve the pertinence of the subsequent recognition process. In addition, in the classification process, the first type of features and the second type of features of the candidate POI sign images are comprehensively considered. Since the two types of features describe the image features from different levels, the content of the candidate POI sign images can be described more accurately and comprehensively, which can improve the accuracy and recall rate of the classification process.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0012] Figure 1A is an example schematic diagram of a POI signboard image according to an embodiment of the present application;

[0013] Figure 1B is an exemplary system architecture of the method and device for processing point of interest information according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of a method for processing point of interest information according to an embodiment of the present application;

[0015] Figure 3is an example schematic diagram of a process for processing point of interest information according to an embodiment of the present application;

[0016] Figure 4 is an example flow chart of a method for processing point of interest information according to another embodiment of the present application;

[0017] Figure 5 is an exemplary schematic diagram of the first type of features of a candidate POI signboard image according to an embodiment of the present application;

[0018] Figure 6 is an example flow chart of constructing and training a classification model according to an embodiment of the present application;

[0019] Figure 7 is a block diagram of a device for processing point of interest information according to an embodiment of the present application; and

[0020] Figure 8 It is a block diagram of an electronic device used to implement the method for processing point of interest information in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] Points of interest (POI) are coordinate point annotation data on an electronic map, used to indicate places represented by the associated coordinate points, such as government departments, commercial institutions (such as gas stations, department stores, supermarkets, restaurants, hotels, convenience stores, etc.), tourist attractions (such as parks, historical sites, etc.), transportation facilities (such as various stations, parking lots, speed cameras, speed limit signs, etc.). When a POI is displayed on an electronic map, it is necessary to obtain relevant information such as the name, category, and geographical location of the POI.

[0023] In one processing method, the name, category and other information of the POI can be determined based on the POI signboard image. Figure 1A , Figure 1A is an example schematic diagram of a POI signboard image according to an embodiment of the present application. Figure 1AAfter seeing the POI sign image shown, the machine can recognize the text information contained in the POI sign image, such as "convenience store", so that the name and category of the POI can be determined based on the recognition result. Exemplarily, the actual street view image can be obtained through various channels, and the actual street view image is used as the image to be processed, and one or more POI sign images are detected and located therefrom using the target detection algorithm. Limited by the detection accuracy of the target detection algorithm, the above detection results may be erroneous, such as misdetecting a billboard image in a street view image as a POI sign image. The above detection errors will lead to invalid recognition processes for subsequent images that do not contain real POI sign information, resulting in low recognition accuracy and efficiency of POI sign images.

[0024] Figure 1B is an exemplary system architecture 100 to which the method and apparatus for processing point of interest information can be applied according to an embodiment of the present application. It should be noted that: Figure 1B What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0025] like Figure 1B As shown, the system architecture 100 according to this embodiment may include multiple terminal devices 110, a network 120 and a server 130. The terminal device 110 may be any terminal device capable of image acquisition, such as a smart phone, a tablet computer, a video camera, a camera, etc., without limitation. The server 130 may be any electronic device with a certain computing capability, without limitation. The following example illustrates the process of interaction between the terminal device 110 and the server 130 via the network 120.

[0026] Exemplarily, the terminal device 110 may be a professional panoramic acquisition device, which is equipped on a motor vehicle such as a car or motorcycle. In some cases, the device may even be manually loaded to collect street images by sweeping the streets of the real world. After the collection, the collected street images and some associated information (such as time information, geographic location information, etc.) are sent to the server 130 via the network 120 for analysis and processing. Similarly, the terminal device 110 may be a user's personal terminal device, and the user may take some street view images in the course of daily work and life and feed them back to the server 130 for analysis and processing. The server 130 extracts effective POI sign information based on the acquired street view image, thereby annotating and updating the POI in the electronic map 140. Furthermore, the electronic map 140 after POI annotation and update can also be pushed to multiple terminal devices (the multiple terminal devices may be the same or different from the above-mentioned terminal device 110) to realize functions such as map display and navigation.

[0027] According to an embodiment of the present application, a method for processing point of interest information is provided. The method is exemplarily described below through a legend. It should be noted that the sequence number of each operation in the following method is only used as a representation of the operation for the purpose of description, and should not be regarded as indicating the execution order of the various operations. Unless explicitly stated, the method does not need to be executed in the order shown.

[0028] Figure 2 is a flowchart of a method for processing point of interest information according to an embodiment of the present application. The method for processing point of interest information can be, for example, Figure 1B Server-side execution shown.

[0029] like Figure 2 As shown, the method may include operations S201 to S205.

[0030] In operation S201, an image to be processed is acquired.

[0031] Exemplarily, the implementation method of obtaining the image to be processed has been illustrated above and will not be repeated here.

[0032] In operation S202, at least one candidate POI signboard image is extracted from the image to be processed.

[0033] Next, for each candidate POI signboard image in the at least one candidate POI signboard image, the following operations S203 to S205 are performed.

[0034] In operation S203, first-category features and second-category features of the candidate POI signboard image are extracted.

[0035] Exemplarily, the first type of features and the second type of features are extracted in different ways, and respectively represent feature information of the candidate POI signboard images at different levels.

[0036] In operation S204, the candidate POI sign image is classified based on the first and second features to determine the type to which the candidate POI sign image belongs.

[0037] In operation S205 , when it is determined that the candidate POI sign image belongs to the predetermined type, the candidate POI sign image is determined as a POI sign image to be identified.

[0038] For example, the predetermined type can be set according to actual business needs. For example, an image containing real POI sign information is set as an image of a predetermined type, or a more fine-grained setting is performed, such as setting an image containing real shop POI sign information as an image of a predetermined type, etc., which is not limited here. Candidate POI sign images that do not belong to the predetermined type can be called invalid images, and there is no need to perform a subsequent recognition process for POI sign content.

[0039] Those skilled in the art can understand that, according to the POI information processing method of the embodiment of the present application, after extracting the candidate POI signboard image from the image to be processed, classification processing is further performed on each candidate POI signboard image to further screen out the candidate POI signboard image that meets expectations (i.e., belongs to a predetermined type) as the POI signboard image to be identified. This can avoid invalid recognition in the later stage due to early detection errors, reduce the time and computing resources occupied by invalid images in the subsequent recognition process, and improve the pertinence of the subsequent recognition process. In addition, in the classification process, the first type of features and the second type of features of the candidate POI signboard image are comprehensively considered. Since the two types of features describe the image features from different levels, the content of the candidate POI signboard image can be described more accurately and comprehensively, which can improve the accuracy and recall rate of the classification process.

[0040] Figure 3 The figure is an example schematic diagram of a process of processing point of interest information according to an embodiment of the present application.

[0041] like Figure 3 As shown, after acquiring the image to be processed 310, an object detection algorithm, such as Fast RCNN (fast region convolutional neural network), Faster RCNN (faster region convolutional neural network), etc., can be used to detect the image to be processed 310. Exemplarily, the image to be processed 310 can be input into the target detection network 320 for processing to obtain a detection result 330. The detection result 330 generates at least one detection box (bounding box) 331 (2 detection boxes in this example) and the category and position of the target object targeted by each detection box in the image to be processed 310. In this example, the POI signboard image is the target object. Based on the above detection results, the local image area surrounded by the at least one detection box 331 can be segmented from the image to be processed as at least one candidate POI signboard image. As shown Figure 3As shown, in this example, two candidate POI signboard images 340 can be obtained. Next, each candidate POI signboard image 340 is classified using a pre-trained classification model 350 .

[0042] According to an embodiment of the present application, when it is determined that the candidate POI sign image 340 belongs to a predetermined type, the candidate POI sign image 340 is determined to be a POI sign image to be identified, and the recognition model 360 can be used to perform text recognition on the candidate POI sign image 340 to obtain a text recognition result for the candidate POI sign image 340. The above recognition process can be implemented, for example, using OCR (optical character recognition) technology. Therefore, the text recognition result can be used to mark the POI for the POI sign image to be identified in the electronic map 370.

[0043] It can be understood that the above process uses a target detection algorithm to detect the target object that is similar to the POI sign in the image to be processed, and the detection accuracy for the real POI sign image is high and the detection speed is fast. In some cases, the above process may also detect images of other objects similar to the POI sign (such as billboards, signboards, etc.) as candidate POI sign images. In order to avoid such detection errors causing loss of accuracy and efficiency in the subsequent recognition process, according to an embodiment of the present application, the classification process is continued for the candidate POI sign image, and the candidate POI sign image is determined to be the POI sign image to be identified based on the classification result, and then the candidate POI sign image is identified and the electronic map is annotated based on the recognition result.

[0044] Figure 4 This is an example flow chart of a method for processing point of interest information according to another embodiment of the present application, which is used to exemplarily illustrate the process of classifying the above operations S203 to S205 in combination with two types of features of the candidate POI image.

[0045] like Figure 4 As shown, the method may include operations S401 to S406.

[0046] In operation S401 , a candidate POI signboard image is acquired.

[0047] The candidate POI signboard images obtained in operation S401 may be each POI signboard image extracted from the image to be processed. The extraction process has been illustrated above and will not be described again here.

[0048] In operation S402, a first type of feature extraction method is used to extract first type of features of the candidate POI signboard image.

[0049] Exemplarily, a manually designed visual feature extraction method can be used to extract the first type of features of the candidate POI signboard image. For example, the first type of features can include color histogram features, color correlation graph features, histogram of oriented gradient (HOG) features, scale-invariant feature transform (SIFT) features, etc., which are used to express image color and texture information. The feature extraction principle of these first type of features is simple, and they can be designed to well characterize the candidate POI signboard image.

[0050] Exemplarily, the first type of features of the POI signboard image may also be determined based on the geographical location information to which the POI signboard image is directed.

[0051] In operation S403, a second type of feature extraction method is used to extract the second type of features of the candidate POI signboard image.

[0052] According to an embodiment of the present application, the above-mentioned process of extracting the second type of features of the candidate POI signboard image may include: using a deep learning model to perform image representation learning on the candidate POI signboard image to obtain the second type of features. Under the deep learning framework, the image representation method is converted into a data-driven learning-based expression, and its model directly accepts the original image signal of the candidate POI signboard image as input, uses the image category information as supervision, and expresses the candidate POI signboard image layer by layer. When there are sufficient training samples and sufficient computing resources, the use of a deep learning framework can facilitate feature learning. For example, a deep convolutional neural network (CNN) can be used to extract the second type of features of the candidate POI signboard image. CNN is an artificial neural network structure designed by inspiration from the visual nervous system in bionics. The convolution kernel is the weight of the neural network, and linear weighting is performed on different image sub-regions (local receptive fields) in the POI signboard image using a shared weight strategy, and nonlinear function mapping is achieved using multi-level processing such as nonlinear activation functions, pooling, and sampling operations. Thereby achieving a good representation of the content of the candidate POI signboard image.

[0053] In operation S404, the first type of features and the second type of features are fused to obtain fused features.

[0054] For example, the first type of features manually designed and extracted according to business needs in the above operation S402 can be called "shallow features", and the second type of features extracted based on the deep learning framework in the above operation S403 can be called "deep features". This operation S404 fuses the two types of features to obtain fused features that can more comprehensively and accurately describe the candidate POI sign image, so that the subsequent classification process for the fused features is more accurate.

[0055] According to an embodiment of the present application, the above-mentioned first-category features and second-category features may both have their own vector representations (embeddings), and the above-mentioned process of fusing the first-category features and the second-category features to obtain fused features may include: concatenating and merging the vector representations of the first-category features and the vector representations of the second-category features to obtain a fused feature vector.

[0056] In operation S405, the fused features are classified using a pre-trained classification model to obtain a classification result of the fused features.

[0057] In operation S406 , the type of the candidate POI signboard image is determined according to the classification result.

[0058] Exemplarily, whether the input image contains real POI sign information can be used as a classification label of a classification model for training to obtain a classification result for characterizing whether the input image contains real POI sign information.

[0059] When the classification result of the fused features indicates that a candidate POI sign image contains real POI sign information, it is determined that the candidate POI sign image belongs to a predetermined type. This indicates that the candidate POI sign image is an image for a real POI sign, and a subsequent recognition process for the POI sign content can be performed. When the classification result of the fused features indicates that a candidate POI sign image does not contain real POI sign information, it is determined that the candidate POI sign image does not belong to a predetermined type. This indicates that the candidate POI sign image is not an image for a real POI sign, but may be an image for a billboard, signboard, etc., and there is no need to perform a subsequent recognition process for the POI sign content. This embodiment re-screens the candidate POI sign images through a classification process to screen out invalid images that do not contain valid POI sign information. The time and computing resources occupied by invalid images in the subsequent recognition process can be reduced, and the recognition processing efficiency and accuracy can be improved.

[0060] Figure 5 is an exemplary schematic diagram of the first type of features of a candidate POI sign image according to an embodiment of the present application.

[0061] like Figure 5As shown, according to an embodiment of the present application, the first type of features of the candidate POI sign image 540 may include one or more of a color distribution feature 541, a texture feature 542, and a geographic location feature 543. Each of the first type of features is exemplarily described below.

[0062] The color distribution feature 541 is one of the most widely used visual features in image retrieval, mainly because color is often closely related to the objects or scenes contained in the image. In addition, compared with other visual features, the color distribution feature is less dependent on factors such as the size, direction, and viewing angle of the image itself, and thus has higher robustness. When extracting the color distribution features of the candidate POI sign image 540, it is necessary to select a suitable color space to describe the features, and it is also necessary to design a certain quantization method to express the color distribution features in vector form.

[0063] For example, the color histogram feature of the candidate POI sign image 540 can be used as the color distribution feature 541 of the candidate POI sign image 540. The color histogram feature can describe the proportion of different colors in the candidate POI sign image. The color histogram feature can be characterized based on different color spaces and coordinate systems, such as the color histogram feature of the R (red) G (green) B (blue) space, the color histogram feature of the H (hue) S (saturation) V (value) space, etc. Calculating the color histogram can divide the color space into several small color intervals, and then obtain the color histogram feature by calculating the number of pixels whose colors fall within each small interval. Figure 5 The example in Figure 2 shows the distribution of the number of pixels in the three color channels of red, green and blue within the predetermined intensity range.

[0064] According to an embodiment of the present application, the HOG features of the candidate POI signboard image 540 can be used as the texture features 542 of the candidate POI signboard image 540. The HOG features are constructed by calculating and counting the gradient direction histograms of the local area of ​​the image. The HOG features can maintain good invariance to both geometric and optical deformations of the image. In implementation, the directional gradient histogram decomposes the image into several image blocks, each of which includes several cell units, calculates the directional gradient histogram of each cell unit, connects the histograms of all cell units in the same image block to form the directional gradient histogram features of the image block, and normalizes them. Finally, the feature descriptions of all image blocks in the image are connected to obtain the HOG features of the entire candidate POI signboard image 540. Figure 5 The example in Figure 2 shows the gradient magnitude in each direction for each cell unit in an image block.

[0065] According to an embodiment of the present application, for example, the candidate POI sign image 540 is extracted from a certain image to be processed, and the POI geographic location information for the image to be processed can be obtained. For example, the geographic location information uploaded by the terminal device when uploading the image to be processed is obtained as the POI geographic location information for the image to be processed. Based on the POI geographic location information, vector representation is performed to determine the geographic location feature 543 of the candidate POI sign image 540. In actual situations, the probability of whether a location has POI sign information can be determined with the help of geographic location information. Using geographic location information as the first type of feature for subsequent classification processes helps to improve the accuracy of the classification model.

[0066] In the method for processing point of interest information according to the embodiment of the present application, before inputting the fusion features of the candidate POI signboard image into the classification model, the classification model may be acquired, for example, by constructing and training the classification model.

[0067] Figure 6 This is an example flowchart of building and training a classification model according to an embodiment of the present application.

[0068] like Figure 6 As shown, the process of constructing and training the classification model may include operations S601 to S606. It should be noted that the process of constructing and training the classification model may be implemented in the same electronic device as operations S201 to S205 above, or in different electronic devices, which is not limited here.

[0069] In operation S601, an initial classification model is constructed.

[0070] In operation S602, a plurality of sample images are acquired.

[0071] The method for acquiring the multiple sample images may be the same as the method for acquiring the image to be processed in the above text, and will not be described in detail here. The multiple sample images may also be acquired from a large number of candidate POI signboard images that have been processed in the past.

[0072] Next, the following operations S603 to S606 are performed for each sample image of the plurality of sample images.

[0073] In operation S603, first-category sample features and second-category sample features of the sample image are extracted.

[0074] The process of extracting the first type of sample features and the second type of sample features of each sample image is similar to the process of extracting the first type of features and the second type of features of each candidate POI sign image described above, and will not be repeated here.

[0075] In operation S604, the first type of sample features and the second type of sample features are fused to obtain sample fusion features of the sample image.

[0076] The process of fusing the first type of sample features and the second type of sample features is similar to the process of fusing the first type of features and the second type of features in the above text, and will not be repeated here.

[0077] In operation S605 , a label is added to the sample image, where the label is used to indicate whether the sample image contains real POI sign information.

[0078] In operation S606, the initial classification model is trained using the sample fusion features and labels of each of the plurality of sample images to obtain a pre-trained classification model.

[0079] It can be understood that in the above training process, since the first type of sample features and the second type of sample features of the sample image respectively represent the features of different levels of the sample image, the fusion features obtained by fusing the two can more comprehensively and accurately describe the content of the sample image. The initial classification model is trained using the fusion features and corresponding labels, so that the classification model can quickly learn to distinguish whether the image contains real POI sign information during the training process.

[0080] Figure 7 It is a block diagram of a device for processing point of interest information according to an embodiment of the present application.

[0081] like Figure 7 As shown, the device 700 for processing point of interest information may include: an acquisition module 710 , an image extraction module 720 , a feature extraction module 730 , a classification module 740 and a determination module 750 .

[0082] The acquisition module 710 is used to acquire the image to be processed.

[0083] The image extraction module 720 is used to extract at least one candidate point of interest (POI) signboard image from the image to be processed.

[0084] The feature extraction module 730 is used to extract the first type of features and the second type of features of each candidate POI signboard image in the at least one candidate POI signboard image.

[0085] The classification module 740 is used to classify the candidate POI sign image based on the first type of features and the second type of features of the candidate POI sign image to determine the type to which the candidate POI sign image belongs.

[0086] The determination module 750 is configured to determine the candidate POI sign image as a POI sign image to be identified when it is determined that the candidate POI sign image belongs to a predetermined type.

[0087] According to an embodiment of the present application, the first type of features mentioned above includes at least one of the following: color distribution features, texture features, and geographic location features.

[0088] Exemplarily, the feature extraction module 730 may include a first extraction submodule, configured to extract a color histogram feature of the candidate POI signboard image as a color distribution feature of the candidate POI signboard image.

[0089] Exemplarily, the feature extraction module 730 may include a second extraction submodule, which is used to extract the directional gradient histogram feature of the candidate POI signboard image as the texture feature of the candidate POI signboard image.

[0090] Exemplarily, the feature extraction module 730 may include a third extraction submodule, configured to obtain POI geographic location information for the image to be processed; and determine the geographic location features of the candidate POI signboard image based on the POI geographic location information.

[0091] According to an embodiment of the present application, the classification module 740 may include: a fusion submodule, used to fuse the first type of features and the second type of features to obtain a fused feature; and a classification processing submodule, used to classify the fused feature using a pre-trained classification model to obtain a classification result of the fused feature, and determine the type of the candidate POI sign image according to the classification result.

[0092] For example, the fusion submodule is used to connect and merge the vector representation of the first type of features and the vector representation of the second type of features to obtain a fused feature vector.

[0093] For example, the classification processing submodule is used to determine that the candidate POI sign image belongs to a predetermined type when the classification result of the fused features indicates that the candidate POI sign image contains real POI sign information.

[0094] According to an embodiment of the present application, the above-mentioned device 700 may also include: a construction module, which is used to construct an initial classification model before the classification module uses a pre-trained classification model to classify the fusion features. A sample acquisition and processing module, which is used to acquire multiple sample images, and perform the following operations for each sample image in the multiple sample images: extract the first type of sample features and the second type of sample features of the sample image, fuse the first type of sample features and the second type of sample features to obtain the sample fusion features of the sample image, and add a label to the sample image, which is used to characterize whether the sample image contains real POI sign information. And, a training module, which is used to train the initial classification model using the sample fusion features and labels of each of the above-mentioned multiple sample images to obtain a pre-trained classification model.

[0095] According to an embodiment of the present application, the feature extraction module 730 may include a fourth extraction submodule, which is used to perform image representation learning on the candidate POI signboard image using a deep learning model to obtain the second type of features.

[0096] According to an embodiment of the present application, the image extraction module 720 is specifically used to detect the image to be processed using a target detection algorithm to generate at least one detection frame in the image to be processed; and to segment the image area surrounded by the at least one detection frame from the image to be processed as at least one candidate POI signboard image.

[0097] According to an embodiment of the present application, the above-mentioned device 700 may also include: a recognition module, which is used to perform text recognition on the POI sign image to be identified after determining that the candidate POI sign image is the POI sign image to be identified, so as to obtain a text recognition result; and a labeling module, which is used to use the text recognition result to label the POI for the POI sign image to be identified in the map.

[0098] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit / sub-unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0099] According to the modules, submodules, units, and subunits of the embodiments of the present application, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present application, any one or more can be split into multiple modules to implement. According to the modules, submodules, units, and subunits of the embodiments of the present application, any one or more can be implemented at least in part as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or encapsulating the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present application, one or more can be implemented at least in part as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0100] According to an embodiment of the present application, the present application further provides an electronic device, a readable storage medium and a computer program product. The computer program product includes a computer program, which implements the method for processing point of interest information provided by the present application when executed by a computer.

[0101] like Figure 8 , is a block diagram of an electronic device according to a method for processing point of interest information according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0102] like Figure 8As shown, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 801 is taken as an example.

[0103] The memory 802 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the method for processing point of interest information provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the method for processing point of interest information provided in the present application.

[0104] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method for processing point of interest information in the embodiment of the present application (for example, the attached Figure 7 The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, the method for processing point of interest information in the above method embodiment is implemented.

[0105] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The electronic device of the method for processing point of interest information may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0107] The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the point of interest information processing method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0108] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0113] According to the technical solution of the embodiment of the present application, after extracting the candidate POI signboard image from the image to be processed, the POI information processing method further performs classification processing on each candidate POI signboard image to further screen out the candidate POI signboard image that meets expectations (i.e., belongs to a predetermined type) as the POI signboard image to be identified. This can avoid invalid recognition in the later stage due to early detection errors, reduce the time and computing resources occupied by invalid images in the subsequent recognition process, and improve the pertinence of the subsequent recognition process. In addition, in the classification process, the first type of features and the second type of features of the candidate POI signboard image are comprehensively considered. Since the two types of features describe the image features from different levels, the content of the candidate POI signboard image can be described more accurately and comprehensively, which can improve the accuracy and recall rate of the classification process.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0115] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for processing point of interest information, comprising: Get the image to be processed; Extracting at least one candidate POI signboard image from the image to be processed, wherein the candidate POI signboard image includes at least one of a real POI signboard image and an approximate POI signboard image, and the approximate POI signboard image includes at least one of a billboard image and a signboard image; For each candidate point of interest POI signboard image in the at least one candidate point of interest POI signboard image, Extracting first and second features of the candidate POI signboard image, wherein the first feature includes a geographic location feature, which is obtained by vectorizing geographic location information uploaded when the terminal uploads the image to be processed, and the second feature includes image representation features extracted using a deep learning model; Based on the fusion features of the first type of features and the second type of features, classifying the candidate POI signboard image to determine whether the candidate POI signboard image is a real POI signboard image; and When it is determined that the candidate point of interest POI signboard image is a real POI signboard image, determining the candidate point of interest POI signboard image as a POI signboard image to be identified; Performing text recognition on the POI sign image to be recognized to obtain a text recognition result; The POI corresponding to the POI sign image to be identified is marked in the map using the text recognition result.

2. The method according to claim 1, wherein: The first category of features also includes at least one of the following: Color distribution characteristics, texture characteristics.

3. The method according to claim 2, wherein: The first type of features of the candidate POI signboard image extracted include: The color histogram features of the candidate POI signboard image are extracted to serve as the color distribution features of the candidate POI signboard image.

4. The method according to claim 2, wherein: The first type of features of the candidate POI signboard image extracted include: The directional gradient histogram features of the candidate POI signboard image are extracted to serve as the texture features of the candidate POI signboard image.

5. The method according to claim 2, wherein: The first type of features of the candidate POI signboard image extracted include: Acquire POI geographic location information for the image to be processed; and Based on the POI geographical location information, the geographical location features of the candidate POI signboard image are determined.

6. The method according to claim 1, wherein: The classifying the candidate POI signboard image based on the first type of features and the second type of features includes: Fusing the first type of features with the second type of features to obtain fused features; Using a pre-trained classification model to classify the fused features to obtain a classification result of the fused features; and Determine whether the candidate point of interest (POI) signboard image is a real POI signboard image according to the classification result.

7. The method according to claim 6, further comprising: Before the fusion feature is classified by using the pre-trained classification model, Build an initial classification model; Acquire multiple sample images; For each sample image of the multiple sample images, extract a first type of sample feature and a second type of sample feature of the sample image, perform fusion processing on the first type of sample feature and the second type of sample feature to obtain a sample fusion feature of the sample image, and add a label to the sample image, wherein the label is used to indicate whether the sample image contains real POI sign information; as well as The initial classification model is trained using the sample fusion features and labels of each of the multiple sample images to obtain the pre-trained classification model.

8. The method according to claim 7, wherein: The step of determining whether the candidate POI signboard image is a real POI signboard image according to the classification result comprises: When the classification result of the fused feature indicates that the candidate point of interest POI signboard image contains real POI signboard information, the candidate point of interest POI signboard image is determined to be a real POI signboard image.

9. The method according to claim 6, wherein: The fusing the first type of features and the second type of features to obtain fused features includes: The vector representation of the first type of features and the vector representation of the second type of features are connected and merged to obtain a fused feature vector.

10. The method according to claim 1, wherein: Extracting at least one candidate point of interest (POI) signboard image from the image to be processed includes: Detecting the image to be processed using an object detection algorithm to generate at least one detection frame in the image to be processed; and An image area surrounded by each of the at least one detection frame is segmented from the image to be processed to serve as the at least one candidate point of interest (POI) signboard image.

11. A point of interest information processing device, comprising: An acquisition module, used for acquiring an image to be processed; An image extraction module, configured to extract at least one candidate POI signboard image from the image to be processed, wherein the candidate POI signboard image includes at least one of a real POI signboard image and an approximate POI signboard image, and the approximate POI signboard image includes at least one of a billboard image and a signboard image; a feature extraction module, configured to extract, for each candidate POI signboard image in the at least one candidate POI signboard image, first and second features of the candidate POI signboard image, wherein the first feature includes a geographic location feature, which is obtained by vectorizing the geographic location information uploaded when the terminal uploads the image to be processed, and the second feature includes an image representation feature extracted using a deep learning model; a classification module, configured to classify the candidate POI signboard image based on the fusion features of the first category of features and the second category of features, so as to determine whether the candidate POI signboard image is a real POI signboard image; A determination module, configured to determine the candidate point of interest POI sign image as a to-be-identified POI sign image when it is determined that the candidate point of interest POI sign image is a real POI sign image; A recognition module, used for performing text recognition on the POI sign image to be recognized to obtain a text recognition result; and The marking module is used to mark the POI of the POI signboard image to be identified in the map using the text recognition result.

12. The device according to claim 11, wherein The first category of features also includes at least one of the following: Color distribution characteristics, texture characteristics.

13. The device according to claim 12, wherein: The feature extraction module comprises: The first extraction submodule is used to extract the color histogram features of the candidate POI signboard image to serve as the color distribution features of the candidate POI signboard image.

14. The device according to claim 12, wherein: The feature extraction module comprises: The second extraction submodule is used to extract the directional gradient histogram features of the candidate POI signboard image to serve as the texture features of the candidate POI signboard image.

15. The device according to claim 12, wherein: The feature extraction module comprises: The third extraction submodule is used to obtain POI geographical location information for the image to be processed; and determine the geographical location features of the candidate POI sign image based on the POI geographical location information.

16. The device according to claim 11, wherein The classification module comprises: a fusion submodule, configured to fuse the first type of features with the second type of features to obtain a fused feature; and The classification processing submodule is used to classify the fused features using a pre-trained classification model to obtain a classification result of the fused features, and determine whether the candidate point of interest POI sign image is a real POI sign image based on the classification result.

17. The apparatus according to claim 16, further comprising: A construction module, used for constructing an initial classification model before classifying the fusion features using the pre-trained classification model; A sample acquisition and processing module, used to acquire a plurality of sample images, and for each of the plurality of sample images, extract a first type of sample feature and a second type of sample feature of the sample image, perform fusion processing on the first type of sample feature and the second type of sample feature to obtain a sample fusion feature of the sample image, and add a label to the sample image, wherein the label is used to indicate whether the sample image contains real POI sign information; as well as The training module is used to train the initial classification model by using the sample fusion features and labels of each of the multiple sample images to obtain the pre-trained classification model.

18. The device according to claim 17, wherein: The classification processing submodule is used to determine that the candidate point of interest POI sign image is a real POI sign image when the classification result of the fusion feature indicates that the candidate point of interest POI sign image contains real POI sign information.

19. The device according to claim 16, wherein: The fusion submodule is used to connect and merge the vector representation of the first type of features and the vector representation of the second type of features to obtain a fused feature vector.

20. The device according to claim 11, wherein The image extraction module is used to detect the image to be processed using a target detection algorithm to generate at least one detection frame in the image to be processed; and to segment the image area surrounded by each of the at least one detection frame from the image to be processed as the at least one candidate point of interest (POI) signboard image.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, which, when executed by the computer, implements the method according to any one of claims 1 to 10.

Images