A problem map screening method and apparatus

Abstract

The application provides a problem map screening method and device, comprising: acquiring a to-be-screened image; labeling the to-be-screened image to obtain a to-be-screened mask image; acquiring a mask image corresponding to a to-be-screened region in the to-be-screened image; determining whether the to-be-screened region is a map according to the mask image corresponding to the to-be-screened region and a preset standard map mask image; if yes, registering the mask image corresponding to the to-be-screened region and the preset standard map mask image to obtain a registered mask image corresponding to the to-be-screened region; and determining whether the to-be-screened region is a problem map according to the registered mask image corresponding to the to-be-screened region and the preset standard map mask image. Since the mask image corresponding to the to-be-screened region is a binary mask image without texture and color information, the application has good recognition effect on to-be-screened images with different textures and colors; and image registration makes it have good recognition effect on to-be-screened images with different deformations and angles.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method and apparatus for screening problem maps. Background Technology

[0002] A large number of problematic maps exist in videos, images, and other media on internet platforms. Currently, map screening on online platforms mostly relies on manual visual inspection. Due to the large amount of data and the complexity of the images, this often consumes a huge amount of manpower and has a high false positive rate. Existing deep learning-based map screening methods mainly include methods based on object detection technology and methods based on image classification technology. Methods based on object detection technology train models to detect areas in the image map that frequently contain problems and determine whether problems exist. Methods based on image classification technology directly classify the image to determine whether it is a map, and then determine whether it is a problematic map based on the feature map output by the model. Because problematic maps on the internet have diverse styles (including color, shape, texture, angle, etc.), complex backgrounds, and unclear differences, these methods cannot adapt to diverse scenarios, leading to many false positives. Summary of the Invention

[0003] In view of this, this application provides a problem map screening method and apparatus to solve the problem that existing technologies cannot effectively identify "problem maps" in complex and diverse scenarios. The technical solution is as follows:

[0004] A problem map screening method includes:

[0005] Acquire the image to be screened;

[0006] Obtain the mask image corresponding to the region to be screened in the image to be screened;

[0007] Based on the mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a map;

[0008] If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened.

[0009] Based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a problematic map.

[0010] Optionally, obtain the mask image corresponding to the region to be screened in the image to be screened, including:

[0011] The image to be screened is segmented using a pre-trained image segmentation model to obtain a mask image corresponding to the image to be screened. The image segmentation model is trained using sample images as training samples and the mask images corresponding to the sample images as training labels. Each image in the sample images includes a standard map or a problem map. The pixel values ​​of the map regions in the mask image are the first gray value, and the pixel values ​​of the non-map regions in the mask image are the second gray value.

[0012] Morphological processing is performed on the mask image corresponding to the image to be screened to obtain the processed mask image;

[0013] Based on the processed mask image, the mask image corresponding to the region to be screened is cropped from the mask image corresponding to the image to be screened.

[0014] Optionally, morphological processing is performed on the mask image corresponding to the image to be screened to obtain a processed mask image, including:

[0015] The first processing core is used to perform a closing operation on the mask image corresponding to the image to be screened, and the mask image after the closing operation is obtained.

[0016] The mask image after the closing operation is eroded using the second operation kernel to obtain the eroded mask image, wherein the first operation kernel is smaller than the second operation kernel;

[0017] Based on the largest connected component in the eroded mask image, remove the non-map regions from the eroded mask image, and then dilate the mask image after removing the non-map regions to obtain the dilated mask image, which is used as the processed mask image.

[0018] Optionally, based on the processed mask image, a mask image corresponding to the region to be screened is cropped from the mask image corresponding to the image to be screened, including:

[0019] Based on the processed mask image, the coordinates of the outermost edge of the region to be screened in the mask image corresponding to the image to be screened are determined. The region to be screened in the mask image corresponding to the image to be screened is expanded and cropped according to a set percentage of the outermost edge coordinates to obtain the mask image corresponding to the region to be screened.

[0020] Optionally, based on the mask image corresponding to the area to be screened and a preset standard map mask image, determine whether the area to be screened is a map, including:

[0021] Determine the intersection-union ratio (IoU) of the mask image corresponding to the area to be screened and the preset standard map mask image;

[0022] If the intersection-union ratio is greater than the first threshold, then the region to be screened is determined to be a map.

[0023] Optionally, the mask image corresponding to the area to be screened is registered with a preset standard map mask image to obtain a registered mask image corresponding to the area to be screened, including:

[0024] The mask image corresponding to the area to be screened and the preset standard map mask image are input into the pre-trained image registration model to obtain the registered mask image corresponding to the area to be screened. The image registration model is trained using the preset standard map mask image and the sample mask image as training samples, and the registered image corresponding to the sample mask image as the sample label.

[0025] Optionally, based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a problematic map, including:

[0026] Calculate the difference between the preset standard map mask image and the registered mask image to obtain the difference image;

[0027] Perform an opening operation on the difference image to obtain the image after the opening operation;

[0028] Based on the image after the opening operation, determine whether the area to be screened is a problem map.

[0029] Optionally, the area to be screened may include areas prone to missing values;

[0030] Problem map screening methods also include:

[0031] If the area to be screened is not a problem map, then the registration field is reversed from the preset standard map mask image to determine the position of the easily missing area in the mask image corresponding to the area to be screened based on the position of the easily missing area in the preset standard map mask image. The registration field represents the spatial coordinate transformation parameters from the mask image corresponding to the area to be screened to the preset standard map mask image.

[0032] Based on the position of the easily missing region in the mask image corresponding to the region to be screened, the easily missing region is cropped out in the mask image corresponding to the region to be screened;

[0033] Determine the edge convex hull and edge contour of the cropped, easily missing areas;

[0034] Calculate the shortest distance from the edge convex hull to the edge profile;

[0035] If the shortest distance is greater than the second threshold, the area to be screened is determined to be a problem map.

[0036] Optionally, the area to be screened includes areas prone to discoloration and areas not prone to discoloration;

[0037] Problem map screening methods also include:

[0038] If the area to be screened is not a problem map, then calculate the color difference span of the non-easily discolored area;

[0039] If the color difference span is less than the third threshold, then calculate the similarity between the color histograms of the easily discolored regions and the non-easily discolored regions.

[0040] If the similarity is less than the fourth threshold, the area to be screened is determined to be a problem map.

[0041] A problem map screening device includes: an image acquisition module, a mask image acquisition module, a map judgment module, an image registration module, and a region screening module.

[0042] The image acquisition module is used to acquire the image to be screened.

[0043] The mask image acquisition module is used to acquire the mask image corresponding to the region to be screened in the image to be screened;

[0044] The map judgment module is used to determine whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image.

[0045] The image registration module is used to register the mask image corresponding to the area to be screened with the preset standard map mask image if the map judgment module determines that the area to be screened is a map, so as to obtain the registered mask image corresponding to the area to be screened.

[0046] The screening module for the area to be screened is used to determine whether the area to be screened is a problematic map based on the registered mask image and the preset standard map mask image corresponding to the area to be screened.

[0047] As can be seen from the above technical solution, the problem map screening method provided in this application first obtains the image to be screened, then obtains the mask image corresponding to the area to be screened in the image to be screened, then determines whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image. If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened, and finally determines whether the area to be screened is a problem map based on the registered mask image corresponding to the area to be screened and the preset standard map mask image. This application first processes the image to be screened into a mask image corresponding to the area to be screened. This mask image is a binary mask image without texture and color information, which enables this application to have a good recognition effect on the image to be screened with different textures and colors. At the same time, this application first determines whether the area to be screened is a map. If so, the mask image corresponding to the area to be screened is registered with a preset standard map mask image. After registration, based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, it is determined whether the area to be screened is a problem map, which improves the screening efficiency. Furthermore, image registration enables this application to have a good recognition effect on the image to be screened with different deformations and angles. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a problem map screening method provided in this application embodiment;

[0050] Figure 2 This is a schematic diagram of the structure of the image segmentation model provided in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of the structure of a problem map screening device provided in an embodiment of this application;

[0052] Figure 4 This is a hardware structure block diagram of a problem map screening device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In view of the problems existing in the prior art, the inventors of this case conducted in-depth research and finally proposed a problem map screening method. The following embodiments will provide a detailed description of the problem map screening method provided in this application.

[0055] Please see Figure 1 The diagram illustrates a flowchart of a problem map screening method provided in an embodiment of this application. This problem map screening method may include:

[0056] Step S101: Obtain the image to be screened.

[0057] In some scenarios, it is necessary to screen images or videos to determine whether the map in the image is a "problem map". In this case, the image to be screened can be obtained through this step. If the video is being screened, each frame of the video can be obtained as the image to be screened. That is, each frame of the video is screened according to the problem map screening method provided in the embodiments of this application.

[0058] Optionally, to facilitate subsequent processing, this step can process the image to be screened to a set size after obtaining it.

[0059] Step S102: Obtain the mask image corresponding to the region to be screened in the image to be screened.

[0060] The image to be screened includes the region to be screened. This step determines the mask image corresponding to the region to be screened. Here, the mask image corresponding to the region to be screened includes the region to be screened and non-regions to be screened. Optionally, the pixel value of the region to be screened is a first grayscale value, and the pixel value of the non-region to be screened is a second grayscale value. Optionally, the first grayscale value is 1, and the second grayscale value is 0.

[0061] Optionally, the process of "obtaining the mask image corresponding to the region to be screened in the image to be screened" includes the following steps:

[0062] Step S1021: Use the pre-trained image segmentation model to segment the image to be screened to obtain the mask image corresponding to the image to be screened. The image segmentation model is trained with sample images as training samples and the mask image corresponding to the sample images as training labels. Each image in the sample images includes a standard map or a problem map. The pixel value of the map region in the mask image is the first gray value, and the pixel value of the non-map region in the mask image is the second gray value.

[0063] Optionally, the image segmentation model described above can be the UNet3+ image segmentation model, which consists of an encoder and a decoder. See [link to relevant documentation]. Figure 2 The diagram shown illustrates the structure of an image segmentation model, including its encoder. and decoder Employ full-scale jump connections; Figure 2 Five encoders are shown, each comprising four downsampling passes, with the resolution of the output feature map reduced by half at each downsampling. Since the feature maps output from the five encoders to the decoder have different resolutions, to ensure that the feature maps output to the same decoder have the same resolution, different encoder layers perform max pooling or bilinear interpolation at different scales to generate feature maps with the same resolution. Here, max pooling is used to reduce the resolution of the feature maps, while bilinear interpolation is used to increase the resolution. This structure can fully utilize low-level details and high-level semantic information while reducing the number of parameters.

[0064] Optionally, the above image segmentation model can be trained with full-scale depth supervision and a hybrid loss function (optionally including focal loss, MS-SSIM loss and IOU loss) can be used to supervise the training of each layer.

[0065] The formula for calculating the ms-ssim loss function is as follows:

[0066]

[0067] In the formula, The value of the ms-ssim loss function. For the number of scales, The mean of the segmentation results (i.e., the mask output by the model). The variance of the segmentation results, The mean of the calibration results (i.e., the manually calibrated mask). The variance of the calibration results, and As weight, and It is a constant decimal (used to prevent the divisor or dividend from being 0). This is the covariance between the segmentation results and the calibration results.

[0068] Optionally, to reduce misidentification caused by oversegmentation (i.e., non-map images being identified as map images), this application can add a classification module after the deepest encoder or decoder to participate in training, so as to determine whether the image input to the model contains a map. If the classification module determines that the image input to the model does not contain a map, the pixels marked as map regions in the segmentation result (i.e. the result obtained based on the encoder and decoder) are cleared, and a mask image of all non-map regions is output.

[0069] Optionally, the process of generating the mask image corresponding to the sample image used during training of the above image segmentation model may include:

[0070] A1. Obtain sample images, where each image in the sample images includes a standard map or a problem map.

[0071] In this step, map materials and standard map materials fed back from the Internet platform can be collected as sample images according to the screening requirements. In other words, this step will obtain many images, each of which must contain a standard map or a "problem map". These many images are the sample images.

[0072] A2. Label the map regions in the sample images.

[0073] In this step, the sample image can be labeled at the pixel level to at least label the map area in the sample image. For example, if the first gray value is 1 and the second gray value is 0, then the pixel value of the map area in the sample image is labeled as 1, and the pixel value of the background area (non-map area) is labeled as 0.

[0074] Optionally, given the complex map edges and high precision requirements, Photoshop annotation can be used in this case.

[0075] A3. Perform settings processing on the marked map area to obtain the processed map area. The settings processing includes, but is not limited to, the following: perspective transformation, random rotation, and random color transformation.

[0076] Here, perspective transformation refers to extracting the labeled map region and then applying a perspective transformation with a random factor of 0.3, enabling the image segmentation model to adapt to maps with certain deformations. Random rotation refers to rotating the labeled map region by a random angle from -90 degrees to 90 degrees. Random color includes random solid color, random inverted color, random gradient color, and random color channel transformation. The purpose of setting the labeled map region in this embodiment is to increase the generalization ability of the image segmentation model under the constraints of data volume and data distribution.

[0077] It should be noted that the above settings are merely examples, and other settings may also be included, which are not limited in this application.

[0078] A4. Obtain the background image, label the pixel values ​​of the background image as the second grayscale value, and apply the processed map area to the background image to obtain the composite image.

[0079] In this step, various background images that do not contain map areas can be collected. After the pixel values ​​of the background images are labeled as the second grayscale values, the processed map areas can be applied to the background images to generate a composite image.

[0080] A5. Perform data augmentation processing on the synthesized image, including but not limited to the following: adding Gaussian noise, randomly adding text occlusion, random translation, and color jitter.

[0081] In this step, data augmentation can be performed on the synthesized image by adding Gaussian noise, randomly adding text occlusion, random translation, color jittering, and other methods.

[0082] It should be noted that the above data augmentation process is only an example. Other data augmentation processes may also be included, and this application does not limit them.

[0083] Optionally, embodiments of this application may further include the following A6:

[0084] A6. Process the composite image and sample image after data augmentation to a set size, and standardize the sample image.

[0085] Optionally, the size can be set to 320*320; the sample image is standardized so that the pixel values ​​are normally distributed near 0, which makes it easier for the image segmentation model to converge.

[0086] The mask images corresponding to the sample images used during training of the image segmentation model can be obtained through A1~A5 or A1~A6 above. This embodiment employs mapping and data augmentation methods to expand the amount of output sample data, and these methods enable the image segmentation model to adapt to different backgrounds, enhancing its generalization ability.

[0087] It should be noted that this step can also use other methods besides image segmentation models to obtain the mask image corresponding to the image to be screened, such as using direct annotation to obtain the mask image corresponding to the image to be screened, etc. This application does not limit this.

[0088] Step S1022: Perform morphological processing on the mask image corresponding to the image to be screened to obtain the processed mask image.

[0089] In one alternative embodiment, the morphological processing procedure may include:

[0090] B1. The first operation core is used to perform a closing operation on the mask image corresponding to the image to be screened, so as to obtain the mask image after the closing operation.

[0091] In this step, the first operation core can be used to perform a closing operation on the mask image corresponding to the image to be screened, so as to fill the holes and discontinuous connections in the mask image corresponding to the image to be screened.

[0092] B2. The mask image after the closing operation is processed by the second operation core and erosion is performed to obtain the eroded mask image. The first operation core is smaller than the second operation core.

[0093] In this step, a second processing core larger than the first processing core can be used to perform erosion processing on the mask image after the closing operation, so as to break the weak connection mask caused by background misidentification at the map edge.

[0094] B3. Based on the largest connected component in the eroded mask image, remove the non-map regions from the eroded mask image, and then dilate the mask image after removing the non-map regions to obtain the dilated mask image, which is used as the processed mask image.

[0095] Here, non-map areas refer to regions where pixel values ​​should be labeled as the second grayscale value but are instead labeled as the first grayscale value. After removing non-map areas, the processed mask image is obtained.

[0096] This step recovers the erosion of B2 by dilating the mask image after removing the non-mapped areas.

[0097] The morphological processing in this step can minimize misjudgments caused by errors in the segmentation process.

[0098] Step S1023: Based on the processed mask image, crop out the mask image corresponding to the region to be screened from the mask image corresponding to the image to be screened.

[0099] In one optional embodiment, the cropping process may include: determining the outermost edge coordinates of the region to be screened in the mask image corresponding to the image to be screened based on the processed mask image, and expanding and cropping the region to be screened in the mask image corresponding to the image to be screened according to a set percentage of the outermost edge coordinates.

[0100] Optionally, in this step, the coordinates of the four points (top, bottom, left, and right) of the outermost edge of the region to be screened in the mask image corresponding to the image to be screened can be calculated based on the processed mask image. The region to be screened in the mask image corresponding to the image to be screened can be cropped after being expanded by 20% according to the obtained coordinate values, so as to obtain the mask image corresponding to the region to be screened.

[0101] This step, which involves morphological processing of the mask image to be screened before cropping, makes the cropped screening area more accurate.

[0102] Step S103: Determine whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image.

[0103] Here, the preset standard map mask image includes a standard map area and a non-standard map area. The pixel value of the standard map area is the first gray value, and the pixel value of the non-standard map area is the second gray value.

[0104] There may be cases where the image to be screened does not include a map. In this case, if the screening area is determined based on the mask image corresponding to the area to be screened, the screening efficiency will be reduced. Therefore, this step can be used to detect whether the area to be screened is a map first. If the area to be screened is detected as a map, then the subsequent steps can be performed.

[0105] Optionally, the process of determining whether the area to be screened is a map in this step includes: determining the intersection-union ratio (IU) of the mask image corresponding to the area to be screened and the preset standard map mask image; if the IU is greater than a first threshold, the area to be screened is determined to be a map; if the IU is less than or equal to the first threshold, the area to be screened is determined not to be a map.

[0106] Step S104: If the area to be screened is a map, then the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened.

[0107] In this step, a preset standard map mask image can be used as a reference to register the mask image corresponding to the area to be screened, so as to correct the deformation, angle, etc. of the mask image corresponding to the area to be screened. Through this step, the registered mask image and the preset standard map mask image are basically aligned.

[0108] In an optional embodiment, the mask image corresponding to the area to be screened can be registered with a preset standard map mask image through a pre-trained image registration model. That is, in this step, the mask image corresponding to the area to be screened can be used as a floating image, and the preset standard map mask image can be used as a fixed image (i.e., a reference image). The floating image and the fixed image are input into the pre-trained image registration model to obtain the registered mask image corresponding to the fixed image, that is, to obtain the registered mask image corresponding to the area to be screened. Here, the image registration model is trained using the preset standard map mask image and the sample mask image as training samples, and the registered image corresponding to the sample mask image as the sample label.

[0109] Optionally, the image registration model mentioned above includes an encoder-decoder network (similar to UNet) and a spatial transformation network (STN). Based on these two networks, the working principle of the image registration model is as follows: a preset standard map mask image is used as a fixed image, and a mask image (in practical applications, this mask image is the mask image corresponding to the screening area; in the model training process, this mask image is the sample mask image) is used as a moving image. After the fixed image and the moving image are concatted, they can be input into the encoder-decoder network to obtain the registration field from the moving image to the fixed image. This registration field is then sampled and input into the spatial transformation network to perform deformation operations on the moving image, thereby obtaining the registration image corresponding to the mask image.

[0110] Step S105: Determine whether the area to be screened is a problematic map based on the registered mask image and the preset standard map mask image corresponding to the area to be screened.

[0111] This step determines whether the area to be screened is complete based on the registered mask image and the preset standard map mask image. If it is incomplete, the area to be screened is determined to be a "problem map"; if it is complete, the area to be screened is determined not to be a "problem map".

[0112] Optionally, this step can subtract the target registration image from the registered mask image corresponding to the region to be screened, and perform relevant morphological processing to determine whether the region to be screened is a "problem map". Specifically, this step can calculate the difference between the preset standard map mask image and the registered mask image to obtain a difference image. This difference image represents the missing region of the region to be screened. Due to certain errors in image segmentation in step S1021 and image registration in step S104, the map edges in the obtained difference image may have small connected set influences. To remove small connected sets, an opening operation can be performed on the difference image to obtain an image after the opening operation. Based on the image after the opening operation, it can be determined whether the region to be screened is a problem map. If the image after the opening operation includes the region marked as a map, it indicates that the region to be screened has a missing region, that is, the region to be screened is incomplete. In this case, the region to be screened can be determined to be a problem map. Conversely, if the image after the opening operation does not include the region marked as a map, it indicates that the region to be screened does not have a missing region, that is, the region to be screened is complete. In this case, the region to be screened can be determined to be a standard map.

[0113] Optionally, if the image after the opening operation includes regions marked as maps, this step can further determine which region is specifically marked as a map. That is, this step can determine the missing regions in the image to be screened based on the image after the opening operation.

[0114] The problem map screening method provided in this application first obtains the image to be screened, then obtains the mask image corresponding to the area to be screened in the image to be screened, then determines whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image. If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened. Finally, the registered mask image corresponding to the area to be screened and the preset standard map mask image determine whether the area to be screened is a problem map. This application first processes the image to be screened into a mask image corresponding to the area to be screened. This mask image is a binary mask image without texture and color information, enabling this application to have good recognition effect on the image to be screened with different textures and colors. At the same time, this application first determines whether the area to be screened is a map. If so, the mask image corresponding to the area to be screened is registered with a preset standard map mask image. After registration, based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, it is determined whether the area to be screened is a problem map, which improves the screening efficiency. Furthermore, image registration enables this application to have good recognition effect on the image to be screened with different deformations and angles. That is, this application can effectively identify "problem maps" in complex and diverse scenes.

[0115] Furthermore, the deep learning-based image segmentation and image registration method provided in this application embodiment can quickly, accurately, and robustly identify "problem maps" in the images to be screened, effectively saving manpower costs and reducing the risks brought by "problem maps".

[0116] In an optional embodiment, considering that some maps may contain small, easily missing areas that are not obvious in the registered mask image, and that step S105 determines whether the area to be screened is a problem map based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, this may lead to a situation where step S105 determines whether the area to be screened is a "problem map" even if the area to be screened does not contain easily missing areas (i.e., the area to be screened is a "problem map"), but step S105 still determines that the area to be screened is not a "problem map". In other words, the above step S105 may make a misjudgment when determining whether the area to be screened is a "problem map". To improve the accuracy of screening maps containing easily missing areas, this embodiment is proposed. When it is determined in step S105 that the area to be screened is not a "problem map", the defect detection method provided in this embodiment is used to further determine whether there are any missing easily missing areas in the area to be screened. If step S105 determines that the area to be screened is not a "problem map", and the defect detection method provided in this embodiment still determines that the area to be screened is not a "problem map", then it is determined that the area to be screened is indeed not a "problem map". Otherwise, the area to be screened is determined to be a "problem map".

[0117] Based on this, in this embodiment, if step S105 determines that the area to be screened is not a "problem map", the following steps can be further performed:

[0118] Step S106: Inversely register the field from the preset standard map mask image to determine the position of the easily missing region in the mask image corresponding to the region to be screened, based on the position of the easily missing region in the preset standard map mask image.

[0119] The registration field represents the spatial coordinate transformation parameters from the mask image corresponding to the area to be screened to the preset standard map mask image.

[0120] It is understandable that the location of the easily missing region in the preset standard map mask image is usually known. In this case, by inversely registering the field from the preset standard map mask image, the location of the easily missing region in the mask image corresponding to the region to be screened can be determined based on the location of the easily missing region in the preset standard map mask image.

[0121] Here, the registration field in this step is the registration field obtained in the above image registration model. Based on this, in an optional embodiment, this step can also be implemented by the above image registration model. That is, in addition to outputting the registered image, the above image registration model can also output the position of the easily missing region in the mask image corresponding to the region to be screened.

[0122] Step S107: Based on the position of the easily missing region in the mask image corresponding to the region to be screened, crop out the easily missing region in the mask image corresponding to the region to be screened.

[0123] Step S108: Determine the edge convex hull and edge contour of the easily missing area after cropping.

[0124] Step S109: Calculate the shortest distance from the edge convex hull to the edge profile.

[0125] The previous step may have identified multiple edge convex hulls. This step can calculate the distance from each edge convex hull to the edge contour to obtain the shortest distance.

[0126] Step S110: If the shortest distance is greater than the second threshold, then the area to be screened is determined to be a problem map.

[0127] If the shortest distance is greater than the second threshold, it can be determined that there is a missing easily missing region in the area to be screened, and the area to be screened is determined to be a "problem image". Otherwise, it is determined that there is no missing easily missing region in the area to be screened, that is, the area to be screened is indeed not a "problem image".

[0128] This application embodiment utilizes concavity and convexity detection to detect areas prone to loss, thereby improving screening accuracy.

[0129] In an optional embodiment, considering that some maps may contain areas prone to color distortion, in a color map, these areas are easily distinguishable from non-color-distorting areas by a significant color difference. Since step S105 is based on a mask image for map screening, and the mask image is a binary grayscale image that does not consider color issues, and the defect detection method provided in the above embodiments of this application also does not consider color issues when determining whether there are easily missing areas in the area to be screened, even if there are color-distorting areas (i.e., the area to be screened is a "problematic map"), steps S105 and S110 will still not determine the area to be screened as a "problematic map." That is, steps S105 and S110 may misjudge whether the area to be screened is a "problematic map." To improve the accuracy of color map screening in this application, this embodiment is proposed. When it is determined in step S105 (or step S110) that the area to be screened is not a "problem map", the color histogram matching method provided in this embodiment is used to further determine whether there is a problem of color difference in easily differentiated areas in the area to be screened. If step S105 (or step S110) determines that the area to be screened is not a "problem map", and the color histogram matching method provided in this embodiment still determines that the area to be screened is not a "problem map", then it is determined that the area to be screened is indeed not a "problem map"; otherwise, it is determined that the area to be screened is a "problem map".

[0130] Based on this, in this embodiment, if step S105 (or step S110) determines that the area to be screened is not a "problem map", the following steps can be further performed:

[0131] Step S111: Calculate the color difference span of the non-easily discolored area.

[0132] Understandably, in some cases, the color difference span between sub-regions within the area to be screened is large, and the focus is not on easily discolored areas. In such cases, determining whether the area to be screened is a "problem map" based on the color difference between easily discolored and non-easily discolored areas is meaningless. Therefore, this embodiment can first calculate the color difference span of non-easily discolored areas through this step.

[0133] Step S112: If the color difference span is less than the third threshold, calculate the similarity between the color histograms of the easily differentiated color regions and the non-easily differentiated color regions.

[0134] If the color difference span of the non-easily color-differentiated area is less than the third threshold, it means that the current area to be screened has not been specifically distinguished from a certain area. In this case, if the color difference between the easily color-differentiated area and the non-easily color-differentiated area is large, the easily color-differentiated area is in violation.

[0135] In this step, the color difference between the easily differentiated and non-easily differentiated regions can be determined by comparing the similarity of their color histograms.

[0136] Step S113: If the similarity is less than the fourth threshold, then the area to be screened is determined to be a problem map.

[0137] In this step, if the similarity between the color histograms of easily discolored regions and non-easily discolored regions is less than the fourth threshold, it indicates that there is a color violation in the easily discolored regions, and the region to be screened is determined to be a problem map.

[0138] This application utilizes color histogram matching to detect easily discolored regions, thereby improving screening accuracy. Furthermore, this embodiment demonstrates good recognition performance for images to be screened that exhibit different deformations, textures, colors, and subtle differences.

[0139] This application also provides a problem map screening device. The problem map screening device provided in this application is described below. The problem map screening device described below can be referred to in correspondence with the problem map screening method described above.

[0140] Please see Figure 3 The diagram shows a structural schematic of the problem map screening device provided in an embodiment of this application, such as... Figure 3 As shown, the problem map screening device may include: a module 601 for acquiring the image to be screened, a module 602 for acquiring the masked image, a module 603 for judging the map, a module 604 for image registration, and a module 605 for screening the area to be screened.

[0141] The image acquisition module 601 is used to acquire the image to be screened.

[0142] The mask image acquisition module 602 is used to acquire the mask image corresponding to the area to be screened in the image to be screened.

[0143] The map judgment module 603 is used to determine whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image.

[0144] The image registration module 604 is used to register the mask image corresponding to the area to be screened with the preset standard map mask image if the map judgment module determines that the area to be screened is a map, so as to obtain the registered mask image corresponding to the area to be screened.

[0145] The screening module 605 is used to determine whether the area to be screened is a problematic map based on the registered mask image and the preset standard map mask image corresponding to the area to be screened.

[0146] The problematic map screening device provided in this application first acquires an image to be screened, then acquires a mask image corresponding to the area to be screened in the image to be screened, then determines whether the area to be screened is a map based on the mask image corresponding to the area to be screened and a preset standard map mask image. If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain a registered mask image corresponding to the area to be screened. Finally, the registered mask image corresponding to the area to be screened and the preset standard map mask image determine whether the area to be screened is a problematic map. This application first processes the image to be screened into a mask image corresponding to the area to be screened. This mask image is a binary mask image without texture and color information, which enables this application to have a good recognition effect on the image to be screened with different textures and colors. At the same time, this application first determines whether the area to be screened is a map. If so, the mask image corresponding to the area to be screened is registered with a preset standard map mask image. After registration, based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, it is determined whether the area to be screened is a problem map, which improves the screening efficiency. Furthermore, image registration enables this application to have a good recognition effect on the image to be screened with different deformations and angles.

[0147] In one possible implementation, the mask image acquisition module 602 may include an image segmentation module, a morphological processing module, and an image cropping module.

[0148] The image segmentation module is used to segment the image to be screened using a pre-trained image segmentation model to obtain a mask image corresponding to the image to be screened. The image segmentation model is trained using sample images as training samples and the mask images corresponding to the sample images as training labels. Each image in the sample images includes a standard map or a problem map. The pixel values ​​of the map regions in the mask image are first gray values, and the pixel values ​​of the non-map regions in the mask image are second gray values.

[0149] The morphological processing module is used to perform morphological processing on the mask image corresponding to the image to be screened, so as to obtain the processed mask image.

[0150] The image cropping module is used to crop out the mask image corresponding to the region to be screened from the mask image corresponding to the image to be screened, based on the processed mask image.

[0151] In one possible implementation, the morphological processing module may include: a closing operation processing module, an erosion processing module, and a non-map region removal and dilation processing module.

[0152] The closing operation processing module is used to perform closing operation processing on the mask image corresponding to the image to be screened using the first operation core, so as to obtain the mask image after closing operation processing.

[0153] The erosion processing module is used to perform erosion processing on the mask image after the closing operation using the second operation core to obtain the eroded mask image, wherein the first operation core is smaller than the second operation core.

[0154] The non-map region removal and dilation module is used to remove non-map regions from the eroded mask image based on the largest connected component in the eroded mask image, and to dilate the mask image after removing non-map regions to obtain the dilated mask image as the processed mask image.

[0155] In one possible implementation, the image cropping module can be used to determine the outermost edge coordinates of the region to be screened in the mask image corresponding to the image to be screened based on the processed mask image, and to expand and crop the region to be screened in the mask image corresponding to the image to be screened according to a set percentage of the outermost edge coordinates, so as to obtain the mask image corresponding to the region to be screened.

[0156] In one possible implementation, the map determination module 603 may include an intersection-union ratio determination module and an intersection-union ratio comparison module.

[0157] The intersection-over-union (IoU) ratio determination module is used to determine the IoU ratio between the mask image corresponding to the area to be screened and the preset standard map mask image.

[0158] The intersection-union ratio (IUU) comparison module is used to determine the region to be filtered out as a map if the IUU is greater than a first threshold.

[0159] In one possible implementation, the image registration module 604 can be specifically used to input the mask image corresponding to the area to be screened and the preset standard map mask image into a pre-trained image registration model to obtain the registered mask image corresponding to the area to be screened. The image registration model is trained using the preset standard map mask image and the sample mask image as training samples, and the registered image corresponding to the sample mask image as the sample label.

[0160] In one possible implementation, the screening module 605 for the area to be screened may include: a difference image calculation module, an opening operation processing module, and a problem map judgment module.

[0161] The difference image calculation module is used to calculate the difference between the preset standard map mask image and the registered mask image to obtain the difference image.

[0162] The opening operation processing module is used to perform opening operations on the difference image to obtain the image after the opening operation.

[0163] The problem map judgment module is used to determine whether the area to be screened is a problem map based on the image after the opening operation.

[0164] In one possible implementation, the area to be screened includes regions prone to missing data. Based on this, the problem map screening device provided in this application may further include: a region prone to missing data location determination module, a region prone to missing data clipping module, a convex hull and contour determination module, a shortest distance calculation module, and a first problem map determination module.

[0165] The module for determining the location of easily missing regions is used to reverse-register the field from a preset standard map mask image if the area to be screened is not a problem map. This is done to determine the location of the easily missing regions in the mask image corresponding to the area to be screened based on the location of the easily missing regions in the preset standard map mask image. The registration field represents the spatial coordinate transformation parameters from the mask image corresponding to the area to be screened to the preset standard map mask image.

[0166] The vulnerable region cropping module is used to crop out vulnerable regions from the mask image corresponding to the region to be screened, based on the position of the vulnerable regions in the mask image corresponding to the region to be screened.

[0167] The convex hull and contour determination module is used to determine the edge convex hull and edge contour of the cropped, easily missing regions.

[0168] The shortest distance calculation module is used to calculate the shortest distance from the edge convex hull to the edge contour.

[0169] The first problem map determination module is used to determine the area to be screened as a problem map if the shortest distance is greater than the second threshold.

[0170] In one possible implementation, the area to be screened includes areas prone to color shifting and areas not prone to color shifting. Based on this, the problem map screening device provided in this application may further include: a color difference span calculation module, a similarity calculation module, and a second problem map determination module.

[0171] Among them, the color difference span calculation module is used to calculate the color difference span of non-easily discolored areas if the area to be screened is not a problem map;

[0172] The similarity calculation module is used to calculate the similarity between the color histograms of easily dissimilar regions and non-easily dissimilar regions if the color difference span is less than the third threshold.

[0173] The second problem map determination module is used to determine the area to be screened as a problem map if the similarity is less than the fourth threshold.

[0174] This application also provides a problem map screening device. Optionally, Figure 4 The hardware structure block diagram of the problem map screening device is shown, with reference to... Figure 4The hardware structure of the problem map screening device may include: at least one processor 701, at least one communication interface 702, at least one memory 703 and at least one communication bus 704;

[0175] In this embodiment of the application, the number of processor 701, communication interface 702, memory 703 and communication bus 704 is at least one, and processor 701, communication interface 702 and memory 703 communicate with each other through communication bus 704.

[0176] The processor 701 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0177] The memory 703 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0178] The memory 703 stores a program, and the processor 701 can call the program stored in the memory 703. The program is used for:

[0179] Acquire the image to be screened;

[0180] Obtain the mask image corresponding to the region to be screened in the image to be screened;

[0181] Based on the mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a map;

[0182] If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened.

[0183] Based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a problematic map.

[0184] Optionally, the refined and extended functions of the program can be found in the description above.

[0185] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the problem map screening method described above.

[0186] Optionally, the refined and extended functions of the program can be found in the description above.

[0187] Finally, it should be noted that in this document, relational terms such as "second" and "etc." are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0188] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0189] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for screening problem maps, characterized in that, include: Acquire the image to be screened; Obtain the mask image corresponding to the region to be screened in the image to be screened; Based on the mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a map; If so, the mask image corresponding to the area to be screened is registered with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened. Based on the registered mask image corresponding to the area to be screened and the preset standard map mask image, determine whether the area to be screened is a problematic map; The step of obtaining the mask image corresponding to the region to be screened in the image to be screened includes: The image to be screened is segmented using a pre-trained image segmentation model to obtain a mask image corresponding to the image to be screened. The mask image corresponding to the image to be screened is subjected to morphological processing to obtain the processed mask image; Based on the processed mask image, the mask image corresponding to the region to be screened is cropped from the mask image corresponding to the image to be screened; The step of performing morphological processing on the mask image corresponding to the image to be screened to obtain the processed mask image includes: The first processing core is used to perform a closing operation on the mask image corresponding to the image to be screened, so as to obtain the mask image after the closing operation. The mask image after the closing operation is eroded using a second operation core to obtain an eroded mask image, wherein the first operation core is smaller than the second operation core; Based on the largest connected component in the eroded mask image, the non-map region in the eroded mask image is removed, and the mask image after removing the non-map region is dilated to obtain the dilated mask image, which is used as the processed mask image. The step of registering the mask image corresponding to the area to be screened with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened includes: The mask image corresponding to the area to be screened and the preset standard map mask image are input into a pre-trained image registration model to obtain a registered mask image corresponding to the area to be screened. The image registration model is trained using the preset standard map mask image and the sample mask image as training samples and the registered image corresponding to the sample mask image as the sample label.

2. The problem map screening method according to claim 1, characterized in that, The image segmentation model is trained using sample images as training samples and mask images corresponding to the sample images as training labels. Each image in the sample images includes a standard map or a problem map. The pixel values ​​of the map regions in the mask image are first grayscale values, and the pixel values ​​of the non-map regions in the mask image are second grayscale values.

3. The problem map screening method according to claim 2, characterized in that, The step of cropping the mask image corresponding to the region to be screened from the mask image corresponding to the image to be screened based on the processed mask image includes: Based on the processed mask image, the coordinates of the outermost edge of the region to be screened in the mask image corresponding to the image to be screened are determined. The region to be screened in the mask image corresponding to the image to be screened is expanded and cropped according to a set percentage of the outermost edge coordinates to obtain the mask image corresponding to the region to be screened.

4. The problem map screening method according to claim 1, characterized in that, The step of determining whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image includes: Determine the intersection-over-union ratio (IoU) of the mask image corresponding to the area to be screened and the preset standard map mask image; If the intersection-union ratio is greater than the first threshold, then the area to be screened is determined to be a map.

5. The problem map screening method according to claim 1, characterized in that, The step of determining whether the area to be screened is a problematic map based on the registered mask image corresponding to the area to be screened and the preset standard map mask image includes: Calculate the difference between the preset standard map mask image and the registered mask image to obtain a difference image; The difference image is subjected to an opening operation to obtain the image after the opening operation; Based on the image after the opening operation, determine whether the area to be screened is a problem map.

6. The problem map screening method according to claim 5, characterized in that, The areas to be screened include areas prone to missing data; The problem map screening method also includes: If the area to be screened is not a problem map, then the registration field is reversed from the preset standard map mask image to determine the position of the easily missing area in the mask image corresponding to the area to be screened based on the position of the easily missing area in the preset standard map mask image. The registration field represents the spatial coordinate transformation parameters from the mask image corresponding to the area to be screened to the preset standard map mask image. Based on the position of the easily missing region in the mask image corresponding to the region to be screened, the easily missing region is cropped out from the mask image corresponding to the region to be screened; Determine the edge convex hull and edge contour of the cropped, easily missing region; Calculate the shortest distance from the edge convex hull to the edge profile; If the shortest distance is greater than the second threshold, then the area to be screened is determined to be a problem map.

7. The problem map screening method according to claim 5 or 6, characterized in that, The areas to be screened include areas prone to discoloration and areas not prone to discoloration. The problem map screening method also includes: If the area to be screened is not a problem map, then calculate the color difference span of the non-easily discolored area; If the color difference span is less than the third threshold, then the similarity of the color histograms of the easily discolored region and the non-easily discolored region is calculated. If the similarity is less than the fourth threshold, then the area to be screened is determined to be a problem map.

8. A problem map screening device, characterized in that, include: The system includes a module for acquiring images to be screened, a module for acquiring masked images, a module for determining maps, a module for image registration, and a module for screening areas to be screened. The image acquisition module is used to acquire the image to be screened; The mask image acquisition module is used to acquire the mask image corresponding to the region to be screened in the image to be screened; The map judgment module is used to determine whether the area to be screened is a map based on the mask image corresponding to the area to be screened and the preset standard map mask image. The image registration module is used to register the mask image corresponding to the area to be screened with the preset standard map mask image if the map judgment module determines that the area to be screened is a map, so as to obtain the registered mask image corresponding to the area to be screened. The screening module for the area to be screened is used to determine whether the area to be screened is a problematic map based on the registered mask image corresponding to the area to be screened and the preset standard map mask image. The step of obtaining the mask image corresponding to the region to be screened in the image to be screened includes: The image to be screened is segmented using a pre-trained image segmentation model to obtain a mask image corresponding to the image to be screened. The mask image corresponding to the image to be screened is subjected to morphological processing to obtain the processed mask image; Based on the processed mask image, the mask image corresponding to the region to be screened is cropped from the mask image corresponding to the image to be screened; The step of performing morphological processing on the mask image corresponding to the image to be screened to obtain the processed mask image includes: The first processing core is used to perform a closing operation on the mask image corresponding to the image to be screened, so as to obtain the mask image after the closing operation. The mask image after the closing operation is eroded using a second operation core to obtain an eroded mask image, wherein the first operation core is smaller than the second operation core; Based on the largest connected component in the eroded mask image, the non-map region in the eroded mask image is removed, and the mask image after removing the non-map region is dilated to obtain the dilated mask image, which is used as the processed mask image. The step of registering the mask image corresponding to the area to be screened with the preset standard map mask image to obtain the registered mask image corresponding to the area to be screened includes: The mask image corresponding to the area to be screened and the preset standard map mask image are input into a pre-trained image registration model to obtain a registered mask image corresponding to the area to be screened. The image registration model is trained using the preset standard map mask image and the sample mask image as training samples and the registered image corresponding to the sample mask image as the sample label.

Images