A deep learning-based picture superposition element completeness judgment method

Abstract

This application provides a deep learning-based method for judging the completeness of overlay elements in images. It extracts key information from the uploaded image and classifies the image based on this information. According to the classification results, it first performs targeted judgments on traffic light images, single-point speed measurement images, or section speed measurement images, excluding problematic images and improving detection efficiency. Then, it uses a watermark information completeness detection model to judge the completeness and correctness of the watermark in the image. The entire process requires no manual intervention, greatly reducing the workload of review staff.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method for judging the completeness of overlay elements in images based on deep learning. Background Technology

[0002] According to the requirements of the evidence collection standard GA / T 832, the information superimposed on the evidence images should at least include: the time of occurrence, the location of occurrence, the behavior code, the behavior description, the image evidence collection equipment number, and anti-counterfeiting information; for motor vehicles violating speed limits, each evidence image should also be superimposed with the speed limit value and the driving speed value; for speeding evidence images captured by section speed measurement, at least the name of the speed measurement section, the distance, the entry time, the exit time, and the average speed should also be included.

[0003] However, in practical applications, it has been found that some devices do not fully comply with the technical specifications when capturing evidence images. Specifically, for example... Figure 1 As shown in the examples in the images, this is reflected in the following aspects:

[0004] (1) Missing or incomplete information. Some evidence images may not have all the necessary information superimposed, such as the time of occurrence, location of occurrence, behavior code, behavior description, image evidence collection device number, or anti-counterfeiting information.

[0005] (2) The speed measurement information is incomplete or not superimposed, such as the speed limit or driving speed.

[0006] (3) Insufficient accuracy of time information. For abnormal behaviors that occur during the driving process of a motor vehicle, the time of occurrence of the driving behavior superimposed on the image may not meet the accuracy requirement of 0.01s.

[0007] (4) Improper placement of superimposed information. Superimposed information may be located in key areas of the image, obscuring specific behavioral evidence, such as vehicle license plates, driver facial features, or traffic lights, thus reducing the evidentiary value of the image.

[0008] If the evidence images are not properly formatted, they cannot be used as evidence in subsequent procedures. Therefore, the current method involves manual review of the evidence images by auditors. This method consumes a significant amount of human resources. Summary of the Invention

[0009] To address the problem that existing technologies cannot automatically identify the completeness and correctness of watermark overlay information on evidence images, this invention provides a deep learning-based method for judging the completeness of overlay elements in images. This method automatically judges the completeness and correctness of overlay fields on evidence images, reducing instances of non-standard evidence collection and alleviating the workload of reviewers.

[0010] The technical solution of this invention is as follows: a method for judging the completeness of image overlay elements based on deep learning, characterized in that it includes the following steps:

[0011] S1: Obtain the image to be judged;

[0012] Locate the monitoring device that captured the image to be judged, and denote it as: the monitoring device used for judgment;

[0013] Acquire all recorded data of abnormal driving behaviors uploaded by the monitoring device used for judgment, denoted as: judgment record data;

[0014] S2: Based on the monitoring equipment used for judgment and the recorded data used for judgment, extract key information related to the image to be judged;

[0015] The key information includes: device number, device type, shooting time, vehicle behavior description, behavior code, and vehicle license plate number;

[0016] S3: Based on the device type and behavior code of the monitoring device used for judgment in the key information, classify the image to be judged by image type;

[0017] The image types correspond one-to-one with the evidence collection types for vehicle driving behavior, specifically including: illegal parking, traffic light, non-traffic light and non-speed measurement, single-point speed measurement, and section speed measurement.

[0018] S4: Read the image type corresponding to the image to be judged;

[0019] If the image type is traffic light type, then execute S6;

[0020] If the image type is a single-point speed measurement or an area speed measurement, then proceed to step S7;

[0021] If the image type is illegal and belongs to any other type besides those mentioned above, proceed to step S8;

[0022] S5: Construct a traffic light watermark occlusion judgment model based on a deep learning algorithm model for object detection; construct a speed measurement violation classification model based on a binary classification algorithm model in deep learning; construct a watermark information completeness detection model based on a deep learning algorithm model;

[0023] S6: Use the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light;

[0024] If no occlusion is formed, proceed to step S8;

[0025] Otherwise, if the image is determined to be obscuring the traffic light due to the superimposed watermark, the subsequent detection of the image will be stopped, and the image will be directly prompted that it has failed the detection.

[0026] S7: Use the trained speed violation classification model to classify and filter single-point speed measurement images or section speed measurement images, and record the classification results output by the model as: identification type;

[0027] Read the behavior code from the key information of the image to be judged to obtain the filing type of the image to be judged;

[0028] Confirm whether the identification type and the filing type of the monitoring device for the uploaded image are consistent. If they are consistent, proceed to step S8.

[0029] Otherwise, it indicates that the filing type of the image to be judged does not match its actual status, and the subsequent detection of the image to be judged will be stopped, and the image will be directly prompted that it has failed the detection;

[0030] S8: The image to be judged is fed into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs the judgment result.

[0031] The judgment results include: whether it is complete, whether it is blurry, and whether it is obscured.

[0032] Its further features are:

[0033] The traffic light watermark occlusion detection model is built based on the YOLOv8 deep learning model.

[0034] The watermark information completeness detection model is constructed based on a convolutional neural network model;

[0035] Step S7 includes the following operations:

[0036] a1: Find splicing features in the image to be judged;

[0037] The splicing features include: feature points, edges, and textures;

[0038] a2: Based on the splicing features, find the boundaries of the spliced ​​images, split the image to be judged, and obtain two split images, denoted as: input image;

[0039] a3: Simultaneously feed the two input images corresponding to the same image to be judged into the trained speed measurement violation classification model;

[0040] a4: The speed violation classification model identifies the background differences between two input images and completes the classification and filtering of the images to be judged;

[0041] When the backgrounds of the two input images are identical, the image type of the image to be judged is single-point speed measurement.

[0042] When the backgrounds of the two input images are different, the image type of the image to be judged is an interval speed measurement image;

[0043] Step S8 specifically includes the following operations:

[0044] b1: Locate the watermark in the image to be judged, extract all text information in the watermark, and record it as: watermark data to be judged;

[0045] b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and get the filing type of the image to be judged;

[0046] b3: Find the preset standard watermark information data based on the filing type of the image to be judged, denoted as: comparison watermark data;

[0047] b4: The watermark data to be judged and the watermark data to be compared are jointly sent into the watermark information completeness detection model for comparison;

[0048] b5: The watermark information completeness detection model outputs the final comparison result.

[0049] This application provides a deep learning-based method for judging the completeness of overlay elements in images. It extracts key information from the uploaded image and classifies the image based on this information. According to the classification results, it first performs targeted judgments on traffic light images, single-point speed measurement images, or section speed measurement images, excluding problematic images and improving detection efficiency. Then, it uses a watermark information completeness detection model to judge the completeness and correctness of the watermark in the image. The entire process requires no manual intervention, greatly reducing the workload of review staff. Attached Figure Description

[0050] Figure 1 Example of an image used for evidence collection with incomplete overlay information;

[0051] Figure 2 This is a flowchart illustrating the image overlay feature completeness judgment method based on deep learning proposed in this application.

[0052] Figure 3 Examples of images used for evidence collection related to section speed measurement. Detailed Implementation

[0053] like Figure 2 As shown, this application includes a method for judging the completeness of image overlay elements based on deep learning, which includes the following steps.

[0054] S1: Obtain the image to be judged;

[0055] Locate the monitoring device that captured the image to be judged, and denote it as: the monitoring device used for judgment;

[0056] Acquire all recorded data of abnormal driving behaviors uploaded by the monitoring equipment used for judgment, denoted as: judgment record data;

[0057] S2: Based on the monitoring equipment and recorded data used for judgment, extract key information related to the image to be judged;

[0058] Key information includes: device number, device type, shooting time, vehicle behavior description, behavior code, and vehicle license plate number.

[0059] S3: Based on the device type and behavior code of the monitoring equipment used for judgment in the key information, classify the image to be judged by image type;

[0060] The types of images correspond one-to-one with the types of evidence collected for vehicle driving behavior, specifically including: illegal parking, traffic light violations, non-traffic light and non-speed measurement violations, single-point speed measurement violations, and section speed measurement violations.

[0061] Traffic light images typically record driving behaviors such as running red lights and failing to follow lane markings. Non-traffic light and non-speed camera images record behaviors not captured by red light or speed camera devices, such as: violating traffic restrictions, crossing lane lines, using a mobile phone while driving, and failing to yield to pedestrians.

[0062] Single-point speed measurement images capture speeding behavior at a single point, while section speed measurement images record speeding behavior captured by section speed measurement devices. The difference lies in the monitoring equipment: single-point speed measurement uses a single device, while section speed measurement uses two devices spaced a distance apart. Therefore, the background locations of images captured by section speed measurement devices are different. In contrast, two images captured by single-point speed measurement devices at very short intervals share the same background location.

[0063] The image to be judged as evidence is an image composed of two images stitched together, such as... Figure 3 The image shown is an example of an evidence image for section speed measurement, composed of images taken by two separate monitoring devices stitched together to form a single evidence image. In practical applications, a common error in section speed measurement and single-point speed measurement evidence images is that the monitoring devices incorrectly label the image type when uploading the evidence image to the traffic management platform.

[0064] A common mistake with evidence images of traffic lights is that when the surveillance equipment overlays watermarks, it does so by superimposing the watermark onto the area of ​​the traffic light, making it impossible to clearly see the status of the traffic light later.

[0065] Therefore, in this method, before judging the integrity and correctness of the watermark, the unique watermark issues of images such as traffic lights, single-point speed measurement images, and section speed measurement images are first checked. After confirming that there are no problems, the integrity and correctness of the watermark are then judged. This can reduce the amount of subsequent calculations and also ensure that all problems of the images to be judged are found, further reducing the workload of the reviewers.

[0066] S4: Read the image type corresponding to the image to be judged;

[0067] If the image type is traffic light type, then execute S6;

[0068] If the image type is a single-point speed measurement or an area speed measurement, then proceed to step S7;

[0069] If the image type is illegal and belongs to any other type besides those mentioned above, proceed to step S8.

[0070] S5: A traffic light watermark occlusion detection model is constructed based on a deep learning algorithm model for object detection; a speed measurement violation classification model is constructed based on a binary classification algorithm model in deep learning; and a watermark information completeness detection model is constructed based on a deep learning algorithm model. The specific model construction and training methods are implemented based on existing technologies. In this embodiment, the traffic light watermark occlusion detection model is constructed based on a YOLOv8 deep learning model; the watermark information completeness detection model is constructed based on a convolutional neural network model.

[0071] S6: Use the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light;

[0072] If no occlusion is formed, proceed to step S8;

[0073] Otherwise, if the image is determined to be obscuring the traffic light due to the superimposed watermark, further detection of the image will be stopped, and the image will be directly indicated as failing the detection.

[0074] In this method, the first step is to detect whether the superimposed watermark of the image to be judged obstructs the traffic light. If it does not obstruct the light, the subsequent judgment is continued; otherwise, the subsequent judgment is not performed.

[0075] Both section speed measurement and single-point speed measurement monitoring devices detect whether vehicles are speeding, and the vehicle driving behavior code type in the watermarks of the evidence images submitted by both types of devices is the same. In practical application, it has been found that there is a certain probability of incorrect device type registration during the initial registration process. For example, a section speed measurement monitoring device might be registered as a single-point speed measurement device. This incorrect device type registration will lead to incorrect watermarks on the evidence documents submitted by the device, thus rendering the evidence images invalid. This method will first identify and resolve this issue of incorrect device type registration.

[0076] S7: Use the trained speed measurement violation classification model to classify and filter single-point speed measurement images or section speed measurement images, and record the classification results output by the model as: recognition type;

[0077] Confirm whether the identification type and the filing type of the monitoring device for the uploaded image are consistent. If they are consistent, proceed with the next step.

[0078] Otherwise, it indicates that the filing type of the image to be judged does not match its actual type, and the subsequent detection of the image to be judged will be stopped, and the image will be directly prompted that it has failed the detection.

[0079] In this method, a binary classification interval speed measurement identification model based on deep learning algorithms is used to filter images of single-point speed measurement and interval speed measurement. Illegal images that do not match the registered device type are removed and reported as anomalies. Only images matching the registered device type are retained for subsequent watermark integrity checks.

[0080] Step S7 includes the following operations:

[0081] a1: Find splicing features in the image to be judged;

[0082] The splicing features include: feature points, edges, and textures;

[0083] a2: Based on the splicing features, find the boundaries of the spliced ​​images, split the image to be judged, and obtain two split images, denoted as: input image;

[0084] a3: Simultaneously feed the two input images corresponding to the same image to be judged into the trained speed measurement violation classification model;

[0085] a4: The speed violation classification model identifies the background differences between two input images and classifies and filters the images to be judged.

[0086] When the backgrounds of the two input images are identical, the image type of the image to be judged is single-point speed measurement.

[0087] When the backgrounds of the two input images are different, the image type of the image to be judged is the interval speed measurement type.

[0088] In this application, the background difference value of the two split images is calculated using the image background difference method. The larger the difference value, the different the backgrounds are, indicating that the images are of the interval speed measurement type. The smaller the difference value, the similar the backgrounds are, indicating that the images are of the single-point speed measurement type, thus ensuring the accuracy of the detection results.

[0089] S8: Input the image to be judged into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs the judgment result.

[0090] The judgment results include: whether it is complete, whether it is blurry, and whether it is occluded.

[0091] Step S8 specifically includes the following operations:

[0092] b1: Locate the watermark in the image to be judged, and extract all text information in the watermark based on the existing Optical Character Recognition (OCR) model, denoted as: watermark data to be judged;

[0093] b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and get the filing type of the image to be judged;

[0094] b3: Find the preset standard watermark information data based on the filing type of the image to be judged, denoted as: comparison watermark data;

[0095] b4: The watermark data to be judged and the watermark data to be compared are both sent into the watermark information completeness detection model for comparison.

[0096] The model automatically analyzes and compares elements such as watermark information, location, and clarity in the image, and compares them with a preset standard for the completeness of watermark data.

[0097] b5: The watermark information completeness detection model outputs the final comparison result;

[0098] The final comparison result is the completeness judgment result of the overlay elements, including detailed information such as whether they are complete, whether they are blurry, and whether they are occluded.

[0099] To ensure the accuracy of the watermark information completeness detection model, the training set is constructed by applying various illegal evidence image overlay templates to the original images without any overlaid information. By controlling the structured input information, positive and negative samples are automatically generated. Positive samples are generated by adding complete watermark information to the original images. The method for adding watermark information to positive samples is as follows:

[0100] It must include: the time of occurrence, the location of occurrence, the behavior code, the behavior description, the image evidence collection device number, and anti-counterfeiting information; for motor vehicles violating speed limits, each evidence image should also include the speed limit value and the driving speed value; for speeding evidence images captured by section speed measurement, it should at least include the name of the speed measurement section, the distance, the entry time, the exit time, and the average speed.

[0101] When generating negative samples, firstly, various templates for overlaying information on illegal evidence images are constructed. Then, incomplete watermark information is overlaid on the original images to generate negative samples, such as... Figure 1 The two examples shown are missing illegal acts, illegal codes, etc., or the speed measurement information is not fully superimposed or not superimposed at all, or the time information is not accurate enough, or the superimposed watermark is improperly positioned.

[0102] After using the technical solution of this invention, key information related to the image to be judged is first extracted, and then the image is classified. Different processing procedures are executed according to the classification results. For images of single-point speed measurement violations and section speed measurement violations, speed measurement violations are first screened; for traffic light violations, traffic light watermark occlusion is first judged. Then, a deep learning model is used to analyze and compare elements such as watermark information, position, and clarity in the image. Finally, the completeness judgment result of the image overlay elements is output, including detailed information such as completeness, clarity, and occlusion. This ensures that the completeness of the overlaid watermark in violation images collected by traffic monitoring equipment can be accurately judged.

Claims

1. A method for judging the completeness of image overlay elements based on deep learning, characterized in that, It includes the following steps: S1: Obtain the image to be judged; Locate the monitoring device that captured the image to be judged, and denote it as: the monitoring device used for judgment; Acquire all recorded data of abnormal driving behaviors uploaded by the monitoring device used for judgment, denoted as: judgment record data; S2: Based on the monitoring equipment used for judgment and the recorded data used for judgment, extract key information related to the image to be judged; The key information includes: device number, device type, shooting time, vehicle behavior description, behavior code, and vehicle license plate number; S3: Based on the device type and behavior code of the monitoring device used for judgment in the key information, classify the image to be judged by image type; The image types correspond one-to-one with the evidence collection types for vehicle driving behavior, specifically including: illegal parking, traffic light, non-traffic light and non-speed measurement, single-point speed measurement, and section speed measurement. S4: Read the image type corresponding to the image to be judged; If the image type is traffic light type, then execute S6; If the image type is a single-point speed measurement or an area speed measurement, then proceed to step S7; If the image type is illegal and belongs to any other type besides those mentioned above, proceed to step S8; S5: Construct a traffic light watermark occlusion judgment model based on a deep learning algorithm model for object detection; construct a speed measurement violation classification model based on a binary classification algorithm model in deep learning; construct a watermark information completeness detection model based on a deep learning algorithm model; S6: Use the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light; If no occlusion is formed, proceed to step S8; Otherwise, if the image is determined to be obscuring the traffic light due to the superimposed watermark, the subsequent detection of the image will be stopped, and the image will be directly prompted that it has failed the detection. S7: Use the trained speed violation classification model to classify and filter single-point speed measurement images or section speed measurement images, and record the classification results output by the model as: identification type; Read the behavior code from the key information of the image to be judged to obtain the filing type of the image to be judged; Confirm whether the identification type and the filing type of the monitoring device for the uploaded image are consistent. If they are consistent, proceed to step S8. Otherwise, it indicates that the filing type of the image to be judged does not match its actual status, and the subsequent detection of the image to be judged will be stopped, and the image will be directly prompted that it has failed the detection; S8: The image to be judged is fed into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs the judgment result. The judgment results include: whether it is complete, whether it is blurry, and whether it is obscured.

2. The method for determining the completeness of image overlay elements based on deep learning according to claim 1, characterized in that: The traffic light watermark occlusion detection model is built based on the YOLOv8 deep learning model.

3. The image overlay feature completeness judgment method based on deep learning according to claim 1, characterized in that: The watermark information completeness detection model is built based on a convolutional neural network model.

4. The image overlay feature completeness judgment method based on deep learning according to claim 1, characterized in that: Step S7 includes the following operations: a1: Find splicing features in the image to be judged; The splicing features include: feature points, edges, and textures; a2: Based on the splicing features, find the boundaries of the spliced ​​images, split the image to be judged, and obtain two split images, denoted as: input image; a3: Simultaneously feed the two input images corresponding to the same image to be judged into the trained speed measurement violation classification model; a4: The speed violation classification model identifies the background differences between two input images and completes the classification and filtering of the images to be judged; When the backgrounds of the two input images are identical, the image type of the image to be judged is a single-point speed measurement image. When the backgrounds of the two input images differ, the image type to be judged is a section speed measurement image.

5. The method for determining the completeness of image overlay elements based on deep learning according to claim 1, characterized in that: Step S8 specifically includes the following operations: b1: Locate the watermark in the image to be judged, extract all text information in the watermark, and record it as: watermark data to be judged; b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and get the filing type of the image to be judged; b3: Find the preset standard watermark information data based on the filing type of the image to be judged, denoted as: comparison watermark data; b4: The watermark data to be judged and the watermark data to be compared are jointly sent into the watermark information completeness detection model for comparison; b5: The watermark information completeness detection model outputs the final comparison result.

Images