Picture superposition element completeness judgment method based on deep learning

Abstract

According to the picture superposition element completeness judgment method based on deep learning provided by the invention, the key information of the to-be-judged picture during uploading is extracted, the pictures are classified based on the key information, and according to the classification result, the signal lamp type pictures and the single-point speed measurement type or interval speed measurement type pictures are subjected to targeted judgment respectively, so that the completeness of the to-be-judged picture is obtained. Pictures with problems are eliminated, so that the detection efficiency is improved; and then the watermark information completeness detection model is utilized to judge the completeness and correctness of the watermark of the to-be-judged picture, and the whole process does not need manual participation, so that the workload of auditing staff is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for determining the completeness of image superposition elements based on deep learning. Background Art

[0002] According to the provisions of the evidence collection specification GA / T 832, the information superimposed on the evidence picture should at least include: the time of occurrence, the place of occurrence, the behavior code, the behavior description, the image evidence equipment number, and anti-counterfeiting information; if a motor vehicle violates the speed limit regulations, each evidence picture should also be superimposed with the speed limit value and the driving speed value; the speeding evidence picture captured by speed measurement in the interval should at least include the speed measurement interval name, distance, entry time, exit time, average speed and other information.

[0003] However, in actual applications, it was found that the evidence pictures captured by some devices did not fully comply with the technical specifications. Figure 1 As shown in the example in the picture, it is reflected in the following aspects: (1) Missing or incomplete information. Some forensic images may not contain all necessary information, such as the time of occurrence, location of occurrence, behavior code, behavior description, image forensic device number, or anti-counterfeiting information.

[0004] (2) The speed measurement information is incomplete or not superimposed, such as the speed limit value or driving speed.

[0005] (3) Insufficient accuracy of time information. For abnormal behaviors that occur during the driving process of a motor vehicle, the time of the driving behavior superimposed on the image may not meet the accuracy requirement of 0.01s.

[0006] (4) Improper superimposition. The superimposed information may be located in a key area of ​​the image, obscuring specific behavioral evidence, such as the vehicle license plate, the driver's facial features, or traffic lights, thus reducing the evidentiary value of the image.

[0007] If the evidence pictures are not standardized, they cannot be used as evidence in subsequent procedures. Therefore, the current method is to have reviewers manually review the evidence pictures. This method consumes a lot of human resources. Summary of the invention

[0008] In order to solve the problem that the integrity and correctness of watermark overlay information on forensic images cannot be automatically identified in the prior art, the present invention provides a method for judging the completeness of image overlay elements based on deep learning, which automatically judges the completeness and correctness of the superimposed fields on forensic images, reduces the irregularities in evidence collection, and reduces the workload of auditors.

[0009] The technical solution of the present invention is as follows: a method for judging the completeness of image superposition elements based on deep learning, characterized in that it comprises the following steps: S1: Get the image to be judged; Find the monitoring device that collects the image to be judged, and record it as: the monitoring device used for judgment; Obtaining all recorded data of abnormal driving behaviors uploaded by the monitoring device for judgment, recorded as: recorded data for judgment; S2: extracting key information related to the image to be judged based on the monitoring device for judgment and the recorded data for judgment; The key information includes: equipment number, equipment type, shooting time, vehicle behavior description, behavior code and vehicle license plate; S3: classifying the image to be judged into image types according to the device type and behavior code of the judgment monitoring device in the key information; The image types correspond to the types of evidence collection for vehicle driving behavior, including: illegal parking, signal light, non-signal light and non-speed measurement, single-point speed measurement, and interval speed measurement; S4: Read the picture type corresponding to the picture to be judged; If the image type belongs to the traffic light category, execute S6; If the image type belongs to the single-point speed measurement category or the interval speed measurement category, execute step S7; If the image type is illegal except for the above, execute step S8; S5: Build a traffic light watermark occlusion judgment model based on the target detection deep learning algorithm model; build a speed measurement violation classification model based on the binary classification algorithm model in the deep learning algorithm; build a watermark information completeness detection model based on the deep learning algorithm model; S6: using the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light; If no occlusion is formed, execute step S8; Otherwise, it is determined that the superimposed watermark of the image to be judged blocks the signal light, and the subsequent detection of the image to be judged is stopped, and a prompt is directly given that the image has failed the detection; S7: using the trained speed measurement violation classification model to classify and screen the single-point speed measurement pictures or the interval speed measurement pictures, and recording the classification result output by the model as: identification type; Reading the behavior code in the key information of the image to be judged, and obtaining the filing type of the image to be judged; Confirm whether the identification type is consistent with the filing type of the monitoring device that uploaded the image, and if they are consistent, execute step S8; Otherwise, it means that the filing type of the image to be judged does not match its actual type, and the subsequent testing of the image to be judged is stopped, and it is directly prompted that the image has not passed the test; S8: sending the image to be judged into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs a judgment result; The judgment result includes: whether it is complete, whether it is blurred, and whether it is blocked.

[0010] It is further characterized by: The signal light watermark occlusion judgment model is built based on the yolov8 deep learning model; The watermark information completeness detection model is constructed based on a convolutional neural network model; Step S7 includes the following operations: a1: Find the splicing features in the image to be judged; The splicing features include: feature points, edges, and textures; a2: Based on the splicing features, find the boundary of the spliced ​​image, split the image to be judged, and obtain two split images, which are recorded as: input image; a3: sending the two input images corresponding to the same image to be judged to the trained speed measurement violation classification model at the same time; a4: The speed measurement violation classification model identifies the background differences between the two input images and completes classification and screening of the images to be judged; When there is no difference in the backgrounds of the two input images, the image type of the image to be determined is a single-point speed measurement type; When the backgrounds of the two input images are different, the image type of the image to be determined is an interval speed measurement image; Step S8 specifically includes the following operations: b1: Find the watermark position in the image to be judged, extract all the text information in the watermark, and record it as: watermark data to be judged; b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and obtain the filing type of the image to be judged; b3: Find the preset standard watermark information data according to the record type of the image to be judged, recorded as: comparison watermark data; b4: sending the watermark data to be judged and the watermark data for comparison to the watermark information integrity detection model for comparison; b5: The watermark information completeness detection model outputs the final comparison result.

[0011] The present application provides a method for judging the completeness of image overlay elements based on deep learning, which extracts key information when the image to be judged is uploaded, classifies the image based on the key information, and first makes targeted judgments on traffic light images, single-point speed measurement images or interval speed measurement images according to the classification results, and excludes images with problems, thereby improving detection efficiency; then, a watermark information completeness detection model is used to judge the completeness and correctness of the watermark of the image to be judged. The entire process does not require human participation, which greatly reduces the workload of the review staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of a forensic picture with incomplete superimposed information; Figure 2 A flowchart of a method for determining the completeness of image superposition elements based on deep learning according to the present application; Figure 3 This is an example of a forensic picture for speed measurement in a section; DETAILED DESCRIPTION

[0013] like Figure 2 As shown, the present application includes a method for determining the completeness of image overlay elements based on deep learning, which includes the following steps.

[0014] S1: Get the image to be judged; Find the monitoring device that collects the image to be judged, and record it as: the monitoring device used for judgment; Obtaining all recorded data of abnormal driving behaviors uploaded by the monitoring device for judgment, recorded as: recorded data for judgment; S2: extracting key information related to the image to be judged based on the monitoring equipment and the recorded data for judgment; Key information includes: equipment number, equipment type, shooting time, vehicle behavior description, behavior code and vehicle license plate.

[0015] S3: classify the image to be judged according to the device type and behavior code of the monitoring device used for judgment in the key information; The image types correspond one-to-one to the types of evidence collected on vehicle driving behaviors, including: illegal parking, traffic light, non-traffic light and non-speed measurement, single-point speed measurement, and interval speed measurement.

[0016] Usually, traffic light images record driving behaviors such as running red lights and not following directions. Non-traffic light and non-speed measurement images record behaviors other than those captured by red light and speed measurement devices, such as driving in prohibited areas, crossing the line, using mobile phones, and not giving way to pedestrians.

[0017] Single-point speed measurement pictures are speeding behaviors captured by single-point speed measurement, while interval speed measurement pictures record speeding behaviors captured by interval speed measurement equipment. The difference between the two is that the monitoring equipment for single-point speed measurement is a single device, while the monitoring equipment for interval speed measurement is two monitoring devices separated by a certain distance, so the backgrounds of a group of pictures captured by the interval speed measurement monitoring equipment are located in different places. However, the backgrounds of two pictures captured by the single-point speed measurement equipment with a short interval are located in the same place.

[0018] The picture to be judged as evidence is a picture of two pictures spliced ​​together, such as Figure 3 The following is an example of an interval speed measurement evidence picture, which is composed of pictures taken by two monitoring devices in front and behind, and then stitched together to form an evidence picture. In actual applications, a common error in interval speed measurement evidence pictures and single-point speed measurement evidence pictures is that the monitoring device mislabels the picture type when uploading the evidence image to the traffic management platform.

[0019] A common mistake in evidence images such as traffic lights is that when the monitoring equipment superimposes the watermark, it superimposes the watermark on the area of ​​the traffic light, resulting in the inability to clearly see the status of the traffic light later.

[0020] Therefore, in this method, before judging the integrity and correctness of the watermark, the unique watermark problems of traffic light pictures, single-point speed measurement pictures, and interval speed measurement pictures are first checked. After confirming that there are no problems, the integrity and correctness of the watermark are judged. This can reduce the subsequent calculation amount and ensure that all problems of the pictures to be judged are found, further reducing the workload of the auditors.

[0021] S4: Read the image type corresponding to the image to be judged; If the image type belongs to the traffic light category, execute S6; If the image type belongs to the single-point speed measurement category or the interval speed measurement category, execute step S7; If the image type is illegal other than the above, execute step S8.

[0022] S5: A traffic light watermark occlusion judgment model is constructed based on the target detection deep learning algorithm model; a speed measurement violation classification model is constructed based on the binary classification algorithm model in the deep learning algorithm; and a watermark information completeness detection model is constructed based on the deep learning algorithm model. The specific model construction method and training method are implemented based on the existing technology. In this embodiment, the traffic light watermark occlusion judgment model is constructed based on the yolov8 deep learning model; and the watermark information completeness detection model is constructed based on the convolutional neural network model.

[0023] S6: Use the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light; If no occlusion is formed, execute step S8; Otherwise, it is determined that the superimposed watermark of the image to be determined blocks the signal light, and the subsequent detection of the image to be determined is stopped, and a prompt is directly given that the image has failed the detection.

[0024] In this method, it is first detected whether the superimposed watermark of the image to be judged blocks the traffic light. If not, the subsequent judgment is continued, otherwise no subsequent detection is performed.

[0025] The detection targets of interval speed measurement monitoring equipment and single-point speed measurement monitoring equipment are both to detect whether the vehicle is speeding. The vehicle driving behavior code type in the watermark of the evidence pictures submitted by the two types of equipment has the same code; in actual applications, it is found that when these two types of equipment are initially filed, there is a certain probability that the type of monitoring equipment will be filed incorrectly. For example: filing an interval speed measurement type monitoring equipment as a single-point speed measurement type equipment. This equipment type filing error will cause the watermark on the evidence file submitted by the collected equipment to be incorrect, which will in turn make the evidence picture invalid. This method will first troubleshoot the problem of this equipment type filing error.

[0026] S7: Use the trained speed measurement violation classification model to classify and screen the single-point speed measurement pictures or the interval speed measurement pictures, and record the classification results output by the model as: identification type; Confirm whether the identification type is consistent with the filing type of the monitoring device that uploaded the image, and if they are consistent, execute the steps; Otherwise, it means that the filing type of the image to be judged does not match its actual type, and the subsequent detection of the image to be judged is stopped, and it is directly prompted that the image has not passed the detection.

[0027] In this method, for single-point speed measurement pictures and interval speed measurement pictures, a two-class interval speed measurement identification model based on a deep learning algorithm is used for screening. Illegal pictures that are inconsistent with the registered equipment type are eliminated, and anomalies are reported. Only pictures that match the registered equipment type are retained for subsequent watermark integrity detection.

[0028] Step S7 includes the following operations: a1: Find the splicing features in the image to be judged; The stitching features include: feature points, edges, and textures; a2: Based on the splicing features, find the boundary of the spliced ​​image, split the image to be judged, and obtain two split images, which are recorded as: input image; a3: Send the two input images corresponding to the same image to be judged to the trained speed violation classification model at the same time; a4: The speed measurement violation classification model identifies the background differences between the two input images and completes the classification and screening of the images to be judged; When there is no difference in the backgrounds of the two input images, the image type of the image to be determined is a single-point speed measurement type; When the backgrounds of the two input images are different, the image type of the image to be determined is the interval speed measurement type.

[0029] In this application, the image background difference method is used to calculate the background difference value of the two split pictures. The larger the difference, the different the background is, which is an interval speed measurement type picture. The smaller the difference, the similar the background is, which is a single-point speed measurement type picture, to ensure the correctness of the detection result.

[0030] S8: Send the image to be judged into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs the judgment result; The judgment results include: whether it is complete, whether it is blurred, and whether it is blocked.

[0031] Step S8 specifically includes the following operations: b1: Find the watermark position in the image to be judged, and extract all the text information in the watermark based on the optical character recognition (OCR) model in the prior art, which is recorded as the watermark data to be judged; b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and obtain the filing type of the image to be judged; b3: Find the preset standard watermark information data according to the record type of the image to be judged, recorded as: comparison watermark data; b4: Send the watermark data to be judged and the watermark data to be compared into the watermark information integrity detection model for comparison; The model automatically analyzes and compares the watermark information, location, clarity and other elements in the image, and compares them with the preset watermark data integrity standards; b5: The watermark information integrity detection model outputs the final comparison result; The final comparison result is the completeness judgment result of the superimposed elements, including detailed information such as whether it is complete, whether it is blurred, and whether there is any obstruction.

[0032] Among them, in order to ensure the accuracy of the detection results of the watermark information completeness detection model, when constructing the model training set, based on the original picture without any superimposed information, various illegal evidence pictures are applied to superimpose information templates, and the positive and negative samples are automatically generated by controlling the structured input information. Add complete watermark information to the original picture to generate a positive sample. The method of adding watermark information to the positive sample is as follows: It must include: time of occurrence, location of occurrence, behavior code, behavior description, image evidence equipment number, and anti-counterfeiting information; if a motor vehicle violates the speed limit regulations, each evidence picture should also have the speed limit value and driving speed value superimposed on it; speeding evidence pictures captured by speed measurement at intervals should at least include information such as the speed measurement interval name, distance, entry time, exit time, average speed, etc.

[0033] When generating negative samples, we first construct templates for superimposing information on various types of illegal evidence images, and then superimpose incomplete watermark information on the original images to generate negative samples, such as Figure 1 In the two examples shown, illegal behaviors, illegal codes, etc. are missing, or the speed measurement information is incompletely or not superimposed, or the time information is not accurate enough, or the superimposed watermark is not in the right position.

[0034] After using the technical solution of the present invention, first extract the key information related to the picture to be judged, then classify the pictures, and execute different processing flows according to the classification results. For single-point speed measurement violation pictures and interval speed measurement violation pictures, first screen the speed measurement violation; for traffic light violation pictures, first perform traffic light watermark occlusion judgment. Then use the deep learning model to analyze and compare the watermark information, position, clarity and other elements in the picture. Finally, output the completeness judgment result of the picture superimposed elements, including completeness, clarity, occlusion and other detailed information. Ensure that the completeness of the superimposed watermark in the violation pictures collected by the traffic technical monitoring equipment can be accurately judged.

Claims

1. A method for judging the completeness of image superposition elements based on deep learning, characterized in that: It includes the following steps: S1: Get the image to be judged; Find the monitoring device that collects the image to be judged, which is recorded as: the monitoring device used for judgment; Obtaining all recorded data of abnormal driving behaviors uploaded by the monitoring device for judgment, recorded as: recorded data for judgment; S2: extracting key information related to the image to be judged based on the monitoring device for judgment and the recorded data for judgment; The key information includes: equipment number, equipment type, shooting time, vehicle behavior description, behavior code and vehicle license plate; S3: classifying the image to be judged into image types according to the device type and behavior code of the judgment monitoring device in the key information; The image types correspond to the types of evidence collection for vehicle driving behavior, including: illegal parking, signal light, non-signal light and non-speed measurement, single-point speed measurement, and interval speed measurement; S4: Read the picture type corresponding to the picture to be judged; If the image type belongs to the traffic light category, execute S6; If the image type belongs to the single-point speed measurement category or the interval speed measurement category, execute step S7; If the image type is illegal except for the above, execute step S8; S5: Build a traffic light watermark occlusion judgment model based on the target detection deep learning algorithm model; build a speed measurement violation classification model based on the binary classification algorithm model in the deep learning algorithm; build a watermark information completeness detection model based on the deep learning algorithm model; S6: using the trained traffic light watermark occlusion judgment model to detect whether the superimposed watermark of the image to be judged occludes the traffic light; If no occlusion is formed, execute step S8; Otherwise, it is determined that the superimposed watermark of the image to be judged blocks the signal light, and the subsequent detection of the image to be judged is stopped, and a prompt is directly given that the image has failed the detection; S7: using the trained speed measurement violation classification model to classify and screen the single-point speed measurement pictures or the interval speed measurement pictures, and recording the classification result output by the model as: identification type; Reading the behavior code in the key information of the image to be judged, and obtaining the filing type of the image to be judged; Confirm whether the identification type is consistent with the filing type of the monitoring device that uploaded the image, and if they are consistent, execute step S8; Otherwise, it means that the filing type of the image to be judged does not match its actual type, and the subsequent testing of the image to be judged is stopped, and it is directly prompted that the image has not passed the test; S8: sending the image to be judged into the trained watermark information completeness detection model to identify whether the watermark information in the image to be judged is complete, and the watermark information completeness detection model outputs a judgment result; The judgment result includes: whether it is complete, whether it is blurred, and whether it is blocked.

2. According to claim 1, a method for judging the completeness of image superposition elements based on deep learning is characterized in that: The traffic light watermark occlusion judgment model is built based on the yolov8 deep learning model.

3. According to claim 1, a method for judging the completeness of image superposition elements based on deep learning is characterized in that: The watermark information completeness detection model is built based on a convolutional neural network model.

4. According to claim 1, a method for judging the completeness of image superposition elements based on deep learning is characterized in that: Step S7 includes the following operations: a1: Find the splicing features in the image to be judged; The splicing features include: feature points, edges, and textures; a2: Based on the splicing features, find the boundary of the spliced ​​image, split the image to be judged, and obtain two split images, which are recorded as: input image; a3: sending the two input images corresponding to the same image to be judged to the trained speed measurement violation classification model at the same time; a4: The speed measurement violation classification model identifies the background differences between the two input images and completes classification and screening of the images to be judged; When there is no difference in the backgrounds of the two input images, the image type of the image to be determined is a single-point speed measurement type; When the backgrounds of the two input images are different, the image type of the image to be determined is an interval speed measurement image.

5. According to claim 1, a method for judging the completeness of image superposition elements based on deep learning, characterized in that: Step S8 specifically includes the following operations: b1: Find the watermark position in the image to be judged, extract all the text information in the watermark, and record it as: watermark data to be judged; b2: Obtain the filing type of the monitoring device that uploaded the image to be judged, and obtain the filing type of the image to be judged; b3: Find the preset standard watermark information data according to the record type of the image to be judged, recorded as: comparison watermark data; b4: sending the watermark data to be judged and the watermark data for comparison to the watermark information integrity detection model for comparison; b5: The watermark information completeness detection model outputs the final comparison result.

Images