Multi-modal large model-based fake license plate vehicle detection method, system, device and medium

Abstract

The application discloses a kind of based on multimodal big model's suit plate vehicle detection method, system, equipment and medium, the method includes the following steps: obtaining the vehicle entry data containing license plate number and this time entry graph;If it is detected that license plate number exists the history record that has not yet exited, corresponding historical entry graph is extracted;Image condition identification and quality assessment are carried out to two graphs, and condition label is generated;Comparison instruction is generated based on condition label, and it and two graphs are input into multimodal big model;Big model carries out semantic level vehicle appearance comparison, and outputs judgment result and confidence;According to judgment result and confidence, execute hierarchical processing: high confidence determines different vehicles, and generates alarm and retains evidence, medium confidence is transferred to artificial review, and determines same vehicle, then normally releases.The application realizes full-automatic, high-precision semantic level suit plate detection, self-adapting complex shooting condition and supporting feedback optimization, significantly reduces the misjudgment rate and system modification cost is low.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and specifically relates to a method, system, equipment and medium for detecting cloned vehicles based on a multimodal large model. Background Technology

[0002] With the rapid advancement of smart community construction, parking lot license plate recognition systems have become standard equipment for vehicle access management in residential communities, industrial parks, commercial complexes, and other similar locations. When a vehicle enters, the camera automatically recognizes the license plate number and reports it to the management platform. Based on this, the system records the vehicle's entry and exit times, calculates parking fees, and manages the parking spaces within the lot.

[0003] However, a violation exists in reality: some individuals forge legitimate license plates, allowing multiple vehicles to share the same license plate number, thereby evading parking fees and traffic violations. This practice not only causes economic losses to parking lots but also poses a serious threat to community safety management. In parking lot scenarios, this behavior manifests as follows: a vehicle with the same license plate, already recorded as having entered the parking lot but not yet exited, is detected re-entering with the same license plate. This abnormal state of entering without exiting and then re-entering is the key characteristic for identifying license plate cloning.

[0004] To identify such cloned vehicles, existing technologies have explored related approaches. For example, Chinese invention patent application CN119942806A discloses a method for detecting cloned vehicles. This method utilizes a parking lot's data collection terminal to gather parking information. When the parking sign of a vehicle to be detected indicates that it is currently parked, the vehicle is determined to be a suspected cloned vehicle. After determining that it is a suspected cloned vehicle, the solution generates warning information based on the target license plate number's entry image, historical entry images, and historical exit images into the current parking lot, allowing staff to conduct manual review.

[0005] However, in actual closed parking management scenarios, the aforementioned existing technologies and existing parking management systems still have the following technical shortcomings: First, the system cannot differentiate between abnormal duplicate reporting and genuine license plate clone entry at the automation level. Existing systems rely solely on license plate number strings for authentication. When duplicate entry with the same license plate is detected, the system cannot distinguish whether it's the same vehicle repeatedly reporting the same entry event due to device latency, network jitter, or other reasons, or whether it's a cloned license plate being used by different vehicles. Since these two situations appear identical at the data level, relying solely on status identifiers or string comparisons is highly prone to misjudgment.

[0006] Secondly, it relies heavily on manual verification, resulting in poor processing efficiency and real-time performance. As mentioned above, even after an anomaly is triggered, the solution still ultimately relies on staff to manually review the images. This method is not only extremely inefficient and lacks real-time performance, but it is also almost impossible to implement effectively in large parking lots and during peak business hours, easily leading to missed detections.

[0007] Finally, traditional image comparison techniques are severely lacking in accuracy and robustness when used to assist in the judgment. Extracting single features for comparison using traditional computer vision algorithms requires the prior establishment of a vehicle feature database, resulting in high deployment costs. Summary of the Invention

[0008] This invention provides a method, system, device, and medium for detecting cloned vehicles based on a multimodal large model. It triggers detection by detecting abnormal states of vehicles entering without exiting and then re-entering during vehicle entry record reporting. It adaptively generates comparison instructions based on conditional labels, uses a multimodal large model to perform semantic-level appearance comparisons between historical and current entry images, and executes hierarchical alarm closed-loop and feedback optimization based on the judgment confidence level. This aims to solve problems such as the inability to automatically identify cloned vehicle anomalies in closed parking lot scenarios, the inability to distinguish between normal repeated system reports and genuine cloned vehicle entry behavior, and the poor robustness and high false positive rate of existing image comparison technologies in complex lighting and occlusion environments.

[0009] To address the aforementioned technical problems, this invention proposes a method for detecting cloned vehicles based on a multimodal large model, comprising the following steps: Obtain vehicle entry data including license plate number and current entry image; If a vehicle license plate number is detected to have a history of not yet leaving the venue, the corresponding historical entry image is extracted. Image condition recognition and quality assessment are performed on the historical entry images and the current entry images to generate condition labels; A comparison instruction is generated based on the conditional labels, and the comparison instruction and two entrance images are input into the multimodal large model. The multimodal large model performs semantic-level vehicle appearance comparison on two entry images and outputs the judgment result and confidence level as to whether they are the same vehicle. Based on the judgment results and confidence levels, a tiered processing method is implemented: if the high confidence level indicates that the vehicle is a different vehicle, a license plate clone alarm is generated and dual-image evidence is retained; if the medium confidence level indicates that the vehicle is transferred to manual review; and if the vehicle is determined to be the same vehicle, the record is updated and the vehicle is released normally.

[0010] Preferably, the method further includes a continuous optimization mechanism based on closed-loop feedback: Record the administrator's confirmation actions on alarm results and manual review results to form a closed-loop feedback data; Based on the closed-loop feedback data, the accuracy rate is calculated according to the label dimension of specific lanes or specific conditions, and the confidence level threshold under the corresponding scenario is dynamically adjusted. Misjudgment cases discovered during manual review are fed back to the template library to correct the comparison instructions under the corresponding condition tags.

[0011] Preferably, the generation of conditional labels specifically includes: Obtain information on the shooting time, lighting conditions, obstruction and damage, and shooting angle of the two entrance photos respectively; Based on the above information, conditional labels reflecting the image environment and quality are generated; If the assessment determines that the occlusion ratio of any entry image exceeds a preset threshold or that abnormal lighting results in low image quality, the input of the multimodal large model is stopped, the vehicle entry data is directly transferred to the manual review queue, and the image quality is marked as insufficient.

[0012] Preferably, the step of generating the comparison instruction based on the condition label specifically includes: A template library containing multiple comparison instruction templates is pre-built; Based on the condition labels, select or assemble the corresponding comparison instruction template from the template library; When the condition label indicates insufficient lighting or inconsistent lighting conditions, the comparison instruction instructs the model to reduce the judgment weight of color features and increase the comparison weight of vehicle outline and structural features. When the condition label indicates that there is partial occlusion, the comparison instruction indicates that the model makes a judgment based only on the visible area and lowers the confidence level accordingly.

[0013] Preferably, the semantic-level vehicle appearance comparison method is as follows: the multimodal large model extracts and analyzes multidimensional visual features from the two entry images according to the comparison instructions; the multidimensional visual features include: vehicle color, vehicle type, body texture and details, and modification and personalization features.

[0014] Preferably, the hierarchical processing specifically includes: If the judgment result is different vehicles and the confidence level is greater than or equal to the first threshold, a license plate cloning alarm is generated, the historical entry image and the current entry image are associated with the alarm ID for persistent storage, and an alarm message containing a comparison of the two images is pushed to the administrator terminal. If the judgment result is the same vehicle and the confidence level is greater than or equal to the first threshold, it is determined to be a normal repeated entry report, the entry record is updated and no alarm is generated. If the confidence level is between the second threshold and the first threshold, it will be transferred to the manual review queue. If the confidence level is less than the second threshold, the vehicle will be treated as a single vehicle and allowed to pass normally, and the judgment details will be recorded in the system log. The second threshold is less than the first threshold.

[0015] Preferably, when the confidence level is between the second threshold and the first threshold, and the application is transferred to the manual review queue, the following processing is also included: The secondary evidence collection mechanism is triggered, and multiple entry images of the corresponding license plate number in the historical records are retrieved from the entry and exit record table of the database. The multiple entry photos, historical entry photos, and this entry photo will be included. Figure 1 The data is then provided to the administrator for manual review, and the review confirmation result is written back to the entry and exit records.

[0016] A second aspect of the present invention also proposes a vehicle cloning detection system based on a multimodal large model, the system being used to implement the vehicle cloning detection method as described in the first aspect of the present invention, comprising: The entry data acquisition module is used to acquire vehicle entry data, including license plate numbers and the current entry map; The historical image extraction module is used to extract the corresponding historical entry image if the license plate number has a history of not yet leaving the venue. The image condition evaluation module is used to perform image condition recognition and quality evaluation on the historical entry image and the current entry image, and generate condition labels. The instruction generation and input module is used to generate a comparison instruction based on the condition label, and input the comparison instruction and two entrance images into the multimodal large model; The large model comparison output module is used to perform semantic-level vehicle appearance comparison on two entry images using the multimodal large model, and output the judgment result and confidence level of whether they are the same vehicle. The alarm and tiered processing module is used to perform tiered processing based on the judgment result and confidence level: if the high confidence level determines that they are different vehicles, a cloned license plate alarm is generated and dual-image evidence is retained; if the medium confidence level determines that they are transferred to manual review; if the determination that they are the same vehicle, the record is updated and the vehicle is released normally.

[0017] A third aspect of the present invention also provides an electronic device comprising: One or more processors; Memory, used to store one or more computer programs; One or more computer programs stored in the memory are executed by the one or more processors, causing the one or more processors to implement the vehicle counterfeiting detection method as described in the first aspect of the present invention.

[0018] In a fourth aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting counterfeit vehicles as described in the first aspect of the present invention.

[0019] Compared with the prior art, the present invention has the following technical effects: 1. The proposed method for detecting cloned vehicles in this invention achieves fully automated, high-precision intelligent detection and closed-loop alarming of cloned vehicles in a closed parking lot environment. This effectively curbs illegal activities such as using cloned license plates to evade parking fees, safeguarding the economic interests and security management order of the parking lot. The detection logic is embedded in the existing entry record processing flow, without affecting normal parking management business processes, and the modification cost is low.

[0020] 2. The proposed method for detecting cloned vehicles in this invention performs image condition recognition and quality assessment on entry images, and dynamically selects or assembles comparison instructions for a multimodal large model based on generated condition labels. This enables the comparison strategy to adaptively adjust the comparison criteria for complex capture scenarios such as nighttime, occlusion, and inconsistent lighting, improving the stability and robustness of judgment in complex scenarios. Simultaneously, the quality assessment mechanism effectively filters out low-quality images, avoiding misjudgments based on unreliable images.

[0021] 3. The vehicle clone detection method proposed in this invention uses a multimodal large model to perform semantic-level overall appearance comparison of two vehicle entry images. It can comprehensively analyze multi-dimensional visual features such as vehicle color, model, body texture and personalized modifications. It can achieve high-precision vehicle identification without the need to establish a vehicle feature database in advance, which greatly improves the accuracy of clone detection.

[0022] 4. The proposed method for detecting cloned vehicles executes high, medium, and low-level branch processing based on the confidence level output by a multimodal large model. Combined with the logic of automatic alarm and retention of dual-image evidence, and secondary evidence collection at medium confidence level followed by manual review, it avoids false alarms or missed alarms caused by a single threshold, achieving controllable management of the false positive rate. Furthermore, the closed-loop feedback mechanism based on manual review results allows the system's threshold setting and comparison instructions to continuously converge and optimize with actual operating data, continuously improving detection efficiency. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the method for detecting counterfeit vehicles according to the present invention; Figure 2 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0025] Example 1 This embodiment presents a method for detecting cloned vehicles based on a multimodal large model, such as... Figure 1 , 2 As shown, it includes the following steps one through six: Step 1: Obtain vehicle entry data including license plate number and current entry image.

[0026] When a vehicle arrives at the parking lot entrance, the underlying device layer of the system (such as a license plate recognition camera) automatically identifies the vehicle's license plate number and captures an image of the vehicle, preferably a front-facing image. Subsequently, the front-end device reports the vehicle entry data to the management platform through a preset entry interface. An example of an entry interface for a parking management system is the / parkingLot / push / carIn interface.

[0027] In this embodiment, the reported vehicle entry data includes, but is not limited to: license plate number (CPH), entry time (InTime), vehicle captured image (usually converted to base64 format for data transmission, serving as the entry image for this instance), lane information (RoadName), and related order number (OrderNo). Obtaining this multi-dimensional entry data not only provides foundational material for subsequent image comparison but also provides data support for system alarm tracing and lane statistics.

[0028] After obtaining the aforementioned vehicle entry data, the entry / exit record status mechanism in the core detection layer uses the currently reported license plate number as input to execute the status detection logic, namely, entry-only detection. Specifically, the determination process involves searching the system's on-site vehicle table for a data entry that matches the currently reported license plate and has not yet generated an exit record.

[0029] If the search result is not found, it means that the current vehicle has entered the parking lot normally. The system will directly write the entry and exit record and update the parking lot vehicle table, and the current business process will end normally. If the search result shows that the vehicle has triggered an abnormal state where the same license plate appears repeatedly without exiting, it is suspected of being a cloned license plate, thus triggering the subsequent image comparison and cloned license plate detection process.

[0030] Step 2: If a historical record of the license plate number not yet appearing is detected, the corresponding historical entry image is extracted.

[0031] After the state machine triggers the card-cloning detection process, the entry image extraction module in the system needs to prepare two image materials for the final comparison: the historical entry image and the current entry image.

[0032] For this entry map, the system directly extracts and obtains it from the vehicle entry data reported in step one. For historical entry images, the system accurately queries and extracts the most recent entry record corresponding to the license plate number that triggered the anomaly from the entry and exit record table in the database, and uses the vehicle capture image stored in that record as the historical entry image.

[0033] It should be noted that both the historical entry images and the current entry images extracted here are pictures captured by cameras at the parking lot entrances and exits, containing overall vehicle appearance information. Obtaining these two images provides a visual basis for subsequent deep semantic feature comparison using a multimodal large-scale model.

[0034] Step 3: Perform image condition recognition and quality assessment on the historical entry image and the current entry image, and generate condition labels.

[0035] Before inputting two images into a multimodal large model for core comparison, it is necessary to perform preprocessing-based conditional recognition and quality assessment on the images. The core purpose is to enable subsequent comparison instructions and strategies to adapt to different shooting conditions and to filter out low-quality images that cannot be reliably compared in advance.

[0036] The generation of conditional labels specifically includes the following steps S31 to S33: S31: Obtain the shooting time, lighting conditions, occlusion and damage status, and shooting angle information for the two entrance images respectively; specifically, the acquisition and evaluation of the aforementioned dimensions include the following specific implementation methods: Shooting time period label: Based on the timestamp information of the image, accurately determine whether the vehicle was captured during the day or at night; Lighting condition assessment: By numerically analyzing parameters such as brightness and contrast of the overall image and local areas, the system identifies whether the image has abnormal lighting conditions such as backlighting, overexposure, underexposure, or low illumination. Occlusion and dirt detection: Evaluate the proportion of the target vehicle body area visible in the whole image to identify whether the vehicle body is occluded by other objects, the camera lens is dirty, or there is severe rain or fog interference; Shooting angle judgment: Based on the relative position of the identified license plate frame and the overall vehicle outline, calculate and determine whether the captured image is a frontal capture, a side capture, or a large-angle tilt capture.

[0037] S32: Based on the aforementioned information such as shooting time, lighting conditions, occlusion and contamination, and shooting angle, conditional labels reflecting the image environment and quality are generated. After the comprehensive evaluation across these multiple dimensions, the evaluation results are integrated to generate structured conditional labels. For example, after recognition, the system can generate conditional labels similar to "Nighttime - Slight Occlusion - Frontal Shooting". These conditional labels will serve as important input parameters for subsequent processing, along with historical entry images and the current entry image. Figure 1 It is used to transfer to a multimodal large model.

[0038] S33: If the occlusion ratio of any entry image exceeds a preset threshold or the lighting is abnormal, resulting in low image quality, the input of the multimodal large model is stopped, the vehicle entry data is directly transferred to the manual review queue, and the image quality is marked as insufficient.

[0039] In this step, when abnormal lighting reaches a certain level, such as severe overexposure or underexposure rendering the core features of the vehicle body virtually unrecognizable, or when the occlusion area is too large, the image is deemed unable to provide credible visual cues. At this point, instead of forcibly comparing using a multimodal large model, the abnormal entry event is directly transferred to the subsequent manual review queue, with the reason for the transfer clearly marked as insufficient image quality. This mechanism effectively avoids misjudgments caused by the system forcibly comparing unreliable image data, ensuring the accuracy of subsequent license plate detection results.

[0040] Step 4: Generate a comparison instruction based on the conditional labels, and input the comparison instruction and two entrance images into the multimodal large model.

[0041] In this step, the previously filtered historical entry images and the current entry image, along with the conditional labels generated in step three, are input into the multimodal large model (visual language model, VLM) for comparison. To enable the large model's comparison strategy to adapt to different shooting conditions, this embodiment employs a dynamic instruction generation mechanism instead of using a single fixed comparison instruction.

[0042] The specific method for generating the comparison instructions is as follows: steps S41 to S44: S41: Pre-build a template library containing multiple alignment instruction templates. An example is shown below: Reference template under ideal conditions Applicable conditions tag: Both images are taken during the day and are unobstructed.

[0043] Comparison instruction template: Please comprehensively analyze the vehicle's appearance information, such as color, model, body texture, window shape, and modification features, to determine whether two images are of the same vehicle, and return the judgment result in JSON format.

[0044] Template for nighttime or low-light conditions Applicable conditions tag: Any image was taken at night.

[0045] Comparison instruction template: Colors are easily distorted under nighttime lighting conditions. Please reduce the weight of color features in the judgment and focus on comparing structural features that are not affected by light, such as headlight outlines, body structure lines, window shapes, and roof additions.

[0046] Template with occlusion conditions Applicable condition label: There is partial occlusion.

[0047] Comparison instruction template: Part of the vehicle body area in the image is obscured. Please make a judgment based only on the appearance features of the visible area, and truthfully reflect the uncertainty caused by the obscuration in the returned results, and appropriately reduce the confidence value.

[0048] Inconsistent lighting condition template Applicable conditions tag: Two images with different lighting conditions (e.g., one during the day and one at night).

[0049] Comparison instruction template: The two images are under different lighting conditions. Please first normalize the appearance information of the two images separately, and then compare them based on the normalized features to avoid directly using the apparent color difference as the basis for judgment.

[0050] S42: After receiving the incoming image and label, select or assemble the corresponding comparison instruction template from the template library according to the condition label. For example, when the condition label is "daytime - unobstructed - frontal shot", the assembled comparison instruction template will instruct the model: "Please comprehensively analyze the appearance information such as vehicle color, model, body texture, window shape, and modification features to determine whether the two images are of the same vehicle, and return the judgment result in JSON format."

[0051] For special and complex shooting environments, this embodiment dynamically adjusts the focus of the instructions through conditional labels: S43: When the condition label indicates insufficient lighting or inconsistent lighting conditions, the comparison instruction instructs the model to reduce the judgment weight of color features and increase the comparison weight of vehicle outline and structural features.

[0052] As a further preferred example, if the condition label indicates that either image was taken at night, since colors are easily distorted under nighttime lighting conditions, the comparison instruction will require the model to focus on comparing structural features unaffected by lighting conditions, such as headlight outlines, body lines, window shapes, and roof additions. If the condition label indicates that the lighting conditions of the two images are inconsistent, the comparison instruction will guide the model to first normalize the appearance information of the two images separately, and then compare them based on the normalized features, in order to avoid misjudgment caused by directly using apparent color difference as a basis for judgment.

[0053] S44: When the condition label indicates that there is local occlusion, the comparison instruction indicates that the model makes a judgment based only on the visible area and lowers the confidence level accordingly.

[0054] Specifically, the instruction will indicate that some areas of the vehicle body in the model image are occluded, requiring the model to accurately reflect the uncertainty caused by the occlusion in the returned structured results, thereby appropriately reducing the confidence value of the final output.

[0055] Through the above mechanism, the multimodal large model, guided by customized comparison instructions, can perform a more accurate semantic-level overall appearance understanding of two images, thereby adapting to various complex real-world parking lot snapshot scenarios.

[0056] Under the ideal conditions template, when the applicable condition label is "daytime - unobstructed - frontal shooting", the instruction design logic of this embodiment is as follows: Under high-quality images, the large model is required to fully extract all available features and output standard structured results. An example of a generated comparison instruction is: "You are now a professional traffic vehicle recognition system. Please comprehensively analyze the two provided vehicle entry images. Please comprehensively compare the overall appearance information of the two vehicles, including but not limited to: vehicle color, model (such as sedan, SUV, etc.), body texture, window shape, and any personalized modification features (such as window tinting, roof rack). Please determine whether the two images are of the same vehicle and return the judgment result strictly in JSON format. The JSON format requirements are as follows: {"isSameVehicle": Boolean value,"confidence": a floating-point number between 0 and 1,"reason":"a string explaining the basis for the judgment"}. Step 5: The multimodal large model performs semantic-level vehicle appearance comparison on the two entry images and outputs the judgment result and confidence level of whether they are the same vehicle.

[0057] Upon receiving the incoming image and corresponding comparison instruction, the multimodal large model will perform overall appearance semantic understanding using non-traditional pixel comparison. Specifically, the semantic-level vehicle appearance comparison method involves the multimodal large model extracting and analyzing multidimensional visual features from the two incoming images based on the comparison instruction. These multidimensional visual features include: vehicle color, vehicle type, body texture and details, and modification and personalization features.

[0058] Guided by adaptive alignment instructions, the comprehensive analysis of the aforementioned multi-dimensional visual features by the multimodal large model specifically includes: Vehicle color comparison: Extract and compare the overall color and color distribution of the vehicle body in two images; Vehicle type comparison: Identify and compare whether two vehicles belong to the same vehicle type category, such as sedan, SUV, or truck; Body texture and detail comparison: In-depth analysis and comparison of subtle structural features such as door line direction, window shape and outline, and bumper features; Modification and personalization feature comparison: Focus on identifying and comparing highly recognizable personalized additions such as exterior wraps, roof racks, and special decorations.

[0059] After performing comprehensive semantic analysis based on the aforementioned multidimensional features, the multimodal large model will output a structured judgment result. Preferably, the judgment result is returned in JSON data format, and includes at least: a Boolean parameter indicating whether they are the same vehicle (isSameVehicle), a confidence score (typically ranging from 0 to 1, with higher values ​​indicating greater certainty from the model), the condition tag used in this comparison (conditionTag), and a text description explaining the basis for the judgment (reason).

[0060] Based on the comparison instruction example under the ideal condition template in step four, the output example of the multimodal large model is as follows: { "isSameVehicle":false, "confidence":0.95, "conditionTag":"Daytime - Unobstructed - Frontal Shot", "reason":"The two cars have different body colors ( Figure 1 It is a white SUV. Figure 2 The vehicle was a black sedan, and the model did not match, therefore they were determined to be different vehicles. } By using a semantic-level comparison and structured output mechanism based on multi-dimensional features, we can not only overcome the limitations of traditional image algorithms that rely on single features and achieve high-precision comparison, but also obtain interpretable and detailed reasoning basis, providing intuitive data support for subsequent hierarchical processing, alarm generation and manual review.

[0061] Step Six: Perform tiered processing based on the judgment results and confidence levels: if the high confidence level indicates that the vehicle is a different vehicle, generate a license plate clone alarm and retain dual-image evidence; if the medium confidence level indicates that the vehicle is transferred to manual review; if the vehicle is determined to be the same vehicle, update the record and allow the vehicle to pass normally.

[0062] In this embodiment, the confidence score output by the large model is combined with multi-level thresholds to perform refined closed-loop management. The hierarchical processing specifically includes: Wherein, the second threshold is less than the first threshold. As a preferred embodiment, the first threshold can be set to 0.85 and the second threshold to 0.5.

[0063] The high confidence level indicates a cloned license plate, confirming it as an unregistered vehicle. If the judgment result is different vehicles and the confidence level is greater than or equal to the first threshold, a license plate cloning alarm is generated, the historical entry image and the current entry image are associated with the alarm ID for persistent storage, and an alarm message containing a comparison of the two images is pushed to the administrator terminal.

[0064] When the model outputs `isSameVehicle=false` and `confidence≥0.85`, the license plate cloning alarm loop is officially triggered. First, the alarm type, license plate number, alarm time, and confidence level are written to the alarm log table. By persistently storing the dual-image evidence, the most direct image evidence is preserved for subsequent evidence collection. Simultaneously, the alarm information is pushed to the administrator's terminal in real time via a message push mechanism and displayed intuitively on the management dashboard, prompting the administrator to intervene quickly.

[0065] The high confidence level is considered normal, confirming that they were in the same vehicle. If the judgment result is the same vehicle and the confidence level is greater than or equal to the first threshold, it is determined to be a normal repeated entry report, the entry record is updated and no alarm is generated.

[0066] When the model outputs `isSameVehicle=true` and `confidence≥0.85`, the current state of entry without exit and re-entry is considered a normal duplicate entry report, such as duplicate pushes caused by license plate recognition device delays or network jitter. In this case, the system only updates the original entry record, the process continues normally, and no alarms are generated to interfere with management personnel.

[0067] Medium confidence level, fuzzy zone: If the confidence level is between the second threshold and the first threshold, it is transferred to the manual review queue. That is, regardless of the judgment result value, as long as the confidence is between 0.5 and 0.85, the large model is considered to have a moderate grasp of the current judgment, requiring manual intervention. Transferring to the manual review queue when the confidence level is between the second threshold and the first threshold also includes the following processing: triggering a secondary evidence collection mechanism, appending and retrieving multiple entry images of the corresponding license plate number from the entry and exit record table in the database; and combining the multiple entry images, historical entry images, and the current entry image. Figure 1 The data is then provided to the administrator for manual review, and the review confirmation result is written back to the entry / exit log. This review result writing-back mechanism serves two purposes: firstly, it determines whether the anomaly should be escalated to a formal alarm; secondly, it serves as closed-loop feedback data for subsequent model performance verification and dynamic optimization of tiered thresholds.

[0068] Low confidence, insufficient certainty, and fallback measures: If the confidence level is less than the second threshold, the system will default to treating the same vehicle as a single vehicle and allowing it to pass normally, recording the judgment details in the system log. That is, when confidence < 0.5, the system considers the model's judgment insufficient. To avoid severely hindering normal vehicle traffic efficiency, the system defaults to allowing the same vehicle to pass normally, but will record the complete judgment details in the system background log, including the conditional labels at the time, comparison images, and low confidence values, for subsequent operational audits and model performance evaluation.

[0069] To avoid the system relying solely on fixed rules configured only once, this embodiment introduces a self-learning and adaptive feedback optimization process. This allows the detection strategy to converge continuously with actual operational data, gradually reducing the false positive rate in specific scenarios. This mechanism specifically includes the following steps: First, the system records the administrator's confirmation actions regarding alarm results and manual review results, forming a closed-loop feedback data. In actual business scenarios, administrators will conduct final verification of the cloned vehicle alarms generated by the system and the tasks assigned to the manual review queue. The system will comprehensively record these confirmation actions by the administrator (e.g., confirming it as cloned vehicle, marking it as a false alarm, or providing supplementary judgment opinions, etc.). These manual confirmation records based on real business scenarios constitute closed-loop feedback data for system optimization.

[0070] Secondly, based on the closed-loop feedback data, the system calculates the accuracy rate according to specific lane or condition label dimensions and dynamically adjusts the confidence level threshold for the corresponding scenario. The system utilizes the collected closed-loop feedback data to calculate the consistency rate between the large model's judgment results and the final human confirmation results, based on dimensions such as lane, time period, or condition label, forming quantifiable detection accuracy data. Based on the accuracy statistics, the system dynamically adjusts the thresholds. For example, when a specific lane is detected, or when the false alarm rate is consistently high under a certain type of condition label (such as a specific lighting environment), the confidence level threshold (i.e., the first threshold or the second threshold) corresponding to that scenario can be adjusted downwards or upwards accordingly. Through this fine-grained dynamic adjustment, the false alarm rate in complex scenarios is effectively controlled.

[0071] Finally, the system feeds back misjudgments discovered during manual review to the template library to correct the comparison instructions under the corresponding condition labels. This step constitutes the iterative mechanism of the comparison instruction templates. The system extracts typical misjudgments discovered during manual review (e.g., the model exhibits a fixed misjudgment pattern under specific lighting or occlusion conditions) and feeds them back to the Prompt template library. System maintainers or algorithm logic can use these real misjudgment cases to supplement or correct the comparison instructions under the corresponding condition labels. For example, if it is found that a certain nighttime reflection is often mistaken for a specific vehicle feature, the instruction template for the corresponding label can be supplemented with a prompt to "ignore specific reflection interference," thereby improving the accuracy of the next comparison of the multimodal large model from the source.

[0072] Example 2 This embodiment is a cloned vehicle detection system based on a multimodal large model. The system is used to implement the cloned vehicle detection method as described in Embodiment 1, including: The entry data acquisition module is used to acquire vehicle entry data, including license plate numbers and the current entry map; The historical image extraction module is used to extract the corresponding historical entry image if the license plate number has a history of not yet leaving the venue. The image condition evaluation module is used to perform image condition recognition and quality evaluation on the historical entry image and the current entry image, and generate condition labels. The instruction generation and input module is used to generate a comparison instruction based on the condition label, and input the comparison instruction and two entrance images into the multimodal large model; The large model comparison output module is used to perform semantic-level vehicle appearance comparison on two entry images using the multimodal large model, and output the judgment result and confidence level of whether they are the same vehicle. The alarm and tiered processing module is used to perform tiered processing based on the judgment result and confidence level: if the high confidence level determines that they are different vehicles, a cloned license plate alarm is generated and dual-image evidence is retained; if the medium confidence level determines that they are transferred to manual review; if the determination that they are the same vehicle, the record is updated and the vehicle is released normally.

[0073] Example 3 This embodiment is an electronic device, including: One or more processors; Memory, used to store one or more computer programs; One or more computer programs stored in the memory are executed by the one or more processors, causing the one or more processors to implement the vehicle counterfeiting detection method as described in Embodiment 1.

[0074] Example 4 This embodiment is a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the method for detecting cloned vehicles as described in Embodiment 1.

[0075] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for detecting cloned vehicles based on a multimodal large model, characterized in that, Includes the following steps: Obtain vehicle entry data including license plate number and current entry image; If a vehicle license plate number is detected to have a history of not yet leaving the venue, the corresponding historical entry image is extracted. Image condition recognition and quality assessment are performed on the historical entry images and the current entry images to generate condition labels; A comparison instruction is generated based on the conditional labels, and the comparison instruction and two entrance images are input into the multimodal large model. The multimodal large model performs semantic-level vehicle appearance comparison on two entry images and outputs the judgment result and confidence level as to whether they are the same vehicle. Based on the judgment results and confidence levels, a tiered processing method is implemented: if the high confidence level indicates that the vehicle is a different vehicle, a license plate clone alarm is generated and dual-image evidence is retained; if the medium confidence level indicates that the vehicle is transferred to manual review; and if the vehicle is determined to be the same vehicle, the record is updated and the vehicle is released normally.

2. The method according to claim 1, characterized in that, The method also includes a continuous optimization mechanism based on closed-loop feedback: Record the administrator's confirmation actions on alarm results and manual review results to form a closed-loop feedback data; Based on the closed-loop feedback data, the accuracy rate is calculated according to the label dimension of specific lanes or specific conditions, and the confidence level threshold under the corresponding scenario is dynamically adjusted. Misjudgment cases discovered during manual review are fed back to the template library to correct the comparison instructions under the corresponding condition tags.

3. The method according to claim 1 or 2, characterized in that, The specific generation condition label is as follows: Obtain information on the shooting time, lighting conditions, obstruction and damage, and shooting angle of the two entrance photos respectively; Based on the above information, conditional labels reflecting the image environment and quality are generated; If the assessment determines that the occlusion ratio of any entry image exceeds a preset threshold or that abnormal lighting results in low image quality, the input of the multimodal large model is stopped, the vehicle entry data is directly transferred to the manual review queue, and the image quality is marked as insufficient.

4. The method according to claim 1 or 2, characterized in that, The specific steps for generating the comparison instruction based on the conditional label are as follows: A template library containing multiple comparison instruction templates is pre-built; Based on the condition labels, select or assemble the corresponding comparison instruction template from the template library; When the condition label indicates insufficient lighting or inconsistent lighting conditions, the comparison instruction instructs the model to reduce the judgment weight of color features and increase the comparison weight of vehicle outline and structural features. When the condition label indicates that there is partial occlusion, the comparison instruction indicates that the model makes a judgment based only on the visible area and lowers the confidence level accordingly.

5. The method according to claim 1 or 2, characterized in that, The semantic-level vehicle appearance comparison method is as follows: the multimodal large model extracts and analyzes the multidimensional visual features of the two entry images according to the comparison instructions; The multidimensional visual features include: vehicle color, vehicle type, body texture and details, and modification and personalization features.

6. The method according to claim 1 or 2, characterized in that, The hierarchical processing specifically includes: If the judgment result is different vehicles and the confidence level is greater than or equal to the first threshold, a license plate cloning alarm is generated, the historical entry image and the current entry image are associated with the alarm ID for persistent storage, and an alarm message containing a comparison of the two images is pushed to the administrator terminal. If the judgment result is the same vehicle and the confidence level is greater than or equal to the first threshold, it is determined to be a normal repeated entry report, the entry record is updated and no alarm is generated. If the confidence level is between the second threshold and the first threshold, it will be transferred to the manual review queue. If the confidence level is less than the second threshold, the vehicle will be treated as a single vehicle and allowed to pass normally, and the judgment details will be recorded in the system log. The second threshold is less than the first threshold.

7. The method according to claim 6, characterized in that, When the confidence level is between the second threshold and the first threshold, the application is transferred to the manual review queue, and the following processing is also included: The secondary evidence collection mechanism is triggered, and multiple entry images of the corresponding license plate number in the historical records are retrieved from the entry and exit record table of the database. The multiple entry photos, historical entry photos, and current entry photos are provided to the administrator for manual review, and the review confirmation results are written back to the entry and exit records.

8. A vehicle clone detection system based on a multimodal large model, characterized in that, The system is used to implement the method for detecting cloned vehicles as described in any one of claims 1-7, including: The entry data acquisition module is used to acquire vehicle entry data, including license plate numbers and the current entry map; The historical image extraction module is used to extract the corresponding historical entry image if the license plate number has a history of not yet leaving the venue. The image condition evaluation module is used to perform image condition recognition and quality evaluation on the historical entry image and the current entry image, and generate condition labels. The instruction generation and input module is used to generate a comparison instruction based on the condition label, and input the comparison instruction and two entrance images into the multimodal large model; The large model comparison output module is used to perform semantic-level vehicle appearance comparison on two entry images using the multimodal large model, and output the judgment result and confidence level of whether they are the same vehicle. The alarm and tiered processing module is used to perform tiered processing based on the judgment result and confidence level: if the high confidence level determines that they are different vehicles, a cloned license plate alarm is generated and dual-image evidence is retained; if the medium confidence level determines that they are transferred to manual review; if the determination that they are the same vehicle, the record is updated and the vehicle is released normally.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs; The feature is that one or more computer programs stored in the memory are executed by the one or more processors, causing the one or more processors to implement the vehicle counterfeiting detection method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for detecting cloned vehicles as described in any one of claims 1-7.

Images