A three-layer dynamic cascading image recognition method

Abstract

The application relates to a three-layer dynamic cascade image recognition method, which realizes high-precision image recognition under a three-layer dynamic cascade model and can identify ambiguous scenes. The method comprises the following steps: inputting an image to be recognized into an edge detection model to output a candidate box, the candidate box comprising a coarse classification label and a first confidence; if the first confidence is not located in a preset double-threshold interval, the coarse classification label is output; if the first confidence is located in the preset double-threshold interval, the intermediate layer feature map obtained by inputting the image to be recognized into the edge detection model is uploaded to a cloud-side fine classification model to output a fine classification label and a softmax probability; if the softmax probability is not less than a first preset threshold, the fine classification label is output; if the softmax probability is less than the first preset threshold and greater than a second preset threshold, the image to be recognized is subjected to RoI cutting, the RoI cutting image and a preset Prompt template are input into a semantic arbitration model, and a natural language conclusion and a second confidence are output; if the second confidence is not less than a third preset threshold, the natural language conclusion is output; and if the second confidence is less than the third preset threshold, manual review is performed.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a three-layer dynamic cascaded image recognition method. Background Technology

[0002] With the rapid development of image recognition technology, in order to ensure its efficient and stable performance in real-world applications, there is a simultaneous demand for "high accuracy, low latency, and low bandwidth" in image recognition. Traditional solutions either deploy lightweight models at the edge, resulting in high false alarm rates due to limited computing power, or upload the entire image to the cloud, where bandwidth and latency become bottlenecks. Existing patents have attempted the "edge screening + cloud backup" model, but they generally suffer from the following drawbacks: high false alarm rate of single edge models (>10%), which cannot meet the hard target of ≤2% for industrial quality inspection; large bandwidth consumption of whole image upload scheme (50–200KB / frame), which can overwhelm the LTE uplink with just 4 channels of 1080p30; static thresholds or fixed intervals fail when the scene changes (backlight, rain, fog, new target), resulting in a surge in cloud call volume; offline full parameter update cycle ≥1 week, relying on manual annotation, which cannot achieve minute-level adaptation for open set targets (new defects and new contraband not covered by the training library); two-level cascading stops at "classification logits", which is powerless for ambiguous scenarios that require semantics + common sense (such as "live work without warning signs").

[0003] A search of existing technical literature revealed a patent application (application number 202510025847.3) entitled "A Method, Device, and System for Assisting Classroom Teaching Quality Assessment Based on Cloud-Edge Collaboration." This patent acquires video image data of students in the classroom in real time from a camera and transmits it to an edge device. On the edge device, a pre-trained student classroom behavior recognition model is used to identify and track student classroom behaviors. During tracking, the recognition results of neighboring frames are fused with the current frame's recognition result to obtain the final recognition result for the current frame. Classroom teaching quality is assessed based on the student classroom behavior recognition results. The student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and employs a knowledge distillation structure. The student classroom behavior recognition results and the assessment results are sent to a cloud video server for storage. However, this patent suffers from problems such as a lack of a semantic arbitration model and insufficient image recognition accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a three-layer dynamic cascaded image recognition method to address the above-mentioned technical problems, so as to achieve high-precision image recognition under the three-layer dynamic cascaded model and be able to identify ambiguous scenes.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] This invention provides a three-layer dynamic cascaded image recognition method, the method comprising:

[0007] S1: Input the image to be identified into the edge detection model and output candidate boxes. The candidate boxes include coarse classification labels and first confidence scores. The edge detection model includes an INT8 quantized YOLOv8-nano model.

[0008] S2: Determine whether the first confidence level is within the preset double threshold interval. If it is not within the preset double threshold interval, use the coarse classification label as the recognition result of the image to be recognized. If it is within the preset double threshold interval, input the image to be recognized into the intermediate layer feature map obtained by the edge detection model and upload it to the cloud-based precision analysis model to output the fine classification label and softmax probability. The cloud-based precision analysis model includes the Swin-T model.

[0009] S3: Determine whether the softmax probability is less than a first preset threshold. If it is not less than the first preset threshold, output the fine classification label as the recognition result of the image to be recognized. If it is less than the first preset threshold and greater than the second preset threshold, perform RoI cropping on the image to be recognized to obtain the RoI cropped image. Input the RoI cropped image and the preset Prompt template into the semantic arbitration model, and output the natural language conclusion and the second confidence. The first preset threshold is greater than the second preset threshold. The semantic arbitration model includes the Qwen-VL-72B model.

[0010] S4: Determine whether the second confidence level is less than the third preset threshold. If it is not less than the third preset threshold, output a natural language conclusion as the recognition result of the image to be recognized. If it is less than the third preset threshold, perform manual review.

[0011] Preferably, the intermediate layer feature map is reconstructed into a 224×224 dimensional pseudo-map within the cloud-based precision model, and the intermediate layer feature map includes 1×4200 dimensional float16 format Neck layer P5 features.

[0012] Preferably, before uploading the intermediate layer feature map obtained by inputting the image to be identified into the edge detection model to the cloud-based precision analysis model, the method further includes: performing Zlib compression and AES-128-GCM encryption on the intermediate layer feature map in sequence.

[0013] Preferably, the region of interest for RoI cropping is determined based on the candidate bounding box.

[0014] Preferably, step S3 further includes stopping the recognition of the image to be recognized if the softmax probability is less than a second preset threshold.

[0015] Preferably, after outputting the fine classification label as the recognition result of the image to be recognized, the method further includes:

[0016] An online distillation method is used to input the fine-class labels as training parameters into the edge detection model for model training and updating.

[0017] Preferably, after outputting the natural language conclusion as the recognition result of the image to be recognized, the method further includes:

[0018] An online distillation method is used to input natural language conclusions as training parameters into the edge detection model and the cloud-based precision analysis model for model training and updating.

[0019] Preferably, the edge detection model is trained and updated using the KL divergence loss at a first preset temperature.

[0020] Preferably, the edge detection model and the cloud-based precision model are trained and updated using the KL divergence loss at a second preset temperature.

[0021] Preferably, the coarse classification label is compared with the fine classification label or the coarse classification label is compared with the natural language conclusion. If the comparison results are inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] This invention provides a three-layer dynamic cascaded image recognition method. The image to be recognized is first detected by an edge detection model deployed on an edge device. If it can be recognized, a corresponding coarse classification label is obtained. If it cannot be recognized or the recognition is abnormal, it is uploaded to a cloud-based precision analysis model deployed in the cloud for detection. This realizes a mode of initial edge screening plus cloud-based fallback. Furthermore, for ambiguous scenarios that require semantic and common sense, a semantic arbitration model is added. This allows images with new targets, new defects, or new semantics to be further identified by the semantic arbitration model even if the cloud-based precision analysis model cannot recognize them, greatly reducing the false alarm rate and false negative rate of image recognition. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a three-layer dynamic cascaded image recognition method in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] Example 1

[0027] like Figure 1As shown, this embodiment proposes a three-layer dynamic cascaded image recognition method, the method comprising:

[0028] S1: Input the image to be identified into the edge detection model and output candidate boxes. The candidate boxes include coarse classification labels and first confidence scores. The edge detection model includes an INT8 quantized YOLOv8-nano model.

[0029] The specific implementation of this step is as follows: As shown in Table 1, the edge detection model deployed on the edge device in this embodiment adopts the YOLOv8-nano model with INT8 quantization. The model is deployed in the Jetson Orin Nano development kit. The entire image to be identified is input into the YOLOv8-nano model, and the output is a candidate box. The candidate box includes a coarse classification label and a first confidence level C∈[0,1]^K.

[0030] Table 1

[0031]

[0032] S2: Determine whether the first confidence level is within the preset double threshold interval. If it is not within the preset double threshold interval, use the coarse classification label as the recognition result of the image to be recognized. If it is within the preset double threshold interval, input the image to be recognized into the intermediate layer feature map obtained by the edge detection model and upload it to the cloud-based precision analysis model to output the fine classification label and softmax probability. The cloud-based precision analysis model includes the Swin-T model.

[0033] The specific implementation of this step is as follows: Preset dual threshold intervals [ , After obtaining the first confidence level C∈[0,1]^K, given the value of [0.35, 0.75], determine whether the first confidence level falls within the preset double threshold interval. If max(C)≤ or ≥ The edge detection model directly outputs a coarse classification label as the recognition result for the image to be recognized. <max(C)< The image to be identified is input into the intermediate layer feature map obtained by the edge detection model and uploaded to the cloud-based precision segmentation model. The upload method is MQTT protocol (running on the TLS 1.3 encryption protocol). The cloud-based precision segmentation model adopts the Swin-T model and is deployed on a GPU / TPU cluster. The Swin-T model reconstructs the features into a 224×224-dimensional pseudo-map and outputs the fine classification label y′ and softmax probability P_swin within 25ms.

[0034] S3: Determine whether the softmax probability is less than a first preset threshold. If it is not less than the first preset threshold, output the fine classification label as the recognition result of the image to be recognized. If it is less than the first preset threshold and greater than the second preset threshold, perform RoI cropping on the image to be recognized to obtain the RoI cropped image. Input the RoI cropped image and the preset Prompt template into the semantic arbitration model, and output the natural language conclusion and the second confidence. The first preset threshold is greater than the second preset threshold. The semantic arbitration model includes the Qwen-VL-72B model.

[0035] The specific implementation of this step is as follows: The softmax probability P_swin is compared with a preset threshold. If P_swin ≥ 0.95, that is, not less than the first preset threshold, the final review is performed directly, and the fine classification label y′ is output as the recognition result of the image to be recognized. If 0.5 ≤ P_swin < 0.95, that is, less than the first preset threshold and greater than the second preset threshold, the RoI of the image to be recognized is cropped to obtain a 224×224-dimensional RoI cropped image. The RoI cropped image and the preset Prompt template are input into the semantic arbitration model, and the natural language conclusion and the second confidence P_LMM are output. The semantic arbitration model adopts the Qwen-VL-72B model, and the GPT-4V model, Intern-VL-40B model, etc. can also be used. The model is deployed in the large model inference pool. The preset Prompt template is shown below:

[0036] Please determine whether the {category} in the image is normal. Only answer "normal / abnormal" with a brief reason, with a confidence level of 0-1.

[0037] S4: Determine whether the second confidence level is less than the third preset threshold. If it is not less than the third preset threshold, output a natural language conclusion as the recognition result of the image to be recognized. If it is less than the third preset threshold, perform manual review.

[0038] The specific implementation of this step is as follows: The second confidence level P_LMM is compared with a preset threshold. If P_LMM≥0.90, the natural language conclusion output by the semantic arbitration model is used as the pseudo-label as the recognition result of the image to be recognized. If P_LMM<0.90, the manual review interface is triggered.

[0039] Example 2

[0040] This embodiment further supplements the description of the three-layer dynamic cascaded image recognition method proposed in Embodiment 1.

[0041] The intermediate layer feature map is reconstructed into a 224×224-dimensional pseudo-map within the cloud-based precision model. The intermediate layer feature map includes 1×4200-dimensional float16 format Neck layer P5 features.

[0042] Before uploading the intermediate layer feature map obtained by inputting the image to be identified into the edge detection model to the cloud-based precision analysis model, the method further includes: performing Zlib compression and AES-128-GCM encryption on the intermediate layer feature map in sequence, with the encrypted data size ≤8KB.

[0043] The region of interest for RoI cropping is determined based on the candidate bounding box.

[0044] The specific implementation of this step is as follows: The system will re-crop a 224×224 real RoI region on the image to be identified according to the region of interest of the cloud-based precision model. The region of interest for RoI cropping is determined based on the candidate bounding box.

[0045] Step S3 further includes that if the softmax probability is less than a second preset threshold, the image to be identified will no longer be identified.

[0046] The specific implementation of this step is as follows: If P_swin < 0.5, that is, less than the second preset threshold, it is directly judged as negative, and there is no need to upload it to the semantic arbitration model. The image to be recognized will no longer be recognized.

[0047] Example 3

[0048] This embodiment further supplements the description of the three-layer dynamic cascaded image recognition method proposed in Embodiment 1.

[0049] After image recognition using a three-layer model (L1 edge detection model, L2 cloud-based precision segmentation model, and L3 semantic arbitration model), the process also includes updating the L1 edge detection model and L2 cloud-based precision segmentation model using a minute-level weakly supervised backflow method, and correcting the preset dual threshold interval based on the weakly supervised backflow signal, as detailed below:

[0050] After outputting the fine-classification labels as the recognition results for the image to be recognized, the following is also included:

[0051] The fine classification labels are used as training parameters to train and update the edge detection model by online distillation. The KL divergence loss at the first preset temperature is used to train and update the edge detection model.

[0052] The specific implementation of this step is as follows: online distillation uses KL divergence loss at temperature T=4.0, and only updates the parameters of the detection head layer of the edge detection model.

[0053] After outputting the natural language conclusion as the recognition result for the image to be recognized, it also includes:

[0054] The natural language conclusions are used as training parameters to train and update the edge detection model and the cloud-based precision analysis model by using an online distillation method. The KL divergence loss at a second preset temperature is used to train and update the edge detection model and the cloud-based precision analysis model.

[0055] The specific implementation of this step is as follows: If P_LMM≥0.90, the natural language conclusion is used as a pseudo-label. The natural language conclusion is used as a training parameter to be input into the edge detection model and the cloud-based precision model for model training and updating. The KL divergence loss at temperature T=6.0 is used to jointly perform online distillation on the detection head layer of the edge detection model and the fully connected layer (FC layer) of the cloud-based precision model, accounting for ≤4% of the total parameters. If P_LMM<0.90, the manual review interface is triggered, and batch distillation is carried out the next day.

[0056] If the coarse classification label is compared with the fine classification label or the coarse classification label is compared with the natural language conclusion, and the comparison results are inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size.

[0057] The specific implementation of this step is as follows: The initial value of the preset dual threshold interval is... =0.35, =0.75; dynamically adjusted by cloud-based feedback signals (coarse classification labels and natural language conclusions) with a step size of 0.02; if a certain type of sample is frequently "misclassified" or "missed" by the marginal model, but the cloud or large model corrects the results, the system will adjust [ , The system adjusts the confidence level of an image based on a certain range to allow more or fewer samples to enter the cloud. For example, if the edge detection model sets the confidence level of the image to be identified to 0.4, the coarse classification label will show "abnormal," and the image to be identified will frequently enter the cloud. However, the L2 cloud-based fine-grained model or the L3 semantic arbitration model in the cloud will almost always determine it as "normal," meaning the fine classification label or natural language conclusion will show "normal." The system then performs an improvement based on a confidence level of 0.4. Values ​​(e.g., from 0.35 to 0.37) reduce invalid uploads; if the edge detection model sets the confidence level of the image to be identified to 0.8, and the coarse classification label shows "abnormal," then manual secondary identification of the coarse classification label is required. If the manual identification result is "normal," then the confidence level should be increased. This allows more high-confidence samples to be reviewed in the cloud.

[0058] As shown in Table 2, the image recognition false alarm rate under the three-layer dynamic cascade after model update using the weak supervision backflow method is ≤0.3% throughout the year, the open set accuracy is ≥89%, and the cloud call rate is ≤10%.

[0059] Table 2

[0060]

[0061] In practical applications, the L1 layer edge detection model in this embodiment uses the following hardware configuration: Jetson Orin Nano 8 GB high-performance AI development kit, Basler 1920×1080 30 fps industrial camera, and 4G module (for remote data transmission of the Jetson Orin Nano development kit); the L2 layer cloud-based precision model uses the following hardware configuration: Alibaba Cloud GPU-T4-2G server; and the L3 layer semantic arbitration model uses the following hardware configuration: NVIDIA A100-80G server.

[0062] As shown in Table 3, after 30 days of continuous operation, the total number of frames was 2.4 × 10. 7 The results are as follows:

[0063] Table 3

[0064]

[0065] Under constraints of bandwidth ≤ 8 KB, latency ≤ 90 ms, and false alarm rate ≤ 0.3%, it achieves open set recognition, semantic reasoning, and minute-level self-evolution, making it suitable for scenarios with high requirements for real-time performance and accuracy, such as industrial quality inspection, smart security, and vehicle-road cooperation.

Claims

1. A three-layer dynamic cascaded image recognition method, characterized in that, include: S1: Input the image to be identified into the edge detection model and output candidate boxes. The candidate boxes include coarse classification labels and first confidence scores. The edge detection model includes an INT8 quantized YOLOv8-nano model. S2: Determine whether the first confidence level is within the preset double threshold interval. If it is not within the preset double threshold interval, use the coarse classification label as the recognition result of the image to be recognized. If it is within the preset double threshold interval, input the image to be recognized into the intermediate layer feature map obtained by the edge detection model, perform Zlib compression and AES-128-GCM encryption in sequence, and then upload it to the cloud-based precision analysis model to output the fine classification label and softmax probability. The cloud-based precision analysis model includes the Swin-T model. S3: Determine whether the softmax probability is less than a first preset threshold. If it is not less than the first preset threshold, output the fine classification label as the recognition result of the image to be recognized. Compare the coarse classification label with the fine classification label. If the comparison result is inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size. If the value is less than the first preset threshold and greater than the second preset threshold, RoI cropping is performed on the image to be recognized to obtain the RoI cropped image. The RoI cropped image and the preset Prompt template are input into the semantic arbitration model, and the natural language conclusion and the second confidence score are output. If the softmax probability is less than the second preset threshold, the image to be recognized is no longer recognized. The first preset threshold is greater than the second preset threshold. The semantic arbitration model includes the Qwen-VL-72B model. S4: Determine whether the second confidence level is less than the third preset threshold. If the second confidence level is not less than the third preset threshold, output the natural language conclusion as the recognition result of the image to be recognized. Compare the coarse classification label with the natural language conclusion. If the comparison result is inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size. If the second confidence level is less than the third preset threshold, perform manual review.

2. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, The intermediate layer feature map is reconstructed into a 224×224-dimensional pseudo-map within the cloud-based precision model. The intermediate layer feature map includes 1×4200-dimensional float16 format Neck layer P5 features.

3. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, The region of interest for RoI cropping is determined based on the candidate bounding box.

4. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, After outputting the fine-classification labels as the recognition results for the image to be recognized, the following is also included: An online distillation method is used to input fine-class labels as training parameters into the edge detection model for model training and updating.

5. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, After outputting the natural language conclusion as the recognition result for the image to be recognized, it also includes: An online distillation method is used to input natural language conclusions as training parameters into the edge detection model and the cloud-based precision analysis model for model training and updating.

6. The three-layer dynamic cascaded image recognition method according to claim 4, characterized in that, The edge detection model is trained and updated using the KL divergence loss at the first preset temperature.

7. The three-layer dynamic cascaded image recognition method according to claim 5, characterized in that, The edge detection model and the cloud-based precision analysis model are trained and updated using the KL divergence loss at the second preset temperature.

Images