Three-layer dynamic cascade image recognition method

Abstract

The invention relates to a three-layer dynamic cascade image recognition method, which realizes high-precision image recognition under a three-layer dynamic cascade model and can recognize ambiguous scenes. The method comprises the steps that a to-be-recognized image is input into an edge detection model, a candidate box is obtained through output, and the candidate box comprises a coarse classification label and a first confidence coefficient; if the first confidence coefficient is not within a preset dual-threshold interval, outputting a coarse classification label, and if the first confidence coefficient is within the preset dual-threshold interval, inputting the image to be recognized into an edge detection model to obtain an intermediate layer feature map, uploading the intermediate layer feature map to a cloud fine classification model, and outputting a fine classification label and a softmax probability; if the softmax probability is not smaller than a first preset threshold value, outputting a fine classification label, and if the softmax probability is smaller than the first preset threshold value and larger than a second preset threshold value, performing RoI cutting on the to-be-recognized image, inputting an RoI cutting image and a preset Prompt template into a semantic arbitration model, and outputting a natural language conclusion and a second confidence coefficient; if the second confidence coefficient is not smaller than a third preset threshold value, outputting a natural language conclusion, and if the second confidence coefficient is smaller than the third preset threshold value, performing manual rechecking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a three-layer dynamic cascading image recognition method. BACKGROUND

[0002] In the current rapid development of image recognition technology, in order to ensure that it can play a role efficiently and stably in actual application scenarios, the demand for high precision, low delay and low bandwidth of image recognition is synchronous. The traditional scheme either deploys a light model on the edge, which leads to a high false alarm rate due to limited computing power, or uploads the entire image to the cloud, which causes bandwidth and delay to become a bottleneck. Existing patents attempt to use the "edge preliminary screening + cloud backup" mode, but generally have the following defects: the false alarm rate of the edge single model is high (>10%), which cannot meet the hard target of industrial quality inspection ≤2%; the whole image uploading scheme consumes a large amount of bandwidth (50-200KB / frame), and 4 routes of 1080p30 can support LTE uplink; static threshold or fixed interval fails when the scene changes (backlight, rain and fog, new target), resulting in a sharp increase in cloud calls; the offline full parameter update cycle is ≥1 week, which relies on manual labeling and cannot achieve minute-level adaptation to open set targets (new defects and new prohibited items not covered by the training library); the two-level cascade stops at "classification logits" and cannot handle ambiguous scenes that require semantics and common sense (such as "live working without warning signs").

[0003] After searching the existing technical literature, it was found that the patent with application number 202510025847.3 and the patent name "Classroom Teaching Quality Auxiliary Evaluation Method, Device and System Based on Cloud Edge Collaboration" acquires video image data of students in the classroom collected by a camera in real time and transmits it to an edge device; in the edge device, a pre-trained student classroom behavior recognition model is used to recognize and track the student classroom behavior of the received video image data, the recognition results of adjacent multiple frames are fused with the current frame recognition result to obtain the final recognition result of the current frame, the classroom teaching quality is evaluated according to the student classroom behavior recognition result, and the student classroom behavior recognition model is obtained by pre-training based on a multi-label object detection algorithm and using a knowledge distillation structure; the student classroom behavior recognition result and the evaluation result are sent to a cloud video server for storage. However, this patent has problems such as lack of semantic arbitration model and insufficient image recognition accuracy. SUMMARY

[0004] Therefore, it is necessary to provide a three-layer dynamic cascading image recognition method to achieve high-precision image recognition under a three-layer dynamic cascading model and to identify ambiguous scenes.

[0005] To achieve the above technical problems, the present application provides the following technical solutions: The application provides a three-layer dynamic cascade image recognition method, which comprises the following steps: S1: inputting an image to be recognized into an edge detection model to output a candidate box, wherein the candidate box comprises a coarse classification label and a first confidence, and the edge detection model comprises an INT8 quantized YOLOv8-nano model; S2: determining whether the first confidence is located in a preset double-threshold interval, if not, taking the coarse classification label as a recognition result of the image to be recognized, and if yes, uploading an intermediate layer feature map obtained by inputting the image to be recognized into the edge detection model to a cloud fine classification model to output a fine classification label and a softmax probability, wherein the cloud fine classification model comprises a Swin-T model; S3: determining whether the softmax probability is less than a first preset threshold, if not, outputting the fine classification label as the recognition result of the image to be recognized, if yes and greater than a second preset threshold, performing RoI cropping on the image to be recognized to obtain an RoI cropped image, inputting the RoI cropped image and a preset Prompt template into a semantic arbitration model to output a natural language conclusion and a second confidence, wherein the first preset threshold is greater than the second preset threshold, and the semantic arbitration model comprises a Qwen-VL-72B model; S4: determining whether the second confidence is less than a third preset threshold, if not, outputting the natural language conclusion as the recognition result of the image to be recognized, and if yes, performing manual review.

[0006] Preferably, the intermediate layer feature map is reconstructed into a 224x224-dimensional pseudo image in the cloud fine classification model, and the intermediate layer feature map comprises a 1x4200-dimensional float16 format Neck layer P5 feature.

[0007] Preferably, before the intermediate layer feature map obtained by inputting the image to be recognized into the edge detection model is uploaded to the cloud fine classification model, the intermediate layer feature map is further subjected to Zlib compression and AES-128-GCM encryption in sequence.

[0008] Preferably, the attention region of the RoI cropping is determined according to the candidate box.

[0009] Preferably, step S3 further comprises if the softmax probability is less than the second preset threshold, the image to be recognized is no longer subjected to recognition.

[0010] Preferably, after the fine classification label is outputted as the recognition result of the image to be recognized, the following steps are further included: The fine classification label is inputted as a training parameter into the edge detection model in an online distillation mode to perform model training and update.

[0011] Preferably, after outputting the natural language conclusion as the recognition result of the image to be recognized, the method further includes: An online distillation method is used to input natural language conclusions as training parameters into the edge detection model and the cloud-based precision analysis model for model training and updating.

[0012] Preferably, the edge detection model is trained and updated using the KL divergence loss at a first preset temperature.

[0013] Preferably, the edge detection model and the cloud-based precision model are trained and updated using the KL divergence loss at a second preset temperature.

[0014] Preferably, the coarse classification label is compared with the fine classification label or the coarse classification label is compared with the natural language conclusion. If the comparison results are inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a three-layer dynamic cascaded image recognition method. The image to be recognized is first detected by an edge detection model deployed on an edge device. If it can be recognized, a corresponding coarse classification label is obtained. If it cannot be recognized or the recognition is abnormal, it is uploaded to a cloud-based precision analysis model deployed in the cloud for detection. This realizes a mode of initial edge screening plus cloud-based fallback. Furthermore, for ambiguous scenarios that require semantic and common sense, a semantic arbitration model is added. This allows images with new targets, new defects, or new semantics to be further identified by the semantic arbitration model even if the cloud-based precision analysis model cannot recognize them, greatly reducing the false alarm rate and false negative rate of image recognition. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a three-layer dynamic cascaded image recognition method in one embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] Example 1 like Figure 1 As shown, this embodiment proposes a three-layer dynamic cascaded image recognition method, the method comprising: S1: Input the image to be identified into the edge detection model and output candidate boxes. The candidate boxes include coarse classification labels and first confidence scores. The edge detection model includes an INT8 quantized YOLOv8-nano model. In this step, the specific implementation is as follows: as shown in Table 1, the edge detection model deployed by the edge device in this embodiment adopts a YOLOv8-nano model quantized by INT8, the model is deployed on a Jetson Orin Nano development kit, the whole image to be recognized is input into the YOLOv8-nano model, and a candidate box is output, which includes a coarse classification label and a first confidence C∈[0,1]^K.

[0019] Table 1

[0020] S2: determining whether the first confidence is located in a preset double-threshold interval, if not, taking the coarse classification label as a recognition result of the image to be recognized, if yes, uploading the intermediate layer feature map obtained by inputting the image to be recognized into the edge detection model to a cloud fine classification model, and outputting a fine classification label and a softmax probability, the cloud fine classification model comprising a Swin-T model; In this step, the specific implementation is as follows: the preset double-threshold interval , ] = [0.35, 0.75], after obtaining the first confidence C∈[0,1]^K, it is determined whether the first confidence is located in the preset double-threshold interval, if max(C)≤ or ≥ , the edge detection model directly outputs the coarse classification label as the recognition result of the image to be recognized, if <max(C)< , the image to be recognized is input into the intermediate layer feature map obtained by the edge detection model and uploaded to the cloud fine classification model, the uploading mode is an MQTT protocol (running on a TLS 1.3 encryption protocol), the cloud fine classification model adopts a Swin-T model, the model is deployed on a GPU / TPU cluster, and the Swin-T model reconstructs the features into a pseudo image of 224×224 dimensions, and outputs a fine classification label y' and a softmax probability P_swin within 25 ms.

[0021] S3: determining whether the softmax probability is less than a first preset threshold, if not, outputting the fine classification label as the recognition result of the image to be recognized, if yes and greater than a second preset threshold, performing RoI cropping on the image to be recognized to obtain an RoI cropped image, inputting the RoI cropped image and a preset Prompt template into a semantic arbitration model, and outputting a natural language conclusion and a second confidence, the first preset threshold being greater than the second preset threshold, and the semantic arbitration model comprising a Qwen-VL-72B model; The implementation of this step is as follows: the softmax probability P_swin is compared with the preset threshold value, if P_swin is greater than or equal to 0.95, that is, not less than the first preset threshold value, the final decision is directly made, that is, the fine classification label y' is output as the recognition result of the image to be recognized, if 0.5≤P_swin<0.95, that is, less than the first preset threshold value and greater than the second preset threshold value, the RoI is cropped to obtain a 224x224 RoI cropped image, the RoI cropped image and the preset Prompt template are input into the semantic arbitration model, and a natural language conclusion and a second confidence P_LMM are output, the semantic arbitration model adopts a Qwen-VL-72B model, and can also adopt a GPT-4V model, an Intern-VL-40B model, etc., the model deployment position is a large model inference pool, and the preset Prompt template is as follows: Please judge whether the {category} in the figure is normal. Only answer normal / abnormal + brief reason, confidence 0-1.

[0022] S4: determining whether the second confidence is less than a third preset threshold value, if not, outputting the natural language conclusion as the recognition result of the image to be recognized, if yes, performing manual review.

[0023] The implementation of this step is as follows: the second confidence P_LMM is compared with the preset threshold value, if P_LMM is greater than or equal to 0.90, the natural language conclusion output by the semantic arbitration model is taken as a pseudo label as the recognition result of the image to be recognized, if P_LMM is less than 0.90, an artificial review interface is triggered.

[0024] Embodiment 2 This embodiment further supplements the three-layer dynamic cascading image recognition method proposed in embodiment 1.

[0025] The intermediate layer feature map is reconstructed into a 224x224 pseudo image in the cloud fine segmentation model, and the intermediate layer feature map includes a 1x4200 float16 format Neck layer P5 feature.

[0026] Before uploading the intermediate layer feature map obtained by inputting the image to be recognized into the cloud fine segmentation model, the intermediate layer feature map is further subjected to Zlib compression and AES-128-GCM encryption, and the encrypted data amount is less than or equal to 8KB.

[0027] The RoI cropped region of interest is determined according to the candidate box.

[0028] The implementation of this step is as follows: the system will re-cut a 224x224 real RoI region on the image to be recognized according to the region of interest of the cloud fine segmentation model, and the RoI cropped region of interest is determined according to the candidate box.

[0029] The step S3 further comprises: if the softmax probability is less than a second preset threshold, no longer performing identification on the image to be identified.

[0030] The specific implementation of this step is as follows: if P_swin<0.5, i.e., less than the second preset threshold, directly judging as negative, without uploading to the semantic arbitration model, and no longer performing identification on the image to be identified.

[0031] Embodiment 3 This embodiment further supplements the three-layer dynamic cascading image identification method proposed in Embodiment 1.

[0032] After image identification by the three-layer model of the L1 layer edge detection model, the L2 layer cloud fine segmentation model, and the L3 layer semantic arbitration model, the method further comprises: using a minute-level weak supervision backflow mode to update the L1 layer edge detection model and the L2 layer cloud fine segmentation model, and correcting the preset double threshold interval according to the weak supervision backflow signal, specifically as follows: After outputting the fine classification label as the identification result of the image to be identified, the method further comprises: The fine classification label is input as a training parameter into the edge detection model for model training and updating by using an online distillation mode, and the edge detection model is trained and updated by using a KL divergence loss of a first preset temperature.

[0033] The specific implementation of this step is as follows: the online distillation uses a KL divergence loss of a temperature T=4.0, and only updates the detection head layer (Detect Head layer) parameters of the edge detection model.

[0034] After outputting the natural language conclusion as the identification result of the image to be identified, the method further comprises: The natural language conclusion is input as a training parameter into the edge detection model and the cloud fine segmentation model for model training and updating by using an online distillation mode, and the edge detection model and the cloud fine segmentation model are trained and updated by using a KL divergence loss of a second preset temperature.

[0035] The specific implementation of this step is as follows: if P_LMM≥0.90, the natural language conclusion is used as a pseudo label, the natural language conclusion is input as a training parameter into the edge detection model and the cloud fine segmentation model for model training and updating by using an online distillation mode, and the KL divergence loss of a temperature T=6.0 is used to jointly perform online distillation on the detection head layer (Detect Head layer) of the edge detection model and the fully connected layer (FC layer) of the cloud fine segmentation model, and the total parameters are ≤4%, if P_LMM<0.90, an artificial review interface is triggered, and batch distillation is performed the next day.

[0036] If the coarse classification label is compared with the fine classification label or the coarse classification label is compared with the natural language conclusion, and the comparison results are inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size.

[0037] The specific implementation of this step is as follows: The initial value of the preset dual threshold interval is... =0.35, =0.75; dynamically adjusted by cloud-based feedback signals (coarse classification labels and natural language conclusions) with a step size of 0.02; if a certain type of sample is frequently "misclassified" or "missed" by the marginal model, but the cloud or large model corrects the results, the system will adjust [ , The system adjusts the confidence level of an image based on a certain range to allow more or fewer samples to enter the cloud. For example, if the edge detection model sets the confidence level of the image to be identified to 0.4, the coarse classification label will show "abnormal," and the image to be identified will frequently enter the cloud. However, the L2 cloud-based fine-grained model or the L3 semantic arbitration model in the cloud will almost always determine it as "normal," meaning the fine classification label or natural language conclusion will show "normal." The system then performs an improvement based on a confidence level of 0.4. Values ​​(e.g., from 0.35 to 0.37) reduce invalid uploads; if the edge detection model sets the confidence level of the image to be identified to 0.8, and the coarse classification label shows "abnormal," then manual secondary identification of the coarse classification label is required. If the manual identification result is "normal," then the confidence level should be increased. This allows more high-confidence samples to be reviewed in the cloud.

[0038] As shown in Table 2, the image recognition false alarm rate under the three-layer dynamic cascade after model update using the weak supervision backflow method is ≤0.3% throughout the year, the open set accuracy is ≥89%, and the cloud call rate is ≤10%.

[0039] Table 2

[0040] In practical applications, the L1 layer edge detection model in this embodiment uses the following hardware configuration: Jetson Orin Nano 8 GB high-performance AI development kit, Basler 1920×1080 30 fps industrial camera, and 4G module (for remote data transmission of the Jetson Orin Nano development kit); the L2 layer cloud-based precision model uses the following hardware configuration: Alibaba Cloud GPU-T4-2G server; and the L3 layer semantic arbitration model uses the following hardware configuration: NVIDIA A100-80G server.

[0041] As shown in Table 3, after 30 days of continuous operation, the total number of frames was 2.4 × 10. 7 The results are as follows: Table 3

[0042] Under the constraints of bandwidth ≤ 8 KB, delay ≤ 90 ms, and false positive rate ≤ 0.3%, open set recognition, semantic inference, and minute-level self-evolution are achieved, which are suitable for industrial quality inspection, intelligent security, vehicle-road cooperation and other scenarios with high requirements for real-time performance and accuracy.

Claims

1. A three-layer dynamic cascaded image recognition method, characterized in that, include: S1: Input the image to be identified into the edge detection model and output candidate boxes. The candidate boxes include coarse classification labels and first confidence scores. The edge detection model includes an INT8 quantized YOLOv8-nano model. S2: Determine whether the first confidence level is within the preset double threshold interval. If it is not within the preset double threshold interval, use the coarse classification label as the recognition result of the image to be recognized. If it is within the preset double threshold interval, input the image to be recognized into the intermediate layer feature map obtained by the edge detection model and upload it to the cloud-based precision analysis model to output the fine classification label and softmax probability. The cloud-based precision analysis model includes the Swin-T model. S3: Determine whether the softmax probability is less than a first preset threshold. If it is not less than the first preset threshold, output the fine classification label as the recognition result of the image to be recognized. If it is less than the first preset threshold and greater than the second preset threshold, perform RoI cropping on the image to be recognized to obtain the RoI cropped image. Input the RoI cropped image and the preset Prompt template into the semantic arbitration model, and output the natural language conclusion and the second confidence. The first preset threshold is greater than the second preset threshold. The semantic arbitration model includes the Qwen-VL-72B model. S4: Determine whether the second confidence level is less than the third preset threshold. If it is not less than the third preset threshold, output a natural language conclusion as the recognition result of the image to be recognized. If it is less than the third preset threshold, perform manual review.

2. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, The intermediate layer feature map is reconstructed into a 224×224-dimensional pseudo-map within the cloud-based precision model. The intermediate layer feature map includes 1×4200-dimensional float16 format Neck layer P5 features.

3. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, Before uploading the intermediate layer feature map obtained by inputting the image to be identified into the edge detection model to the cloud-based precision analysis model, the process also includes: performing Zlib compression and AES-128-GCM encryption on the intermediate layer feature map in sequence.

4. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, The region of interest for RoI cropping is determined based on the candidate bounding box.

5. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, Step S3 further includes that if the softmax probability is less than a second preset threshold, the image to be identified will no longer be identified.

6. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, After outputting the fine-classification labels as the recognition results for the image to be recognized, the following is also included: An online distillation method is used to input the fine-class labels as training parameters into the edge detection model for model training and updating.

7. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, After outputting the natural language conclusion as the recognition result for the image to be recognized, it also includes: An online distillation method is used to input natural language conclusions as training parameters into the edge detection model and the cloud-based precision analysis model for model training and updating.

8. The three-layer dynamic cascaded image recognition method according to claim 6, characterized in that, The edge detection model is trained and updated using the KL divergence loss at the first preset temperature.

9. The three-layer dynamic cascaded image recognition method according to claim 7, characterized in that, The edge detection model and the cloud-based precision analysis model are trained and updated using the KL divergence loss at the second preset temperature.

10. The three-layer dynamic cascaded image recognition method according to claim 1, characterized in that, If the coarse classification label is compared with the fine classification label or the coarse classification label is compared with the natural language conclusion, and the comparison results are inconsistent, the preset dual threshold interval is dynamically corrected according to the first confidence level with a preset step size.

Images