Target detection optimization method based on multi-modal large model
By combining the small object detection model with multimodal large model, fast edge screening and cloud-based secondary reasoning, the performance instability and false detection of the object detection model in new scenarios is solved, and efficient object detection optimization is achieved.
Patent Information
- Application Number
- CN202510237659.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-03
- Publication Date
- 2025-08-08
AI Technical Summary
The existing target detection model has unstable performance when migrating new scenarios, and its dependence on texture feature recognition leads to high error detection rates. Multimodal large models lack target positioning capabilities and are difficult to directly apply to target detection.
Combining the object detection small model and multimodal large model, lightweight object detection models are deployed at the edge to quickly filter, multimodal large model is deployed in the cloud for secondary reasoning, and using the powerful scene understanding ability of multimodal large model, false detection and filtering is performed by introducing coordinate information of the object detection box.
It improves the overall performance of target detection, reduces the error detection rate, improves the detection accuracy and response speed of the system, and significantly improves the detection effect in complex scenarios.
Smart Images

Figure CN120451610A_ABST
Abstract
Description
Technical Field
[0001] The present invention is applied to the field of target detection, and specifically is a target detection optimization method based on a multimodal large model. Background Art
[0002] The object detection task involves taking a given input image, processing it through an algorithm, and outputting a rectangular box representing the coordinate positions of all specified objects contained in the image. The processing flow generally involves collecting a large number of training images and labeling the position coordinates and categories of all objects contained in the image. Deep learning-based object detection models are relatively mainstream, and generally include two-stage methods, single-stage methods, anchor-free methods, and transformer-based methods. The advantages of this object detection model are its lightweight and fast computational speed, but there are several typical problems: the model's generalization performance depends on the richness and volume of the training data, and its performance is particularly unstable when migrating to new scenes; the network lacks reasoning capabilities and relies solely on target texture features for detection and recognition, making it difficult to eliminate falsely detected targets with similar textures.
[0003] The mainstream framework of large multimodal models consists of four parts: a modal encoder, an adapter, a large language model, and a modal decoder. The modal encoder and modal decoder are designed for specific modal data, such as images and audio. Large multimodal models have strong cross-modal understanding capabilities. They can combine image content with text descriptions to enhance the output of visual tasks. Large multimodal models effectively improve the model's representational capabilities through joint learning of large-scale data. This pre-training method enables the model to demonstrate stronger generalization capabilities when faced with complex and changing application scenarios. However, the current mainstream large multimodal models have lower target positioning capabilities than small target detection models, suffer from significant false detection and missed detection issues, and are difficult to directly apply to solve target detection problems.
[0004] Traditional target detection algorithms (such as Faster RCNN, YOLO, SSD, etc.) usually have a relatively small model size and a limited number of parameters, which makes it impossible for such methods to effectively learn sufficiently rich data features when processing complex scenes and diversified data; the methods mainly rely on manually designed feature extractors or predefined anchor boxes, which have limited data expression capabilities, unstable model performance when migrating to new scenes, and weak generalization performance. In response to the above problems, the present invention proposes a method that, based on the technical architecture of a multimodal large model, additionally introduces the output value of the target detection model as input for comprehensive reasoning. This method can fully utilize the powerful reasoning and scene understanding capabilities of the large model, make up for the limitation of target detection only for texture feature recognition, and at the same time solve the lack of positioning capabilities of the multimodal large model. The overall model deployment method is that the target detection small model has a small amount of computation and is deployed on the edge of a limited computing power platform, giving priority to ensuring the detection rate of the system and achieving the purpose of quickly and efficiently filtering invalid data. The preliminarily screened data is uploaded to the cloud in a timely manner, and reasoning is performed through the modified multimodal large model to further delete the false detection targets, thereby achieving an improvement in the overall target detection performance.
[0005] To address these issues, we propose a method based on a large multimodal model to optimize the performance of target detection algorithms. Building on the output of a small target detection model and combining it with the representational power of a large multimodal model, we can further eliminate falsely detected targets with similar textures. For example, for fireworks detection tasks, this method can effectively filter out falsely detected targets such as car lights and red flags that are clearly not fireworks. These falsely detected targets are characterized by being close to the fireworks targets in high-dimensional space based on texture features, but significantly different in semantic space. The combination of a small target detection model and a large multimodal model effectively leverages the strengths of both to comprehensively improve the performance of detection tasks. Summary of the Invention
[0006] The technical problem to be solved by this invention is to address the shortcomings of the existing technology and provide a target detection optimization method based on a multimodal large model. This method is based on the multimodal large model architecture and combines the output of the target detection small model to further optimize the target detection accuracy. This method is applicable to any target detection network structure, can fully utilize the computing power resources of the cloud and edge, and use the powerful scene understanding and reasoning capabilities of the multimodal large model to make up for the shortcomings of the traditional target detection small model that relies on texture features for detection and positioning.
[0007] To solve the above technical problems, the present invention provides an object detection optimization method based on a multimodal large model, which specifically includes the following steps:
[0008] Get video stream data;
[0009] The image after the video stream is extracted is input into the preset target detection model, and the coordinate information of the target detection box contained in the image is output;
[0010] If the confidence of the target detection frame is greater than the preset value, the current image and the target detection frame coordinate information are uploaded to the multimodal large model;
[0011] The multimodal large model receives coordinate information, image information, and text prompt words, and determines and outputs whether the current image contains the specified target.
[0012] As a possible implementation method, the following steps are further included: if the multimodal large model determines that the output is included, the final alarm is considered to be a true alarm and is pushed to the business system for display; if the multimodal large model determines that the output is not included, the final alarm is considered to be a false alarm and is not pushed.
[0013] As a possible implementation, further, the multimodal large model annotation method includes at least one of the following:
[0014] Provide a detailed description of the image content, with the prompt being "Please describe the content of the image";
[0015] Describe the objects contained in the image. The prompt is "Please list the objects contained in the image in order."
[0016] Option question and answer annotation, the prompt is "Please determine whether the image contains the preset target."
[0017] As a possible implementation method, further, the interaction requires the multimodal large model to perform option question and answer annotation, determine whether the image contains the preset target step, pre-annotate the image through the target detection model and the open source multimodal large model, list a limited number of target types to be identified {C1, C2, ..., Cm}, and then undergo manual review.
[0018] As a possible implementation method, the multimodal large model is further modified based on the Qwen2-VL multimodal large model, specifically including:
[0019] The detection frame coordinates calculated at the edge end are converted into position coding features through the position encoder;
[0020] The image is converted into image features through the image encoder and adapted to the language model space through the adapter;
[0021] The position encoding features, image encoding features, and text tag features are concatenated and sent to the language model for inference decoding to output the final option answer.
[0022] As a possible implementation method, further, the multimodal large model is fine-tuned and trained using the Lora method, and only the language large model part is fine-tuned.
[0023] As a possible implementation method, the target detection model is further deployed at the edge to quickly calculate and filter invalid data, thereby reducing the frequency of calling the multimodal large model.
[0024] As a possible implementation method, further, a multimodal large model is deployed in the cloud to receive images, target detection frame coordinate information and text prompts uploaded by the edge, perform secondary calculations and output judgment results.
[0025] As a possible implementation method, further, the detection frame coordinates are mapped to feature points using a Fourier method to generate position coding features.
[0026] As a possible implementation, further, the target detection model input is an image, and the output is the coordinates and confidence information of the specified target in the image, expressed in the form of x, y, w, h, conf, where x and y are the relative coordinate values of the center point of the target box, w and h are the width and height of the target box, and conf is the probability that the model considers the corresponding coordinates to be the specified target.
[0027] The present invention adopts the above technical solution and has the following beneficial effects:
[0028] 1. The coordinate output of target detection is introduced as an additional input to the multimodal large model. The powerful representation capabilities of the multimodal large model and the better detection capabilities of the small target detection model are comprehensively utilized to improve the overall system detection performance.
[0029] 2. The small target detection model is deployed on the edge, which has low computing power requirements. It can quickly calculate and filter invalid data, and then feed it into the multimodal large model for calculation, greatly reducing the call frequency of the large model and ensuring high concurrency of the overall system.
[0030] 2. The present invention has no special restrictions on target detection methods and is highly flexible. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0032] Figure 1 Schematic diagram of the process of an embodiment of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0034] Example 1
[0035] A target detection optimization method based on a multimodal large model specifically includes the following steps:
[0036] Get video stream data;
[0037] The image after the video stream is extracted is input into the preset target detection model, and the coordinate information of the target detection box contained in the image is output;
[0038] If the confidence of the target detection frame is greater than the preset value, the current image and the target detection frame coordinate information are uploaded to the multimodal large model;
[0039] The multimodal large model receives coordinate information, image information, and text prompt words, and determines and outputs whether the current image contains the specified target.
[0040] If the multimodal large model determines that the output contains it, the final alarm is considered a true alarm and is pushed to the business system for display; if the multimodal large model determines that the output does not contain it, the final alarm is considered a false alarm and is not pushed.
[0041] The multimodal large model receives coordinate information, image information, and text prompt words, and determines and outputs whether the current image contains the multimodal large model annotation method text prompt words in the specified target step, including at least:
[0042] Interaction requires a large multimodal model to describe and annotate image content in detail;
[0043] Interaction requires a large multimodal model to enumerate and describe the objects contained in the image;
[0044] The interaction requires a large multimodal model to perform option question and answer annotation to determine whether the image contains the preset target.
[0045] The interaction requires a large multimodal model to annotate the question and answer options and determine whether the image contains the preset target. The image is pre-annotated through the target detection model and the open source multimodal large model, and a limited number of target types to be identified {C1, C2, ..., Cm} are listed, which are then manually reviewed.
[0046] The multimodal large model is modified based on the Qwen2-VL multimodal large model, specifically including:
[0047] The detection frame coordinates calculated at the edge end are converted into position coding features through the position encoder;
[0048] The image is converted into image features through the image encoder and adapted to the language model space through the adapter;
[0049] The position encoding features, image encoding features, and text tag features are concatenated and sent to the language model for inference decoding to output the final option answer.
[0050] The multimodal large model is fine-tuned using the Lora method, and only the language large model part is fine-tuned.
[0051] The target detection model is deployed at the edge to quickly calculate and filter invalid data, reducing the frequency of calling large multimodal models.
[0052] The multimodal large model is deployed in the cloud to receive images, target detection box coordinate information, and text prompts uploaded by the edge, perform secondary calculations, and output judgment results.
[0053] The detection frame coordinates are mapped to feature points using the Fourier method to generate position encoding features.
[0054] The target detection model takes an image as input and outputs the coordinates and confidence information of the specified target in the image, expressed in the form of x, y, w, h, conf, where x and y are the relative coordinate values of the center point of the target box, w and h are the width and height of the target box, and conf is the probability that the model believes that the corresponding coordinates are the specified target.
[0055] Example 2
[0056] like Figure 1 As shown, the present invention provides a method for optimizing target detection performance based on a cloud-edge system by combining a large multimodal model with a small target detection model. The steps are briefly as follows:
[0057] 1. Pull camera video stream data through the edge gateway
[0058] 2. After the video stream is framed, the image is input into the lightweight target detection model deployed by the edge gateway, and the coordinate information of the target detection box contained in the output image is
[0059] 3. If the confidence of the target detection frame is greater than the preset value, the current image and the target detection frame coordinate information will be uploaded to the cloud for secondary calculation.
[0060] 4. A large multimodal model is deployed in the cloud. It receives coordinate information, image information, and text prompts calculated by the edge, and outputs an answer as to whether the current image contains the specified target. The answer is either "yes" or "no." If the answer is "yes," the final alarm is considered true and pushed to the business system for display. If the answer is "no," the final alarm is considered false and not pushed.
[0061] The following is a detailed description of the process:
[0062] Edge object detection model inference module:
[0063] This patent is not targeted at a specific target detection task, such as pedestrian detection, fireworks detection, etc., and the target detection network design is not limited, such as YOLOX, YOLOV6, etc. The target detection network input is an image, and the output is the coordinates and confidence information of the specified target in the image, expressed in the form of (x, y, w, h, conf), where x, y are the relative coordinate values of the center point of the target box, w, h are the width and height of the target box, and conf is the probability that the model believes the corresponding coordinates are the specified target.
[0064] The current image I and the output results of the target detection model (x, y, w, h, conf) are sent to the cloud for secondary calculation.
[0065] Cloud-based large model inference module:
[0066] Image data is collected and annotated using three question-and-answer methods: the first provides a detailed description of the image content, with the prompt "Please describe the content of the image"; the second describes the objects contained in the image, with the prompt "Please list the objects contained in the image in order"; and the third uses a multiple-choice question-and-answer method, with the prompt "Please determine whether the image contains object X. Simply answer [yes] or [no] without further explanation," where X can be any object. For the third type of question-and-answer annotation, images are pre-annotated using an object detection model and an open-source multimodal large model, listing a finite number of target object types {C1, C2, ..., Cm}. This is then manually reviewed to reduce annotation costs. The third type of annotation is automatically generated from the second type of annotation, meaning that m different question-and-answer pairs are generated for each image.
[0067] The present invention is based on the Qwen2-VL multimodal large model for transformation, wherein the detection frame coordinates (x, y, w, h) calculated at the edge end are converted into position coding features through a position encoder, and image I is converted into image features through an image encoder, and is adapted to the language large model space through an adapter. The three types of information, position coding features, image coding features, and text mark features, are spliced and sent to the language large model for inference decoding. The model outputs the final option answer: included, not included. The detection coordinates (x, y, w, h) are mapped to feature points using the Fourier method to generate position coding features.
[0068] The large model is fine-tuned using the data annotated in S21, using the Lora method to fine-tune the training, and only the language large model is fine-tuned. This step results in a fine-tuned large model, and then proceeds to step S25 to deploy inference.
[0069] The detection frame coordinates, image, and prompt "Please determine whether the image contains target X. Answer [Contains] or [Does not contain] without further explanation. The coordinates (xi, yi, wi, hi) may contain target X" obtained in step S11, where (xi, yi, wi, hi) are the coordinates of the i-th target in the image, are fed as input to the multimodal large model obtained by fine-tuning in step S23. X is set according to the specific task. For example, if the target task is fireworks detection, X = fireworks.
[0070] The large model outputs the [Include] option, and the alarm data is eventually pushed to the business system; the large model outputs the [Not Included] option, and the alarm is not pushed to the business system.
[0071] Example 3
[0072] A smoke and fire detection optimization method based on multimodal large model
[0073] This example describes a method for optimizing fireworks detection based on a multimodal large-scale model. The system architecture consists of two parts: the edge and the cloud. A lightweight object detection model is deployed on the edge to rapidly process video stream data and initially filter out images that may contain fireworks. A multimodal large-scale model is deployed on the cloud to perform secondary inference on images uploaded by the edge, further filtering out falsely detected objects and improving detection accuracy.
[0074] Edge object detection module
[0075] Video stream data acquisition: Pull video stream data from the camera in real time through the edge gateway.
[0076] Video stream frame extraction: Extract frames from the video stream at a fixed frame rate (such as 1 frame per second) to generate an image sequence.
[0077] Object detection model inference: The extracted image is fed into a lightweight object detection model (such as YOLOv6) deployed on the edge. The object detection model outputs the coordinates (x, y, w, h, conf) of the object bounding box detected in the image, where (x, y) represents the center coordinates of the object bounding box, (w, h) represents the width and height of the object bounding box, and conf represents the confidence level.
[0078] Preliminary screening: If the confidence of the target detection box is greater than the preset threshold (such as 0.7), the current image and target box coordinate information are uploaded to the cloud for secondary calculation.
[0079] Cloud-based multimodal large model inference module
[0080] Data reception and preprocessing: The cloud receives images and target bounding box coordinates uploaded by the edge. The target bounding box coordinates are mapped to feature points using Fourier transform to generate position encoding features. The image is converted into image features using an image encoder. The text prompt (e.g., "Please determine whether the image contains fireworks. Answer only [includes] or [does not contain]") is converted into text tag features using a text encoder.
[0081] Multimodal feature fusion: After splicing the position encoding features, image encoding features and text tag features, they are input into a fine-tuned multimodal large model (based on the Qwen2-VL model) for inference.
[0082] Model inference and output: The multimodal large model infers based on input features and outputs two options: "Contains" or "Does Not Contain." If the output is "Contains," the image is considered to contain fireworks and an alert is sent to the business system. If the output is "Does Not Contain," the image is considered to contain no fireworks and no alert is sent.
[0083] Model training and fine-tuning
[0084] Data annotation: To train a large multimodal model, the image data must be annotated first. The annotation methods include:
[0085] Image content description: The prompt is "Please describe the content in the image", which is used to generate a detailed description of the image content.
[0086] Target list description: The prompt is "Please list the targets contained in the image in order", which is used to generate a list of all targets in the image.
[0087] Option question-answer annotation: The prompt is "Please determine whether the image contains fireworks targets. Answer only [contains] or [does not contain]", which is used to generate binary classification question-answer pairs.
[0088] Model fine-tuning: Use labeled data to fine-tune the multimodal large model, and use the Lora method to fine-tune only the language part of the large model to reduce training costs and improve model performance.
[0089] 5. System performance optimization
[0090] Collaboration between the edge and the cloud: The target detection model at the edge is responsible for quickly filtering out invalid data, reducing the frequency of calling large multimodal models on the cloud, and improving the overall concurrent processing capabilities of the system.
[0091] Optimizing the positioning capability of the multimodal large model: By introducing the coordinate information of the target detection box, the multimodal large model can better understand the location information of the target in the image, improve positioning accuracy, and reduce false detections and missed detections.
[0092] Experimental results
[0093] Improved detection accuracy: In fireworks detection tasks, traditional object detection models have a high false positive rate, especially in scenarios with objects with similar textures, such as car lights and red flags. By introducing a large multimodal model for secondary inference, the false positive rate is significantly reduced, and detection accuracy is improved by 15%.
[0094] System response speed: Because the edge-side target detection model can quickly filter out invalid data, the overall response speed of the system has increased by 20%. Especially in high-concurrency scenarios, the system can still maintain a high processing efficiency.
[0095] Example 4
[0096] Pedestrian Detection Optimization Method Based on Multimodal Large Model
[0097] This example describes a pedestrian detection optimization method based on a multimodal large model. The system architecture is similar to that of Example 1, consisting of two components: the edge and the cloud. A lightweight object detection model is deployed on the edge to rapidly process video stream data and initially filter out images that may contain pedestrians. A multimodal large model is deployed on the cloud to perform secondary inference on images uploaded by the edge, further filtering out falsely detected objects and improving detection accuracy.
[0098] Edge object detection module
[0099] Video stream data acquisition: Pull video stream data from the camera in real time through the edge gateway.
[0100] Video stream frame extraction: Extract frames from the video stream at a fixed frame rate (such as 1 frame per second) to generate an image sequence.
[0101] Object detection model inference: The extracted image is fed into a lightweight object detection model deployed on the edge (such as YOLOX). The object detection model outputs the coordinates (x, y, w, h, conf) of the object bounding box detected in the image, where (x, y) represents the center coordinates of the object bounding box, (w, h) represents the width and height of the object bounding box, and conf represents the confidence level.
[0102] Preliminary screening: If the confidence of the target detection box is greater than the preset threshold (such as 0.6), the current image and target box coordinate information are uploaded to the cloud for secondary calculation.
[0103] Cloud-based multimodal large model inference module
[0104] Data reception and preprocessing: The cloud receives images and target bounding box coordinates uploaded by the edge. The target bounding box coordinates are mapped to feature points using Fourier transform to generate position encoding features. The image is converted into image features using an image encoder. The text prompt (e.g., "Please determine whether the image contains a pedestrian target. Answer only [includes] or [does not contain]") is converted into text tag features using a text encoder.
[0105] Multimodal feature fusion: After splicing the position encoding features, image encoding features and text tag features, they are input into a fine-tuned multimodal large model (based on the Qwen2-VL model) for inference.
[0106] Model reasoning and output: The multimodal large model infers based on input features and outputs two options: "Contains" or "Does not contain." If the output is "Contains," the image is considered to contain a pedestrian target and an alert is sent to the business system. If the output is "Does not contain," the image is considered to contain no pedestrian target and no alert is sent.
[0107] Model training and fine-tuning
[0108] Data annotation: To train a large multimodal model, the image data must be annotated first. The annotation methods include:
[0109] Image content description: The prompt is "Please describe the content in the image", which is used to generate a detailed description of the image content.
[0110] Target list description: The prompt is "Please list the targets contained in the image in order", which is used to generate a list of all targets in the image.
[0111] Option question-answer annotation: The prompt is "Please determine whether the image contains pedestrian targets. Answer only [contains] or [does not contain]", which is used to generate binary classification question-answer pairs.
[0112] Model fine-tuning: Use labeled data to fine-tune the multimodal large model, and use the Lora method to fine-tune only the language part of the large model to reduce training costs and improve model performance.
[0113] System performance optimization
[0114] Collaboration between the edge and the cloud: The target detection model at the edge is responsible for quickly filtering out invalid data, reducing the frequency of calling large multimodal models on the cloud, and improving the overall concurrent processing capabilities of the system.
[0115] Optimizing the positioning capability of the multimodal large model: By introducing the coordinate information of the target detection box, the multimodal large model can better understand the location information of the target in the image, improve positioning accuracy, and reduce false detections and missed detections.
[0116] Experimental results
[0117] Improved detection accuracy: In pedestrian detection tasks, traditional object detection models have a high false positive rate, especially in complex backgrounds (such as trees and billboards). By introducing a large multimodal model for secondary inference, the false positive rate is significantly reduced, and detection accuracy is improved by 12%.
[0118] System response speed: Because the edge-side target detection model can quickly filter out invalid data, the overall system response speed has increased by 18%. Especially in high-concurrency scenarios, the system can still maintain a high processing efficiency.
[0119] The above are embodiments of the present invention. For ordinary technicians in this field, based on the teachings of the present invention, all equivalent changes, modifications, substitutions and variations made within the scope of the patent application of the present invention without departing from the principles and spirit of the present invention should be covered by the scope of the present invention.
Claims
1. A target detection optimization method based on a multimodal large model, characterized in that: The specific steps include: Get video stream data; The image after the video stream is extracted is input into the preset target detection model, and the coordinate information of the target detection box contained in the image is output; If the confidence of the target detection frame is greater than the preset value, the current image and the target detection frame coordinate information are uploaded to the multimodal large model; The multimodal large model receives target detection frame coordinate information, image information, and text prompt words, and determines and outputs whether the current image contains the specified target.
2. The target detection optimization method based on a multimodal large model according to claim 1, characterized in that: It also includes the following steps: if the multimodal large model determines that the output contains the specified target, the final alarm is considered a true alarm and is pushed to the business system for display; if the multimodal large model determines that the output does not contain the specified target, the final alarm is considered a false alarm and is not pushed.
3. The target detection optimization method based on a multimodal large model according to claim 1, characterized in that: The multimodal large model is modified based on the Qwen2-VL multimodal large model, specifically including: The detection frame coordinates calculated at the edge end are converted into position coding features through the position encoder; The image is converted into image features through the image encoder and adapted to the language model space through the adapter; The position encoding features, image encoding features, and text tag features are concatenated and sent to the language model for inference decoding to output the final option answer.
4. The target detection optimization method based on a multimodal large model according to claim 3, characterized in that: The multimodal large model annotation method includes at least one of the following: Provide a detailed description of the image content, with the prompt being "Please describe the content of the image"; Describe the objects contained in the image. The prompt is "Please list the objects contained in the image in order." Option question and answer annotation, the prompt is "Please determine whether the image contains the preset target." 5. The target detection optimization method based on a multimodal large model according to claim 1 or 3, characterized in that: The interaction requires a large multimodal model to perform option question-answer annotation, determine whether the image contains a preset target step, pre-annotate the image through a target detection model and an open source multimodal large model, list a limited number of target types to be identified {C1, C2, ..., Cm}, and then undergo manual review.
6. The target detection optimization method based on a multimodal large model according to claim 1 or 3, characterized in that: The multimodal large model is fine-tuned and trained using the Lora method, and only the language large model part is fine-tuned.
7. The target detection optimization method based on a multimodal large model according to claim 1, characterized in that: The target detection model is deployed at the edge to quickly calculate and filter invalid data, reducing the frequency of calling the multimodal large model.
8. The target detection optimization method based on a multimodal large model according to claim 1, characterized in that: The multimodal large model is deployed in the cloud and is used to receive images, target detection frame coordinate information and text prompts uploaded by the edge end, perform secondary calculations and output judgment results.
9. The object detection optimization method based on a multimodal large model according to claim 1, characterized in that: The detection frame coordinates are mapped to feature points using a Fourier method to generate position coding features.
10. The target detection optimization method based on a multimodal large model according to claim 1, characterized in that: The target detection model takes an image as input and outputs the coordinates and confidence information of the specified target in the image, expressed in the form of x, y, w, h, conf, where x and y are the relative coordinate values of the center point of the target box, w and h are the width and height of the target box, and conf is the probability that the model believes that the corresponding coordinates are the specified target.
Citation Information
Cited By
Cloud-edge collaborative abnormal behavior character recognition method, device and system
CN121121872A
Multi-mode campus monitoring method and system based on target preprocessing
CN121259697A
Monitoring video analysis method and system
CN121505524A
Image positioning calibration method and device, electronic equipment and computer readable medium
CN121937535A