A collaborative reasoning method for large-scale vision-language models in cloud-edge systems

Abstract

This invention discloses a large-scale visual-language model collaborative inference method for cloud-edge-device systems. Its features include: deploying a high-performance cloud-based LVLM model and a cloud-based archive in the cloud; deploying an edge-based LVLM model and an edge-based archive at the edge, wherein the edge LVLM model incorporates a retrieval-enhanced generation algorithm, and the edge archive and the cloud archive can dynamically interact in terms of knowledge; a task scheduler evaluates the complexity of received query tasks based on a task allocation algorithm, and allocates the query tasks to the edge for local processing or to the cloud for processing based on the complexity evaluation results. This invention can improve the inference efficiency within cloud-edge-device systems while maintaining high accuracy, effectively solving the technical problem of low inference efficiency caused by limited computing resources, intermittent communication windows, and bandwidth limitations of edge devices.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method for collaborative reasoning of large-scale visual-language models for cloud-edge systems. Background Technology

[0002] The rapid proliferation of distributed intelligent devices in recent years stems from the ever-growing demand for real-time sensing and intelligent analysis, bringing unprecedented opportunities for multimodal data acquisition and processing. Technology companies are dedicated to building large-scale distributed device networks, generating massive amounts of high-resolution multimodal data through embedded sensors. Meanwhile, breakthroughs in large-scale visual-language models (LVLMs) have demonstrated their superior capabilities in processing complex multimodal data, indicating their enormous potential in enhancing edge intelligence. This trend raises a crucial question: how can these computationally intensive AI systems be efficiently deployed on resource-constrained edge devices to fully leverage their analytical capabilities?

[0003] However, practical applications face three core limitations. First, edge computing resources are extremely limited. Although current new-generation smart terminals are equipped with dedicated computing platforms to support AI operations, their computing power is typically only equivalent to that of mobile devices. Furthermore, local storage space must be prioritized for storing raw sensor data, making it difficult to support the complex calculations required for large-scale LVLMs, directly limiting inference performance. Second, due to the dynamic nature of device deployment environments, network connections with the cloud are intermittent, and data exchange can only be completed within a limited time window, posing a challenge to continuous data processing and inference tasks. Third, network bandwidth between devices and the cloud is limited, while the massive amounts of multimodal data generated on the device side result in low data transmission efficiency, severely restricting real-time processing and analysis capabilities. This problem is particularly prominent for applications requiring the processing of high-resolution data.

[0004] Existing solutions have significant shortcomings in addressing the aforementioned challenges. While cloud-centric deployments are not limited by local computing resources and can leverage powerful cloud computing capabilities, they struggle to efficiently transmit massive amounts of device data due to limited network bandwidth and insufficient connection stability. The latency from data acquisition to analysis result output increases significantly, failing to meet real-time inference requirements. On the other hand, purely local deployments, while reducing reliance on network bandwidth, are limited by the limited computing resources of terminal devices, preventing the execution of large-scale LVLMs and thus restricting intelligent inference capabilities. Existing collaborative inference methods attempt to allocate resources between edge devices and cloud servers, but they rely on continuous and stable network connections, which is inconsistent with the dynamically changing network conditions in real-world application scenarios. Summary of the Invention

[0005] This invention provides a collaborative reasoning method for large-scale visual-language models in cloud-edge-device systems, aiming to effectively solve the aforementioned technical problems.

[0006] According to a first aspect of the present invention, the present invention provides a large-scale visual-language model collaborative reasoning method for cloud-edge systems, characterized in that it includes: deploying a high-performance cloud LVLM model and a cloud archive in the cloud; deploying an edge LVLM model and an edge archive at the edge, wherein the edge LVLM model incorporates a retrieval-enhanced generation algorithm, and the edge archive and the cloud archive are capable of dynamic knowledge interaction; a task scheduler performs complexity evaluation on received query tasks based on a task allocation algorithm, and allocates the query tasks to the edge for local processing or to the cloud for processing according to the complexity evaluation results; wherein each query consists of an image and a natural language instruction.

[0007] Furthermore, both the cloud-based LVLM model and the edge LVLM model include a visual query module, an instruction query module, and a data fusion module. The visual query module converts images into feature vectors based on a visual embedding model and then compares their image similarity with images stored in the archive. The instruction query module calculates the similarity of images that match the instructions based on a text embedding model. The data fusion module integrates image similarity and instruction similarity to select the most matching single relevant instruction from the archive to generate a comprehensive search result.

[0008] Furthermore, the cloud-based archive stores a large collection of images. , No. One image There exists a set of instruction-answer pairs .

[0009] Furthermore, the edge archive is a subset of the cloud archive, storing data most relevant to the current task at the edge, and the stored data is highly optimized based on a dynamic adjustment algorithm.

[0010] Furthermore, the high optimization involves adjusting the position of each image in the queue based on the usage pattern of each image in the edge archive: each time the edge processes a query task, the first K retrieved images are moved to the front of the queue, indicating that they have been recently accessed; conversely, images that rarely appear in the results are gradually moved to the back of the queue over time; when the edge archive needs to be updated, images and corresponding instructions are removed from the tail of the queue to make room for the edge archive to introduce new images and corresponding instructions, which are then placed at the front of the queue to ensure that high-demand data always takes priority.

[0011] Furthermore, the complexity assessment includes a matching test and a cognitive test. The matching test is used to assess whether the retrieved relevant content is sufficient to support reasoning before reasoning begins. It includes two steps: filtering low-relevance records and checking the number of remaining contents. If the matching test is passed, the task will be processed by the edge device. If the matching test is not passed, the task scheduler will transmit the query task to the cloud for processing by the cloud LVLM model. The cognitive test is used to quickly assess the credibility of the answer given by the edge LVLM model. If the credibility assessment is passed, the answer will be taken as the final answer. If the credibility assessment is not passed, the task scheduler will transmit the query task to the cloud for processing by the cloud LVLM model.

[0012] Furthermore, the specific process of filtering low-relevance records is as follows: the task scheduler filters out low-relevance records from the retrieved relevant content, and each relevant record... Both image similarity scores and instruction similarity scores are associated. Any record with an image similarity score or instruction similarity score lower than the corresponding similarity threshold will be removed from the relevant content.

[0013] The specific process of checking the remaining content quantity is as follows: the scheduler checks the filtered content. Does it contain at least 1 record, of which This lower bound ensures that LVLM has sufficient context to complete reliable reasoning, if If the edge archive lacks the necessary information to process the query, then the query is considered invalid. It will be cached and await transmission to the cloud for processing; otherwise, it will enter the edge inference process. The content will be displayed during the assembly stage and tasks. Integration.

[0014] Furthermore, the credibility assessment process is as follows:

[0015] During the LVLM model inference process, each newly generated label is tracked. And record its probability. ,in A high probability means that LVLM has a high confidence level in the selection of that label. The final output consists of multiple labels, and the geometric mean of the probabilities of all newly generated labels is calculated as the overall confidence score.

[0016]

[0017] in, It is the length of the output response. It is the number of input tags;

[0018] Then, this Values ​​and thresholds Compare. If If the result is not strong enough, it indicates that the edge-end LVLM model is not confident in its inference results, so the result is discarded and the query is returned. Place it in the transmission buffer and wait for communication with the cloud; otherwise, the result is accepted as the final answer and no further transmission is required.

[0019] According to a second aspect of the invention, the invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method as described above.

[0020] According to a third aspect of the invention, the invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0021] Through one or more embodiments of the above-described embodiments of the present invention, at least the following technical effects can be achieved: The large-scale visual-language model inference method for cloud-edge systems proposed in this invention establishes an interactive dynamic knowledge base between the edge and the cloud, adaptively introducing retrieval-enhanced generation (RAG) technology to the edge LVLM, enabling the edge to process more inference tasks locally while transmitting fewer data samples to the cloud. Furthermore, the task allocation algorithm and customized hierarchical communication mechanism of this method can save bandwidth and improve network utilization. In summary, this method can improve inference efficiency within cloud-edge systems while maintaining high accuracy, effectively solving the problem of low inference efficiency caused by limited computing resources of edge devices, intermittent communication windows, and bandwidth limitations in existing technologies. Attached Figure Description

[0022] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0023] Figure 1 This is a framework diagram of the large-scale visual-language model collaborative reasoning method for cloud-edge-device systems provided in an embodiment of the present invention;

[0024] Figure 2 This diagram illustrates the test accuracy and inference time of the collaborative reasoning method provided in this embodiment of the invention on different datasets.

[0025] Figure 3 This diagram illustrates the test accuracy and inference time of the collaborative inference method provided in this embodiment of the invention under different bandwidth settings on different datasets.

[0026] Figure 4This diagram illustrates the test accuracy of the collaborative reasoning method provided in the embodiments of the present invention under different search settings. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0029] The following description, in conjunction with the accompanying drawings, introduces a communication edge-cloud segmentation inference method for large-scale language models provided by the present invention.

[0030] like Figure 1 As shown in the figure, the implementation process of the communication edge cloud segmentation inference method for large-scale language models provided by this embodiment of the invention is as follows:

[0031] Deploy high-performance cloud-based LVLM models and a comprehensive cloud archive for cloud environments.

[0032] Deploy compact edge LVLM models at the edge and lightweight edge archives that can dynamically interact with cloud archives. Introduce retrieval augmented generation (RAG) algorithms to the edge LVLM models, enabling the edge to handle more inference tasks locally while transmitting fewer data samples to the cloud.

[0033] The task scheduler evaluates the complexity of received query tasks based on a task allocation algorithm, and allocates the query tasks to the edge for local processing or to the cloud for processing through a hierarchical communication mechanism based on the complexity evaluation results.

[0034] Based on the above settings, each query From an image and a natural language instruction Composition, that is ( The goal of the collaborative reasoning method in this embodiment is to follow instructions. For images It enables interpretation and achieves fast and reliable inference under resource constraints at the edge.

[0035] The relevant content of the search results It is a set of records . No. records Includes the following:

[0036] Related images An image from the archive that matches the query. Similar images.

[0037] Related instructions : with query command Closely related instructions.

[0038] Relevant true values :against The real answers are manually annotated.

[0039] In this embodiment, the cloud archive stores a large collection of images. Formally, for the first One image There exists a set of instruction-answer pairs Each real value has been carefully annotated by human experts. The goal of this archive is retrieval and querying. The most relevant content.

[0040] Edge archives must be highly optimized to adapt to resource-constrained edge environments. In fact, edge archives are a subset of cloud archives, containing only data most relevant to the current task at the edge. Because the trajectory and operational goals at the edge are relatively fixed, querying... It also possesses a degree of predictability. Therefore, a carefully selected subset of cloud-based archives can support stable onboard inference while minimizing storage overhead.

[0041] However, determining which part of the cloud archive should be uploaded to the edge archive is not easy. Furthermore, if the scope of tasks at the edge changes over time, the archive also needs to be adjusted accordingly. To address these issues, this embodiment designs a dynamic adjustment algorithm: tracking the usage pattern of each image in the edge archive and dynamically adjusting the position of each image in the queue based on the usage pattern.

[0042] Specifically, each time the edge device processes a query task, the retrieved top... Images that have been recently accessed are moved to the front of the queue. Conversely, images that rarely appear in the results are gradually moved to the back of the queue over time. When the archive needs updating, the replacement module removes entries from the tail of the queue (i.e., least recently used entries). These outdated or unused entries are discarded to free up space. Subsequently, newly introduced images and instructions are loaded into the archive and placed at the front of the queue, ensuring that high-demand data always takes precedence.

[0043] Because of each query It is multimodal and includes a newly captured image. Therefore, the LVLM model uses separate modules to process visual and text queries separately in order to jointly identify relevant content. The LVLM model in this embodiment includes a visual query module, an instruction query module, and a data fusion module.

[0044] The visual query module includes:

[0045] Based on visual embedding model The image is converted into a feature vector. Specifically, each image in the knowledge base... All are embedded and normalized to unit vectors:

[0046]

[0047] These normalized vectors are stacked to form a matrix:

[0048]

[0049] The matrix is ​​stored in the archive for later retrieval.

[0050] When querying images Upon arrival, the embedded model Generate a normalized query vector Then calculate. and Cosine similarity of each column:

[0051]

[0052] Get the query image Similarity score between the stored image and the image.

[0053] Instruction query module:

[0054] Natural language instructions in the knowledge base are processed through a similar process, using a text embedding model. Embedding is then performed. However, to reduce computational and storage overhead, the instructions are first deduplicated, as different images may share the same instructions. After removing duplicates, a unique set of instructions is obtained. and maintain a mapping table. This records which instructions belong to which images. Then, similarity is calculated in the same way. .

[0055] Data fusion module:

[0056] While a single image may correspond to multiple instructions, including all instructions in the prompt can overwhelm the LVLM, thus degrading inference performance. Too many irrelevant instructions increase computational burden and dilute the critical context required by the LVLM. To balance coverage and relevance, the data fusion module integrates... and To generate comprehensive search results. For each image in the cloud archive, the data fusion module refers to the mapping table. Find and query commands Most similar instruction:

[0057]

[0058] The data fusion module selects from the archives and... The best-matching single relevant instruction is ultimately returned. 1 related record.

[0059] Due to limited computing and storage resources, the size of edge archives and edge LVLMs is constrained. Therefore, edge-side inference may not always provide the correct answer. To address this, this embodiment proposes a task scheduling algorithm to determine which tasks should be retained for local inference and which tasks cannot be completed at the edge and need to be sent to the cloud for processing.

[0060] Before inference begins, the scheduler performs a matching test to evaluate the retrieved relevant content. Is it sufficient to support reasoning? The matching test consists of two steps:

[0061] (1) Filtering low-relevance records: The scheduler filters out low-relevance records from the retrieved relevant content. Each relevant record Both are associated with two similarity scores: image similarity and instruction similarity. Two thresholds are defined. and , Used for image similarity, Used for instruction similarity. Any record with an image or instruction similarity below the corresponding threshold will be removed from the relevant content.

[0062] (2) Check the amount of remaining content: The scheduler checks the filtered content. Does it contain at least records (of which) This lower bound ensures that LVLM has sufficient context to perform reliable reasoning. If If so, it is assumed that the edge archive lacks the necessary information to process the query. In this case, the query... It will be cached and await transmission to the cloud, where a more comprehensive archive and a more powerful LVLM can handle the inference task. If the filtered relevant content... If the matching test is passed, the edge inference process begins. The content will be displayed during the assembly stage and tasks. Integration.

[0063] While the relevant content helps compensate for the limitations of LVLM models in terms of scale and generalization ability at edge computing endpoints, the final results still cannot be fully trusted without additional verification steps. Therefore, this embodiment introduces a cognitive test to quickly assess the credibility of the LVLM answers.

[0064] During LVLM inference, each newly generated tag is tracked. And record its probability. ,in A high probability means that LVLM has a high level of confidence in the choice of that label. Since the final output consists of multiple labels, the geometric mean of the probabilities of all newly generated labels is calculated as the overall confidence score:

[0065]

[0066] in, It is the length of the output response. This is the number of input tags.

[0067] Then, this Values ​​and thresholds Compare. If This indicates that the LVLM at the edge is not confident enough in its inference results. The system will discard the result and query... Place it in the transmission buffer, awaiting communication with the cloud. Otherwise, the result is accepted as the final answer, requiring no further transmission.

[0068] In this embodiment, unlike the edge-side inference process, the cloud does not perform matching tests on the retrieved content. For each received query, the cloud archive returns... One related record. All. All records are assumed to be relevant and are therefore automatically included as input in the cloud LVLM.

[0069] Each query and its corresponding content are added to a task queue. The inference module is activated when the task queue in the cloud is not empty. By separating the retrieval process from LVLM inference, the cloud can quickly respond to newly arriving queries during short communication windows, allowing more data to be downloaded from the edge before the link is lost.

[0070] The LVLM inference process in the cloud is similar to that at the edge. Because the cloud typically has more powerful and comprehensive resources, cognitive testing is omitted. Therefore, the inference result becomes the final result without additional verification. This approach ensures that the cloud can efficiently process queries and provide reliable inference results, while maximizing the use of each communication window.

[0071] In some embodiments of the present invention, the method is tested and verified by setting up experiments.

[0072] Experimental setup:

[0073] A large-scale visual-language model collaborative inference system for cloud-edge-device applications is deployed on an H3C UniServer R5300 G3 server, equipped with eight NVIDIA GeForce RTX 3090 GPUs, two Intel Xeon Silver4210R processors (10 cores each, 2.84 GHz), and 8 x 32 GB DDR4 memory, running Ubuntu 18.04.6LTS. The software stack includes Python 3.12.4 and PyTorch 2.5.1. A kernel-based virtual machine (KVM) is used to simulate the edge and cloud. The link between the edge and cloud is simulated using Open vSwitch. Traffic conditions are controlled by the tc tool. The edge and cloud run on different RTX 3090 GPUs within the R5300 G3 server. The computing power of the edge and cloud is set to 1.33 TFLOPS and 35.6 TFLOPS, respectively.

[0074] Dataset:

[0075] Experiments used the RSVQA dataset to evaluate the accuracy of the inference system. RSVQA is a visual question-answering dataset, divided into low-resolution (RSVQA-LR) and high-resolution (RSVQA-HR) datasets. RSVQA-LR is one of the earliest VQA datasets for low spectral resolution, containing Sentinel-2 images with a resolution of 256×256 pixels, sourced from OpenStreetMap. RSVQA-HR, as a high-spectral resolution dataset, consists of aerial RGB images taken by USGS, with a resolution of 512×512 pixels.

[0076] In addition, three classification datasets—RESISC45, AID, and WHU-RS19—were used to evaluate the accuracy of the proposed framework. RESISC45 is a large-scale image dataset comprising 45 distinct land use categories, with 700 images of 256×256 pixels each, totaling 31,500 images. The AID dataset is a large-scale collection of aerial images covering 30 scene categories, containing 10,000 images. WHU-RS19 is another image dataset covering 19 scene categories, with approximately 50 images per category.

[0077] Model:

[0078] A pre-trained Qwen2.5-VL 2B model is used at the edge, and a pre-trained Qwen2.5-VL 7B model is used in the cloud. Qwen2.5-VL integrates a vision-language adapter for improved efficiency and supports multi-image functionality. For the image and instruction embedding models used in dynamic archives, GeoRSCLIP is used to generate embedding vectors, specifically using the ViT-B-32 architecture.

[0079] Baseline: This approach is compared with several LVLM deployment baselines:

[0080] Edge Only: This baseline uses LVLMs on the edge for local inference only. All inference tasks are completed at the edge and do not involve the cloud.

[0081] Cloud-only: The edge only performs data collection, transmitting all data to the cloud via a wireless link, and then the cloud uses cloud-based LVLM to perform inference on each query.

[0082] Tabi: A multi-level cascaded high-efficiency model service system proposed by Wang et al., which uses methods such as early return of simple queries, attention mechanism word pruning, and weighted multi-level ensemble learning.

[0083] PETALS: A system proposed by Borzunov et al., which enables multiple devices to collaborate on inference and fine-tune large language models, and can run LLM inference on consumer-grade GPUs.

[0084] Hyperparameter settings:

[0085] The number of relevant records, K, was set to 5. In the matching test, the image threshold T_M was set to 0.8, the instruction threshold T_I to 0.94, and the minimum number of relevant records after filtering, T_K, was set to 3. The confidence threshold T_"Conf" in the cognitive test was set to 0.75. To reasonably evaluate the performance of this method, in the simulated scenario, the network interruption time was set to 60 seconds, the communication window to 3 seconds, and the network bandwidth to 30 Mbps. Data was transmitted using zlib compression, and one image was captured every 2 seconds at the edge.

[0086] All data from the training dataset were imported into the cloud archive. Initially, the edge archive used 20 randomly selected images from the training set and their associated instruction-answer pairs. Therefore, the edge archive contained far fewer images than the cloud archive and needed to be updated incrementally during operation to meet query requirements. During the experiment, the maximum number of images in the edge archive was limited to 20.

[0087] Experimental results:

[0088] like Figure 2 As shown, our method was evaluated against four baselines on both low-resolution (LR) and high-resolution (HR) RSVQA datasets. Our method demonstrated superior inference accuracy across all test scenarios, achieving 83.3% accuracy on the LR dataset and 78.8% accuracy on the HR dataset, significantly outperforming other baseline methods, none of which reached 70% accuracy.

[0089] Further analysis of the inference latency from task acquisition to result generation was conducted, and the time statistics are summarized in the figures. Three time metrics were investigated: average processing time, median processing time, and maximum processing time. The figures show that our method achieves a significant efficiency improvement in average processing time compared to other baselines. While the theoretical minimum latency for edge-only processing is approximately 1.6 seconds (as demonstrated by the “edge-only” baseline), our method maintains competitive average latencies of 6.3 seconds (LR) and 7.9 seconds (HR), representing reductions of 76%–95% compared to cloud-dependent methods. Notably, our method's median processing time of 1.6 seconds on both datasets is close to the theoretical optimum, indicating stable time performance for most samples even with occasional outliers. Maximum latency analysis provides important insights into the system's robustness in the worst-case scenario. Our method's maximum latencies of 61.1 seconds (LR) and 64.2 seconds (HR) are significantly lower than the extreme values ​​in other methods, such as Tabi's 336.4 seconds and the cloud-only method's 1,045.8 seconds.

[0090] In summary, the collaborative reasoning method provided by this invention effectively balances enhanced accuracy and reduced inference time, making it a highly efficient spaceborne inference solution for resource-constrained edge environments. By simultaneously optimizing accuracy and speed, this method ensures reliable performance while minimizing reliance on cloud-based processing, thereby enhancing the overall performance of the edge-cloud collaborative reasoning framework.

[0091] Based on any of the above embodiments, another embodiment of the present invention provides an electronic device, which may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor may invoke logical instructions in the memory to execute the above method.

[0092] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, embodiments of the present invention also provide a storage medium storing a plurality of instructions adapted for loading by a processor to execute the communication edge-cloud segmentation inference method for large language models as provided in the above embodiments.

[0094] On the other hand, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] In summary, although the present invention has been disclosed above with reference to preferred embodiments, the above preferred embodiments are not intended to limit the present invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for collaborative reasoning of large-scale vision-language models in cloud-edge-device systems, characterized in that, include: Deploy high-performance cloud-based LVLM models and cloud-based archives in the cloud; An edge LVLM model and an edge archive are deployed at the edge. The edge LVLM model introduces a retrieval-enhanced generation algorithm, and the edge archive and the cloud archive can perform dynamic knowledge interaction. The task scheduler evaluates the complexity of received query tasks based on a task allocation algorithm, and allocates the query tasks to the edge for local processing or to the cloud for processing based on the complexity evaluation results. Each query consists of an image and a natural language instruction. The cloud archive stores a large collection of images. , No. One image There exists a set of instruction-answer pairs , The edge archive is a subset of the cloud archive, storing data most relevant to the current task at the edge. The stored data is highly optimized using a dynamic adjustment algorithm, whereby the position of each image in the queue is adjusted based on its usage pattern within the edge archive. The complexity assessment includes matching tests and cognitive tests. The matching test is used to evaluate whether the retrieved relevant content is sufficient to support inference before inference begins. It includes two steps: filtering low-relevance records and checking the number of remaining contents. If the matching test is passed, the task will be processed by the edge. If the matching test is not passed, the task scheduler will transmit the query task to the cloud for processing by the cloud LVLM model. The cognitive test is used to quickly assess the credibility of the answer given by the edge LVLM model. If the credibility assessment is passed, the answer is taken as the final answer. If the credibility assessment is not passed, the task scheduler transmits the query task to the cloud for processing by the cloud LVLM model.

2. The method according to claim 1, characterized in that, Both the cloud-based LVLM model and the edge LVLM model include a visual query module, an instruction query module, and a data fusion module. The visual query module converts images into feature vectors based on a visual embedding model and then compares them with images stored in the archive. The instruction query module calculates the similarity of images that match the instruction based on a text embedding model. The data fusion module integrates image similarity and instruction similarity to select the most matching single relevant instruction from the archive, thereby generating comprehensive search results.

3. The method according to claim 1, characterized in that, The height optimization specifically refers to: Each time a query task is processed at the edge, the first K retrieved images are moved to the front of the queue, indicating that they have been accessed recently. Conversely, images that rarely appear in the results are gradually moved to the back of the queue over time. When the edge archive needs to be updated, images and corresponding instructions are removed from the tail of the queue to make room for the edge archive to introduce new images and corresponding instructions, which are then placed at the front of the queue to ensure that high-demand data always takes priority.

4. The method according to claim 1, characterized in that, The specific process for filtering low-relevance records is as follows: The task scheduler filters out low-relevance records from the retrieved relevant content, and each relevant record... Both image similarity scores and instruction similarity scores are associated. Any record with an image similarity score or instruction similarity score lower than the corresponding similarity threshold will be removed from the relevant content. The specific process for checking the remaining content quantity is as follows: The scheduler checks the filtered content. Does it contain at least 1 record, of which This lower bound ensures that LVLM has sufficient context to complete reliable reasoning, if If the edge archive lacks the necessary information to process the query, then the query is considered invalid. It will be cached and await transmission to the cloud for processing; otherwise, it will enter the edge inference process. The content will be displayed during the assembly stage and tasks. Integration.

5. The method according to claim 1, characterized in that, The credibility assessment process is as follows: During the LVLM model inference process, each newly generated tag is tracked. And record its probability. ,in A high probability means that LVLM has a high confidence level in the selection of that label. The final output consists of multiple labels, and the geometric mean of the probabilities of all newly generated labels is calculated as the overall confidence score. in, It is the length of the output response. It is the number of input tags; Then, this Values ​​and thresholds Compare, if If the result is not strong enough, it indicates that the edge-end LVLM model is not confident in its inference results, so the result is discarded and the query is returned. Place it in the transmission buffer and wait for communication with the cloud; otherwise, the result is accepted as the final answer and no further transmission is required.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1 to 5.

7. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images