Distributed system and collaborative reasoning method applied to the distributed system, electronic device

Abstract

Embodiments of the present application provide a kind of distributed system and the collaborative inference method applied to distributed system, electronic equipment, involve collaborative inference technical field, the method comprises: in response to the processing request of collaborative inference task, receiving the target feature embedding vector sent by second type node;In the plurality of prototype vectors of prototype vector library, determine the target prototype vector matched with target feature embedding vector;Multiple prototype vectors correspond to multiple incremental categories one by one, prototype vector is generated according to the feature embedding vector of multiple support samples belonging to the same incremental category, feature embedding vector is extracted in the process that first model is used parameter freezing to support sample and executes inference task, first model is identical with second model, incremental category is associated with the original output category of second model;The incremental category corresponding to target prototype vector is determined as the inference result of collaborative inference task.The present application embodiment can reduce communication and computing load, improve collaborative inference efficiency.

Description

Technical Field

[0001] This application relates to the field of collaborative reasoning technology, and in particular to a distributed system and a collaborative reasoning method and electronic device applied to the distributed system. Background Technology

[0002] For future 6G wireless networks, mobile devices (such as terminals and satellites) and edge / cloud nodes (such as edge / cloud servers and ground stations) urgently need collaborative reasoning capabilities to efficiently process sensory data and make intelligent decisions in a distributed environment. However, limited by computing power, energy consumption, and communication bandwidth, traditional centralized reasoning methods require uploading large amounts of raw data to ground servers for unified processing, resulting in high communication overhead, high latency, and low system response efficiency. Furthermore, the performance of models already deployed on the terminal side significantly degrades when faced with new subcategories or changing environments. Summary of the Invention

[0003] This application provides a distributed system and a collaborative reasoning method and electronic device applied to the distributed system, in order to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a collaborative reasoning method applied to a distributed system, the distributed system including a first type of nodes and a second type of nodes, the method being applied to the first type of nodes, the method comprising: In response to the processing request of the collaborative reasoning task, the system receives the target feature embedding vector sent by the second type of node; the target feature embedding vector is generated by the second type of node during the process of performing a reasoning task on the query sample using the second model, and the second model is a deep learning model statically deployed on the second type of node; Among multiple prototype vectors in the prototype vector library, a target prototype vector that matches the target feature embedding vector is determined; the multiple prototype vectors correspond one-to-one with multiple incremental categories, and the prototype vectors are generated based on the feature embedding vectors of multiple support samples belonging to the same incremental category. The feature embedding vectors are extracted during the inference task performed on the support samples using a first model with frozen parameters. The first model is the same as the second model, and the incremental category is associated with the original output category of the second model. The incremental category corresponding to the target prototype vector is determined as the reasoning result of the collaborative reasoning task.

[0005] In some implementations, the method further includes, before receiving the target feature embedding vector sent by the second type of node: The first model is used to perform an inference task on each of the support samples belonging to the incremental category, and the feature embedding vector generated during the inference task is extracted. Based on the feature embedding vectors of multiple supporting samples belonging to the same incremental category, a prototype vector of the incremental category is generated and added to the prototype vector library.

[0006] In some implementations, determining the target prototype vector that matches the target feature embedding vector from among multiple prototype vectors in the prototype vector library includes: Calculate the distance between the target feature embedding vector and each of the prototype vectors in the prototype vector library; The prototype vector that is closest to the target feature embedding vector is determined as the target prototype vector.

[0007] In some implementations, generating the prototype vector of the incremental category based on the feature embedding vectors of multiple support samples belonging to the same incremental category includes: Find the vector center of the feature embedding vectors of multiple support samples; The vector center is used as the prototype vector of the incremental category.

[0008] In some implementations, the incremental category includes at least one of the following: a subcategory of the original output category, a nearest neighbor category of the original output category, and an instance of the original output category.

[0009] Secondly, embodiments of this application provide a collaborative reasoning method applied to a distributed system, the distributed system including a first type of nodes and a second type of nodes, the method being applied to the second type of nodes, the method comprising: In response to the processing request of the collaborative reasoning task, the statically deployed second model is used to perform the reasoning task on the query sample; Extract the feature embedding vectors generated during the inference task to obtain the target feature embedding vector; The target feature embedding vector is sent to the first type of node, and the target feature embedding vector is used as query information for the first type of node to perform collaborative reasoning tasks.

[0010] In some implementations, the method further includes, before sending the target feature embedding vector to the first type of node: Based on the results of the inference task, it is determined that there exists any original output category in the second model whose confidence level is higher than a preset threshold.

[0011] Thirdly, embodiments of this application provide a distributed system, including: The second type of node is used to respond to the processing request of the collaborative reasoning task, perform a reasoning task on the query sample using a statically deployed second model; extract the feature embedding vector generated during the reasoning task to obtain the target feature embedding vector; and send the target feature embedding vector to the first type of node, wherein the target feature embedding vector is used as query information for the first type of node to perform the collaborative reasoning task. The first type of node is configured to, in response to the processing request, receive the target feature embedding vector sent by the second type of node; determine a target prototype vector that matches the target feature embedding vector from multiple prototype vectors in the prototype vector library; the multiple prototype vectors correspond one-to-one with multiple incremental categories, the prototype vectors are generated based on the feature embedding vectors of multiple support samples belonging to the same incremental category, the feature embedding vectors are extracted during the inference task performed on the support samples using a first model with frozen parameters, the first model is the same as the second model, and the incremental category is associated with the original output category of the second model; the incremental category corresponding to the target prototype vector is determined as the inference result of the collaborative inference task.

[0012] In some implementations, the first type of node is a ground station, and the second type of node is an on-orbit remote sensing satellite.

[0013] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0014] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0015] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0016] Based on the aforementioned distributed system and the collaborative reasoning method and electronic device applied to the distributed system, this application has at least the following beneficial effects or advantages: By pre-building a prototype vector library, the first model, identical to the second type of node and with its parameters frozen, processes the support samples of the incremental category, generating prototype vectors from the library. This enables feature representation of the incremental category. Upon receiving a collaborative inference task request, the second type of node only needs to extract the target feature embedding vector of the query sample and send it to the first type of node, without uploading the original query sample data, significantly reducing the communication load and data transmission cost in the distributed system. Simultaneously, the first type of node directly matches the target feature embedding vector with the prototype vectors in the library to quickly output collaborative inference results, without needing to retrain or update the parameters of the statically deployed second model on the second type of node. This effectively reduces the cost and cycle of model iteration and deployment, improving the inference efficiency for new inference tasks based on the collaborative inference capabilities of the distributed system without altering the original inference capabilities of the second type of node.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0019] Figure 1 This invention provides a flowchart illustrating a collaborative reasoning method for a distributed system executed on one side of a first type of node, according to an embodiment of this application. Figure 2 A flowchart illustrating a collaborative reasoning method applied to a distributed system on the side of a second type of node, according to an embodiment of this application, is shown. Figure 3 This is a schematic diagram illustrating the prediction results of a portion of qualitative analysis samples by the collaborative reasoning method for distributed systems provided in this application embodiment. Figure 4 A schematic diagram of the classification confusion matrix on a test set image for the collaborative reasoning method for distributed systems provided in this application embodiment; Figure 5 A schematic diagram illustrating the F1 score prediction for each class of the collaborative reasoning method for distributed systems provided in this application embodiment; Figure 6 A schematic diagram illustrating the t-SNE visualization dimensionality reduction result of the collaborative reasoning method for distributed systems provided in this embodiment of the application. Figure 7 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. It should be noted that the application scenarios or application examples provided in this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions.

[0022] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0023] The distributed system described in this application includes a first type of node and a second type of node. The first type of node is an edge node / cloud node, including but not limited to: ground stations, edge servers, edge computing nodes, central servers, central cloud servers, and cloud servers, possessing abundant computing and storage resources. The second type of node is a terminal node, referring to devices on the edge side, including but not limited to: mobile devices, IoT sensors, smart cameras, drones, or orbiting satellites. Terminal node devices typically have limited computing power, energy consumption, or communication bandwidth. Models deployed on the terminal side usually only possess coarse-grained recognition capabilities. When faced with new subcategories or changing environments, such as new inference tasks in the network that are not directly supported by the original deep neural network models on the terminal side, their performance significantly degrades.

[0024] Many related technologies rely on collecting sufficient data centrally at edge nodes / cloud nodes, retraining the model, and then distributing the optimized model to terminal nodes. This process is not only time-consuming and energy-intensive, but also fails to meet the real-time and resource-constrained online adaptive requirements. Therefore, this application proposes an edge-to-end collaborative inference enhancement technology that, without retraining the terminal model, fully utilizes limited new samples and a distributed collaborative mechanism to dynamically enhance inference capabilities, while significantly reducing the system's communication and computational load.

[0025] See Figure 1 The flowchart shown illustrates a collaborative reasoning method applied to distributed systems. This method is applied to the first type of nodes in a distributed system, such as edge nodes / cloud nodes. The method specifically includes the following steps: Step 101: Use the first model with frozen parameters to perform an inference task on the support samples belonging to the incremental category and extract the feature embedding vectors generated during the inference task.

[0026] The first model is identical to the second model statically deployed on the second type of nodes. The second type of nodes are terminal nodes in the distributed system, on which the second model with frozen parameters is deployed. The first type of nodes, however, can obtain a copy of the model with the same model structure and parameters as the second model—that is, the first model. Both the first and second models are deep learning models, such as deep neural network models. The first model can be used to perform inference tasks, including but not limited to: classification tasks, object matching tasks, and instance recognition tasks. After performing the inference task, the second model can output the probability parameters of each original output category.

[0027] For example, the first type of node is a ground station, and the second type of node is an on-orbit remote sensing satellite. The ground station sends the pre-trained second model to the remote sensing satellite. After the remote sensing satellite deploys the second model locally, it can use the second model to perform classification tasks on the remote sensing satellite image and output the probability parameters of each original output category contained in the remote sensing satellite image. These parameters can be specifically implemented as probability or confidence level.

[0028] The incremental category is associated with the original output category of the first and second models. The incremental category is the category that needs to be identified in response to a new inference task. This new inference task includes, but is not limited to: further subdividing at least one original output category to identify subcategories; target matching tasks with categories similar to the original output category; and instance identification tasks for a specific instance of the original output category. Accordingly, the incremental category is the subcategory, the nearest neighbor category, or the instance.

[0029] In some implementations, the prototype vector of the incremental category can be calculated using a small number of support samples, which can be collected from different second-class nodes. For example, if the original inference task supported by the first and second models is image recognition, used to identify whether an image contains an aircraft, and the incremental category can be a specific aircraft model, then the support samples can be images of that specific aircraft model taken from different shooting angles, different shooting distances, etc., with a number of no less than 30. It is understood that the above design of the category and the number of support samples is only for illustration and does not constitute a limitation on the embodiments of this application.

[0030] In some implementations, the first type of node first collects a small set of supporting samples that constitute all incremental categories. The first type of node then uses a first model (a copy of the second model) to infer each supporting sample. During inference, the parameters of the second model are also frozen, thus ensuring consistency with the statically deployed first model on the first type of node and preventing changes in the mapping of the embedding vector space. The feature embedding vector can be an intermediate layer result of the second model; for example, it can be extracted using a feature layer prior to the last fully connected layer.

[0031] Step 102: Generate prototype vectors for incremental categories based on the feature embedding vectors of multiple support samples belonging to the same incremental category, and add them to the prototype vector library.

[0032] After extracting the feature embedding vectors of each support sample, the prototype vector, or prototype center, of the incremental category is calculated based on the feature embedding vectors of multiple support samples belonging to the same incremental category. The prototype vector library stores multiple prototype vectors that correspond one-to-one with multiple incremental categories.

[0033] The prototype vector library serves as the baseline information for the first type of nodes to collaborate with the second type of nodes in performing inference on the query sample. The query sample is the sample to be identified in the new inference task. It should be noted that the second type of nodes do not need to send the query sample to the first type of nodes; instead, they can only send the target feature embedding vector extracted by the second type of nodes after performing the original inference task on the query sample using the second model. The aforementioned collaborative inference performed by the first type of nodes on the query sample refers to the first type of nodes and the second type of nodes jointly performing collaborative inference on the query sample. Specifically, the first type of nodes performs collaborative inference by matching the target feature embedding vector of the query sample in the prototype vector library.

[0034] The embodiments of this application can significantly reduce communication and computing load and improve collaborative inference efficiency. This collaborative mechanism enables the distributed system as a whole to possess new inference capabilities (such as fine classification, object matching, or instance recognition) that the original terminal model did not have, without relying on retraining or modifying the terminal model. This solves the problems of poor timeliness and deployment difficulties caused by the reliance on retraining when facing new tasks in existing technologies. It improves the flexibility and scalability of the edge-to-edge collaborative inference system. When the distributed system receives a new inference task (such as fine classification, object matching, or instance recognition) that is not directly supported by the original deep learning model on the terminal side, it only needs to calculate and add a new prototype vector using a small number of samples at the edge / cloud, without updating or replacing the deployed terminal model.

[0035] It is understood that in some embodiments of this application, the construction and updating of the prototype vector library can be placed on nodes other than the first type of nodes and the second type of nodes. In other embodiments, the first type of nodes may include multiple node instances that communicate with each other, and the construction and updating of the prototype vector library can be deployed in multiple node instances to be implemented jointly.

[0036] In some implementations, reference Figure 1 The method also includes: Step 103: In response to the processing request of the collaborative reasoning task, receive the target feature embedding vector sent by the second type of node; the target feature embedding vector is generated by the second type of node during the process of performing the reasoning task on the query sample using the second model.

[0037] Step 104: Among the multiple prototype vectors in the prototype vector library, determine the target prototype vector that matches the target feature embedding vector.

[0038] Step 105: Determine the incremental category corresponding to the target prototype vector as the reasoning result of collaborative reasoning.

[0039] The target feature embedding vector is calculated by the second type of node. The second type of node uses the parameter-frozen second model to perform the original inference task, extracting the feature embedding vector during the inference process to obtain the target feature embedding vector. The extraction method / location of the target feature embedding vector is the same as that used by the first type of node to extract any supporting sample. Both the first and second type of nodes have parameter-frozen models, thus avoiding changes in the mapping representation of the vector embedding space caused by model parameter updates. This ensures that for the same query sample, both the first and second type of nodes can extract the same target feature embedding vector.

[0040] This application embodiment extracts the target feature embedding vector of the query sample through a second type of node, and executes a specific new inference task on a first type of node to obtain the result of the new inference task. By uploading only a lightweight feature embedding vector, the bandwidth consumption of wireless communication is greatly reduced. At the same time, it can fully utilize the advantages of distributed computing in a distributed system, reduce redundant computing overhead at the edge / cloud, and also achieve enhanced collaborative inference capabilities without retraining, breaking through the limitations of terminal models.

[0041] In some implementations, step 104, determining the target prototype vector that matches the target feature embedding vector among multiple prototype vectors in the prototype vector library, includes: calculating the distance between the target feature embedding vector and each prototype vector in the prototype vector library; and determining the prototype vector that is closest to the target feature embedding vector as the target prototype vector.

[0042] For example, distance can be calculated by calculating cosine similarity, with the closest distance indicating the highest similarity. This application embodiment utilizes edge / cloud nodes to construct a prototype vector library based on a small number of samples, performing rapid similarity calculations on the target feature embedding vectors uploaded by terminal nodes to obtain results for new inference tasks, thus improving computational speed.

[0043] In some implementations, step 102 generates a prototype vector for the incremental category based on the feature embedding vectors of multiple support samples belonging to the same incremental category, including: solving for the vector center of the feature embedding vectors of the multiple support samples; and using the vector center as the prototype vector of the incremental category.

[0044] For example, the vector center can be the geometric center of the vector, which is obtained by calculating the arithmetic mean of the feature embedding vectors of all samples of the same incremental category. Alternatively, the vector center can also be calculated by clustering algorithms or other methods. This application does not specifically limit this.

[0045] This application also provides a collaborative reasoning method for a distributed system, which includes a first type of node and a second type of node. The method is applied to the second type of node. It is understood that all embodiments of this application are based on the same inventive concept, and the specific implementation methods and technical effects of each embodiment can be used as references to each other.

[0046] refer to Figure 2 In the second type of node, the collaborative reasoning method applied to distributed systems includes the following steps: Step 201: In response to the processing request of the collaborative reasoning task, perform the reasoning task on the query sample using the statically deployed second model; Step 202: Extract the feature embedding vectors generated during the inference task to obtain the target feature embedding vector; Step 203: Send the target feature embedding vector to the first type of node. The target feature embedding vector is used as query information for the first type of node to perform collaborative reasoning tasks.

[0047] A collaborative inference request is used to request the execution of a new inference task on the query sample. This new inference task differs from the original inference task of the statically deployed second model on the second-type nodes. Upon receiving the collaborative inference request, the second-type nodes use the second model to execute the original inference task process on the query sample. Instead of directly feeding back the inference result to the first-type nodes, they send the feature embedding vectors extracted during the inference task to the first-type nodes. The method for extracting the target feature embedding vector is exactly the same as the method used by the first-type nodes to extract the feature embedding vectors of the supporting samples. In some implementations, the first-type nodes may communicate with the second-type nodes beforehand to send the extraction method (such as the number of layers for extracting the feature embedding vectors) to the second-type nodes. After receiving the target feature embedding vector, the first-type nodes use it as query information and match it in the prototype vector library to obtain the inference result of the new inference task (i.e., the processing result of the collaborative inference request).

[0048] In some implementations, before sending the target feature embedding vector to the first type of node, the method further includes: determining, based on the results of the inference task, that there exists a confidence level higher than a preset threshold for any of the original output categories in the original output categories of the second model.

[0049] The second type of node performs the original inference task and obtains the result of the original inference task. The result of the original inference task includes the confidence scores of all original output categories. If any original output category in the second model has a confidence score higher than a preset threshold, then that original output category is considered to have been identified.

[0050] This application embodiment achieves selective data reporting by setting confidence levels on the terminal side, thus avoiding resource waste caused by uploading all features.

[0051] This application provides a distributed system including a first type of node and a second type of node.

[0052] The second type of node is used to respond to the processing request of the collaborative reasoning task, and to perform the reasoning task on the query sample using the statically deployed second model; extract the feature embedding vector generated during the reasoning task to obtain the target feature embedding vector; and send the target feature embedding vector to the first type of node, which is used as the query information for the first type of node to perform the collaborative reasoning task. The first type of node is used to respond to the processing request and receive the target feature embedding vector sent by the second type of node; among multiple prototype vectors in the prototype vector library, the target prototype vector that matches the target feature embedding vector is determined; multiple prototype vectors correspond one-to-one with multiple incremental categories. The prototype vectors are generated based on the feature embedding vectors of multiple support samples belonging to the same incremental category. The feature embedding vectors are extracted during the inference task performed on the support samples using the first model with frozen parameters. The first model is the same as the second model, and the incremental category is associated with the original output category of the second model; the incremental category corresponding to the target prototype vector is determined as the inference result of the collaborative inference task.

[0053] In some implementations, the first type of node is a ground station, and the second type of node is an in-orbit remote sensing satellite.

[0054] The following describes a specific implementation of an embodiment of this application.

[0055] This embodiment provides a general implementation process for an edge-side collaborative inference enhancement technology solution based on few-shot prototype learning. This embodiment defines two key entities: the "terminal" refers to edge-side devices, including but not limited to IoT sensors, smart cameras, drones, or satellites. These devices typically have limited computing power, energy consumption, or communication bandwidth. The "edge / cloud" refers to central servers or edge servers with abundant computing and storage resources, including but not limited to ground stations, edge computing nodes, or central cloud servers.

[0056] Step A1: When new inference tasks emerge in the network that are not directly supported by the existing deep neural network models on the terminal side, build a prototype library for the edge / cloud. In a preferred embodiment, the edge / cloud server first performs a prototype library preparation step. This step includes steps A11 to A14: Step A11: For one or more new inference tasks (such as fine classification, object matching, or instance recognition) that are not directly supported by the existing deep neural network models on the terminal side, the edge / cloud first collects a small set of samples that constitute all new categories or new instances based on the task. ; in, The total number of categories or instances of the aforementioned new reasoning tasks. For the first A small sample set of each target category or instance.

[0057] Step A12: At the edge / cloud, use a copy of the deep neural network model with the same network structure and parameters as the model deployed on the terminal. Iterate through each category And extract a small number of corresponding samples. Centralize all samples Feature embedding : ; Step A13: Obtain the prototype center of the category by calculating the arithmetic mean of the feature embeddings of all samples in the category. ; Step A14: Calculate the prototype centers for all categories Collect into a prototype library And stored at the edge / cloud.

[0058] Step A2: When the terminal receives a collaborative reasoning request for new sample data, it calculates the feature embedding based on the deployed deep neural network model; In a preferred embodiment, when a terminal (e.g., a satellite or an IoT camera) receives a collaborative inference request for new sample data, to ensure feature consistency in the subsequent prototype matching process, the terminal freezes the parameters of its deployed deep neural network model, so that the model retains only coarse-grained classification and general feature extraction capabilities. The terminal then uses the newly acquired images... The newly acquired image is input into the frozen model, a complete forward propagation is performed, and the feature embeddings are extracted from the intermediate layers of the model. .

[0059] Step A3: The terminal evaluates the confidence level of the feature embedding. When the confidence level is higher than the preset threshold, the corresponding feature embedding is transmitted to the edge / cloud. In a preferred embodiment, forward propagation is performed in step S2 and the feature embedding is calculated. At the same time, the terminal simultaneously obtains information from the aforementioned model. The final output layer obtains one or more coarse categories. confidence level Subsequently, the terminal compared the aforementioned confidence level with a preset threshold. Compare the results to perform a filtering operation.

[0060] If and only if At that time, the terminal performs an upload operation, embedding the features of the newly acquired images. The data is transmitted to the edge / cloud via a wireless link. If the above conditions are not met, the terminal discards the feature embeddings of the newly acquired image. .

[0061] Step A4: The edge / cloud calculates the inference results by performing similarity or distance measurements with the prototype centers in the prototype library; In a preferred embodiment, when the edge / cloud receives feature embeddings from the terminal... Then, from its prototype library Search all Prototype Center The edge / cloud computes feature embeddings from the newly acquired images. With each prototype in the prototype library The distance or similarity between the features is used to determine the feature embedding of the newly acquired images. Subcategories .

[0062] Specifically, the calculation process employs either similarity or distance metrics.

[0063] In a preferred approach, cosine similarity is used. Perform similarity measurement and find the prototype index with the highest similarity at the edge / cloud. :

[0064]

[0065] In another preferred approach, Euclidean distance is used. Perform distance measurement to find the prototype index with the smallest distance at the edge / cloud. :

[0066]

[0067] Edge / cloud will index the prototype Corresponding subcategories The result of (or instance recognition / target matching) is output as the final inference result.

[0068] Step A5: Update the prototype library based on the inference results.

[0069] In a preferred embodiment, to enable the prototype library to adapt to slow changes in the environment or data and improve inference accuracy, the edge / cloud can utilize newly confirmed samples. Update the prototype library.

[0070] For example, this update could use an exponential moving average algorithm. Definition For category The old prototype, As a new prototype, The update rate constant is greater than 0 and less than 1, and the update formula is:

[0071] Edge / cloud usage cover This completes a smooth update.

[0072] Based on the above embodiments, the following describes the specific application process in satellite scenarios. Communication bandwidth and onboard computing resources are particularly limited in satellite scenarios. Traditional remote sensing image analysis methods heavily rely on downloading massive amounts of raw image data (GB or even TB levels) to ground stations for processing. This not only consumes extremely valuable and expensive satellite-to-ground communication link bandwidth but also results in significant latency, failing to meet the requirements for low-latency inference. Simultaneously, the limited onboard computing and storage capabilities of satellites make it difficult to run multiple large-scale deep learning models in orbit, let alone support on-orbit model fine-tuning or retraining for sudden and diverse new tasks. Therefore, how to enable on-orbit satellites to quickly support and execute new fine-grained classification tasks issued by ground stations without increasing satellite payload or consuming downlink bandwidth is a key challenge in achieving efficient satellite-to-ground collaborative inference.

[0073] This embodiment provides a specific application of the method of this application in the scenario of satellite-ground collaborative remote sensing image classification, which significantly reduces the communication and computing load in satellite-ground collaborative inference tasks, and enhances satellite-ground collaborative inference capabilities without relying on retraining or modifying the terminal model.

[0074] In this embodiment, the terminal is an in-orbit remote sensing satellite with limited computing and communication resources. The edge / cloud is a ground-based central station with powerful computing servers and massive storage systems. The collaborative inference request is a classification task of remote sensing images acquired by the satellite, and it is a new inference task not directly supported by the terminal's existing deep neural network model. The deep neural network model is a frozen Vision Transformer model pre-trained on the large ImageNet-21k dataset.

[0075] The method disclosed in this embodiment can be described in detail through the following scenario simulation and steps: Step B1: When new inference tasks appear in the network that are not directly supported by the existing deep neural network models on the terminal side, a ground station prototype library is built. The aforementioned ground station loads a copy of the same deep neural network model as the satellite. In the scenario simulation, the aforementioned deep neural network model is the ViT-B / 16 model.

[0076] The aforementioned ground station replaces the header of a copy of the same deep neural network model as the satellite, making it a feature extractor. In the scene simulation, this feature extractor outputs a 768-dimensional feature embedding vector.

[0077] The aforementioned ground stations collect a small number of samples that constitute all new categories for the new inference task. In the scene simulation, taking the 21 subcategories in the UCMerced Land Use satellite remote sensing dataset as an example, 30 images are selected from each category as a small number of samples to constitute all new categories for the aforementioned inference task. The aforementioned Merced Land Use satellite remote sensing dataset contains 2100 256*256 pixel remote sensing images, divided into 21 scene categories, with 100 images in each category. The image resolution is 1 foot, suitable for high-resolution remote sensing image scene classification.

[0078] The ground station extracts feature embeddings from a small number of samples across all new categories, calculates their arithmetic mean to obtain the prototype centers of all new categories, and stores them in its prototype library. In the scenario simulation, 30 images are randomly selected from each of the 21 subcategories as a small number of samples for all new categories. Their arithmetic mean is calculated to obtain the prototype centers of all new categories, and stored in their prototype library. In the scenario simulation, the remaining images in the dataset serve as a test set to simulate new sample data collected by the in-orbit satellite.

[0079] Step B2: The satellite in orbit calculates its feature embedding based on the deployed deep neural network model and selects the feature embeddings that need to be transmitted to the ground station; The aforementioned in-orbit satellites acquired new high-resolution remote sensing images, which underwent image preprocessing. In the scene simulation, image preprocessing was performed to reduce the dimensionality to 224*224, which is the input dimension of the ViT-B / 16 model. This process can obtain one or more low-dimensional image patches through image cropping or transformation algorithms.

[0080] The aforementioned satellite in orbit inputs the pre-processed images into the deployed deep neural network model and performs a forward propagation.

[0081] Optionally, if the original coarse classification head of the aforementioned deep neural network model has the ability to classify the parent class of the new category, then it outputs the new high-resolution remote sensing image and the confidence score of the parent class. For example, if the original coarse classification head can output the parent class "aircraft," and the new task is to classify specific aircraft models, such as those containing subclasses like "Boeing 737" and "Boeing 747," then it outputs the confidence score of each model. Optionally, it further includes: setting a threshold to... The satellite will have confidence. With preset threshold Compare; and, when the above confidence levels Higher than The above targets are determined to be high-value targets. The feature embeddings output by the feature extractor are collected and transmitted to the ground station via a satellite-to-ground link. The above threshold is a constant greater than 0 and less than or equal to 1, which can be selected and determined according to the collaborative reasoning task. This application does not impose any restrictions on it.

[0082] Step B3: The ground station calculates the cosine similarity with the prototype center in the prototype library and outputs the inference result.

[0083] The aforementioned ground station receives feature embeddings from in-orbit satellites, retrieves the prototype centers of all new categories from the prototype library, performs similarity or distance metric calculations, and outputs the inference results. In the scenario simulation, the ground station calculates the cosine similarity score between the feature embeddings and the prototype centers of all new categories, finds the new category index corresponding to the maximum similarity, and maps it to the inference results.

[0084] Optionally, the ground station compares the inference results with real labels manually annotated by operations personnel and updates the prototype library using the newly confirmed samples.

[0085] To verify the effectiveness of the proposed solution, the following explanation is based on the scenario simulation results: As described above in this embodiment, compared to the traditional method which requires transmitting a 256x256 pixel original image (approximately 192KB, and potentially hundreds of MB in real remote sensing scenarios), the method in this application only requires transmitting a 768-dimensional feature embedding of approximately 3KB. This achieves an order-of-magnitude reduction in communication data volume, saving over 98% in this example, and thus optimizing the cloud computing load from complete reception of the neural network forward propagation to efficient vector similarity calculation, greatly improving the efficiency of collaborative inference.

[0086] Figure 3 The presentation showcases prediction results for some qualitative analysis samples, displaying 10 images, all belonging to the "agricultural" category, as query samples. Each incremental category has 30 supporting samples. The "agricultural" category is a new incremental category not directly supported by the original network model on the terminal side. Although these supporting sample images differ significantly in texture, color, and shooting angle, the method provided in this application embodiment can output correct prediction results.

[0087] Figure 4This is the classification confusion matrix of the method in this embodiment on the aforementioned test set images. The vertical axis of the confusion matrix represents the true (annotated) category label, and the horizontal axis represents the predicted category label from collaborative inference. The classification results for a total of 21 category labels are mostly concentrated on the main diagonal, indicating a high degree of consistency between the true and predicted labels. The method in this embodiment achieves an overall accuracy of up to 90.88% on the entire test set, fully verifying the effectiveness and inference performance of this embodiment.

[0088] Figure 5 This embodiment of the application describes the F1 score for each class, given that 30 images are randomly selected from each original output class as a small sample for all new classes. The F1 score is a combined evaluation metric of precision and recall, used to measure the robust performance of the model across each class. Figure 5 As can be seen from the bar chart, the method of this application embodiment achieved an F1 score greater than 0.8 in the vast majority of categories, i.e., 80%. Even Figure 4 Even for categories with slight confusion (such as "dense residential" and "medium residential"), the F1 score remained above 0.75. This further demonstrates the effectiveness and robustness of the embodiments of this application under conditions where samples of the new category are scarce.

[0089] Figure 6 This is the t-SNE visualization dimensionality reduction result in the aforementioned feature embedding space. The t-SNE is a nonlinear dimensionality reduction technique that projects high-dimensional feature embeddings onto a two-dimensional plane for observation. Figure 6 In the diagram, scatter dots of different colors represent feature embeddings from test samples of different categories. Dots with thick black borders represent prototype centers of each new category. Scatter dots of the same color within the same category spontaneously cluster into clearly identifiable clusters in the feature embedding space. These prototype centers all accurately fall within the center or high-density region of their corresponding category clusters. This diagram visually confirms the technical premise of the embodiments of this application: that the terminal's frozen model already possesses the ability to extract highly separable features; therefore, matching algorithms based on similarity or distance metrics in the edge / cloud are efficient and feasible.

[0090] The above Figures 4-6 L1 to L21 are 21 different category labels, specifically: L1: agricultural, L2: airplane, L3: baseball diamond, L4: beach, L5: buildings, L6: chaparral, L7: dense residential area, L8: forest, L9: freeway, L10: golf course, L11: harbor, L12: intersection, L13: medium-density residential area, L14: mobile home park, L15: overpass, L16: parking lot, L17: river, L18: runway, L19: sparse residential area, L20: storage tanks, L21: tennis court.

[0091] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 7 As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more. In a specific implementation, the electronic device may also include a communication interface 703 for communicating with external devices and performing data exchange and transmission.

[0092] In practical implementation, if the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0093] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0094] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0095] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0096] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0097] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0098] It should be understood that the aforementioned processor can be a CPU (Central Processing Unit), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0099] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0100] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0103] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0104] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0105] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0107] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A collaborative reasoning method applied to distributed systems, characterized in that, The distributed system includes a first type of node and a second type of node. The method is applied to the first type of node, and the method includes: In response to the processing request of the collaborative reasoning task, the system receives the target feature embedding vector sent by the second type of node; the target feature embedding vector is generated by the second type of node during the process of performing a reasoning task on the query sample using the second model, and the second model is a deep learning model statically deployed on the second type of node; Among multiple prototype vectors in the prototype vector library, a target prototype vector that matches the target feature embedding vector is determined; the multiple prototype vectors correspond one-to-one with multiple incremental categories, and the prototype vectors are generated based on the feature embedding vectors of multiple support samples belonging to the same incremental category. The feature embedding vectors are extracted during the inference task performed on the support samples using a first model with frozen parameters. The first model is the same as the second model, and the incremental category is associated with the original output category of the second model. The incremental category corresponding to the target prototype vector is determined as the reasoning result of the collaborative reasoning task.

2. The method according to claim 1, characterized in that, Before receiving the target feature embedding vector sent by the second type of node, the method further includes: The first model is used to perform an inference task on each of the support samples belonging to the incremental category, and the feature embedding vector generated during the inference task is extracted. Based on the feature embedding vectors of multiple supporting samples belonging to the same incremental category, a prototype vector of the incremental category is generated and added to the prototype vector library.

3. The method according to claim 2, characterized in that, The step of determining the target prototype vector that matches the target feature embedding vector from multiple prototype vectors in the prototype vector library includes: Calculate the distance between the target feature embedding vector and each of the prototype vectors in the prototype vector library; The prototype vector that is closest to the target feature embedding vector is determined as the target prototype vector.

4. The method according to claim 2, characterized in that, The step of generating the prototype vector of the incremental category based on the feature embedding vectors of multiple support samples belonging to the same incremental category includes: Find the vector center of the feature embedding vectors of multiple support samples; The vector center is used as the prototype vector of the incremental category.

5. The method according to claim 1, characterized in that, The incremental category includes at least one of the following: a subcategory of the original output category, a nearest neighbor category of the original output category, or an instance of the original output category.

6. A collaborative reasoning method applied to distributed systems, characterized in that, The distributed system includes a first type of node and a second type of node. The method is applied to the second type of node, and the method includes: In response to the processing request of the collaborative reasoning task, the statically deployed second model is used to perform the reasoning task on the query sample; Extract the feature embedding vectors generated during the inference task to obtain the target feature embedding vector; The target feature embedding vector is sent to the first type of node, and the target feature embedding vector is used as query information for the first type of node to perform collaborative reasoning tasks.

7. The method according to claim 6, characterized in that, Before sending the target feature embedding vector to the first type of node, the method further includes: Based on the results of the inference task, it is determined that there exists any original output category in the second model whose confidence level is higher than a preset threshold.

8. A distributed system, characterized in that, Includes first-class nodes and second-class nodes, wherein: The second type of node is used to respond to the processing request of the collaborative reasoning task, perform a reasoning task on the query sample using a statically deployed second model; extract the feature embedding vector generated during the reasoning task to obtain the target feature embedding vector; and send the target feature embedding vector to the first type of node, wherein the target feature embedding vector is used as query information for the first type of node to perform the collaborative reasoning task. The first type of node is configured to, in response to the processing request, receive the target feature embedding vector sent by the second type of node; determine a target prototype vector that matches the target feature embedding vector from multiple prototype vectors in the prototype vector library; the multiple prototype vectors correspond one-to-one with multiple incremental categories, the prototype vectors are generated based on the feature embedding vectors of multiple support samples belonging to the same incremental category, the feature embedding vectors are extracted during the inference task performed on the support samples using a first model with frozen parameters, the first model is the same as the second model, and the incremental category is associated with the original output category of the second model; the incremental category corresponding to the target prototype vector is determined as the inference result of the collaborative inference task.

9. The system according to claim 8, characterized in that, The first type of node is a ground station, and the second type of node is an in-orbit remote sensing satellite.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5 or 6-7.

Images