A semantic consistency verification-based edge-cloud collaborative robot memory construction method and system and a storage medium

Abstract

This invention relates to the field of edge-cloud collaborative robot technology, and discloses a method, system, and storage medium for constructing edge-cloud collaborative robot memory based on semantic consistency verification. Addressing the problem of difficulty in evaluating the semantic fidelity of edge models under limited bandwidth, which easily leads to long-term memory fragmentation, this invention generates a question-and-answer database from the cloud and distributes it. The edge device answers based solely on its own generated initial subtitles under image masking conditions, and the semantic consistency score is calculated by comparing the answers. Simultaneously, a dynamic trigger threshold is calculated by combining a bandwidth debt queue and real-time channel gain. When the score falls below this threshold, the neighboring sequence of abnormal frames is extracted and sent back to the cloud for batch reconstruction, overwriting the original memory of the edge device. This invention achieves online unsupervised quantization of edge device cognition under constrained networks, adaptively eliminates memory gaps, and achieves a globally optimal balance between memory quality and communication overhead, enabling the construction of high-quality contextual memories for robots in complex task scenarios.

Description

Technical Field

[0001] This invention relates to the field of edge-cloud collaborative robot technology, and in particular to a method, system and storage medium for constructing memory for edge-cloud collaborative robots based on semantic consistency verification. Background Technology

[0002] With the development of embodied intelligence technology, mobile robots are evolving from simple environmental perception relying on geometric information to deeper semantic understanding. In complex task scenarios such as embodied question answering, robots need to continuously transform visual observations into retrieveable contextual memories during long-term operation in order to respond to users' natural language queries based on historical scenarios in subsequent moments. This process relies on visual language models to map high-dimensional images into semantic text captions and vector embeddings, and then store them in a vector database along with spatial pose and timestamps. The semantic quality of the entries in the memory directly determines the accuracy of downstream question answering tasks. Therefore, how to build high-fidelity long-term semantic memories under the limited computing power and communication resources on the robot's edge has become a key issue in this field.

[0003] Existing solutions can be broadly categorized into three approaches. One approach involves a large-scale visual language model in the cloud processing the entire video stream uploaded by the robot. While this method achieves high semantic generation quality, it requires a significant and continuous consumption of uplink communication bandwidth, making it unsustainable in constrained network environments. Furthermore, the large number of redundant frames uploaded results in severe resource waste. Another approach relies entirely on a lightweight visual language model deployed on the robot's edge for local inference. Although this eliminates dependence on communication links, its limitations in model parameter size make it prone to semantic illusions and omissions of crucial information in complex or long-tailed scenarios, compromising the quality of entries stored in the memory bank. Edge-cloud collaborative solutions, falling between these two approaches, attempt to pre-screen at the edge to reduce unnecessary uploads. However, existing triggering mechanisms often employ low-level physical heuristics, such as pixel differences or optical flow changes, to determine whether to send the current frame back to the cloud for processing. These rules are semantically agnostic, making it impossible to assess online whether the edge model has experienced cognitive biases in the current scene—a static image containing crucial semantic changes might be missed due to minor pixel differences, while insignificant lighting fluctuations might trigger a large number of invalid uploads. Furthermore, the approach of statically optimizing small edge models based on offline knowledge distillation is also unable to cope with unknown scenarios in open worlds that exceed the training distribution. Once a small model fails at a critical moment, the system lacks online error correction capabilities, which will lead to a permanent break in the semantic memory of the corresponding time period.

[0004] Therefore, how to accurately quantify the semantic fidelity of edge models in an online, unsupervised manner under the condition of strictly limited communication resources, and adaptively trigger on-demand cloud reconstruction to eliminate memory gaps, while ensuring that long-term communication overhead does not exceed the system budget, is a technical problem that urgently needs to be solved in the memory construction of edge-cloud collaborative robots. Summary of the Invention

[0005] To address the technical problem that existing edge-cloud collaborative robots, under conditions of limited communication resources, struggle to accurately quantify the semantic fidelity of edge models online and in an unsupervised manner, leading to long-term semantic memory gaps and an inability to balance memory quality and communication overhead, the present invention aims to provide an edge-cloud collaborative robot memory construction method, system, and storage medium based on semantic consistency verification.

[0006] This invention provides a memory construction method for edge-cloud collaborative robots based on semantic consistency verification, comprising the following steps: acquiring multimodal observation data of the robot, wherein the multimodal observation data includes at least visual images; generating initial semantic captions and vector embeddings for the visual images in the current time slot using a lightweight visual language model deployed on the mobile robot to construct initial memory units and storing the initial memory units in a local vector database; extracting keyframes from consecutive observation frames at a preset frequency and uploading them to a cloud server; generating authoritative semantic representations for the keyframes using a large-scale visual language model on the cloud server; and constructing a plain text question bank containing several verification questions and standard answers based on the multidimensional semantic elements of the authoritative semantic representations; sending the plain text question bank to the mobile robot via a downlink communication link; and, under the condition of shielding the original visual images, using a text question-answering probe model, using only the initial semantic captions as the sole source of contextual information to answer the questions in the plain text question bank. The system answers verification questions in the library to obtain predicted answers. These predicted answers are then compared one by one with the corresponding standard answers to calculate the semantic consistency quality score of the lightweight visual language model in the current time slot, enabling online unsupervised quantification of cognitive biases in edge models. A bandwidth debt queue is maintained and updated to track the system's cumulative bandwidth overrun. A dynamic trigger threshold is calculated based on the Lyapunov optimization framework, combining the current system bandwidth debt queue value with the real-time wireless channel gain. The semantic consistency quality score is compared with the dynamic trigger threshold. If the semantic consistency quality score is lower than the dynamic trigger threshold, the corresponding abnormal frame and its temporal neighborhood sequence are extracted and sent back to the cloud server for batch reconstruction. The high-fidelity memory unit generated by the cloud reconstruction is used to overwrite the initial memory unit in the corresponding time slot in the vector database. Otherwise, the initial memory unit is retained, thereby achieving a globally optimal balance between memory quality and communication overhead over a long operating period.

[0007] As a preferred embodiment of the present invention, the multimodal observation data is represented as triples. ,in Indicates the current time slot The acquired visual image, These are the robot's current spatial coordinates. For timestamps; the formal expression of the initial memory unit is as follows: ,in For lightweight visual language models Visual images The generated initial semantic captions, For initial semantic subtitles The embedding vector representation obtained after encoding by the text embedding model.

[0008] As a preferred embodiment of the present invention, the step of generating authoritative semantic representations of the keyframes using a large-scale visual language model on the cloud server, and constructing a plain text question bank containing several verification questions and standard answers based on the multi-dimensional semantic elements of the authoritative semantic representations, includes: the large-scale visual language model on the cloud server generating high-quality semantic subtitles for the received keyframe images. This serves as the authoritative semantic representation of the scene in that frame; based on the semantic subtitles... The multidimensional semantic elements contained therein are automatically extracted and a question bank is constructed. The multidimensional semantic elements include at least key entities, attribute features, spatial relationships, and action states. For the first A verification question, This is the standard answer to the question. This represents the total number of verification questions. This design enables the verification mechanism to comprehensively detect potential defects in edge-end subtitles at various semantic levels.

[0009] As a preferred embodiment of the present invention, the mobile robot terminal utilizes a text-based question-answering probe model to answer verification questions in the plain text question bank and calculate scores, using only the initial semantic subtitles as the sole source of contextual information. Specifically, this includes: the mobile robot terminal using a text-based question-answering probe model to input the initial semantic subtitles. Confirmation problem The predicted answer is obtained by inference. ,in Define the answer similarity function. Through calculation The overall semantic accuracy on each verification question is quantified to obtain the semantic consistency quality score for the current time slot. : As a preferred technical solution of the present invention, the answer similarity function The calculation method includes any of the following: predicting the answer Compared with the standard answer Each statement is encoded into a dense vector using a pre-trained sentence embedding model, and the cosine similarity between the two vectors is calculated as a similarity score. The text reasoning capabilities of a lightweight language model on the mobile robot are then used to input the predicted answer. Compared with the standard answer The paired text is judged. If the two are determined to be semantically equivalent, the binary score 1 is output; otherwise, 0 is output.

[0010] As a preferred embodiment of the present invention, the bandwidth debt queue is implemented through a virtual queue. Real-time tracking is performed, and the update rule after the end of each time slot is as follows: in, For the current time slot Bandwidth debt queue value, To control the binary decision variable of whether to trigger cloud-based reconstruction, For the physical layer communication overhead of a single cloud reconstruction, This represents the upper limit of the long-term average bandwidth quota allocated by the system to the robot. This design effectively quantifies the system's historical bandwidth overruns.

[0011] As a preferred embodiment of the present invention, the step of calculating the dynamic triggering threshold based on the Lyapunov optimization framework, by combining the current system's bandwidth debt queue value and real-time wireless channel gain, specifically involves: minimizing the drift-penalty expression to obtain the dynamic triggering threshold in a closed-form analytical expression. : in, These are parameters used to adjust the trade-off between memory quality and communication stability. The data volume of a single frame image. The total number of frames in the continuous frame sequence to be transmitted back to the cloud for batch reconstruction. For wireless channel gain, For the robot's transmission power, It is the sum of environmental noise and interference from other equipment; when At that time, decision variables Set the value to 1 to trigger the cloud-based reconstruction.

[0012] As a preferred embodiment of the present invention, the step of extracting the corresponding abnormal frames and their continuous frame sequences within their temporal neighborhood, transmitting them back to the cloud server for batch reconstruction, and using the high-fidelity memory units generated by the cloud reconstruction to cover the initial memory units of the corresponding time slots in the vector database includes: a large visual language model on the cloud server processing the received data including abnormal frames. A continuous frame sequence is used to generate reconstructed semantic subtitles and reconstructed vector embeddings frame by frame, forming high-fidelity memory units. These high-fidelity memory units are then sent to a mobile robot, where they are overwritten with the original initial memory units in the corresponding time slots of the robot's vector database by replacing them according to timestamps or frame identifiers. This local context reconstruction strategy effectively eliminates semantic memory gaps and ensures the temporal continuity of the memory bank.

[0013] This invention also provides an edge-cloud collaborative robot memory construction system for implementing the aforementioned edge-cloud collaborative robot memory construction method based on semantic consistency verification. The system includes a mobile robot and a cloud server, with a wireless communication link established between them. The mobile robot is equipped with a lightweight visual language model, a vector database, a text question-answering probe model, a bandwidth debt queue module, and a dynamic threshold decision module. These modules are used to collect multimodal environmental observation data, generate local initial memory units, perform plain text answer verification under isolated image conditions, calculate dynamic trigger thresholds and reconstruction decisions, and store and update the local vector database. The cloud server is deployed with a large-scale visual language model, used to extract multidimensional semantic elements from received keyframes and generate a plain text question bank containing verification questions and standard answers, as well as to perform high-fidelity batch reconstruction of received abnormal continuous frame sequences.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described edge-cloud collaborative robot memory construction method based on semantic consistency verification.

[0015] Compared with the prior art, the present invention has the following beneficial effects: By employing the technical solution provided in this invention, a teacher-student semantic consistency verification mechanism based on plain text question answering is established between the edge and the cloud. This enables online unsupervised quantitative evaluation of the semantic fidelity of lightweight edge models, allowing the system to accurately identify cognitive biases of edge models in complex scenarios without relying on manual annotation. This overcomes the limitation of semantic agnosticism in triggering methods based on underlying physical heuristics. Upon detecting semantic biases, the system does not merely repair a single abnormal frame but actively extracts consecutive frame sequences within the temporal neighborhood of that frame for batch reconstruction in the cloud and overwrites the original memory, effectively eliminating long-term semantic memory gaps caused by edge model failures. Simultaneously, by introducing a dynamic triggering threshold based on Lyapunov optimization, the time-varying physical channel capacity is deeply coupled with the system's historical bandwidth debt, resulting in a closed-loop adaptive decision criterion that can be computed in real time at the edge. This allows the triggering timing of cloud reconstruction to be dynamically adjusted according to changes in network status and resource consumption, achieving an asymptotically global optimal balance between memory quality and communication overhead over a long operating period. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the method described in this invention; Figure 2 This is an architectural block diagram of the edge-cloud collaborative robot memory construction system described in this invention; The following are the reference numerals in the attached diagram: 1. Mobile robot terminal; 2. Cloud server terminal; 3. Lightweight visual language model; 4. Large visual language model; 5. Vector database; 6. Text question answering probe model; 7. Wireless communication link; 8. Bandwidth debt queue module; 9. Dynamic threshold decision module. Detailed Implementation

[0017] To make the objectives, technical solutions, and technical effects of this invention clearer, the specific embodiments of this invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of protection of this invention.

[0018] like Figure 1As shown, the overall process of the method described in this invention includes the following steps: In each runtime slot, the mobile robot terminal 1 collects multimodal observation data containing visual images, spatial coordinates, and timestamps; the lightweight visual language model 3 at the edge generates initial semantic captions and vector embeddings for the current image, forming memory units and storing them in the vector database 5; the edge extracts keyframes at a preset sampling frequency and uploads them to the cloud server terminal 2 via the wireless communication link 7; the large visual language model 4 at the cloud server terminal 2 generates high-quality captions for the received keyframes, and constructs a question bank containing several verification questions and standard answers based on the multidimensional semantic elements in the captions, which is then sent to the mobile robot terminal 1 via the downlink; the mobile robot terminal 1, after shielding the original image... Under these conditions, the text question-answering probe model 6 uses only its own generated subtitles as context to answer questions in the question bank in plain text, and compares the predicted answers with the standard answers to calculate the semantic consistency quality score; the bandwidth debt queue module 8 maintains and updates the bandwidth debt queue value based on historical decisions, and the dynamic threshold decision module 9 calculates the dynamic trigger threshold by combining the current bandwidth debt queue value, real-time channel gain and quality score through the Lyapunov optimization framework; when the quality score is lower than the dynamic threshold, the system extracts abnormal frames and their continuous frame sequences in the time neighborhood and sends them back to the cloud server 2 for high-fidelity batch reconstruction and overwrites the original memory in the vector database 5, otherwise the edge subtitles are retained; after the decision is completed, the bandwidth debt queue is updated and the system enters the next time slot.

[0019] See Figure 2 The edge-cloud collaborative robot memory construction method based on semantic consistency verification described in this invention is applicable to scenarios where mobile robots need to build high-quality contextual memories during long-term operation. The entire system is physically divided into two layers: a mobile robot terminal 1 and a cloud server terminal 2. The mobile robot terminal 1 is equipped with a lightweight visual language model 3, responsible for collecting environmental observation data and performing real-time primary semantic description generation. The cloud server terminal 2 deploys a large-scale visual language model 4, possessing high-fidelity deep semantic understanding and complex reasoning capabilities, used for refined reconstruction of questionable memory content at the edge. The two ends are connected via a wireless communication link 7, which is limited by time-varying channel conditions and limited bandwidth.

[0020] In terms of time, the system divides the robot's entire operating cycle into... There are three time slots of equal length, and the duration of each time slot is denoted as . In the first Within each time slot, The raw observations collected by the robot were organized into a triple. ,in This represents the visual image acquired in the current time slot. and These are the height and width of the image, respectively; This refers to the robot's current spatial coordinates, which can be obtained by the robot's simultaneous localization and mapping (SMR) module. In some implementation scenarios... It can also be extended to a high-dimensional pose vector that includes the orientation angle; This is the timestamp corresponding to the observation.

[0021] The core objective of the system is to utilize visual language models to interpret high-dimensional images. Mapped to highly condensed semantic text captions Combined with spatial coordinates With timestamp They are stored together in vector database 5, forming a complete memory unit. Define the generating function. The formal expression of a memory unit is then: in semantic subtitles The result obtained after text embedding model encoding The embedded vector is a key index for subsequent semantic retrieval. When a user initiates a natural language query, the system encodes the query statement into a vector and performs a nearest neighbor search in the vector database 5 to locate the memory unit most relevant to the query semantics.

[0022] The key decision the system faces in each time slot is choosing which end of the model should perform memory generation; this choice is determined by a binary decision variable. Control. When At this time, the system adopts an edge strategy, that is, it calls the lightweight visual language model 3 deployed on the mobile robot terminal 1 to process the current image, and the generated memory unit is denoted as ,in This is a text embedding function. In one specific embodiment of the invention, the lightweight edge model can employ a visual language model with approximately three billion parameters, which can generate a basically usable scene description with extremely low computational cost in typical scenarios. When At this time, the system adopts a cloud strategy, uploading the current frame image to the cloud server 2 via wireless communication link 7, where it is processed by a large visual language model 4. The resulting memory unit is denoted as... Large cloud-based models significantly outperform small edge models in terms of semantic understanding depth and generation quality, but their use comes at the cost of consuming uplink communication bandwidth.

[0023] It should be noted that, within the overall framework of this invention, the memory construction phase is decoupled from the subsequent memory retrieval phase for embodied question-answering tasks. Memory construction occurs during the robot's daily inspections or autonomous exploration, aiming to continuously transform the environmental information perceived along the way into structured semantic vectors and store them in the vector database 5. Memory retrieval, on the other hand, is triggered when a user initiates a natural language question, retrieving relevant memory units from the vector database 5 to assist in generating an accurate answer. Because the two phases are asynchronous and decoupled, the semantic quality of the content stored in the memory bank largely determines the final accuracy of the question-answering task.

[0024] To characterize memory quality, this invention uses the memory generated by a large-scale cloud model as a semantic benchmark, defining the quality of a single-frame memory unit as the degree of consistency between its contained semantic information and this benchmark. Specifically, a quality assessment operator is defined. To measure memory units With cloud benchmark The semantic similarity between them is then reflected in the decision variables. Next, the The memory quality function for each time slot is expressed as: because It is itself a semantic benchmark, and has The above formula can be simplified to: in This refers to the semantic captions generated at the edge. With cloud-based ideal subtitles The semantic similarity between them is an metric that intuitively reflects the perceptual fidelity of small edge models in the current scene. When When the value approaches 1, it means that the smaller model's understanding of the current scene is highly consistent with that of the larger model. In this case, retaining the edge captions can achieve near-optimal memory quality; conversely, when... If the value is too low, it indicates that the small model has produced significant semantic bias in the current scenario, and it is necessary to trigger cloud reconstruction to restore memory quality.

[0025] At the communication level, set at time The robot's transmission power is The wireless channel gain is The sum of environmental noise and interference from other equipment is If the system decides to upload the image for cloud reconstruction, it needs to transmit the current frame and its temporal neighborhood. Frame image data, each frame is [size missing] According to Shannon's channel capacity theorem, data transmission must occur within the deadline. This is completed internally, thus yielding the physical layer transmission feasibility constraints: This constraint ensures that, under given channel conditions, the amount of data uploaded does not exceed the physical link's capacity limit. Meanwhile, considering the limited communication resources during long-term robot operation, the upper limit of the long-term average bandwidth quota allocated to this robot by the system is set as follows: The standard network resource usage required for a single upload is The system also needs to meet the following long-term bandwidth quota constraints: Based on the above objective function and constraints, the core optimization problem of this invention can be formalized as: finding an optimal set of decision sequences under constraints of physical transmission feasibility, long-term bandwidth quota, upper bound of transmission power, and discrete values ​​of decision variables. The goal is to maximize the expected memory quality throughout the entire runtime. This optimization problem is expressed as follows: The constraints are: Solving the aforementioned optimization problem faces two main difficulties: first, the unpredictability of future environmental changes and channel conditions makes it impossible to determine the globally optimal sequence in advance; second, the discrete values ​​of the decision variables place the problem within the category of mixed-integer nonlinear programming, which conventional continuous convex optimization solvers cannot directly handle. To address this difficulty, this invention introduces a semantic consistency verification mechanism to obtain... The system performs online estimation and utilizes Lyapunov optimization theory to transform long-term constraints into instantaneous decision rules that can be solved frame by frame, thereby achieving asymptotically global optimal online control at the edge with extremely low computational overhead.

[0026] The implementation of the semantic consistency verification mechanism is explained in detail below.

[0027] During system operation, the actual perceived fidelity The reason it cannot be directly obtained is that its definition relies on the output of a large model in the cloud as a benchmark, and uploading each frame to the cloud for comparison is precisely the high-bandwidth operation that this invention aims to avoid. Therefore, this invention designs a teacher-student collaborative quality verification mechanism based on plain text question answering, which estimates the semantic quality of edge-end memory online in a low-overhead manner.

[0028] The mechanism operates in three stages. In the teacher-generated question stage, the edge samplers take data from consecutive observation frames at a pre-set fixed frequency, at time... A keyframe The keyframe is uploaded to cloud server 2 via wireless communication link 7. The upload frequency of this frame is much lower than the robot's image acquisition frame rate, thus the resulting bandwidth overhead is controllable. The large visual language model 4 on cloud server 2 acts as a "teacher," performing two operations on the received keyframe: firstly, generating high-quality semantic subtitles. On the one hand, it serves as the authoritative semantic representation of the scene in that frame; on the other hand, based on the key entities, attribute features, spatial relationships, and action states appearing in the subtitles, it automatically extracts and constructs a semantic representation containing... A question bank of verification questions and their standard answers. Its formal expression is: in For the first A verification question, This is the standard answer to the question. These are preset hyperparameters, and their values ​​should be selected to balance the coverage of the evaluation with computational cost. The questions are designed to encompass semantic features across different dimensions, such as spatial relationships, object attributes, entity existence, and ongoing actions and events within the scene, thereby ensuring multi-faceted detection of the semantic completeness of edge-end subtitles. Once the question bank is built, it will be available in the cloud. The downlink data packets for the question-and-answer pairs are sent to the mobile robot terminal 1. Since the content of the question bank is in plain text format, its data volume is much smaller than that of image data, and the bandwidth occupied by downlink transmission is almost negligible.

[0029] In the student's response phase, the edge-end mini-model acts as the "student." A key design principle in this invention is that the student's responses must be completed under conditions of isolation from the original visual image. Specifically, the lightweight visual language model 3 on the mobile robot end 1 processes the same frame of image... Generate its own semantic captions independently: This subtitle represents the edge memory content to be evaluated. Subsequently, the mobile robot 1 uses the text-based question-answering probe model 6 to answer each question in the question bank. This probe model can directly reuse the text reasoning module of the lightweight visual language model 3, without requiring the deployment of a dedicated question-answering model. During the answering process, the system uses the original image... Complete masking; text question answering probe model 6 only uses subtitles generated at the edge. Based on this as the sole source of contextual information, the answer to each question can be inferred: The intention behind this design is: if the subtitles are at the edge... The image was captured completely and accurately. If the key semantic information in the text is present, then the various verification questions generated by the cloud based on truth-based subtitles can be answered correctly using only this subtitle; conversely, if... The presence of semantic omissions, descriptive biases, or illusory content will inevitably reveal biased answers on the corresponding dimensions of the questions. The constraint of masking the original image ensures that the answering process strictly relies on the information quality of the subtitles themselves, rather than bypassing the subtitles to directly obtain answers from the image. This allows the answering results to truly reflect the semantic fidelity of the subtitles.

[0030] In the quality quantification estimation stage, the system compares students' predicted answers with the standard answers provided by teachers one by one, and calculates... The overall semantic accuracy on each question is used to quantify the quality of the captions at the edges of the current frame. A similarity function for the answers is defined. This function can be implemented based on the cosine distance of the embedding vectors, or it can use a lightweight language model to determine the semantic equivalence of two answers and output a binary or continuous score. Based on this, the quality score of the edge-end subtitles is calculated. The calculation is as follows: This score constitutes the perceptual fidelity of the small model within a local time window. A reliable numerical estimate. When When the value is close to 1, it indicates that the semantic output of the small edge model in the current scenario is highly consistent with that of the large cloud model, and the corresponding edge memory can be trusted and retained in the vector database 5; while when If the value is significantly low, it indicates that the small model has a cognitive bias in the current scenario, and the corresponding edge memory is at risk of semantic distortion, requiring subsequent corrective measures.

[0031] The quality score generated by the above verification mechanism This provides a quantitative basis for subsequent adaptive reconstruction triggering decisions. The following explains how this invention utilizes Lyapunov online optimization theory to transform the aforementioned long-term stochastic optimization problem into a frame-by-frame solvable deterministic decision rule.

[0032] To meet long-term bandwidth quota constraints, this invention introduces a virtual queue. To track the system's cumulative bandwidth overrun in real time, this virtual queue is maintained by the Bandwidth Debt Queue Module 8. The update rules for this virtual queue are as follows: Among them, the communication time overhead of a single cloud reconstruction Rigorous definition by Shannon's channel capacity theorem: queue The physical meaning can be understood as the system's cutoff time. The "bandwidth debt." When a system performs a cloud-based refactoring in a certain time slot but its communication overhead exceeds the current quota, the excess will accumulate to... In the middle, the queue value increases; conversely, when the system chooses to retain edge memory in the time slot, the queue will gradually decrease. A larger queue value indicates a more severe historical bandwidth overrun, and subsequent decisions should be more conservative to avoid further debt accumulation. At each time step... The complete state of the system is defined as a triple. Each of these includes the current bandwidth debt level, real-time channel gain, and the quality score obtained from the most recent semantic consistency check.

[0033] Based on this, a quadratic Lyapunov function is defined to quantify the network congestion level of the system: This function is in When it reaches its minimum value of zero, it follows The value increases monotonically with increasing , and therefore can be used as a benchmark to measure the degree to which a system deviates from its steady state. Further, we define single-step conditional Lyapunov drift. Used to characterize the current state Expected increase in debt after implementing the decision: The solution approach of this invention seeks an optimal balance between the two competing objectives of "stabilizing bandwidth debt" and "maximizing memory quality." To this end, the original optimization objective is transformed into minimizing the following "drift-penalty" expression: in These are parameters used to adjust the algorithm's pursuit of memory quality. When the value is large, the system tends to prioritize high-quality memory and allow for higher bandwidth debt fluctuations in the short term. When the value is small, the system becomes more conservative, prioritizing the stability of bandwidth consumption.

[0034] The drift term is then bounded upwards to derive a deterministic expression solvable in real time. The two sides of the virtual queue update equation are expanded by squares, and inequalities are used... We can obtain: Expand the right end: Dividing both sides by 2 and taking the conditional expectation, we can obtain the strict upper bound of the drift term: in Let this be a positive constant constrained by the maximum single communication cost of the system; once the system parameters are determined, this value is a known bounded quantity. Substituting this upper bound into the drift-penalty expression, we obtain: According to Lyapunov's optimization theorem, greedily minimizing the above upper bound in each time slot guarantees the asymptotic optimality of the original long-run average optimization problem. Furthermore, note that in the current time slot... State variables , and Since all variables are locally observable in real time rather than random variables, the originally complex cross-time-slot stochastic optimization problem is equivalently reduced to solving the following deterministic, frame-independent scalar minimization problem: Due to decision variables There are only two possible values, and the globally optimal solution can be obtained directly through enumeration and comparison using the dynamic threshold decision module 9. This requires no invocation of any iterative solver, resulting in extremely low computational overhead, making it perfectly suitable for real-time execution on resource-constrained edge environments. The objective function values ​​corresponding to the two decisions are calculated separately: when That is, when retaining edge memory: when That is, when cloud-based reconstruction is triggered: The necessary and sufficient condition for the system to choose to perform cloud-based reconstruction is: ,Right now: By rearranging the terms of the above inequality, we can obtain the semantic quality score. Judgment criteria: Physical layer communication overhead Substituting the expression, we finally obtain the dynamic trigger threshold. Closed-form analytic expression: The dynamic threshold It has a clear physical meaning and adaptive adjustment capability. (When bandwidth debt queue) When the threshold is large, meaning the system has accumulated significant communication overruns, the threshold is... Automatic reduction means that cloud-based reconstruction will only be triggered when the semantic quality at the edge is severely degraded, thereby prompting the system to digest existing debt and restore communication margin; conversely, when... When the threshold is lower, the threshold increases accordingly, allowing the system to more actively utilize cloud resources to improve memory quality. Channel conditions The impact is reflected in the Shannon capacity term of the denominator: when channel conditions are good, the time overhead of a single transmission is reduced. Lowering the threshold makes it more lenient; as the channel deteriorates, transmission overhead increases, and the threshold tightens accordingly. Trade-off parameters. Its function is to regulate the global balance between the system's pursuit of memory quality and its conservatism regarding communication stability. The larger the value, the more aggressive the pursuit of quality and the higher the overall threshold. The smaller the value, the stricter the bandwidth control and the lower the overall threshold.

[0035] In actual judgment execution, whenever the consistency verification module outputs a quality score... Then, the dynamic threshold decision module 9 immediately compares it with the threshold calculated based on the current system state. Compare. If If an unacceptable semantic deviation is found at the edge in the current scene, the system immediately triggers an active sampling and backhaul process. In this invention, the backhaul scope is not limited to a single abnormal frame, but simultaneously extracts the consecutive frame sequences in the immediate and subsequent neighborhoods of the abnormal frame on the timeline, totaling... Frames are uniformly uploaded via wireless communication link 7 to a large visual language model 4 on cloud server 2 for high-fidelity batch reconstruction. The cloud generates high-quality semantic subtitles and vector embeddings for each frame, replacing the original edge-end memories in the corresponding time slots of the vector database 5 through overwriting. This local context reconstruction strategy effectively eliminates semantic memory gaps that may result from the failure of small models in key interaction events, ensuring the continuity and integrity of the memory bank in the temporal dimension. Or, due to extremely congested network conditions... If the value drops below zero, the system retains the edge captions as final memory and does not initiate additional communication overhead.

[0036] In a specific implementation example, this invention is deployed in a mobile robot experimental environment based on a simulation platform. The system interface displays a monitoring interface for the system memory reconstruction process, showing when cloud reconstruction is triggered during robot inspection and the frame range covered by the reconstruction. The simulator's real-time display shows the robot's operation in the virtual environment. Significant differences exist between the caption annotations of the edge-side small model and the cloud-side large model; some semantic information is correctly recognized only by the cloud-side large model, while the edge-side small model cannot capture it. Therefore, the difference in perception capabilities between the two types of models in complex scenarios underlies the necessity of the consistency verification mechanism proposed in this invention to trigger reconstruction.

[0037] In a preferred embodiment of the present invention, the verification questions generated during the teacher-generated question phase cover the following dimensions: existence verification of objects appearing in the scene, such as confirming whether a specific category of object exists in the scene; verification of the visual attributes of objects, such as color, shape, material, or size; verification of spatial relationships between objects, such as relative position or containment relationship; and verification of actions or events occurring in the scene. This multi-dimensional question design enables the verification mechanism to comprehensively detect potential defects in edge-end subtitles at various semantic levels, rather than relying solely on a single indicator for judgment. Hyperparameters The value of can be adjusted in actual deployment based on the computing power margin and response latency requirements of the edge terminal. In typical implementation scenarios, it is usually set to . Using an integer between three and eight can balance evaluation accuracy and computational efficiency.

[0038] In another embodiment of the invention, the answer similarity function The implementation can be achieved using one of the following methods: Encode the predicted answer and the standard answer into vectors using a pre-trained sentence embedding model, and calculate the cosine similarity between them as a score; or utilize the reasoning capabilities of a lightweight language model at the edge, inputting the paired text of the predicted answer and the standard answer, and having it determine whether they are semantically equivalent and output a binary score. The former is more computationally efficient and suitable for scenarios where edge computing power is extremely limited; the latter can capture finer-grained semantic differences to a certain extent and is suitable for scenarios with higher requirements for evaluation accuracy. Both methods are within the scope of protection of this invention.

[0039] In yet another embodiment of the invention, a virtual queue initial value Setting it to zero indicates that the system has no bandwidth debt at startup. As runtime progresses, the evolution trajectory of the queue is entirely determined by the historical decision sequence and actual channel conditions. In the initial stage of robot deployment, due to... Smaller, dynamic threshold With relatively relaxed bandwidth, the system can utilize the ample communication budget during this phase to perform more frequent cloud-based refactoring, quickly establishing a high-quality initial memory. However, as runtime increases, if bandwidth consumption continues to exceed the quota, As the accumulation progresses, the threshold tightens, and the system automatically shifts to a conservative mode that relies primarily on peripheral memory, triggering necessary reconstruction only when severe cognitive biases occur in small models. This natural transition from radical to conservative is an inherent characteristic of the Lyapunov framework, requiring no manually set phased strategy switching rules.

[0040] In weighing parameters Regarding the selection of [specific components], the present invention provides the following guiding principles. The physical meaning of can be understood as the maximum incremental bandwidth debt that the system is willing to bear to improve memory quality by one unit. In deployment scenarios with relatively abundant bandwidth resources, this can be appropriately increased. The value of should be chosen to obtain a higher upper bound on memory quality; in scenarios where bandwidth resources are extremely scarce, a smaller value should be selected. To ensure the long-term communication stability of the system. In practical engineering deployments, a small number of offline simulation experiments can be used to determine a system that achieves the optimal quality-cost tradeoff within the target communication budget. value.

[0041] It should be further noted that the sampling frequency of keyframes and the generation frequency of the verification question bank in the method of this invention can be flexibly configured according to actual deployment requirements. In scenarios with relatively stable environmental changes, the sampling frequency can be appropriately reduced to further save uplink bandwidth; in scenarios with drastic environmental changes or high task criticality, the sampling frequency can be increased to shorten the detection latency of semantic bias. Furthermore, when the system detects multiple consecutive time slots... When all values ​​remain at a high level, the subsequent sampling frequency can be adaptively reduced without affecting memory quality assurance, thereby further optimizing the efficiency of communication resource utilization.

[0042] At the system implementation level of this invention, the edge-side vector database 5 can employ a lightweight embedding retrieval engine that supports incremental updates and overwrite operations. This engine needs to support targeted entry replacement based on timestamps or frame identifiers to facilitate the memory overwrite process after cloud reconstruction. (Text embedding function) A pre-trained multilingual sentence encoder can be used to map semantic captions to a unified dense vector space. During the retrieval phase, the user's natural language query, after being mapped by the same encoder, is matched against the nearest neighbor of the memory vectors stored in the vector database 5. This returns the set of memory units with the highest semantic relevance. The question-answering generation module then integrates these memories with the query content to generate the final answer. Because the semantic quality of the entries in the vector database is guaranteed during the memory construction phase through the consistency verification and on-demand reconstruction mechanism of this invention, the memory units obtained during the retrieval phase have high reliability, thereby effectively improving the accuracy of the downstream embodied question-answering task.

[0043] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification, characterized in that, Includes the following steps: Acquire multimodal observation data of the robot, the multimodal observation data including at least visual images, generate initial semantic captions and vector embeddings for the visual images of the current time slot through a lightweight visual language model deployed on the mobile robot, so as to construct an initial memory unit and store the initial memory unit in a local vector database; Keyframes are extracted from continuous observation frames at a preset frequency and uploaded to the cloud server. The cloud server generates authoritative semantic representations of the keyframes using a large-scale visual language model. Based on the multi-dimensional semantic elements of the authoritative semantic representations, a plain text question bank containing several verification questions and standard answers is constructed. The plain text question bank is sent to the mobile robot terminal via a downlink communication link. Under the condition of shielding the original visual image, the mobile robot terminal uses a text question-answering probe model to answer the verification questions in the plain text question bank using only the initial semantic captions as the sole source of contextual information, and obtains the predicted answers. The predicted answer is compared with the corresponding standard answer one by one, and the semantic consistency quality score of the lightweight visual language model in the current time slot is calculated. Maintain and update a bandwidth debt queue for tracking the cumulative bandwidth overrun of the system. Combine the current bandwidth debt queue value of the system with the real-time wireless channel gain and calculate the dynamic trigger threshold based on the Lyapunov optimization framework. The semantic consistency quality score is compared with the dynamic trigger threshold. If the semantic consistency quality score is lower than the dynamic trigger threshold, the corresponding abnormal frame and its continuous frame sequence in the time neighborhood are extracted and sent back to the cloud server for batch reconstruction. The high-fidelity memory unit generated by the cloud reconstruction is used to cover the initial memory unit of the corresponding time slot in the vector database. Otherwise, the initial memory unit is retained.

2. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 1, characterized in that, The multimodal observation data are represented as triples. ,in Indicates the current time slot The acquired visual image, These are the robot's current spatial coordinates. For timestamps; the formal expression of the initial memory unit is as follows: ,in For lightweight visual language models Visual images The generated initial semantic captions, For initial semantic subtitles The embedding vector representation obtained after encoding by the text embedding model.

3. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 1, characterized in that, The process involves generating authoritative semantic representations of the keyframes using a large-scale visual language model on the cloud server, and constructing a plain text question bank containing several verification questions and standard answers based on the multi-dimensional semantic elements of the authoritative semantic representations. Large-scale visual language models on cloud servers generate high-quality semantic captions for received keyframe images. , serving as the authoritative semantic representation of the scene in that frame; Based on the semantic subtitles The multidimensional semantic elements contained therein are automatically extracted and a question bank is constructed. The multidimensional semantic elements include at least key entities, attribute features, spatial relationships, and action states. For the first A verification question, This is the standard answer to the question. This represents the total number of confirmatory questions.

4. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 3, characterized in that, The mobile robot terminal utilizes a text-based question-answering probe model, using only the initial semantic captions as the sole source of contextual information to answer verification questions in the plain text question bank and calculate scores. Specifically, this includes: The mobile robot uses a text-based question-answering probe model to input initial semantic captions. Confirmation problem The predicted answer is obtained by inference. ,in ; Define the answer similarity function Through calculation The overall semantic accuracy on each verification question is quantified to obtain the semantic consistency quality score for the current time slot. : 。 5. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 4, characterized in that, The answer similarity function The calculation method includes any of the following: Predict the answer Compared with the standard answer The two sentences are encoded into dense vectors using a pre-trained sentence embedding model, and the cosine similarity between them is calculated as the similarity score. Leveraging the text reasoning capabilities of a lightweight language model on a mobile robot, input the predicted answer. Compared with the standard answer The paired text is judged. If the two are determined to be semantically equivalent, the binary score 1 is output; otherwise, 0 is output.

6. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 1, characterized in that, The bandwidth debt queue is accessed via a virtual queue. Real-time tracking is performed, and the update rule after the end of each time slot is as follows: in, For the current time slot Bandwidth debt queue value, To control the binary decision variable of whether to trigger cloud-based reconstruction, For the physical layer communication overhead of a single cloud reconstruction, This is the upper limit of the long-term average bandwidth quota allocated by the system to this robot.

7. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 6, characterized in that, The dynamic triggering threshold is calculated based on the Lyapunov optimization framework by combining the current system's bandwidth debt queue value and real-time wireless channel gain. By minimizing the drift-penalty expression, the dynamic triggering threshold of the closed-form analytical expression is obtained. : in, These are parameters used to adjust the trade-off between memory quality and communication stability. The data volume of a single frame image. The total number of frames in the continuous frame sequence to be transmitted back to the cloud for batch reconstruction. For wireless channel gain, For the robot's transmission power, It is the sum of environmental noise and interference from other equipment; when At that time, decision variables Set the value to 1 to trigger the cloud-based reconstruction.

8. The method for constructing the memory of an edge-cloud collaborative robot based on semantic consistency verification according to claim 1, characterized in that, The process of extracting the corresponding abnormal frames and their consecutive frame sequences within their temporal neighborhood, transmitting them back to the cloud server for batch reconstruction, and using the high-fidelity memory units generated by the cloud reconstruction to overwrite the initial memory units of the corresponding time slots in the vector database includes: The large-scale visual language model on the cloud server processes a total of [number] received frames, including those with anomalous characteristics. A continuous frame sequence is used to generate reconstructed semantic subtitles and reconstructed vector embeddings frame by frame, forming a high-fidelity memory unit; The high-fidelity memory unit is sent to the mobile robot terminal, and the original initial memory unit written to the corresponding time slot in the vector database of the mobile robot terminal is overwritten by the targeted replacement method according to the timestamp or frame identifier, so as to eliminate semantic memory gaps.

9. A memory construction system for edge-cloud collaborative robots, used to implement the memory construction method for edge-cloud collaborative robots based on semantic consistency verification as described in any one of claims 1 to 8, characterized in that, The system includes a mobile robot terminal (1) and a cloud server terminal (2), and a wireless communication link (7) is established between the mobile robot terminal (1) and the cloud server terminal (2); the mobile robot terminal (1) is equipped with a lightweight visual language model (3), a vector database (5), a text question answering probe model (6), a bandwidth debt queue module (8), and a dynamic threshold decision module (9), which are used to collect environmental multimodal observation data, generate local initial memory units, verify plain text answers under isolated image conditions, calculate dynamic trigger thresholds and reconstruction decisions, and store and update the local vector database; the cloud server terminal (2) is equipped with a large visual language model (4), which is used to extract multidimensional semantic elements from the received keyframes and generate a plain text question bank containing verification questions and standard answers, and to perform high-fidelity batch reconstruction of the received abnormal continuous frame sequences.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the edge-cloud collaborative robot memory construction method based on semantic consistency verification as described in any one of claims 1 to 8.

Images