A robot multi-camera shared frame resource scheduling and distribution method and system

Abstract

The application discloses a kind of robot multi-camera shared frame resource scheduling and distribution method and system, wherein method steps include: the frame of multiple cameras acquisition is registered as shared frame resource and generates standardization description information, based on frame resource description information, consumer ability parameter and credit state executes access matching, determines distribution object after same frame resource fan-out distribution to multiple consumer, sets reference count before distribution according to total number of distribution object;According to the consumption completion state or abnormal state of consumer update reference count, when count is zero and synchronization condition is satisfied, recycle frame resource;While running state control and fault isolation are carried out to consumer, dynamically update credit state and feedback to access matching link, realize the binding linkage of fault isolation and frame life cycle.

Description

Technical Field

[0001] This invention relates to the field of robot vision data processing technology, and in particular to a method and system for scheduling and distributing shared frame resources among multiple robot cameras. Background Technology

[0002] With the rapid development of robotics technology, modern robots are typically equipped with multiple cameras, including front-view, rear-view, left-view, right-view, fisheye, and depth cameras, to meet diverse task requirements such as remote driving, environmental perception, inspection recording, local debugging, and fault analysis. In actual operation, image frames captured by the same camera often need to simultaneously serve multiple consumer ends, including local display, remote preview, video encoding, local storage, UVC output, screenshotting, and diagnostics, forming a typical "one source, multiple ends" concurrent consumption scenario. Therefore, how to efficiently achieve the sharing, scheduling, and distribution of multiple camera frames among multiple consumer ends has become a key technical requirement in the field of robot vision data processing.

[0003] Currently, the existing field of robot vision data processing commonly adopts solutions such as independent copy distribution, single video pipeline, remote preview, shared buffer mechanism, and local recording. Although these solutions can solve single-point problems such as display, transmission, storage, or buffer sharing, they still have significant shortcomings when facing concurrent scenarios with multiple cameras and multiple consumer terminals on a robot.

[0004] Existing solutions typically make distribution decisions directly at the camera acquisition link, with each consumer independently acquiring data. The lack of a unified frame resource description and shared import entry point leads to redundant memory copying, format conversion, and encoding when multiple consumers concurrently consume the same camera frame stream, significantly increasing CPU, memory bandwidth, and power consumption. Furthermore, existing solutions lack a unified capability registration and admission mechanism for consumers. Different consumers have varying requirements for format, resolution, latency, and queue depth, resulting in incompatible frames or frames exceeding latency limits still being pushed to consumers, increasing operational risks. Moreover, multiple consumers sharing the same acquisition queue lacks consumer-level backpressure isolation and abnormal release mechanisms. When a consumer processes slowly, experiences network congestion, or abnormally exits, it can block the main camera acquisition link and affect other normal consumers.

[0005] Existing shared buffer mechanisms (such as DMA-BUF and shared memory) only provide low-level zero-copy transmission capabilities and lack unified resource governance for the camera frame lifecycle. They cannot solve the problems of reference counting management, deterministic reclamation, and synchronization mark release when multiple consumers share the same frame, leading to premature release or long-term occupation of frame resources and uncertain reclamation. In addition, in scenarios where multiple cameras are working simultaneously, existing solutions lack global resource scheduling across cameras and consumers. There is a lack of unified buffer, encoder channel, and network bandwidth allocation mechanisms among tasks such as local display, remote streaming, encoding storage, and diagnostic frame extraction, which exacerbates resource contention and conflicts when multiple cameras are running concurrently.

[0006] This shows that current technologies are still insufficient to meet the needs of robots for efficient, stable, and controllable visual data processing in scenarios with multiple cameras and multiple consumer terminals operating concurrently. Summary of the Invention

[0007] Therefore, the main objective of this invention is to provide a method and system for scheduling and distributing shared frame resources for multiple cameras in a robot, in order to solve the problems of repeated processing by multiple consumer terminals and blockage of the main link for camera acquisition.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for scheduling and distributing shared frame resources of multiple cameras in a robot is provided, comprising:

[0009] Pre-configuration steps: Pre-configure the capability parameters and operating constraint parameters of each consumer terminal, and complete the global resource initialization allocation for multiple cameras;

[0010] The single-frame scheduling step performs the following processing on each camera frame:

[0011] Register shared frame resources and generate shared frame resource description information including frame attributes, shared buffer handle, synchronization flag and reference count;

[0012] Based on the shared frame resource description information, consumer capability parameters, and consumer credit status, admission matching is performed to determine consumers that can be directly shared and distributed; consumers that do not meet the direct admission conditions but are allowed to be downgraded are included in the downgraded distribution path.

[0013] The same shared frame resource is distributed to all distributable consumers. Before distribution, the reference count of the shared frame resource is set according to the total number of distributable consumers.

[0014] Monitor the running status of each consumer. Decrement the reference count when a consumer completes consumption or encounters an abnormal state. When the reference count reaches zero and the synchronization condition is met, reclaim the shared frame resources.

[0015] The system manages and isolates faults at each consumer end, dynamically updates the credit status of each consumer end and feeds it back to the access matching process, and links the fault isolation mechanism with the lifecycle management of shared frame resources.

[0016] In an optional implementation, the frame attributes include at least one of the following: camera identifier, frame number, timestamp, image format, resolution, and planar layout information; the shared frame resource description information further includes at least one of the following: step size, offset, number of planes, buffer modifier, color space, rotation direction, production completion synchronization fence, and lifecycle status; the shared buffer handle is at least one zero-copy buffer handle of DMA-BUF file descriptor, shared memory handle, and GPU texture handle.

[0017] In an optional implementation, the consumer-side capability parameters include at least one of the following: acceptable image format, resolution, and shared buffer import capability; the operational constraint parameters include at least one of the following: maximum latency, queue depth, priority, and frame dropping strategy; the consumer-side capability parameters also include at least one of the following: frame rate, planar layout requirements, buffer modifier requirements, whether color conversion is required, whether degradation to copy path is allowed, user visibility, task mode, and subscription relationship.

[0018] In an optional implementation, the credit status of the consumer is used to characterize whether the consumer queue is congested, whether there are consecutive timeouts, whether it is in an abnormal isolation state or in the recovery observation phase.

[0019] In an optional implementation, the admission matching step includes:

[0020] When a consumer supports the image format, resolution, planar layout, and shared buffer type of the current shared frame resource, and the current queue depth of the consumer has not reached the upper limit of the queue depth, the latency does not exceed its maximum latency constraint, and the credit status is in a normal state, it is determined to be a consumer that can be directly shared and distributed.

[0021] If a consumer does not support the current shared buffer format, has a resolution mismatch, or does not meet the latency conditions, and the consumer allows for downgrading, it will be included in the downgraded distribution path; otherwise, the distribution of this frame resource to the consumer will be refused.

[0022] When determining the distribution targets, the system sorts and filters based on subscription relationships, task modes, user visibility, consumer priority, and consumer credit status to generate the final distribution set.

[0023] In an optional implementation, the degraded distribution path includes at least one of the following: format conversion, scaling, post-encoding distribution, and CPU copy path; the consumer end included in the degraded distribution path decrements the reference count of the shared frame resource after completing consumption, timeout, or abnormal exit.

[0024] In an optional implementation, the abnormal state includes at least one of: consumer timeout, abnormal exit, and consumption failure; the synchronization condition is that the production completion synchronization mark or synchronization fence corresponding to the shared frame resource has been released; when the consumer triggers an abnormal state, the abnormal isolation process is first performed on the consumer, and then the reference count of the corresponding shared frame resource is decremented.

[0025] In an optional implementation, the steps of runtime management and fault isolation include:

[0026] Back pressure isolation: When the consumer queue depth reaches the upper limit, traffic control and frame dropping are performed on the consumer; the frame dropping strategy includes at least one of the following: retain the latest frame and discard the old frame, prioritize key frames, reduce the frame rate, fix the frame rate, and extract frames proportionally.

[0027] Degradation Isolation: When a consumer times out continuously, the credit status of that consumer is set to an abnormal isolation state, the distribution of shared frame resources to it is suspended, and a recovery observation timer is started.

[0028] Recovery Observation: During the recovery observation period, if the queue depth and timeout count of the consumer meet the recovery conditions, the credit status of the consumer will be restored to normal and it will be allowed to re-enter the admission matching process; if it has not returned to normal by the end of the recovery observation period, the abnormal isolation status will be maintained and the abnormal information will be recorded.

[0029] In an optional implementation, the method steps further include:

[0030] Global resource allocation and scheduling: Based on camera priority, consumer priority, user visibility, and task mode, dynamically adjust the allocation of shared buffer pool, encoder channel, and network bandwidth; when system resources are contested, prioritize the resource supply of high-priority consumers corresponding to high-priority cameras, and allow low-priority consumers to enter a degraded path or waiting state when resources are insufficient.

[0031] To achieve the above objectives, according to another aspect of the present invention, a shared frame resource scheduling and distribution system for performing any of the above methods is also provided, comprising:

[0032] Multi-camera acquisition module, used to acquire raw image frames from multiple cameras;

[0033] The multi-camera global scheduler is used to dynamically adjust the allocation of shared buffer pool, encoder channels and network bandwidth based on camera priority, consumer priority, user visibility and task mode. When system resources are contested, priority is given to ensuring the resource supply of high-priority consumers corresponding to high-priority cameras, and low-priority consumers are allowed to enter a degraded path or wait state when resources are insufficient.

[0034] The shared frame resource registration module is used to register frames acquired by the camera as shared frame resources and generate shared frame resource description information that includes frame attributes, shared buffer handles, synchronization flags and reference counts.

[0035] The consumer-side capability management module is used to pre-configure the capability parameters and operational constraint parameters of each consumer.

[0036] The consumer access and downgrade module is used to perform access matching based on shared frame resource description information, consumer capability parameters, and consumer credit status to determine consumers that can be directly shared and distributed; and to include consumers that do not meet the direct access conditions but are allowed to be downgraded in the downgraded distribution path.

[0037] A multi-consumer fan-out module is used to distribute the same shared frame resource to all distributable consumers, and to set the reference count of the shared frame resource according to the total number of distributable consumers before distribution;

[0038] The reference counting and recycling module is used to monitor the running status of each consumer and decrement the reference count when the consumer completes consumption or encounters an abnormal state; when the reference count reaches zero and the synchronization condition is met, the shared frame resources are recycled.

[0039] The back pressure and fault isolation module is used to manage and isolate faults in the operation of each consumer terminal, dynamically update the credit status of the consumer terminal and feed it back to the consumer terminal access and degradation module, and bind the fault isolation mechanism with the lifecycle management of shared frame resources.

[0040] The robot multi-camera shared frame resource scheduling and distribution method and system provided by this invention cleverly transforms the traditional "push mode" oriented towards camera acquisition links into a "pull mode" oriented towards matching shared frame resources with consumer capabilities, and constructs a complete resource governance closed loop of "unified description → capability access → fan-out sharing → reference count recycling → back pressure isolation → global scheduling". This system ensures that once a camera frame is acquired, it is registered as a standardized resource object that can be shared, tracked, and negotiated. Distribution decisions are no longer directly oriented towards the acquisition link, but rather towards the admission matching results between the shared frame resource description information and the consumer's capability parameters. Simultaneously, consumer-level backpressure isolation, fault isolation, and reference counting recycling are bound to the same shared frame resource lifecycle. This limits the impact of slow or abnormal consumer failures to their own scope, preventing them from blocking the main camera acquisition link. This eliminates the problem of duplicate processing by multiple consumers, significantly reduces system overhead, avoids the erroneous push of frames with mismatched capabilities, reduces operational risks, prevents slow or abnormal consumers from blocking the main camera acquisition link, and achieves deterministic recycling of frame resources. This avoids memory leaks or premature release, thereby solving the resource contention and conflict problems when multiple cameras are running concurrently to a certain extent. Attached Figure Description

[0041] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0042] Figure 1 This is a schematic diagram of the overall architecture logic of the robot multi-camera shared frame resource scheduling and distribution method and system of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the matching relationship between the shared frame resource description table and the consumer capability table of the present invention.

[0044] Figure 3 This is a schematic diagram of the single-frame multi-consumer distribution and reference counting recycling process of the present invention;

[0045] Figure 4 This is a schematic diagram of the slow consumer back pressure isolation and degradation processing flow of the present invention;

[0046] Figure 5 This is a schematic diagram illustrating the global resource scheduling logic concept for multi-camera systems according to the present invention.

[0047] Explanation of reference numerals in the attached figures

[0048] Multi-camera acquisition module 1, shared frame resource registration module 2, shared frame resource description table 3, consumer capability management module 4, consumer capability table 5, consumer admission and degradation module 6, multi-consumer fan-out module 7, reference counting and recycling module 8, back pressure and fault isolation module 9, multi-camera global scheduler 10, statistics and auditing module 11, UVC consumer 21, encoder consumer 22, storage consumer 23, local display consumer 24, WebRTC consumer 25, diagnostic consumer 26. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with embodiments, so as to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this application are merely some embodiments of the present invention, and not all embodiments. It should be noted that, for those skilled in the art, the embodiments and features in the embodiments of this application can be combined with each other without departing from the concept of the present invention and without conflict. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the disclosure and protection scope of the present invention.

[0050] Furthermore, the terms "first," "second," "S100," "S200," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such features can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those described herein. At the same time, the stages described in each step are not necessarily to be implemented in the same step; it should be understood that the implementation order of the contents of each step stage can be adjusted and interchanged without violating the inventive concept, so that embodiments of the invention described herein can be implemented in orders other than those described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arrange," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this case based on the specific circumstances and in conjunction with existing technology.

[0051] To address the issues of redundant processing across multiple consumer devices and blockage in the main camera acquisition link, such as... Figures 1 to 5 As shown, this invention proposes a robot multi-camera shared frame resource scheduling and distribution system, which includes: a multi-camera acquisition module 1, a shared frame resource registration module 2, a consumer capability management module 4, a consumer admission and degradation module 6, a multi-consumer fan-out module 7, a reference counting and recycling module 8, a backpressure and fault isolation module 9, a multi-camera global scheduler 10, and a statistics and auditing module 11. It should be understood that the above modules are used to explain this solution, but do not limit the modules to be implemented with independent hardware, independent processes, or in a fixed order.

[0052] Among them, such as Figure 1As shown in the example, the camera frames output by the multi-camera acquisition module 1 are registered as shared frame resources by the shared frame resource registration module 2 and written into the shared frame resource description table 3; the consumer capability management module 4 is used to maintain the consumer capability table 5; the consumer admission and degradation module 6 is used to perform admission judgment based on the description table 3 and the capability table 5; the multi-consumer fan-out module 7 is used to distribute the same frame resource to at least two of the UVC consumer 21, encoder consumer 22, storage consumer 23, local display consumer 24, WebRTC consumer 25, and diagnostic consumer 26; the reference counting and recycling module 8 is used to manage the frame resource lifecycle; the back pressure and fault isolation module 9 is used to be responsible for the isolation and degradation of slow consumers; and the statistics and auditing module 11 is used to record the frame rate, latency, frame loss, degradation path, and abnormal status of each consumer.

[0053] To address the technical problems in existing technologies where multiple consumer terminals concurrently consume the same camera frame stream, leading to redundant memory copies, format conversions, and encoding due to each terminal independently retrieving copies from the acquisition link and the lack of a unified frame resource description and shared import entry point, this invention proposes a unified abstraction of the original frames acquired by the camera into standardized resource objects that can be shared and imported by multiple consumer terminals. Therefore, this invention proposes a unified description and registration scheme for shared frame resources. By establishing resource description information including frame attributes, shared buffer handles, synchronization markers, and reference counts, the underlying frame data acquires negotiable, distributable, and traceable upper-layer resource semantics, thus laying the data foundation for zero-copy sharing.

[0054] Specifically, in this example, the shared frame resource registration module 2 is used to establish shared frame resources for each frame. Examples of shared frame resources include at least: camera identifier, frame number, timestamp, image format, resolution and stride, offset, number of planes, shared buffer handle, buffer modifier, color space and rotation direction, production completion synchronization flag or fence, reference count, and lifecycle status. In this example, the shared buffer handle includes at least one of: a DMA-BUF file descriptor, a shared memory handle, or an equivalent zero-copy buffer handle.

[0055] The following combination Figures 1 to 3 Taking a four-camera remote preview and local display scene as an example, this invention illustrates the unified description and registration process of shared frame resources. Corresponding to the above system, this invention proposes a method for scheduling and distributing shared frame resources among multiple robots, with example steps including:

[0056] Step S100: After the camera frames are acquired, the multi-camera acquisition module 1 establishes or selects a corresponding frame resource pool for each camera; the shared frame resource registration module 2 establishes a shared frame resource containing the above example fields for each frame.

[0057] Step S110: The shared frame resource registration module 2 writes the above information into the shared frame resource description table 3 and publishes the frame availability event to the consumer capability management module 4.

[0058] Through the above steps, this case essentially proposes the core design concept of "unifying the description of multiple camera frames into a shared frame resource." By using the shared frame resource description table 3, all subsequent consumer terminals will no longer directly retrieve data from the camera acquisition link, but instead request imports from this standardized description information. This fundamentally eliminates the redundancy of each consumer terminal independently maintaining frame format information, enabling multiple heterogeneous consumer terminals to share and use the same frame data based on the same description, thereby solving the technical problem of repeated copying across multiple consumer terminals.

[0059] On the other hand, in optional implementations, the shared buffer handle in the shared frame resource can be replaced with other equivalent zero-copy buffer handles such as GPU texture handles. On platforms that do not support zero-copy, it can be entirely replaced by CPU copy path plus frame reference management, still falling within the scope of the shared frame resource lifecycle management of this solution. In addition, the shared frame resource description table and the consumer-side capability table can be replaced with equivalent descriptor objects, capability description structures, or key-value records, as long as the core structure of unified description and shared import is retained.

[0060] Furthermore, to address the issue of incompatible frames or frames exceeding latency limits being pushed to consumers due to the lack of unified capability registration and admission judgment at the consumer end, thus increasing operational risks, this example preferably incorporates an admission matching mechanism between shared frame resource description information and consumer capability parameters before distribution. Therefore, this proposal suggests a consumer capability admission and degradation matching scheme. Distribution decisions are not directly based on the camera acquisition link, but rather on the aforementioned admission matching results. Only consumers meeting the direct sharing conditions are identified as distribution targets, while incompatible consumers are downgraded or rejected, thereby completing capability adaptation before the frame enters the consumer end.

[0061] like Figure 2 As shown, the shared frame resource description table 3 and the consumer capability table 5 form a matching relationship, and the consumer access and downgrade module 6 performs access judgment based on the matching result.

[0062] Specifically, in this example, the consumer capability management module 4 pre-registers or dynamically registers the capability tables 5 for each consumer. Each consumer declares its acceptable formats, resolutions, frame rates, planar layouts, buffer modifiers, maximum latency, queue depth, priority, whether it supports shared buffer imports, whether color conversion is required, and whether it allows downgrading to a copy path. The consumer admission and downgrading module 6 performs admission judgments based on the shared frame resource description table 3, the consumer capability table 5, and the credit status of each consumer: when a consumer supports the current frame resource format and its maximum latency, priority, and queue status meet preset conditions, the consumer is added to the distribution list; when a consumer does not support the current shared buffer format or does not meet the latency conditions, it determines whether downgrading is allowed. If downgrading is allowed, format conversion, scaling, encoding, and then distribution or CPU copy path are performed, and the reason for downgrading is recorded; if downgrading is not allowed, the consumer is rejected and the reason for rejection is recorded.

[0063] The following example, using a four-camera remote preview and local display scenario, illustrates the specific implementation of admission matching. The corresponding method and steps include...

[0064] Step S120: Consumer capability management module 4 pre-registers the capability tables of each consumer terminal 5.

[0065] Step S130: Consumer admission and downgrade module 6 performs admission judgment based on description table 3, capability table 5, and consumer credit status. When both local display consumer 24 and WebRTC consumer 25 meet the admission conditions (support for current frame format, resolution matching, latency within the threshold, and normal credit status), both are simultaneously added to the normal distribution list.

[0066] Step S140: If downgrading is allowed, perform format conversion, scaling, encoding and then distribute or copy the path to the CPU, and record the reason for downgrading; if downgrading is not allowed, reject the consumer and record the reason for rejection.

[0067] Through the above design, this case actually proposes the core concept of "distribution decision based on access matching results rather than camera acquisition link". In the prior art, the consumer usually subscribes to data directly from the acquisition link, resulting in frames with incompatible formats or excessive latency still being pushed; while this invention, by introducing capability access judgment before distribution, transfers the distribution control from the acquisition link to a matching layer of shared frame resources and consumer capabilities, thereby only pushing compatible frames to the target consumer, effectively reducing operational risks.

[0068] Furthermore, to ensure the system maintains overall availability even in complex scenarios with concurrent heterogeneous consumer terminals, this invention proposes a closed-loop design concept of "capability admission and degradation fault tolerance." When a consumer's capabilities are mismatched, the system does not simply reject it, but maintains the consumer's availability through degradation paths (format conversion, scaling, CPU copying, etc.), while recording the reasons for degradation for subsequent auditing.

[0069] For example, unlike the consumer combination in the example above, in inspection mode, the same camera frame is consumed simultaneously by encoder consumer 22 and UVC consumer 21. Encoder consumer 22 directly imports shared frame resources and outputs H.265 bitstream when shared buffer import is supported; UVC consumer 21 outputs shared frame resources as USB Video Class video frames to an external host when the current image format and buffer layout are supported.

[0070] If the UVC consumer 21 does not support the current modifier or resolution, the consumer admission and degradation module 6 performs format conversion, scaling, or CPU copying of the degradation path and records the reason for degradation. In this case, the UVC consumer 21 obtains frame data through the degradation path instead of directly sharing the original buffer. This ensures stable system operation even in complex scenarios with heterogeneous concurrent consumers.

[0071] In addition, in an optional implementation, admission determination can be performed by a centralized scheduler, or each consumer can apply for admission based on the frame description broadcast and have it confirmed by the arbitration module. As long as the core structure of the admission matching result between the shared frame resource description information and the consumer capability information is retained, it is acceptable.

[0072] Furthermore, to address the technical problem of premature release or prolonged occupation of frame resources and uncertain recycling when multiple consumers share the same frame due to the lack of unified resource governance for the camera frame lifecycle, this invention introduces a deterministic recycling mechanism combining fan-out distribution and reference counting in scenarios where the same shared frame resource is consumed by multiple consumers. Therefore, this invention proposes a multi-consumer fan-out and reference counting management scheme for the same frame resource. Before fanning the frame resource to multiple consumers, a reference count is set, and the reference count is deterministically decremented by monitoring the completion, timeout, or abnormal status of each consumer. The frame resource is only recycled after the reference count reaches zero and the synchronization condition is met, thereby achieving a closed-loop lifecycle in multi-consumer sharing scenarios.

[0073] Specifically, in this example, the multi-consumer fanout module 7 generates a consumer distribution list for each frame based on subscription relationships, task modes, user visibility, consumer priority, and consumer credit status. Before the frame resource is distributed to multiple consumers, the reference counting and recycling module 8 increments the reference count to the number of consumers. After each consumer completes consumption, it returns a completion event, and the reference count is decremented. When a consumer times out, exits abnormally, or fails to consume, the backpressure and fault isolation module 9 isolates the consumer, records the exception, and then releases the consumer's reference. When the reference count is zero and the relevant synchronization flag (fence or completion event) has been released, the reference counting and recycling module 8 reclaims the frame resource into the camera frame resource pool; otherwise, it continues to wait for other consumers to complete consumption or time out and release the resource.

[0074] The following example, using the four-camera remote preview and local display scenario as an example, illustrates the specific implementation of fan-out distribution and reference counting garbage collection. The corresponding method example steps include:

[0075] Step S150: The multi-consumer fan-out module 7 generates a consumer distribution list for each frame.

[0076] Step S160: The reference counting and recycling module 8 increases the reference count to the number of consumers (for example, if both the local display consumer 24 and the WebRTC consumer 25 are admitted, the reference count is set to 2).

[0077] Step S170: The multi-consumer fan-out module 7 distributes the same frame resource to the local display consumer 24 and the WebRTC consumer 25.

[0078] Step S180: Reference counting and recycling module 8 monitors the status of each consumer: when a consumer completes consumption, it releases the consumer reference; when a consumer times out or exits abnormally, the back pressure and fault isolation module 9 isolates the consumer and records the abnormality, and then releases the consumer reference.

[0079] Step S190: When the reference count is zero and the associated synchronization flag has been released, the reference counting and recycling module 8 reclaims the frame resources into the frame resource pool.

[0080] The above process design embodies the core design concept of this invention: "multi-consumer fan-out and deterministic reclamation using reference counting." By fanning out the same frame resource to multiple consumers and setting a reference count before distribution, the system can achieve zero-copy sharing of "one source, multiple terminals." Through the dual constraints of reference counting and synchronization marking, the system can avoid premature release of frame resources (leading to data corruption) or long-term occupation (leading to memory leaks), thereby solving the technical problem of uncertain resource reclamation in multi-consumer sharing scenarios.

[0081] In addition, in optional implementations, reference counting can be replaced by a set of consumers to be completed, token recycling, or generation-based recycling; synchronization markers can be replaced by fences, completion events, or timestamp watermarks, as long as the core structure of recycling frame resources is retained after all consumers have completed the task and the synchronization conditions are met.

[0082] Furthermore, to address the technical problem of slow or abnormal consumers blocking the main camera acquisition link and affecting other normal consumers due to multiple consumers sharing the same acquisition queue and lacking consumer-level backpressure isolation and abnormal release mechanisms, this invention aims to bind backpressure isolation, reference counting recycling, and consumer credit status to the same shared frame resource lifecycle. Accordingly, this invention further proposes a consumer-level backpressure isolation and credit status feedback linkage scheme. Each consumer is independently configured with operational constraint parameters and frame dropping strategies. When a consumer becomes congested or abnormal, isolation or degradation is implemented, and the credit status is dynamically updated and fed back to the admission matching stage, thus limiting the impact of the fault to its own scope.

[0083] Specifically, in this example, the backpressure and fault isolation module 9 can independently set the queue depth, maximum latency, consumer credit status, and frame dropping strategy for each consumer. For the local display consumer 24 and the WebRTC consumer 25, a strategy of retaining the latest frame and discarding old frames can be adopted; for the storage consumer 23, a strategy of prioritizing key frames, queue alarms, and downsampling can be adopted; for the UVC consumer 21, a fixed frame rate output strategy can be adopted; and for the diagnostic consumer 26, a strategy of proportional frame dropping or event-triggered frame dropping can be adopted.

[0084] When a consumer's queue depth reaches its limit or experiences consecutive timeouts, the system instructs the consumer to implement traffic control and frame dropping, or to set its credit status to an abnormal isolation state and suspend distribution. Once the consumer returns to normal, the system enters a recovery observation phase. After its queue depth and number of timeouts meet the recovery conditions, it is allowed to re-enter the admission matching process. This dynamic update of the credit status is fed back to the admission matching stage in real time, achieving a closed-loop linkage between runtime control and distribution decisions.

[0085] For example, such as Figure 4 As shown, in this example, the backpressure and fault isolation module 9 sets the queue depth, maximum latency, consumer credit status, and frame dropping policy for each consumer. In one implementation, the maximum latency threshold of the WebRTC consumer 25 can be set to 100ms to 500ms, and the queue depth can be set to 1 to 5 frames; the queue depth of the storage consumer 23 can be set to 30 to 300 frames; and the diagnostic consumer 26 receives frame resources at a frame dropping ratio of 1 / 5 to 1 / 30.

[0086] When a storage consumer 23 experiences a slow disk write, causing its queue depth to exceed a threshold, the backpressure and fault isolation module 9 performs frame reduction storage or suspends subscription based on the consumer's priority, task mode, and credit status. Simultaneously, the multi-camera acquisition module 1 continues to display the latest frame resources to the local consumer 24 and the WebRTC consumer 25. If the storage consumer 23 recovers, the consumer capability management module 4 re-registers its queue status. The system first enters a recovery observation phase, and once its queue depth and timeout count meet the recovery conditions, it is re-added to the normal distribution list.

[0087] In this embodiment, when the WebRTC consumer 25 slows down due to network congestion, it processes the queue according to the strategy of retaining the latest frame and discarding the old frame. The local display consumer 24 is not affected by the network congestion and continues to display the current camera image.

[0088] Through the above process design, this invention achieves the core concept of "consumer-level backpressure isolation and credit status bound to the same shared frame resource lifecycle." In existing technologies, multiple consumers share the same acquisition queue, and if any consumer is blocked, the entire link will be brought down. However, this invention configures independent operating constraints and frame dropping strategies for each consumer, and links backpressure isolation with reference counting reclamation (synchronously releasing the reference when an abnormal consumer is isolated). This strictly limits the impact of slow consumer failures to their own scope, preventing them from blocking the main camera acquisition link and ensuring the continuous service of other normal consumers.

[0089] In addition, in optional implementations, strategies such as retaining the latest frame / discarding old frames, prioritizing key frames, downsizing, fixing the frame rate, and proportionally extracting frames can be replaced or combined with each other; the credit status of the consumer can be represented by any indicator or combination thereof, such as queue depth, number of consecutive timeouts, frame drop rate, or comprehensive score, as long as the core structure that binds back pressure isolation with the lifecycle of shared frame resources is retained.

[0090] Furthermore, to address the technical problem of intensified contention for shared buffers, encoder channels, and network bandwidth during concurrent previewing, storage, streaming, and diagnostics of multiple cameras due to the lack of global resource scheduling across cameras and consumer devices, this invention introduces a unified global resource scheduling mechanism in multi-camera scenarios. Accordingly, this invention proposes a global resource scheduling scheme for multiple cameras, where a global scheduler uniformly allocates resources based on camera priority, consumer priority, user visibility, and task mode. During resource contention, high-priority paths are prioritized, thereby reducing resource conflicts during concurrent multi-camera operation.

[0091] Specifically, in multi-camera scenarios, the system can maintain an independent frame resource pool for each camera, while managing the number of shared buffers, total memory usage, number of encoder channels, number of WebRTC sessions, and available network bandwidth through the multi-camera global scheduler 10. The global scheduler 10 allocates resources based on camera priority, consumer priority, user visibility, task mode, or security level. When system resources are contested, priority is given to ensuring resource supply for high-priority consumers corresponding to high-priority cameras, allowing low-priority consumers to enter a degraded path or wait state when resources are insufficient.

[0092] For example, such as Figure 5 As shown, in a multi-camera scenario, the system maintains an independent frame resource pool for each camera, and manages the number of shared buffers, total memory usage, number of encoder channels, number of WebRTC sessions, and available network bandwidth through the multi-camera global scheduler 10. When resources are insufficient, the global scheduler 10 allocates resources according to camera priority, consumer priority, user visibility, task mode, or security level.

[0093] In one implementation, when the remote control page only displays some of the cameras, the system prioritizes maintaining the WebRTC consumer 25 for the visible cameras; when the user performs multi-camera screenshots, a set of frames with similar timestamps are selected from the most recently available frames of each camera to generate a composite image.

[0094] Through the above process design, this invention realizes the core concept of "global resource scheduling across cameras and consumer terminals." It solves the problem of buffer, encoder, and network bandwidth conflicts caused by the lack of unified resource governance across cameras and consumer terminals in existing technologies. This invention uses a global scheduler to uniformly allocate resources and dynamically adjust supply based on priority, effectively reducing resource contention in multi-camera concurrent scenarios and improving the overall resource utilization of the system.

[0095] In a further embodiment, the present invention also demonstrates the application of shared frame resource description information in cross-camera scenarios. For example, after the system receives a multi-camera composite screenshot request, the multi-consumer fan-out module 7 selects a set of frames with the closest timestamps from the most recent available shared frame resources of the four cameras and sends them to the composite consumer to generate a 2×2 mosaic; after generation, the references of each frame are released, and the composite image can be used for on-site recording, fault review, or remote support. In other embodiments, the composite consumer can generate 1×N, 2×2, 3×3, or fisheye panoramic layouts; when a certain camera is temporarily unavailable, the composite image can display a placeholder image, the last available frame, or fault prompt information, and the camera unavailability status is recorded in the statistics and auditing module 11.

[0096] The above process demonstrates the derivative benefits brought about by the unified description of shared frame resources in this invention. Since the frames from each camera are uniformly described as standardized resource objects containing timestamps and camera identifiers, the system can quickly retrieve and match frame groups with similar timestamps across cameras, thereby supporting advanced applications such as multi-camera synchronous synthesis, and further verifying the technical feasibility of the unified resource description mechanism.

[0097] In optional implementations, the global scheduling criteria can be selected from one or a combination of dimensions such as camera priority, consumer priority, user visibility, task mode, security level, battery level, and temperature. It can also be entirely replaced by an equivalent implementation based on priority queues or weighted fair scheduling. Furthermore, this solution can be deployed entirely within a single process on the robot's end-user side, or it can be replaced by a separate acquisition and distribution process, or a distributed implementation of acquisition and consumption across devices, as long as the core structures of unified description of shared frame resources, capability admission, fan-out, reference counting recycling, and backpressure isolation are retained.

[0098] In summary, the robot multi-camera shared frame resource scheduling and distribution method and system provided by this invention achieves the following overall technical effects:

[0099] 1. Eliminate the defects of redundant processing by multiple consumer terminals and reduce system overhead: By sharing the unified description and fan-out distribution of frame resources, multiple consumer terminals can share and import the same frame data based on the same resource description. This avoids the repeated memory copying, repeated format conversion and repeated encoding caused by each consumer terminal independently taking copies from the acquisition link in the traditional solution, which significantly reduces CPU usage, memory bandwidth consumption and system power consumption.

[0100] 2. Avoid erroneous push of mismatched frames and reduce operational risks: By performing admission matching before distribution, only consumers with compatible formats, acceptable latency, and normal credit status are included in the distribution target. Incompatible consumers are downgraded or rejected, which fundamentally avoids the push of incompatible or excessively delayed frames to consumers, thereby improving the stability and reliability of system operation.

[0101] 3. Prevent slow or abnormal consumers from blocking the main camera acquisition link: Through consumer-level backpressure isolation, dynamic credit status updates, and feedback linkage mechanisms, when a consumer is slow, experiences network congestion, or exits abnormally, the system can promptly implement traffic control, frame loss handling, or suspend subscriptions. Furthermore, the fault handling of that consumer is synchronized with the release of the reference count of shared frame resources, ensuring that other normal consumers are not affected and that the main camera acquisition link remains unobstructed.

[0102] 4. Achieve deterministic reclamation of frame resources to avoid memory leaks or premature release: Through the dual constraints of reference counting and synchronization markers (fence / completion event), it is ensured that frame resources are only reclaimed to the frame resource pool after all consumers have completed consumption or abnormal release and all underlying hardware asynchronous operations have been completed. This solves the problems of premature release or long-term occupation of frame resources and uncertain reclamation in multi-consumer shared scenarios.

[0103] 5. Reduce resource conflicts when multiple cameras are running concurrently: By using a global scheduler to uniformly allocate shared buffers, encoder channels and network bandwidth across cameras and consumer terminals, and dynamically adjust resource supply according to priority and task mode, resource competition and conflicts when multiple cameras are previewing, storing, streaming and diagnosing at the same time are effectively alleviated.

[0104] 6. Maintain system availability and diagnosability: By recording degradation paths, rejection reasons, and abnormal states, the system can still maintain availability through degradation processing when the shared buffer is unavailable, the consumer is incompatible, or the consumer is abnormal, and provides data basis for subsequent fault diagnosis and optimization.

[0105] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

[0106] Those skilled in the art will understand that, besides implementing the system, apparatus, unit, and its modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and its modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0107] Furthermore, all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0108] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A method for scheduling and distributing shared frame resources among multiple cameras in a robot, comprising: Pre-configuration steps: Pre-configure the capability parameters and operating constraint parameters of each consumer terminal, and complete the global resource initialization allocation for multiple cameras; The single-frame scheduling step performs the following processing on each camera frame: Register shared frame resources and generate shared frame resource description information including frame attributes, shared buffer handle, synchronization flag and reference count; Based on the shared frame resource description information, consumer capability parameters, and consumer credit status, admission matching is performed to determine consumers that can be directly shared and distributed; consumers that do not meet the direct admission conditions but are allowed to be downgraded are included in the downgraded distribution path. The same shared frame resource is distributed to all distributable consumers. Before distribution, the reference count of the shared frame resource is set according to the total number of distributable consumers. Monitor the running status of each consumer. Decrement the reference count when a consumer completes consumption or encounters an abnormal state. When the reference count reaches zero and the synchronization condition is met, reclaim the shared frame resources. The system manages and isolates faults at each consumer end, dynamically updates the credit status of each consumer end and feeds it back to the access matching process, and links the fault isolation mechanism with the lifecycle management of shared frame resources.

2. The method according to claim 1, wherein the frame attributes include: At least one of the following: camera identifier, frame number, timestamp, image format, resolution, and planar layout information; The shared frame resource description information also includes at least one of the following: step size, offset, number of planes, buffer modifier, color space, rotation direction, production completion synchronization fence, and lifecycle status; the shared buffer handle is at least one of the following zero-copy buffer handles: DMA-BUF file descriptor, shared memory handle, and GPU texture handle.

3. The method according to claim 1, wherein the consumer-side capability parameters include: Acceptable image format, resolution, shared buffer import capability, etc.; the running constraint parameters include at least one of maximum latency, queue depth, priority, and frame dropping strategy; the consumer capability parameters also include at least one of frame rate, planar layout requirements, buffer modifier requirements, whether color conversion is required, whether degradation to copy path is allowed, user visibility, task mode, and subscription relationship.

4. The method according to claim 1, wherein the credit status of the consumer is used to characterize whether the consumer queue is congested, whether it has consecutive timeouts, whether it is in an abnormal isolation state or in the recovery observation phase.

5. The method according to claim 1, wherein the admission matching step comprises: When a consumer supports the image format, resolution, planar layout, and shared buffer type of the current shared frame resource, and the current queue depth of the consumer has not reached the upper limit of the queue depth, the latency does not exceed its maximum latency constraint, and the credit status is in a normal state, it is determined to be a consumer that can be directly shared and distributed. If a consumer does not support the current shared buffer format, has a resolution mismatch, or does not meet the latency conditions, and the consumer allows for downgrading, it will be included in the downgraded distribution path; otherwise, the distribution of this frame resource to the consumer will be refused. When determining the distribution targets, the system sorts and filters based on subscription relationships, task modes, user visibility, consumer priority, and consumer credit status to generate the final distribution set.

6. The method according to claim 1, wherein the degradation distribution path comprises: At least one of the following paths: format conversion, scaling, post-encoding distribution, and CPU copy; for consumers included in the degradation distribution path, the reference count of the shared frame resource is decremented after consumption is completed, timeout occurs, or abnormal exit occurs.

7. The method according to claim 1, wherein the abnormal state includes: The synchronization condition is that the production completion synchronization mark or the synchronization fence corresponding to the shared frame resource has been released; at least one of the following: consumer timeout, abnormal exit, or consumption failure; When an abnormal state is triggered on the consumer, the abnormal isolation process is first performed on the consumer, and then the reference count of the corresponding shared frame resource is decremented.

8. The method according to claim 1, wherein the steps of operational control and fault isolation include: Back pressure isolation: When the queue depth of the consumer reaches the upper limit, traffic control and frame dropping are performed on the consumer. Frame dropping strategies include at least one of the following: retaining the latest frame and discarding old frames, prioritizing key frames, reducing the frame rate, fixing the frame rate, and proportionally removing frames. Degradation Isolation: When a consumer times out continuously, the credit status of that consumer is set to an abnormal isolation state, the distribution of shared frame resources to it is suspended, and a recovery observation timer is started. Recovery observation: During the recovery observation period, if the queue depth and timeout count of the consumer meet the recovery conditions, the credit status of the consumer will be restored to normal and it will be allowed to re-enter the admission matching process; If the patient has not recovered by the end of the observation period, the abnormal isolation status will be maintained and the abnormal information will be recorded.

9. The method according to any one of claims 1 to 8, further comprising the steps of: Global resource allocation and scheduling: Based on camera priority, consumer priority, user visibility, and task mode, dynamically adjust the allocation of shared buffer pool, encoder channel, and network bandwidth; when system resources are contested, prioritize the resource supply of high-priority consumers corresponding to high-priority cameras, and allow low-priority consumers to enter a degraded path or waiting state when resources are insufficient.

10. A shared frame resource scheduling and distribution system for performing the method of any one of claims 1 to 9, comprising: Multi-camera acquisition module, used to acquire raw image frames from multiple cameras; A multi-camera global scheduler is used to dynamically adjust the allocation of shared buffer pool, encoder channels and network bandwidth based on camera priority, consumer priority, user visibility and task mode. When system resources are in contention, priority is given to ensuring the resource supply for high-priority consumers corresponding to high-priority cameras, while low-priority consumers are allowed to enter a degraded path or wait state when resources are insufficient. The shared frame resource registration module is used to register frames acquired by the camera as shared frame resources and generate shared frame resource description information that includes frame attributes, shared buffer handles, synchronization flags and reference counts. The consumer-side capability management module is used to pre-configure the capability parameters and operational constraint parameters of each consumer. The consumer access and downgrade module is used to perform access matching based on shared frame resource description information, consumer capability parameters, and consumer credit status to determine consumers that can be directly shared and distributed; and to include consumers that do not meet the direct access conditions but are allowed to be downgraded in the downgraded distribution path. A multi-consumer fan-out module is used to distribute the same shared frame resource to all distributable consumers, and to set the reference count of the shared frame resource according to the total number of distributable consumers before distribution; The reference counting and recycling module is used to monitor the running status of each consumer and decrement the reference count when the consumer completes consumption or encounters an abnormal state; when the reference count reaches zero and the synchronization condition is met, the shared frame resources are recycled. The back pressure and fault isolation module is used to manage and isolate faults in the operation of each consumer terminal, dynamically update the credit status of the consumer terminal and feed it back to the consumer terminal access and degradation module, and bind the fault isolation mechanism with the lifecycle management of shared frame resources.

Images