Video data distribution method and device, equipment and medium

Abstract

The invention discloses a video data distribution method, device and equipment and a medium, and the device comprises a camera sharing service which is used for responding to camera opening requests of a plurality of clients and distributing video data collected by cameras to the plurality of clients. According to the invention, the ability of multiple clients to access the same camera data source concurrently is realized through the camera sharing service, and the method has the advantages of improving the utilization rate of system resources and reducing performance overhead.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video data distribution method, device, equipment and medium. Background Art

[0002] With the advancement of intelligent vehicles, smart mobility, and the rapid development of AI-powered cockpits, the interactive experience in cockpits is becoming increasingly rich and engaging. During this interaction, user input is based on contextual information, which is then processed and delivered to the user. Data from onboard cameras is a key source of this information. Because different businesses may utilize camera footage, such as footage from inside or outside the vehicle, there are scenarios where multiple business segments need to share camera data, driven by considerations such as hardware cost and software performance.

[0003] Since the current mainstream cockpit operating systems are all based on the Android system, the in-vehicle Android system's current support for cameras is mainly based on two frameworks: the Camera2 framework and the EVS (Emulated Vehicle Service) framework. In most current cockpit systems, camera services are implemented based on these two frameworks. However, both solutions have significant limitations in practical applications. While the Camera2 framework is fully functional, it cannot enable multiple clients to access the same camera data simultaneously; while the EVS framework supports multi-channel distribution, it lacks effective resource management and performance optimization mechanisms. Especially in high-concurrency scenarios, existing solutions struggle to achieve efficient and stable video data distribution, resulting in wasted system resources and performance bottlenecks. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a video data distribution method, apparatus, device and medium to solve at least one defect in the prior art.

[0005] The present invention provides a video data distribution device, which includes:

[0006] The camera sharing service is used to respond to camera opening requests from multiple clients and distribute video data collected by the camera to the multiple clients.

[0007] In one embodiment of the present invention, the data distribution device further includes:

[0008] The camera sharing SDK is used to realize the connection between the client and the camera sharing service, and to transmit the client's camera opening request, the client's camera closing request and the data receiving buffer created by the client to the camera sharing service.

[0009] In one embodiment of the present invention, the camera sharing SDK further includes:

[0010] The data filling interface is used to transfer the data receiving buffer created by the client to the camera sharing service, so that the camera sharing service fills the video data into the data receiving buffer.

[0011] In one embodiment of the present invention, the camera sharing SDK further includes:

[0012] A connection interface for connecting the client to the camera sharing service;

[0013] A camera opening request transmission interface, used for sending the camera opening request to the camera sharing service;

[0014] The camera shutdown request transmission interface is used to send the camera shutdown request to the camera sharing service.

[0015] In one embodiment of the present invention, the camera sharing service includes:

[0016] A data packaging management module is used to fill the video data into the data receiving buffer after receiving the data receiving buffer to complete the video data distribution to the multiple clients; wherein, the data receiving buffer is created by the client after the client receives the message of successful camera opening feedback from the camera sharing service, and is passed to the data packaging management module.

[0017] In one embodiment of the present invention, the data packaging management module is also used to classify the data receiving buffers created by multiple clients according to the frame rate to obtain multiple groups, each group corresponding to the same frame rate; and when distributing video data, the video data is filled into the data receiving buffers created by multiple clients according to the frame rates of the multiple groups.

[0018] In one embodiment of the present invention, the camera sharing service further includes:

[0019] The camera management module is used to perform at least one of camera management and video data storage management.

[0020] In one embodiment of the present invention, the camera management includes camera opening management, and the camera opening management includes:

[0021] In response to a camera opening request from a first client among the multiple clients, the first client controls the camera to open, and returns a first message of successful camera opening to the first client;

[0022] Respond to a camera opening request of a second client among multiple clients, and return a second message to the second client indicating that the camera has been opened successfully; wherein the camera opening request of the second client is a request initiated by the client when the camera has been opened, and the second message is generated under the condition that the camera does not perform an opening action based on the camera opening request of the second client.

[0023] In one embodiment of the present invention, the camera management further includes camera shutdown management, and the camera shutdown management includes:

[0024] Determine whether an element in a task list is zero, and control the camera to be turned off when the element in the task list is zero; wherein, the task list is generated based on the camera opening requests of the multiple clients, and the task list stores a unique identifier of each client; when a camera closing request from a client is received, the unique identifier of the client that sends the camera closing request is deleted from the task list.

[0025] In one embodiment of the present invention, the video data storage management includes:

[0026] The video data collected by the camera is stored in the shared memory area, so that the video data is distributed from the shared memory area to the data receiving buffer created by the client.

[0027] In one embodiment of the present invention, the camera sharing service further includes:

[0028] An agent module is configured to receive the camera opening request, the camera closing request, and a data receiving buffer transmitted by a client, and forward the camera opening request, the camera closing request, and the data receiving buffer transmitted by the client to a camera engine module;

[0029] The camera engine module is used to receive the camera opening request, camera closing request and data receiving buffer transmitted by the proxy module, and send the camera opening request, camera closing request and data receiving buffer transmitted by the proxy module to the camera management module and the data packaging management module.

[0030] In one embodiment of the present invention, distributing video data collected by a camera to multiple clients includes:

[0031] Convert the camera data into an EGLImageKHR texture and render the EGLImageKHR texture into a data receiving buffer created by the client.

[0032] The present invention provides a video data distribution method, which includes:

[0033] In response to camera opening requests from multiple clients, the video data collected by the camera is distributed to the multiple clients through the camera sharing service.

[0034] The present invention provides a video data distribution device, comprising:

[0035] one or more processors; and

[0036] The memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the video data distribution device implements the method described above.

[0037] The present invention provides a machine-readable medium having instructions stored thereon. When executed by one or more processors, the instructions cause the processors to execute the method described above.

[0038] Beneficial effects of the present invention:

[0039] A video data distribution device according to the present invention includes a camera sharing service for distributing video data collected by a camera to multiple clients in response to camera opening requests from the clients. This camera sharing service enables multiple clients to concurrently access the same camera data source, improving system resource utilization and reducing performance overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0041] In the attached figure:

[0042] Figure 1 This is a principle block diagram of a video data distribution device according to an embodiment of the present invention;

[0043] Figure 2 A flowchart of opening a camera according to an embodiment of the present invention;

[0044] Figure 3 This is a flowchart of the internal initialization of the camera sharing service CameraShareService according to one embodiment of the present invention;

[0045] Figure 4A schematic diagram of a data distribution process according to an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing a memory according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments. The details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. The following embodiments and features therein may be combined with one another without conflict.

[0048] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be changed at will, and the component layout may also be more complicated.

[0049] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0050] See also Figure 1 , Figure 1 This is a principle block diagram of a video data distribution device according to an embodiment of the present invention. The video data distribution device includes:

[0051] The camera sharing service CameraShareService is used to respond to camera opening requests from multiple clients and distribute the video data collected by the camera to multiple clients.

[0052] The CameraShareService is a middleware module that runs independently within the operating system kernel layer. It can be implemented as an Android system service and communicates with clients through the Binder mechanism. This service serves as an interface between hardware resources and upper-layer applications, providing exclusive camera management. The CameraShareService is integrated into the vehicle's Android system as an executable program. It automatically starts at boot via the init.rc file and provides camera data acquisition capabilities upon startup. The CameraShareService is responsible for implementing all interfaces of the CameraShare SDK. Based on interface calls passed by clients through the CameraShare SDK, it performs call merging and data management, converting these calls into calls to the CameraDriver. The service then operates the camera hardware according to the interfaces provided by the CameraDriver. Video data distribution is accomplished by establishing a shared memory area, enabling multiple clients to directly access the same memory space.

[0053] Specifically, when a client initiates a camera open request, the physical camera is activated and a video data acquisition channel is established. The video data is captured by the camera and stored in a shared memory area. This data is then distributed to each client's data receive buffer based on the client's registered data receive buffer.

[0054] The present invention realizes the ability of multiple clients to concurrently access the same camera data source through the camera sharing service CameraShareService, which has the advantages of improving system resource utilization and reducing performance overhead.

[0055] See also Figure 1 , the data distribution device also includes:

[0056] The camera sharing SDK is used to implement the connection between the client and the camera sharing service CameraShareService, and to pass the client's camera opening request, the client's camera closing request, and the data receiving buffer Surface created by the client to the camera sharing service CameraShareService.

[0057] The CameraShare SDK (Software Development Kit) is a software development kit (SDK) specifically designed for communication between clients and the CameraShareService. It implements cross-process communication by encapsulating the underlying communication protocol. Its purpose is to establish a standardized communication channel and eliminate direct dependencies between clients and servers.

[0058] The camera sharing SDK uses Android's Binder-IPC cross-process communication mechanism to ensure communication between the client and the camera sharing service CameraShareService. In addition, the camera sharing SDK defines a unified interface call through AIDL (Android Interface Definition Language), and then compiles AIDL into a backend SDK that can be directly used by Java / Kotlin and C++. The Java language SDK is integrated into the Java / Kotlin client in the form of a jar package; the C++ language SDK is provided as an SO library + header file for C++ clients to link and call. Since the compilation process is based on the Android source code, the code does not need to be adapted. The direct compiled output is the SDK that responds to the corresponding platform, and a unified calling method is output to the outside world, which facilitates upgrades, maintenance, and cross-platform porting.

[0059] Camera open request and camera close request are camera operation instructions initiated by the client to the camera sharing service CameraShareService. They can be transmitted using asynchronous message queues. The request sources of different clients can be distinguished by message identifiers, realizing unified collection and combined processing of multi-client requests.

[0060] The data receiving buffer Surface is the storage medium of video data, which can be implemented using the Android Surface class. The shared memory mechanism is used to achieve efficient transmission of video frame data, and a dedicated channel is established for the camera sharing service CameraShareService to push video data to the client.

[0061] Specifically, the client initiates a connection request by calling the interface of the camera sharing SDK to establish a communication link with the camera sharing service CameraShareService. When access to camera resources is required, the client sends a camera open request in a standardized format through the SDK. The request is encapsulated into a serializable data packet for transmission. After the connection between the client and the camera sharing service CameraShareService is established, the client creates a data receiving buffer Surface and passes the data receiving buffer Surface object to the camera sharing service CameraShareService through the interface provided by the SDK. After the camera sharing service CameraShareService obtains the data receiving buffer Surface, it establishes a video data filling channel to complete data distribution through the video data filling channel.

[0062] This invention uses the camera sharing SDK to convert client requests into standardized operation instructions. The camera sharing service, CameraShareService, centrally processes multiple requests and uniformly distributes video data streams, avoiding resource conflicts caused by multiple clients directly operating the hardware. Furthermore, the data receiving buffer, Surface, is passed from the client to the CameraShareService, ensuring efficient video data transmission and avoiding data competition among multiple clients.

[0063] In one embodiment, the camera sharing SDK includes:

[0064] The data filling interface AddDataSurface is used to pass the data receiving buffer Surface created by the client to the camera sharing service CameraShareService, so that the camera sharing service CameraShareService fills the video data into the data receiving buffer Surface.

[0065] The AddDataSurface data filling interface is used to transfer data receiving buffer objects between the client and the CameraShareService service. The client creates a data receiving buffer Surface and passes it to the CameraShareService via a cross-process interface. The CameraShareService then fills the data receiving buffer with video data. Furthermore, the data receiving buffer Surface provides a data availability notification mechanism. Once the CameraShareService fills the buffer with data, the client is immediately notified and can access the data.

[0066] The present invention establishes a direct data channel between the client and the camera sharing service CameraShareService through a standardized interface, solving the problem of low efficiency caused by redundant video data transmission paths in multi-client scenarios.

[0067] In one embodiment, the camera sharing SDK further includes: a connection interface Connect, a camera opening request transmission interface OpenCamera, and a camera closing request transmission interface CloseCamera;

[0068] The Connect interface is a standardized interface for establishing a communication channel between a client and the CameraShareService. It is used to connect a client to the CameraShareService. It is also used to initialize the communication connection between the client and the CameraShareService. After a successful connection, it checks the connection status with the CameraShareService. If a disconnect occurs, it reconnects the CameraShareService and client to improve communication robustness.

[0069] The OpenCamera camera open request transmission interface is a dedicated interface for clients to send camera open requests to the CameraShareService camera sharing service. When a client initiates a camera open request to the CameraShareService camera sharing service, this interface requires the camera name and the client's unique identifier (process number concatenated with process name). The name corresponds to the actual physical camera. The desired frequency (corresponding to the frame rate) and data format of received data are also required. This interface returns the result of the operation, indicating either a successful or failed camera open.

[0070] The CloseCamera interface is a dedicated interface for clients to send camera close requests to the CameraShareService service. The client initiates a camera close request to the CameraShareService service, which closes the corresponding camera based on the camera name and the client's unique identifier (process ID concatenated with process name).

[0071] Please continue reading Figure 1 , the camera sharing service CameraShareService includes:

[0072] The data packaging management module DataPackingManager is used to fill the video data into the data receiving buffer Surface after receiving it, so as to complete the video data distribution to multiple clients; wherein, the data receiving buffer Surface is created by the client after the client receives the camera opening success message feedback from the camera sharing service CameraShareService, and is passed to the data packaging management module DataPackingManager.

[0073] Among them, the data packaging management module DataPackingManager is a processor logic unit used to control the transmission path of video data to the client. It can be implemented using a multi-threaded task scheduler and realize video data distribution by creating an independent data filling thread.

[0074] Specifically, after receiving confirmation that the camera is turned on successfully, the client generates a data receiving buffer Surface and transmits it to the DataPackingManager module via a standardized communication channel. Upon detecting a valid data receiving buffer Surface object, the DataPackingManager module starts the corresponding data filling thread and writes the video stream captured by the camera into the data receiving buffer Surface at the required frame rate.

[0075] After the data packaging management module DataPackingManager is started, it creates a data filling thread. When camera data arrives, the data filling thread obtains the camera data and fills the data according to different frequencies. When there are too many clients receiving data, one data filling thread cannot meet the filling performance requirements. In this case, you can appropriately increase the filling threads. When some clients turn off the camera and stop receiving data, you can also appropriately recycle some data filling threads to reduce performance overhead.

[0076] The present invention reduces server resource usage by actively creating a data receiving buffer Surface on the client side, optimizes the data transmission path by using the centralized distribution control of the data packaging management module DataPackingManager, and achieves a balance between improving video stream transmission efficiency and hardware resource usage.

[0077] In one embodiment, the data packaging management module DataPackingManager is also used to classify the data receiving buffers created by multiple clients according to the frame rate to obtain multiple groups, and each group corresponds to the same frame rate; and when distributing video data, the video data is filled into the data receiving buffers created by multiple clients according to the frame rates of the multiple groups.

[0078] Before filling the video data into the data storage area Surface created by multiple clients, group them according to the frame rate requirements declared by the clients when registering the data storage area Surface, and store the client identifiers of the same frame rate value into the same data set. In this way, when distributing data, different data receiving buffer Surfaces can be filled according to different frequencies.

[0079] By grouping the data receiving buffers of multiple clients into Surfaces by frame rate, the system can execute independent data distribution processes for different groups. For high-frame-rate groups, a high-frequency data update mechanism is used, such as triggering a texture rendering operation every 10 milliseconds to ensure real-time performance. For low-frame-rate groups, a low-frequency update mechanism is used, such as performing data padding every 100 milliseconds, to reduce redundant calculations. During the data distribution process, the processing flows of each group do not interfere with each other. High-frame-rate groups do not need to wait for the processing of low-frame-rate groups to complete, thus eliminating the overall delay caused by frame rate differences.

[0080] The present invention ensures that high-frame-rate clients obtain timely data updates by grouping processing according to frame rate requirements, while preventing low-frame-rate clients from occupying too many computing resources, thereby improving the response efficiency and resource utilization of the overall system.

[0081] Please continue reading Figure 1 , the camera sharing service CameraShareService also includes:

[0082] The camera management module CameraManager is used to perform at least one of camera management and video data storage management.

[0083] Camera management refers to the centralized control of camera opening and closing operations. Video data storage management refers to the establishment of a shared memory area as a transit storage for video data. This can be implemented using a ring buffer structure, allowing multiple clients to simultaneously read the same raw data.

[0084] The camera management module CameraManager is used to maintain the camera opening status, record which cameras are currently open and which cameras are closed, and then create a shared memory area to store data, fill the video data obtained from the camera into the shared memory area, and then notify the data packaging data packaging management module DataPackingManager to retrieve the corresponding cached data from the shared memory area and distribute it to each client.

[0085] When receiving open requests from multiple different clients to open the same camera, the corresponding open requests will be merged and only one actual open action will be performed. When receiving close instructions from multiple clients to close the same camera, the instructions will also be merged and only one close action will be performed.

[0086] In one embodiment, camera management includes camera opening management, which includes: responding to a camera opening request of a first client among multiple clients to control camera opening, and returning a first message of successful camera opening to the first client; responding to a camera opening request of a second client among multiple clients, and returning a second message of successful camera opening to the second client to complete the merging processing of camera opening requests of multiple clients; wherein, the camera opening request of the second client is a request initiated by the client when the camera has been turned on, and the second message is generated under the condition that the camera does not perform an opening action based on the camera opening request of the second client.

[0087] The first client's camera open request refers to the first request from multiple clients that meets the open conditions and triggers the camera hardware to start. If the camera is detected to be in operation, this request is classified as the first open request. The second client's camera open request refers to the request initiated by a subsequent client when the camera has been called. If the camera is detected to be occupied, this request is classified as the second open request.

[0088] Specifically, each physical camera corresponds to an active list, which stores the unique identifier of each client. When a camera is first requested to be opened by a client, the unique identifier of the client is added to the active list. When other clients request to open the camera later, the active list directly returns a successful opening status and adds the corresponding unique identifier to the active list.

[0089] When the first client initiates a camera open request, the system detects that the camera is inactive, marks the request as the first open request, starts the camera hardware resources, and returns a first message containing the actual hardware operation results. When a subsequent client initiates an open request, the system detects that the camera is already occupied and marks such a request as the second open request. It returns a second message containing only the logical open success and adds the client identifier to the task list. By distinguishing between physical and logical calls, all subsequent requests do not need to repeatedly initialize the camera hardware, but instead share the established video data acquisition channel. When any client releases the camera, the system only shuts down the hardware resources if it is the last client in the task list; otherwise, the camera remains running.

[0090] This invention effectively eliminates the hardware resource competition that arises when multiple clients concurrently open the same camera, reduces the number of times the camera interface must be repeatedly opened, and reduces the extra consumption of system memory and computing resources. By decoupling logical and physical calls, the camera hardware operation frequency is reduced from a frequency related to the number of clients to a fixed frequency, while ensuring that each client can normally obtain video data. This significantly improves resource management efficiency in multi-client scenarios.

[0091] See also Figure 2 , Figure 2 This is a flow chart of opening a camera according to an embodiment of the present invention. Figure 2 In the process, the client triggers the camera opening function and sends an opening request to the camera sharing service CameraShareService through the camera opening request transmission interface OpenCamerar. When the camera sharing service CameraShareService receives the camera opening request, it first verifies the permission of the camera opening request. If the verification fails, it sends the opening result to the client. The client determines that the opening fails based on the opening result and ends the process. After the permission verification passes, the corresponding camera status is detected. If the camera status is open, the opening result is returned to the client. If the opening is successful, the client creates a data receiving buffer Surface and passes the data receiving buffer Surface to the camera sharing service CameraShareService. The camera sharing service CameraShareService classifies the data receiving buffer Surface and then fills the video data into the data receiving buffer Surface. If the camera is not open, the camera opening action is executed to open the camera through the camera driver CameraDriver, and the opening result is returned to the client through the camera sharing service CameraShareService.

[0092] In one embodiment, camera management includes camera shutdown management, and camera shutdown management includes:

[0093] Determine whether the elements in the task list are zero, and control the camera to be turned off when the elements in the task list are zero; wherein the task list is generated based on camera opening requests from multiple clients, and the task list stores the unique identifier of each client; when a camera closing request from a client is received, the unique identifier of the client that sent the camera closing request is deleted from the task list.

[0094] The task list is used to record the set of clients that currently have permission to open the camera. Whether the element is zero is determined by checking whether the number of client unique identifiers stored in the task list is zero. A unique identifier is an identification code used to distinguish different clients.

[0095] When a camera shutdown instruction is received from a client, the system finds the active list of the corresponding camera and removes the unique identifier of the client requesting shutdown from the active list. The system then checks the number of elements in the active list. If it is not zero, no further action is taken. If it is zero, the system actually shuts down the camera. This means that only the last shutdown request actually calls the CameraDriver to shut down the camera. This ensures that camera resources remain open only when there are active clients requesting them, avoiding idle resources caused by abnormal client exits or failure to release resources, and reducing system power consumption.

[0096] In one embodiment, camera management includes permission management. Specifically, permission management is to determine whether the camera has permission to open based on the unique identifier passed by the client when an open request is received. There are two types of open permissions, one is to only access data, and the other is to open and access data. A client that only has permission to access data does not actually open the camera after requesting to open. Only after a client with real permission to open calls to open, the camera will be truly opened, and at this time the data will be given to each client. A module that does not have any permissions will not perform any operation and will return an open failure. The advantage of this is that for the vehicle-mounted system, for the dormant STR (Suspend to RAM) scenario, dark current leakage caused by the camera can be avoided as much as possible.

[0097] In one embodiment, the video data storage management includes: storing the video data collected by the camera in a shared memory area, so that the video data is distributed from the shared memory area to a data receiving buffer Surface created by the client.

[0098] When a camera is turned on, a shared memory area will be created for the camera. The shared memory area can be a RingBuffer (ring buffer) with a default size of 3 (configurable). The three buffers in the shared memory area RingBuffer are memory spaces allocated by Ion. When the camera management module CameraManager obtains the camera data, it fills the data into the RingBuffer and then notifies the data packaging management module DataPackingManager for subsequent data processing. After the camera is closed, the corresponding shared memory area will be reclaimed.

[0099] After the video data captured by the camera is written to the shared memory area, the raw data is directly read through memory mapping. This data is then converted into a format compatible with graphics rendering and transferred in batches to a pre-created data receiving buffer, a Surface, on each client. During this process, the shared memory area serves as a single data source, and all clients obtain a copy of the data from it, eliminating the need to maintain a separate data cache for each client on the server. When multiple clients request video streams of the same resolution, only a single copy of the data in the shared memory area is generated, enabling concurrent access by multiple clients through memory address mapping.

[0100] The present invention uniformly stores original data in a shared memory area and realizes zero-copy transmission through a memory mapping mechanism, pushing data directly to the client's data receiving buffer Surface, reducing memory usage and intermediate links in the data transmission path.

[0101] Please continue reading Figure 1 ,The camera sharing service CameraShareService also includes: a proxy module Proxy and a camera engine module CameraEngine;

[0102] The proxy module Proxy is used to receive camera opening requests, camera closing requests, and the data receiving buffer passed by the client, and forward the camera opening requests, camera closing requests, and the data receiving buffer passed by the client to the camera engine module CameraEngine. The proxy module Proxy implements the camera sharing CameraShare SDK and acts as a server, providing an AIDL Server-side implementation to the outside world, directly responding to the client's call, and then forwarding the call to the camera engine module CameraEngine for subsequent processing. At the same time, it also returns the result processed by the camera engine module CameraEngine to the client. During initialization, the corresponding server-side Binder object is added to the ServiceManager for the camera sharing CameraShareSDK to obtain the call.

[0103] The camera engine module, CameraEngine, is used to receive camera open requests, camera close requests, and data receive buffers passed by the Proxy module, and send the camera open requests, camera close requests, and data receive buffers passed by the Proxy module to the camera management module, CameraManager, and the data packaging management module, DataPackingManager. The camera engine module, CameraEngine, is responsible for all scheduling and instruction processing. After receiving instructions from the Proxy module, it forwards the instructions to the Data Packaging Management Module, DataPackingManager, and the camera management module, CameraManager, respectively, based on the instruction type. For example, instructions for operating the camera, such as camera open requests and camera close requests, will be passed to the Data Packaging Management Module, DataPackingManager, for processing, while data filling will be handled by the Data Packaging Management Module, and the results of the processing by the CameraManager and Data Packaging Management Modules will be returned to the camera engine module, which will be passed back to the Proxy module.

[0104] The CameraShareService startup process is as follows: it automatically starts at boot by configuring init.rc and completes startup at a very early stage, subsequently providing the ability to obtain camera data. It is responsible for implementing all interfaces of the CameraShare SDK, and based on the interface calls passed by the client through the CameraShare SDK, it performs call merging, data management, and other operations, and then converts the calls to the Camera Driver. It also completes camera hardware operations based on the interfaces provided by the CameraDriver. Figure 3 , Figure 3 This is a flowchart of the internal initialization of the camera sharing service CameraShareService according to an embodiment of the present invention. Figure 3As shown in the figure, after the camera sharing service CameraShareService is started, the proxy module Proxy is created first, and then the proxy module Proxy is initialized first. After the proxy module Proxy is initialized, the service proxy is added to the ServiceManager, and then the camera engine module CameraEngine, the camera management module CameraManager, and the module data packaging management DataPackingManager are created in turn and initialized. In this way, the startup process of the program is completed.

[0105] In one embodiment, distributing video data collected by a camera to multiple clients includes:

[0106] Convert the camera data into an EGLImageKHR texture and render the EGLImageKHR texture to the data receiving buffer Surface created by the client.

[0107] Among them, the EGLImageKHR texture refers to a graphics image object created through the EGL extension interface, which can be implemented using the eglCreateImageKHR function in the OpenGL ES graphics library. This object can be shared across processes and supports hardware accelerated processing.

[0108] The raw video data collected by the camera is first converted into the EGLImageKHR texture format. This process is completed by calling the hardware decoding unit of the graphics processing unit GPU (Graphics Processing Unit, referred to as GPU), so that the image processing is transferred from the central processing unit CPU (Central Processing Unit, referred to as CPU) to the graphics processor. The generated texture is directly passed to the data receiving buffer corresponding to each client through the cross-process sharing feature of EGLImageKHR. In the rendering stage, the texture mapping function of OpenGL ES is used to batch render the same texture data into the data receiving buffer associated with multiple clients, without the need to generate a separate copy for each client. Since the EGLImageKHR texture only retains one instance in memory, multiple clients render the picture by referencing this instance, avoiding repeated data storage and transmission operations.

[0109] The present invention realizes the parallel and efficient rendering of the same camera data by multiple clients, reduces data copying, reduces the frequency of data transmission between the GPU and the CPU, fully utilizes hardware performance, improves efficiency, reduces performance overhead, and efficiently completes data distribution.

[0110] See also Figure 4 , Figure 4FIG. 1 is a schematic diagram of a data distribution process according to an embodiment of the present invention. Figure 4 As shown, the CameraShareService controls the camera to open and capture video data, then fills the video data into the shared memory area RingBuffer. It then distributes the video data to the data storage area Surface passed by the client. The CameraShareService then creates an EGL rendering environment. Within the EGL (Embedded-System Graphics Library) environment, it passes the cached data buffer passed by the camera management module CameraManager to the GPU (Graphics Processing Unit) via DMA (Direct Memory Access) to generate an EGLImageKHR (EGLImage on the Android platform) texture. The EGLImageKHR texture is then uploaded to the GPU, which then renders the texture to the client's data storage area Surface via OpenGL rendering. The client then processes the video data. Data storage areas Surfaces of the same type (with the same frame rate requirements) are filled at the same frequency. This way of filling can reuse filling threads as much as possible to avoid unnecessary overhead. In addition, the use of DMA Buffer and GPU rendering filling can quickly complete data filling and reduce the pressure on the CPU (Central Processing Unit).

[0111] In one embodiment, before filling the video data into the associated data receiving buffers of multiple clients, it also includes: classifying the data receiving buffers associated with multiple clients according to the frame rate requested when registering the data receiving buffer associated with each client, to obtain multiple groups, and each group corresponds to the same frame rate; when distributing data, filling the video data into the data receiving buffers associated with multiple clients based on the frame rates corresponding to the multiple groups.

[0112] Before filling the video data into the associated data receiving buffers of multiple clients, the clients are grouped according to the frame rate requirements declared by the clients when registering the data receiving buffers, and the client identifiers with the same frame rate value are stored in the same data set.

[0113] By grouping the data receive buffers associated with multiple clients by frame rate, the system can execute independent data distribution processes for different groups. For high-frame-rate groups, a high-frequency data update mechanism is used, such as triggering a texture rendering operation every 10 milliseconds to ensure real-time performance. For low-frame-rate groups, a low-frequency update mechanism is used, such as performing data padding every 100 milliseconds, to reduce redundant calculations. During the data distribution process, the processing flows of each group do not interfere with each other. High-frame-rate groups do not need to wait for the processing of low-frame-rate groups to complete, thus eliminating the overall delay caused by frame rate differences.

[0114] The present invention ensures that high-frame-rate clients obtain timely data updates by grouping processing according to frame rate requirements, while preventing low-frame-rate clients from occupying too many computing resources, thereby improving the response efficiency and resource utilization of the overall system.

[0115] In one embodiment, for multiple groups, when the frame rates corresponding to the multiple groups are the same, a single-thread batch rendering method is used to fill the video data into the data receiving buffer associated with the multiple clients; when the frame rates corresponding to the multiple groups are different, a set thread is used for rendering for the high frame rate group, and the video data is filled into the data receiving buffer associated with the corresponding client; for the low frame rate group, a thread pool method is used for rendering, and the video data is filled into the data receiving buffer associated with the corresponding client.

[0116] A thread pool is a collection of reusable threads that can be implemented using a fixed number of threads in conjunction with a task queue. It dynamically allocates idle threads to process rendering tasks in low-frame-rate groups, avoiding the resource consumption of frequently creating and destroying threads.

[0117] When it is detected that the frame rates of multiple groups are consistent, the system integrates the group data of the same frame rate into the same batch processing queue, and a single thread completes texture rendering and cache filling in sequence, reducing the synchronization waiting time caused by multi-threaded concurrent operations. When there is a difference in frame rate, the high frame rate group is assigned to an independent thread to perform the rendering task. The priority of this thread is set higher than that of ordinary threads to ensure that high frame rate data is processed first in each frame cycle. The low frame rate group is submitted to the task queue of the thread pool and is processed in sequence by the idle threads in the pool according to the first-in-first-out principle, reducing the system resource utilization by reusing threads. As a result, the data filling tasks of different frame rate groups are isolated to different execution units to avoid high priority tasks being blocked by low priority tasks.

[0118] In one embodiment, the present invention further provides a video data distribution method, comprising:

[0119] In response to camera opening requests from multiple clients, the video data collected by the camera is distributed to the multiple clients through the camera sharing service.

[0120] It should be noted that the video data distribution method provided in the above embodiment and the video data distribution device provided in the above embodiment are based on the same concept. The specific manner in which the various modules and units perform their operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the video data distribution device provided in the above embodiment can, as needed, allocate the aforementioned functions to different functional modules, i.e., divide the internal structure of the device into different functional modules to perform all or part of the functions described above. This is not a limitation herein.

[0121] An embodiment of the present invention also provides a device comprising: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, enables the device to implement the video data distribution method in the above embodiment.

[0122] The embodiments of the present invention further provide one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the video data distribution method in the above embodiments.

[0123] Figure 5 FIG1 shows a schematic diagram of a computer system structure suitable for implementing a memory according to an embodiment of the present invention. It should be noted that Figure 5 The computer system of the memory shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0124] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage portion into the random access memory (RAM) 503, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0126] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the video data distribution method embodied above. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component and / or installed from removable media 511. When the computer program is executed by the central processing unit (CPU) 501, the various functions defined in the system of the present invention are performed.

[0127] It should be noted that the computer-readable medium shown in the embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM) 503, a read-only memory (ROM) 502, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0129] The units involved in the embodiments of the present invention may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not limit the units themselves.

[0130] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When executed by a computer processor, the computer program causes the computer to perform the aforementioned video data distribution method. The computer-readable storage medium may be included in the memory described in the above embodiments, or may exist independently and not be incorporated into the memory.

[0131] Another aspect of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video data distribution method provided in each of the above embodiments.

[0132] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A video data distribution device, characterized in that: The video data distribution device comprises: The camera sharing service is used to respond to camera opening requests from multiple clients and distribute video data collected by the camera to the multiple clients.

2. The video data distribution device according to claim 1, wherein The data distribution device further includes: The camera sharing SDK is used to realize the connection between the client and the camera sharing service, and to transmit the client's camera opening request, the client's camera closing request and the data receiving buffer created by the client to the camera sharing service.

3. The video data distribution device according to claim 2, wherein: The camera sharing SDK also includes: The data filling interface is used to transfer the data receiving buffer created by the client to the camera sharing service, so that the camera sharing service fills the video data into the data receiving buffer.

4. The video data distribution device according to claim 3, wherein: The camera sharing SDK also includes: A connection interface for connecting the client to the camera sharing service; A camera opening request transmission interface, used for sending the camera opening request to the camera sharing service; The camera shutdown request transmission interface is used to send the camera shutdown request to the camera sharing service.

5. The video data distribution device according to claim 1, wherein The camera sharing service includes: A data packaging management module is used to fill the video data into the data receiving buffer after receiving the data receiving buffer to complete the video data distribution to the multiple clients; wherein, the data receiving buffer is created by the client after the client receives the message of successful camera opening feedback from the camera sharing service, and is passed to the data packaging management module.

6. The video data distribution device according to claim 5, wherein: The data packaging management module is also used to classify the data receiving buffers created by multiple clients according to the frame rate to obtain multiple groups, each group corresponding to the same frame rate; and when distributing video data, fill the video data into the data receiving buffers created by multiple clients according to the frame rates of the multiple groups.

7. The video data distribution device according to claim 5, wherein: The camera sharing service also includes: The camera management module is used to perform at least one of camera management and video data storage management.

8. The video data distribution device according to claim 7, wherein: The camera management includes camera opening management, and the camera opening management includes: In response to a camera opening request from a first client among the multiple clients, the first client controls the camera to open, and returns a first message of successful camera opening to the first client; Respond to a camera opening request of a second client among multiple clients, and return a second message to the second client indicating that the camera has been opened successfully; wherein the camera opening request of the second client is a request initiated by the client when the camera has been opened, and the second message is generated under the condition that the camera does not perform an opening action based on the camera opening request of the second client.

9. The video data distribution device according to claim 8, wherein: The camera management also includes camera shutdown management, and the camera shutdown management includes: Determine whether an element in a task list is zero, and control the camera to be turned off when the element in the task list is zero; wherein, the task list is generated based on the camera opening requests of the multiple clients, and the task list stores a unique identifier of each client; when a camera closing request from a client is received, the unique identifier of the client that sends the camera closing request is deleted from the task list.

10. The video data distribution device according to claim 7, wherein: The video data storage management includes: The video data collected by the camera is stored in the shared memory area, so that the video data is distributed from the shared memory area to the data receiving buffer created by the client.

11. The video data distribution device according to claim 5, wherein The camera sharing service also includes: An agent module is configured to receive the camera opening request, the camera closing request, and a data receiving buffer transmitted by a client, and forward the camera opening request, the camera closing request, and the data receiving buffer transmitted by the client to a camera engine module; The camera engine module is used to receive the camera opening request, camera closing request and data receiving buffer transmitted by the proxy module, and send the camera opening request, camera closing request and data receiving buffer transmitted by the proxy module to the camera management module and the data packaging management module.

12. The video data distribution device according to claim 1 or 5, characterized in that: Distribute video data collected by cameras to multiple clients, including: Convert the camera data into an EGLImageKHR texture and render the EGLImageKHR texture into a data receiving buffer created by the client.

13. A video data distribution method, characterized in that: The video data distribution method includes: In response to camera opening requests from multiple clients, the video data collected by the camera is distributed to the multiple clients through the camera sharing service.

14. A video data distribution device, characterized in that: include: one or more processors; and A memory for storing one or more programs, which, when executed by the one or more processors, causes the video data distribution device to implement the method according to claim 12.

15. A machine-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the processors to perform the method of claim 12 .

Images