Video data processing method, vehicle and computer readable storage medium

Abstract

The embodiment of the invention relates to the technical field of vehicle-mounted video processing, and discloses a video data processing method, a vehicle and a computer readable storage medium, the method is applied to the vehicle running a real-time operating system and a general operating system, and the method comprises the following steps: a first service obtains video data collected by a camera device, stores the video data in a shared memory, and stores the shared memory in the shared memory; wherein the first service runs in a real-time operating system, the second service acquires video data from the shared memory, and the second service runs in a universal operating system; the second service generates one or more copied video data according to the video data, and stores each copied video data to a corresponding cache space for a third service associated with each cache space to use; wherein each cache space is associated with a third service, and the third service runs in the universal operating system. The length of a cross-system data transmission channel is reduced, and the problems of video data jamming and frame loss can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of vehicle-mounted video processing, and in particular to a video data processing method, a vehicle, and a computer-readable storage medium. Background Art

[0002] The vehicle can be equipped with a dual-system operating system (RTOS) (e.g., QNX) and a general-purpose operating system (e.g., Android). The RTOS can be a microkernel system that drives the vehicle's camera, receives the video stream data captured by the camera, and transmits the captured video stream data to the general-purpose operating system. The general-purpose operating system and entertainment operating system perform recording and preview.

[0003] In the solution provided by the real-time operating system in the related technology, which transmits the video data collected by the camera device to the general operating system for recording and previewing, there are many cross-system video data transmission channels and many video data replication times, which leads to high system load and causes video freezes and frame drops. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a video data processing method, a vehicle, and a computer-readable storage medium, which are used to solve the problem of video freeze and frame loss caused by high system load in cross-system video data transmission in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a video data processing method is provided, which includes: a first service obtains video data collected by a camera device, and stores the video data in a shared memory, wherein the first service runs on the real-time operating system, and a second service obtains the video data from the shared memory, and the second service runs on a general-purpose operating system; the second service generates one or more copies of video data based on the video data, and stores each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with one third service, and the third service runs on the general-purpose operating system.

[0006] According to another aspect of an embodiment of the present invention, a vehicle is provided, comprising: a time domain controller, the time domain controller comprising a memory and a processor running a general-purpose operating system and a real-time operating system on the time domain controller, wherein the memory stores computer-executable instructions, and when the executable instructions are called by the processor, the processor performs the following operations:

[0007] The video data captured by the camera device is obtained by a first service running in a real-time operating system, and the video data is stored in a shared memory; the video data is obtained from the shared memory by a second service running in a general-purpose operating system; one or more copies of the video data are generated according to the video data by the second service, and each copy of the video data is stored in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with one third service, and the third service runs in the general-purpose operating system.

[0008] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction causes a processor to perform the following operations: obtaining video data captured by a camera device through a first service running in a real-time operating system, and storing the video data in a shared memory; obtaining the video data from the shared memory through a second service running in a general-purpose operating system; generating one or more copies of video data based on the video data through the second service, and storing each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with one third service, and the third service runs in the general-purpose operating system.

[0009] The video data processing method, vehicle, and computer-readable storage medium provided by the present invention shorten the video data transmission channel by running a second service in a general-purpose operating system. The first service directly acts as a client of the first service of the real-time operating system and obtains video data from the shared memory of the first and second services. The second service then copies the video data obtained from the shared memory into one or more copies and stores each copy in a cache space corresponding to a third service for use by the third service, which runs on the general-purpose operating system. Therefore, for different third services of the same camera device, there is no need to establish independent channels to obtain video data from the first service, reducing the number of channels for cross-system data transmission and, in turn, the total number of video data copies. This can reduce the system load caused by cross-system video data transmission, improve overall system performance, and avoid freezes and frame drops in preview video data, as well as frame drops in recorded video.

[0010] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0012] Figure 1 A schematic diagram of cross-system video data transmission in the prior art;

[0013] Figure 2 A schematic flow chart of a first embodiment of a video data processing method provided by the present invention;

[0014] Figure 3 for Figure 2 The schematic diagram of the principle corresponding to the video data processing method shown;

[0015] Figure 4 A schematic flow chart of a second embodiment of the video data processing method provided by the present invention;

[0016] Figure 5 for Figure 4 Schematic diagram of the video data copying process in the video data processing method shown;

[0017] Figure 6 This is a structural diagram of an embodiment of a time domain controller running a real-time operating system and a general operating system provided by the present invention. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0019] Currently, vehicle operating systems typically utilize a dual operating system (RTOS) and a general-purpose operating system (GOS). These dual operating systems can run on a time-domain controller. The RTOS (e.g., QNX) can be used for critical functions requiring high real-time performance, while the GOS (e.g., Android) can be used for infotainment and other non-critical applications.

[0020] The vehicle can be equipped with one or more cameras, which can be driven by a real-time operating system (RTOS) and communicate with it in real time. The vehicle can also be equipped with a general-purpose operating system (GOS), which can run applications (APPs) that interact with the user. Currently, the real-time operating system and the GOS can transmit video data from the RTOS to the GOS through the vehicle imaging system (AIS).

[0021] In the prior art, an APP is run in a general-purpose operating system, and the application uses the native camera architecture of the general-purpose operating system to obtain video data from the real-time operating system across the system.

[0022] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the transmission of inter-system video data in a vehicle in the prior art. Figure 1 As shown, a hardware virtualizer (Hypervisor) 102 runs on hardware (e.g., a time domain controller, including various processors and memories) 101 in a vehicle. The HHV 102 is used to run a general-purpose operating system 11 and a real-time operating system 12 independently of each other, and to isolate the general-purpose operating system 11 and the real-time operating system 12.

[0023] The general operating system 11 may be, for example, an Android system. The general operating system 11 may include a kernel layer 111, a hardware abstraction layer (HAL) 112, a framework layer 113, and an application layer 114. The application layer 114 may run DVR applications (such as Figure 1 Applications running in the application layer 114 can create one or more Camera instances based on service requirements such as preview and recording. For example, for the same camera device, one Camera instance may be used for preview and another for recording. Each Camera instance obtains video data across the system through one of the aforementioned channels.

[0024] The APP in the general operating system 11 (such as the Android operating system based on the Linux kernel) uses the channel formed by the Camera instance, the Camera.server of the framework layer, the V4L2 interface of the general operating system kernel, the Automotive Imaging System (Ais) client and the Ais server (AisServer) in the real operating system to realize cross-system video data transmission between the general operating system and the real-time operating system.

[0025] V4L2_kernel (Video for Linux 2) is a kernel driver framework for video devices provided by the Linux kernel. It runs in the kernel layer 111 of the general operating system and is responsible for low-level communication with the camera hardware.

[0026] The Camera HAL on the HAL layer 112 implements the function of Camera.provider. In addition, the HAL layer 112 also runs Ais_V4L2_proxy, which shares memory with AisServer in the real-time operating system.

[0027] The framework layer 113 runs Camera.server and provides the Camera API for the camera device function to the APP. The application layer 114 can run application programs, such as the Digital Video Recorder (DVR) application program.

[0028] The driver of the camera 13 runs in the real-time operating system 12. The AisServer process of the real-time operating system 12 is responsible for actually controlling the video stream control of the Camera hardware. Ais provides a native interface to the AisClient, and the Ais_v4l2_proxy of the general operating system 11 serves as the AisClient. Each channel of each camera 13 corresponds to an AisClient. When the AisClient requests video data from the AisServer, it usually needs to apply for a buffer from the general operating system 11. The general operating system and the real-time operating system share a set of physical hardware under the Hypervisor architecture. Therefore, the AisClient can pass the handle of the above-mentioned cache space to the AisServer. For each video image frame, the AisServer scans the video data of the frame line by line from the camera hardware. During this process, there will be a copy of the video image frame (the first copy).

[0029] After Ais_V4L2_proxy applies for the video image frame, it will notify Camera.provider as a consumer through V4L2-kernel. When consuming, Camera.provider will use color conversion and downscaling (C2D) hardware acceleration to copy the data on the source buffer of Ais_V4L2_proxy to the target buffer. The target buffer is taken from the Surface passed down by the App. There is also a copy (second copy) in the process from the source buffer to the target buffer.

[0030] The function of the Camera.server is to control the legality of apps' use of the Camera function and to extract the cache space in the Surface of the recording or preview service passed down by the app, encapsulate it, and pass it to the Camera.provider. Video data is not copied between the app and the Camera.server. After the app receives the image frame, it needs to add a watermark to the recorded file. During this process, the video data in the target buffer is copied (a third copy). In other words, each channel has three copies of the video data.

[0031] In summary, the aforementioned channels are relatively long, involving numerous processes. Video data is copied three times per channel, and preview and recording of each camera require separate channels. Therefore, for a vehicle with a four-channel setup, if both preview and recording are enabled simultaneously, there are eight channels, and the system may instantly copy data 3 × 8 = 24 times. Each video frame contains millions of rows of pixels, resulting in a significant amount of video data. This places a significant load on the vehicle's dual operating systems, impacting their performance and causing issues such as freezes and frame drops in preview and recorded video.

[0032] In order to improve the above problems, the present invention provides the following solutions.

[0033] Figure 2 The flowchart of the first embodiment of the video data processing method of the present invention is shown. The method is executed by a domain controller in a vehicle. The domain controller runs a general operating system and a real-time operating system. Figure 2 As shown, the method includes the following steps:

[0034] Step 210: The first service obtains the video data collected by the camera device and stores the video data in the shared memory, wherein the first service runs on a real-time operating system.

[0035] The camera device can be installed on the vehicle. In some application scenarios, the camera device can include four camera devices installed at the left front, right front, left rear, and right rear positions of the vehicle. Each camera device can include a camera and other image sensors.

[0036] The real-time operating system may be, for example, the QNX operating system. The first service may be, for example, an automotive imaging system server (AisServer). The first service may obtain captured video data from a camera device and store the video data in a shared memory. In some application scenarios, the video data may also be pre-processed (e.g., format converted) and the pre-processed video data may be stored in the shared memory.

[0037] Step 220: The second service obtains the video data from the shared memory, where the second service runs on a general operating system.

[0038] The second service may be a digital video recorder server (DVR server). The second service may serve as a client of the first service and directly interact with the first service, for example, requesting video data from the first service.

[0039] The second service can interact with the first service through the car imaging system protocol and share the shared memory with the first service. The second service can access the shared memory and obtain video data from the shared memory.

[0040] In some implementations, the second service runs on a framework layer of a general-purpose operating system.

[0041] In these embodiments, the same second service may provide data services for different third services at the application layer.

[0042] In the Camera2 architecture, the preview channel and the recording channel cannot share video data from V4L2. This is because the recording service in this framework typically needs to record a complete period of video data, close the channel, and then reopen the channel for recording. However, previewing requires continuous operation and cannot be closed and reopened during previewing. Therefore, it is impossible to share the same Ais Camera channel for preview and recording.

[0043] Therefore, the inventors creatively implemented a second service at the framework layer to provide video data to different third-party services at the application layer. For each camera, the second service establishes a channel with the first service to enable cross-system video data transmission. The second service then provides video data to one or more third-party services for that camera.

[0044] Setting the above-mentioned second service in the above-mentioned framework layer is beneficial to shortening the channel for cross-system data transmission on the one hand, and on the other hand, there is no need to establish a separate channel for cross-system video data transmission for each third service, which is beneficial to reducing the number of cross-system video data transmission channels required for multiple third services, thereby reducing the number of data replication times.

[0045] In some embodiments, step 220 includes:

[0046] The second service accesses the shared memory based on the hardware abstraction layer service interface related to the automobile imaging system.

[0047] The general operating system's hardware abstraction layer (HAL) can run the Ais_hidl_service HAL service related to the vehicle imaging system. This service provides a HAL service interface related to the vehicle imaging system. This interface provides a standardized way to interact with the underlying hardware or other system components. Through this interface, the second service can access the shared memory.

[0048] Step 230: The second service generates one or more copies of video data based on the video data, and stores each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with a third service, and the third service runs on a general operating system.

[0049] The second service can copy the video data obtained from the shared space. The number of copies can be equal to the number of third services using the video data. The third service can be provided by an app running on a general operating system (e.g., an app corresponding to a DVR). In some application scenarios, the third service can include previewing, recording, and taking photos.

[0050] As an implementation method, the preview service may correspond to the preview cache space, the video recording may correspond to the video recording cache space, and the photo taking may correspond to the photo taking cache space.

[0051] If the one or more third services include a preview service and a recording service, the second service may copy the video data obtained from the shared space into two copies, sending one copy of the video data to the preview cache space and the other copy of the video data to the recording cache space.

[0052] As an implementation method, the preview service and the photo service can share a cache space. That is, the photo service can obtain the video data of the corresponding frame from the cache space corresponding to the preview service to generate the image.

[0053] The preview service can display the preview screen using the video data in the preview cache space, and the recording service can record the video data in the recording cache space.

[0054] Please refer to Figure 3 , Figure 3 for Figure 2 The schematic diagram of the principle of the video data transmission method shown in FIG. Figure 3 As shown, a hardware virtualizer (Hypervisor) 302 is running on hardware (such as a time domain controller, including various processors and memories) 301 in the vehicle. The hardware virtualizer 302 is used to run a general operating system 31 and a real-time operating system 32 that are independent of each other.

[0055] The general operating system 31 may include a kernel layer 311, a HAL layer 312, a framework layer 313, and an application layer 314. The application layer 314 may run a DVR application. This application may receive instructions from a third service. The third service may be one or more of recording, previewing, and taking photos.

[0056] The real-time operating system 32 is used to drive the camera device 33 and obtain video data captured by the camera device 33. The video data may be composed of multiple frames of video images. It is understood that the camera device can continuously capture video data, and the continuously captured video data is transmitted across the system in the form of a video data stream.

[0057] A first service (AisServer) runs in a real-time operating system 32. A second service is set up in the framework layer 313 of the general operating system 31. The second service and the first service exchange information using the automotive imaging system protocol, and the second service acts as a client of the first service. The first and second services share memory.

[0058] like Figure 3 As shown, the HAL layer 313 of the general operating system runs the hardware abstraction layer service Ais_hidl_service related to the car imaging system, and the second service can access the above shared memory through the service interface provided by Ais_hidl_service. Figure 3 As shown, the cross-system transmission channel implemented between the first service and the second service only includes the first service, Ais_hidl_service and the second service. Figure 1 For the channels shown, Figure 3 The channel shown includes fewer nodes, involves fewer processes, and has a shorter channel length.

[0059] Furthermore, the second service can replicate the acquired video data according to the number of third services and send the replicated video data to the cache space of each third service. Each third service can use the video data in the corresponding cache space to implement the functionality of that third service. If the third service is a preview, the replicated video data can be retrieved from the preview cache space to generate a preview video. If the third service is a video recording service, the replicated video data can be retrieved from the video recording cache space for recording.

[0060] For each camera, if the preview service and the recording service are running simultaneously, the video data replication in this solution occurs when the first service stores the video data in the shared memory. The second service replicates the video data once for the preview service and once for the recording service, resulting in two copies of the data. Therefore, for each camera, there are a total of three replications. For a vehicle with four cameras, there are a total of 3 × 4 = 12 replications. Figure 1 In the illustrated solution, four cameras simultaneously run preview and recording services, requiring 24 video data copies. This reduces the number of video data copies by half, further reducing the amount of copied video data that needs to be cached, instantly saving tens of millions of pixel reads. This reduces system load and provides users with a better video preview and / or recording experience.

[0061] The video processing method provided in this embodiment operates a second service within a general-purpose operating system, acting directly as a client of a first service within a real-time operating system. The second service obtains video data from the shared memory of the first and second services, thereby shortening the video data transmission channel. The second service then copies the video data obtained from the shared memory into one or more copies, and stores each copy in a cache space corresponding to a third service for use by the third service, which operates within the general-purpose operating system. Therefore, different third services of the same camera device do not need to establish independent channels to obtain video data from the first service, thus reducing the number of channels for cross-system data transmission and, in turn, the total number of video data copies. This can reduce the system load caused by cross-system video data transmission, improve overall system performance, and avoid issues such as freezes and frame drops in preview video data, as well as frame drops in recorded video.

[0062] Figure 4 This is a flow chart of a second embodiment of the video data processing method provided by the present invention. Figure 4 Steps 410 to 420 shown are similar to Figure 2 The specific implementation of steps S210 to S220 is the same or similar and will not be described in detail here. The method includes but is not limited to the following step 430.

[0063] Step 430: The second service uses the GPU to generate one or more copies of video data based on the video data, and sends each copy of the video data to its corresponding cache space; for use by one or more third services associated with the cache space; wherein each cache space corresponds to a third service, and the third service runs on a general operating system.

[0064] The second service can make the shared space accessible to the GPU and use the GPU to copy the video data into one or more copies. The copies are then sent to the one or more cache spaces. This alleviates CPU pressure in the time domain controller, reduces pressure on the dual operating systems, and improves video frame freezes and frame drops.

[0065] In some implementations, step 430 may include the following sub-steps:

[0066] First, the second service sets the shared memory usage flag to hardware texture, binds the shared memory to EGLImageKHR, and sets it as OES texture so that the data in the shared memory can be accessed and rendered by the GPU;

[0067] Secondly, the second service uses the GPU to call the graphics interface to render the OES texture, thereby copying the video data into multiple copies, and respectively storing the multiple copies of the video data in their corresponding cache spaces.

[0068] The second service can request a memory space (e.g., GraphicBuffer) to store image data. This memory space can be the shared memory. The second service can set the shared memory's usage flag to HW_TEXTURE, indicating that the shared memory will be used as a hardware texture. This means that the data in the shared memory can be directly used by the GPU as a texture for rendering, without requiring additional copying operations, thereby improving performance.

[0069] The second service binds the shared memory to an EGLImageKHR object. EGLImageKHR is an extension in EGL (OpenGL ES Native Platform) that allows applications to create an image object that can retrieve image data from various sources, including the GraphicBuffer. Binding the shared memory to an EGLImageKHR object enables OpenGL ES to use the video data in the shared memory.

[0070] This allows the GPU to import video data in shared memory bound to the EGLImageKHR object into the OpenGL ES context for rendering as an OES texture when calling OpenGL ES for rendering. This texture can be external, meaning it is not generated by the current context but imported from another source. OES textures support rendering into multiple copies.

[0071] Please refer to Figure 5 , Figure 5 for Figure 4Schematic diagram of video data copying process in the video data processing method shown in FIG. Figure 5 As shown, the first service 51 in the real-time operating system can copy the video data obtained from the camera device to the shared memory 52 by hardware copying. The second service sets the usage flag of the shared memory 52 to a hardware texture, binds the shared memory to EGLImageKHR, and sets it to an OES texture. When the GPU calls OpenGL ES for rendering, it renders the video data in the shared memory as an OES texture, and renders it to one or more of the preview cache space 53, the video cache space 54, and the photo cache space 55. Assuming that the video service and the preview service are started at the same time, the GPU copies the above video data to the preview cache space 53 and the video cache space 54 by rendering OES textures. Therefore, for one camera device, if the video service and the preview service of the camera device are started at the same time, there will be three video data copies. Compared to Figure 1 For the solution, the number of video data copies is reduced and the channel length is also reduced.

[0072] For one camera device, if the recording service, preview service and photo service of the camera device are started at the same time, there will be 4 video data copies.

[0073] In this way, the second service can render each frame of video data as an OES texture to the cache space of the above-mentioned third services, making full use of the performance of the GPU and achieving efficient copying of the video data.

[0074] In some embodiments, the video data processing method further includes: the second service obtaining one or more Surfaces corresponding to the third service, binding each Surface to the EGL environment, wherein each Surface corresponds to a cache queue, and the cache queue includes a producer and a consumer; and using the GPU to call the graphics interface to render the OES texture, copying the video data into multiple copies, and storing the multiple copies of the video data in their respective corresponding cache spaces, including:

[0075] The second service uses the GPU to render OES textures, and as a producer, copies the video data to the cache space corresponding to each Surface for consumption by consumers of the cache queue corresponding to each Surface.

[0076] The second service can obtain the Surface of each of the third services. Surface is a buffer managed by the SurfaceFlinger service, which is the component that manages and synthesizes graphics output on the display device.

[0077] Each third-party service corresponds to a Surface. Each Surface corresponds to a BufferQueue. BufferQueue is a producer-consumer queue used to manage graphics buffers. Producers add buffers to the buffer queue, while consumers remove buffers from the buffer queue for processing, such as display or encoding.

[0078] For each Surface, the second service can use the above method as a producer to copy the video data to a buffer (which can be regarded as a cache space) in the cache queue of the Surface.

[0079] EGL is the interface between OpenGL ES and the native window system, responsible for managing the connection between the OpenGL ES context and the Surface.

[0080] When a frame of data arrives from a video stream, the second service loads it into the OpenGL ES environment as an OES texture. Using OpenGL ES functions, the second service renders this OES texture onto all previously bound Surfaces. In other words, the same frame of image data is copied and rendered into the buffers of third services, such as the preview service, the video service, and the photo service, where it is consumed by consumers in the cache queues of these third services.

[0081] If the third service is a preview, the consumer may be SurfaceFlinger, which obtains a buffer from the BufferQueue, reads video image frame data therefrom, and then synthesizes the video image frame data into final screen display content.

[0082] If the third service is video recording, the consumer can be a video data encoder. The video data encoder obtains a buffer from the BufferQueue, reads video image frame data from it, compresses and encodes the video image frame data, and generates a compressed video stream that conforms to a specific format.

[0083] Through the above-mentioned implementation, video data can be efficiently provided to different third services simultaneously.

[0084] In some embodiments, the video data processing method further includes the following steps:

[0085] First, the application determines one or more third services indicated by the service request operation according to the received service request operation.

[0086] Secondly, create Surfaces corresponding to one or more third services, and a cache queue corresponding to each Surface.

[0087] Finally, the third service information is sent to the second service, where the third service information includes the identifiers of one or more third services, information of the Surface corresponding to the one or more third services, and information of the cache queue corresponding to each Surface.

[0088] An application can receive a user's service request operation. For example, receiving a user trigger operation to activate controls for one or more third services displayed in a DVR application interface can be considered as receiving a service request operation. The application can determine the one or more third services selected by the service request operation and then create Surfaces for the one or more third services and a corresponding cache queue for each Surface.

[0089] It is understood that each Surface has a first identifier, which is used to distinguish the Surfaces corresponding to different third services. The cache queue corresponding to each Surface has a second identifier, which is used to distinguish different cache queues. The Surface information includes the first identifier and location information corresponding to the Surface, and the cache queue information includes the second identifier and location information corresponding to the cache queue.

[0090] The identifier of the third service is used to distinguish different third services. The application can send the third service information to the second service through the second service interface. The second service reads the identifiers of one or more third services, as well as the information of each third service's Surface and each cache queue from the third service information.

[0091] Users can send service request operations for different third services at different times. After each time receiving the above service request operation, the application can determine the third service requested by the service request operation, create the Surface and corresponding cache queue of the third service, and generate information about the third service and send it to the second service.

[0092] By creating a Surface and a corresponding cache queue for the third service triggered by the user, and sending the third service information including the Surface information and the cache queue information to the second service, the second service can conveniently copy the video data for the above-mentioned third service to realize the function of the third service.

[0093] Figure 6 The schematic diagram of the structure of an embodiment of the time domain controller of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the time domain controller.

[0094] like Figure 6As shown, the time domain controller may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0095] Processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 708. Communication interface 604 is used to communicate with other devices, such as clients or other server network elements. Processor 602 is used to execute program 610, which may specifically perform the steps described in the above-mentioned embodiment of the video data processing method.

[0096] Specifically, the program 610 may include program code including computer-executable instructions.

[0097] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in processor 602 may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0098] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0099] The program 610 may be specifically called by the processor 602 to cause the processor 602 to perform the following operations:

[0100] Acquire video data collected by the camera device through a first service running in the real-time operating system, and store the video data in a shared memory;

[0101] Obtaining the video data from the shared memory via a second service running in the general operating system;

[0102] The second service generates one or more copies of video data based on the video data, and stores each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with a third service, and the third service runs on a general operating system.

[0103] In an optional manner, the second service runs on a framework layer of a general operating system.

[0104] In an optional manner, obtaining video data from a shared memory by a second service running in a general operating system includes:

[0105] The shared memory is accessed through the second service based on a hardware abstraction layer service interface related to the automobile imaging system.

[0106] In an optional manner, generating one or more copies of video data according to the video data by the second service, and storing each copy of the video data in a corresponding cache space, includes:

[0107] The second service utilizes the GPU to generate one or more copies of video data according to the video data, and sends each copy of the video data to its corresponding cache space.

[0108] In an optional manner, the second service generates one or more copies of video data based on the video data using the GPU, and sends each copy of the video data to a corresponding cache space, including:

[0109] Set the shared memory usage flag to hardware texture through the second service, bind the shared memory to EGLImageKHR, and set it to OES texture so that the data in the shared memory can be accessed and rendered by the GPU;

[0110] The second service uses the GPU to call the graphics interface to render the OES texture, thereby copying the video data into multiple copies, and storing the multiple copies of the video data in their corresponding cache spaces.

[0111] In an optional manner, the program 610 may be specifically called by the processor 602 to enable the processor 602 to further execute:

[0112] Obtain one or more Surfaces corresponding to the third service through the second service, and bind each Surface to the EGL environment, where each Surface corresponds to a cache queue, and the cache queue includes a producer and a consumer;

[0113] The second service uses the GPU to call the graphics interface to render the OES texture, copying the video data into multiple copies, and storing the multiple copies of the video data in their respective corresponding cache spaces, including:

[0114] The second service uses the GPU to render OES textures, and as a producer, copies the video data to the cache space corresponding to each Surface for consumption by consumers of the cache queue corresponding to each Surface.

[0115] In an optional manner, the program 610 may be specifically called by the processor 602 to enable the processor 602 to further execute:

[0116] Determining, by an application program according to a received service request operation, one or more third services indicated by the service request operation, wherein the application program runs on a general operating system;

[0117] Create, through the application, a Surface corresponding to one or more third services, and a cache queue corresponding to each Surface;

[0118] The third service information is sent to the second service through the application. The third service information includes the identifiers of one or more third services, the Surface information corresponding to the one or more third services, and the cache queue information corresponding to each Surface.

[0119] In an optional manner, the general operating system includes the Android operating system, and the real-time operating system includes the QNX operating system.

[0120] The above solution shortens the video data transmission channel by running the second service in a general-purpose operating system. The first service directly acts as a client of the real-time operating system's first service, obtaining video data from the shared memory of the first and second services. The second service then copies the video data obtained from the shared memory into one or more copies, and stores each copy in a cache space corresponding to a third service for use by the third service, which runs on the general-purpose operating system. Therefore, different third services of the same camera device do not need to establish independent channels to obtain video data from the first service, reducing the number of channels for cross-system data transmission and, in turn, the total number of video data copies. This can reduce the system load caused by cross-system video data transmission, improve overall system performance, and avoid issues such as freezes and frame drops in preview video data, as well as frame drops in recorded video.

[0121] An embodiment of the present invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction runs on a steering actuator or a steering control, the steering actuator executes the vehicle steering control method in any of the above method embodiments.

[0122] The executable instructions can be used to cause the processor to perform the following operations:

[0123] Acquire video data collected by the camera device through a first service running in the real-time operating system, and store the video data in a shared memory;

[0124] Obtaining the video data from the shared memory via a second service running in the general operating system;

[0125] The second service generates one or more copies of video data based on the video data, and stores each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with a third service, and the third service runs on a general operating system.

[0126] In an optional manner, the second service runs on a framework layer of a general operating system.

[0127] In an optional manner, obtaining video data from a shared memory by a second service running in a general operating system includes:

[0128] The shared memory is accessed through the second service based on a hardware abstraction layer service interface related to the automobile imaging system.

[0129] In an optional manner, generating one or more copies of video data according to the video data by the second service, and storing each copy of the video data in a corresponding cache space, includes:

[0130] The second service utilizes the GPU to generate one or more copies of video data according to the video data, and sends each copy of the video data to its corresponding cache space.

[0131] In an optional manner, the second service generates one or more copies of video data based on the video data using the GPU, and sends each copy of the video data to a corresponding cache space, including:

[0132] Set the shared memory usage flag to hardware texture through the second service, bind the shared memory to EGLImageKHR, and set it to OES texture so that the data in the shared memory can be accessed and rendered by the GPU;

[0133] The second service uses the GPU to call the graphics interface to render the OES texture, thereby copying the video data into multiple copies, and storing the multiple copies of the video data in their corresponding cache spaces.

[0134] In an optional manner, the executable instructions may be specifically configured to cause the processor 602 to further perform the following operations:

[0135] Obtain one or more Surfaces corresponding to the third service through the second service, and bind each Surface to the EGL environment, where each Surface corresponds to a cache queue, and the cache queue includes a producer and a consumer;

[0136] The second service uses the GPU to call the graphics interface to render the OES texture, copying the video data into multiple copies, and storing the multiple copies of the video data in their respective corresponding cache spaces, including:

[0137] The second service uses the GPU to render OES textures, and as a producer, copies the video data to the cache space corresponding to each Surface for consumption by consumers of the cache queue corresponding to each Surface.

[0138] In an optional manner, the executable instructions may be specifically configured to cause the processor 602 to further perform the following operations:

[0139] Determining, by an application program according to a received service request operation, one or more third services indicated by the service request operation, wherein the application program runs on a general operating system;

[0140] Create, through the application, a Surface corresponding to one or more third services, and a cache queue corresponding to each Surface;

[0141] The third service information is sent to the second service through the application. The third service information includes the identifiers of one or more third services, the Surface information corresponding to the one or more third services, and the cache queue information corresponding to each Surface.

[0142] In an optional manner, the general operating system includes the Android operating system, and the real-time operating system includes the QNX operating system.

[0143] The above solution shortens the video data transmission channel by running the second service in a general-purpose operating system. The first service directly acts as a client of the real-time operating system's first service, obtaining video data from the shared memory of the first and second services. The second service then copies the video data obtained from the shared memory into one or more copies, and stores each copy in a cache space corresponding to a third service for use by the third service, which runs on the general-purpose operating system. Therefore, different third services of the same camera device do not need to establish independent channels to obtain video data from the first service, reducing the number of channels for cross-system data transmission and, in turn, the total number of video data copies. This can reduce the system load caused by cross-system video data transmission, improve overall system performance, and avoid issues such as freezes and frame drops in preview video data, as well as frame drops in recorded video.

[0144] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0145] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present invention may be practiced without these specific details. Similarly, in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. The claims that follow the detailed description are hereby expressly incorporated into that detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0146] Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively changed and installed in one or more devices different from the embodiments. The modules, units, or components in the embodiments can be combined into one module, unit, or component, and furthermore, they can be divided into multiple submodules, subunits, or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

[0147] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A video data processing method, applied to a vehicle running a real-time operating system and a general operating system, the method comprising: The first service obtains video data collected by the camera device and stores the video data in a shared memory, wherein the first service runs on the real-time operating system. A second service obtains the video data from the shared memory, the second service running on a general operating system; The second service generates one or more copies of video data based on the video data, and stores each copy of the video data in a corresponding cache space for use by a third service associated with each cache space; wherein each cache space is associated with a third service, and the third service runs on the general operating system.

2. The method according to claim 1, characterized in that The second service runs on the framework layer of the general operating system.

3. The method according to claim 2, characterized in that The second service obtains the video data from the shared memory, including: The second service accesses the shared memory based on a hardware abstraction layer service interface related to the automobile imaging system.

4. The method according to claim 1, characterized in that: The second service generates one or more copies of video data according to the video data, and stores each copy of the video data in a corresponding cache space, including: The second service generates the one or more copies of the video data according to the video data by using the GPU, and sends each copy of the video data to the corresponding cache space.

5. The method according to claim 4, characterized in that The second service generates the one or more copies of the video data according to the video data by using the GPU, and sends each copy of the video data to the corresponding cache space, including: The second service sets the usage flag of the shared memory to a hardware texture, binds the shared memory to an EGLImageKHR, and sets the shared memory to an OES texture, so that the data in the shared memory can be accessed and rendered by the GPU; The second service uses the GPU to call a graphics interface to render an OES texture, thereby copying the video data into multiple copies, and respectively stores the multiple copies of the video data in the corresponding cache spaces.

6. The method according to claim 5, characterized in that The method further comprises: The second service obtains the Surface corresponding to the one or more third services respectively, and binds each Surface to the EGL environment, wherein each Surface corresponds to a cache queue, and the cache queue includes a producer and a consumer; The second service uses the GPU to call a graphics interface to render an OES texture, so as to copy the video data into multiple copies, and stores the multiple copies of the video data in the corresponding cache spaces, respectively, including: The second service utilizes the GPU to render OES textures, and as the producer, copies the video data to the cache space corresponding to each Surface for consumption by consumers of the cache queue corresponding to each Surface.

7. The method according to claim 6, characterized in that Also includes: The application determines one or more third services indicated by the service request operation according to the received service request operation, wherein the application runs on a general operating system; The application creates the Surface corresponding to the one or more third services respectively, and a cache queue corresponding to each of the Surfaces; The application sends third service information to the second service, where the third service information includes the respective identifiers of the one or more third services, the information of the Surface corresponding to the one or more third services, and the information of the cache queue corresponding to each of the Surfaces.

8. The method according to any one of claims 1 to 7, characterized in that: The general operating system includes an Android operating system, and the real-time operating system includes a QNX operating system.

9. A vehicle, comprising a time domain controller, the time domain controller comprising a memory and a processor of a general operating system and a real-time operating system running on the time domain controller, wherein: The memory stores computer executable instructions, and when the executable instructions are executed by the processor, the video data processing method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, wherein the computer-readable storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to execute the video data processing method according to any one of claims 1 to 8.

Images