Camera data processing system architecture and method, controller, vehicle and medium

Abstract

The invention provides a system architecture and method for camera data processing, a controller, a vehicle and a medium, and relates to the technical field of data processing.The system architecture comprises an EVS native architecture of an Android system, a camera service process and a camera application program, and the EVS native architecture comprises the EVS service process and an EVS manager; the camera service process interacts with the EVS manager through a first data interface, the EVS service process interacts with the EVS manager through a second data interface, and the service logic of the camera service process and the service logic of the EVS service process are mutually independent; the camera service process is used for receiving the video stream asynchronously sent by the EVS manager and sending the video stream to the camera application program for rendering; the camera application is used to process the rendered video stream, the processing including storing the current video frame. Function development is carried out in a camera application program, multiple functions of photographing, video recording, video processing and the like are supported, and the user interaction capability of an EVS architecture is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a system architecture, method, controller, vehicle, and medium for camera data processing. Background Art

[0002] In the camera system of a mobile phone, the image capture and processing link usually includes a camera application at the application layer, which interacts with the hardware driver through the Android camera interface. The camera hardware is connected to the underlying driver through the Hardware Abstraction Layer (HAL for short). The data stream is forwarded by HAL to the application layer for display or processing. The camera startup process is relatively complex, requiring loading the camera hardware driver, configuring image parameters, initializing the image processing module, etc. Therefore, there is often a long delay when the camera starts. In addition, since the mobile phone usually requires higher-quality image processing, such as functions like taking pictures and recording videos, the data bandwidth requirement for the video stream is relatively high.

[0003] Different from the camera system of a mobile phone, the EVS (Exterior View System) architecture is being more and more widely used in vehicle systems, especially for functions such as real-time video monitoring and rearview imaging. The design goal of the EVS system is fast startup and low latency, and it can provide real-time vehicle views. Usually, a more simplified architecture is adopted, focusing on fast startup and the lowest memory bandwidth occupancy to ensure that the vehicle can quickly respond to the driver's needs. In the EVS system, the camera image processing chain is relatively streamlined, mainly ensuring low latency and fast response through efficient video stream transmission and rendering.

[0004] However, the implementation of the native EVS application is still relatively imperfect. The native application only supports basic image preview functions and lacks advanced functions (such as taking pictures, recording videos, image processing, etc.). The flexibility and scalability of EVS are limited and cannot fully meet the requirements of vehicle application scenarios. Summary of the Invention

[0005] One of the purposes of this application is to provide a system architecture for camera data processing to solve the problem that the native application in the existing EVS architecture only supports basic image preview functions, has a single function, and poor flexibility and scalability. The second purpose of this application is to provide a method for camera data processing to implement taking pictures or recording videos in the EVS architecture. The third purpose is to provide a controller; the fourth purpose is to provide a vehicle; the fifth purpose is to provide a storage medium; the sixth purpose is to provide a program product.

[0006] To achieve the above purposes, the technical solutions adopted in this application are as follows:

[0007] The first aspect of the present application provides a system architecture for camera data processing. The system architecture includes: the EVS native architecture of the Android system, a camera service process, and a camera application. The EVS native architecture includes an EVS service process and an EVS manager;

[0008] The camera service process interacts with the EVS manager through a first data interface, and the EVS service process interacts with the EVS manager through a second data interface. The business logics of the camera service process and the EVS service process are independent of each other;

[0009] The camera service process is used to receive the video stream asynchronously sent by the EVS manager and send the video stream to the camera application for rendering;

[0010] The camera application is used to process the rendered video stream, and the processing includes storing the current video frame.

[0011] According to the above technical means, CameraServer is introduced into the native EVS architecture and runs in parallel with EvsServer, and the system realizes the isolation of business logics. The business of CameraServer and the original EVS architecture will not interfere with each other, thereby improving the stability and scalability of the system. Advanced functions such as taking pictures and recording videos are encapsulated and extended in CameraApp, so that new functions can be flexibly added as needed without affecting the stability of the original system.

[0012] Further, the video stream includes video data collected by a camera of a driver monitoring system (DMS) and / or video data collected by a camera of a cockpit monitoring system (IMS).

[0013] Further, the configuration file of the EVS native architecture includes camera information of the IMS and / or camera information of the DMS.

[0014] Further, the first data interface is the IEvsCameraStream interface.

[0015] The second aspect of the present application provides a method for camera data processing, which is applied to the system architecture according to any one of the first aspect. The method includes:

[0016] In response to a user's operation of taking a picture or starting video recording, the camera application sends a request instruction to the camera service process, and the request instruction includes a camera identifier of the camera;

[0017] After receiving the request instruction, the camera service process sends an initialization signal to the Evs hardware enumeration component of the EVS native architecture;

[0018] The Evs hardware enumeration component enumerates the camera according to the initialization signal to determine the target camera corresponding to the camera identifier;

[0019] The Evs hardware enumeration component acquires the video stream of the target camera and sends the video stream to the EVS manager;

[0020] The EVS manager transmits the video stream to the camera service process through an asynchronous mechanism, and the asynchronous mechanism means that the EVS manager performs other tasks during the process of transmitting the video stream;

[0021] The camera service process transmits the video stream to the camera application;

[0022] The camera application displays the rendered video stream on the screen and processes the rendered video stream, and the processing includes storing the current video frame.

[0023] Further, the EVS manager transmits the video stream to the camera service process through an asynchronous mechanism, including:

[0024] The EVS manager puts the video frame data of the video stream into the buffer through an asynchronous mechanism;

[0025] The camera service process uses the IEvsCameraStream interface to extract the video frame data from the buffer at a preset time interval or based on an event.

[0026] Further, the processing of the rendered video stream by the camera application includes:

[0027] The camera application obtains the current rendering texture of the video stream through a graphics rendering library and saves the rendering texture as an image or a video file.

[0028] Further, the camera service process transmits the video stream to the camera application, including:

[0029] The camera service process renders the video stream into the display buffer Surface provided by the camera application.

[0030] In a third aspect, the present application provides a controller, including: a memory, a processor;

[0031] The memory stores computer execution instructions;

[0032] The processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of the second aspect.

[0033] Fourth aspect, the present application provides a vehicle, including a vehicle body, a controller, a camera and a display device arranged in the vehicle body, and the controller is used to implement the method described in any item of the second aspect.

[0034] Fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any item of the second aspect.

[0035] Sixth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method described in any item of the second aspect.

[0036] Advantages of the present application:

[0037] (1) Through customized development, the present application encapsulates and expands advanced functions such as taking pictures and recording videos in CameraApp. This architecture design enables new functions to be flexibly added as needed without affecting the business processes of the original EVS architecture, maintaining system stability.

[0038] (2) The system architecture for camera data processing of the present application can achieve fast response to IMS and DMS cameras and reduce the system processing time. In real-time monitoring and image preview, it can provide low latency, meeting the requirements of vehicle safety and driver monitoring.

[0039] (3) Compared with the traditional mobile phone Camera architecture, the system architecture for camera data processing of the present application focuses on low consumption of memory bandwidth. It can minimize the consumption of memory bandwidth to the greatest extent without sacrificing image quality, improving the overall system performance.

[0040] (4) By introducing CameraServer into the native EVS architecture and running it in parallel with EvsServer, the system realizes the isolation of business logic. CameraServer does not interfere with the business of the original EVS architecture, thus improving the stability and scalability of the system. Description of the drawings

[0041] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments in line with the present application, and are used together with the specification to explain the principles of the present application.

[0042] Figure 1 It is a schematic structural diagram of the system architecture for camera data processing provided by the present application;

[0043] Figure 2 It is a schematic flowchart of a method embodiment for camera data processing provided by the present application;

[0044] Figure 3 The EVS application logic diagram that provides a camera list for this application;

[0045] Figure 4 The structural schematic diagram of the electronic device provided for this application.

[0046] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0047] The following will illustrate the implementation manners of this application with reference to the drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and not for limiting the protection scope of this application.

[0048] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of this application in a schematic manner. Therefore, only the components related to this application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0049] First, introduce the proprietary terms related to this application.

[0050] In the Android Automotive environment, EVS (Exterior View System) is a video stream processing system designed specifically for in-vehicle applications, mainly used for data transmission between cameras and display devices, and usually involves the processing of external cameras (such as rearview mirrors, surround view systems, etc.). Different from the Camera API on mobile phones, EVS mainly optimizes for low latency, fast startup, and low memory bandwidth consumption to ensure the efficiency and responsiveness of the in-vehicle system when processing video streams in real time. The EVS architecture usually includes the following core components:

[0051] The Evs Server is a core service in the native EVS architecture, responsible for managing video streams, displays, and device control. It handles the interface interaction between video devices (such as cameras) and display devices (such as screens). The evsServer usually runs on an embedded system or hardware platform, processing real-time video data streams.

[0052] The Evs manager is a management component in the EVS system, responsible for interacting with the Hardware Abstraction Layer (HAL) and performing hardware-related initialization and management tasks. The Evs manager is mainly used to schedule and manage hardware resources such as cameras and display devices, and provides interfaces for other system components to access and control hardware resources.

[0053] The Evs app is an application in the EVS architecture, usually customized and replaced by OEM manufacturers according to requirements. The Evs app is the part that executes the business logic in the native EVS system, responsible for implementing video stream processing, display, and interaction, and supporting the image preview function.

[0054] The Hardware Abstraction Layer (HAL) is responsible for communicating with physical hardware (such as cameras and displays), providing standard interfaces, enabling upper-layer applications to control the hardware without directly operating on the underlying hardware.

[0055] SampleDriver is the implementation of the driver layer in the EVS system. It is a driver sample program based on v4l2 (Video4Linux2), used to interact with camera hardware and display devices. It provides the core functions of camera device operations and display device operations for the HAL layer. SampleDriver is a bridge between the hardware and higher-level applications (such as the Camera Server), responsible for obtaining video streams from the camera and transmitting them to the display device.

[0056] Due to the extremely high real-time requirements of in-vehicle applications, the EVS architecture provides an optimized video processing path to ensure real-time display and control. Especially when the vehicle is reversing or parking, the images of cameras such as rearview mirrors need to be presented quickly. Compared with the Camera on mobile phones, EVS is more refined in using hardware resources, capable of processing video streams with lower power consumption and bandwidth consumption, thus extending the usage time of in-vehicle devices and reducing the system burden.

[0057] Although the EVS system provides excellent performance in the in-vehicle environment, the native EVS app mainly provides a simple image preview function and lacks complex functions such as taking photos, recording videos, and image processing.

[0058] In view of this, the present application provides a system architecture for camera data processing, which is improved on the basis of the Android native EVS architecture. A camera service process (Camera Server) and a camera application (CameraApp) are added. CameraServer and Evs manager interact through a data interface, so that the self-developed camera application can obtain the video stream in Evs manager through CameraServer, and then perform function operations such as taking pictures, recording videos, and image processing. Compared with the native EVS architecture, which only supports the preview function, it can support a variety of function operations, providing a better user experience for drivers. Compared with the camera link architecture on the mobile phone side, it can achieve fast and low-latency image processing, optimizing the processing performance of the in-vehicle system for the camera video stream.

[0059] The camera service process (Camera Server) refers to a self-developed middleware service process that is responsible for coordinating the camera and the display, obtaining data frames through the standard interface defined by Google, and rendering the data frames to the surface through EGL.

[0060] The camera application (CameraApp) refers to the front-end application that finally interacts with the user, provides a graphical user interface (GUI), and allows the user to control the functions of the camera through a touch screen or other input devices.

[0061] The system architecture for camera data processing provided by the present application can be applied in the vehicle field, and can also be applied in devices such as ships, airplanes, mobile phones, and virtual reality devices.

[0062] Taking the application in the vehicle field as an example below, the technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0063] Figure 1 It is a schematic structural diagram of the system architecture for camera data processing provided by the present application. As Figure 1 shown, the system architecture includes: the EVS native architecture of the Android system (the dotted box in the figure), the camera service process (Camera Server), and the camera application (Camera App). The EVS native architecture includes an EVS service process (Evs Server), an Evs manager (Evsmanager), and a hardware abstraction layer (EvaHal);

[0064] The EvaHal layer of the EVS native architecture is responsible for providing a unified hardware access interface to the upper layer (such as the EVS manager) while hiding the details of the underlying hardware.

[0065] The camera service process (Camera Server) interacts with the EVS manager (Evs manager) through the first data interface, and the EVS service process (Evs Server) interacts with the EVS manager (Evs manager) through the second data interface. The business logic of the service processes of EvsServer and Camera Server are independent of each other. Evs Server focuses on the acquisition and processing of video streams, and transmits them to various applications in the vehicle system that need to use video stream data, while Camera Server is responsible for the transmission of video streams and passing them to the upper-level camera application (Camera App), which does not affect the original Evs Server business.

[0066] The camera server process is used to receive the video stream sent asynchronously by the EVS manager and send the video stream to the camera application for rendering;

[0067] The camera application (Camera APP) is used to process the rendered video stream, including storing the current video frame. When the user selects the photo function, only the current video frame is stored. When the user selects the video function, the current video frame is continuously stored to generate a video file.

[0068] The native EVS architecture usually only supports the preview function of the video stream, which is mainly used to display video data in real time. The system architecture of camera data processing provided in this embodiment separates the functions and business logic of Evs Server and Camera Server independently, so that each component can focus on its core function and reduce functional coupling. For example, EvsServer is only responsible for processing and transmitting conventional video streams, while Camera Server handles the transmission and rendering of video streams required by users in Camera App, avoiding complex coupling problems. Through this architectural design, function development is carried out in Camera App, which can support multiple functions such as taking pictures, recording, and video processing in addition to preview. This extension makes the system not only limited to displaying video content, but also can store, play back and process content. Users can directly take pictures, record videos, adjust exposure and other operations through Camera App, which improves the user interaction capability of the system.

[0069] In some embodiments, the video stream obtained in the Evs manager includes the video data collected by the cameras of the Driver Monitoring System (DMS) and / or the video data collected by the cameras of the In-Vehicle Monitoring System (IMS).

[0070] In some embodiments, the configuration file of the EVS native architecture includes the camera information of the IMS and / or the camera information of the DMS. The camera information is loaded in the configuration file to enable the opening of the IMS camera and / or the DMS camera.

[0071] In some embodiments, the first data interface between the Camera Server and the Evs manager is set as the IEvsCameraStream interface. In the original EVS architecture, the second data interface between the EVS Server and the Evs manager is the IEvsCameraStream interface. Using the unified IEvsCameraStream interface standardizes the data exchange between different components. Whether it is the Camera Server or the EVS Server, they can perform efficient data interaction with the EVS Manager through the standard interface. This can reduce the complexity of interface design and simplify system integration. Through two independent IEvsCameraStream interfaces (for the Camera Server and the EVS Server respectively), the system can flexibly process different types of video streams, and the business logics between them do not interfere with each other.

[0072] The system architecture for processing camera data can be designed on the native EVS architecture through the following steps:

[0073] The first step: It is necessary to transform the native Evs architecture to enable it to open the IMS and / or DMS cameras.

[0074] Specifically, add the IMS and / or DMS camera configuration information to the configuration file (Config file) of the EVS native architecture. Each camera configuration information includes information such as cameraId (camera identifier), resolution, and frame rate.

[0075] In the EVS architecture, the EvsEnumeratorHw service component is responsible for enumerating the available hardware devices in the system. Add the IMS and / or DMS cameras to the initialization function of the EvsEnumerator object (Evs hardware enumeration component) that implements IEvsEnumerator in the EvsEnumeratorHw service entity.

[0076] The second step: Add an interface for interacting with the Camera Server in the Evs Manager adaptation.

[0077] It is necessary to add the adaptation of the data interface of the Camera Server in the Evs Manager, especially the interfaces of IMS and DMS cameras, such as operations like openDisplay() and openCamera(). These operations are used for functions such as turning on the camera and starting preview. As needed, interfaces such as closeCamera() and closeDisplay() can also be adapted so that users can turn off the camera stream and display stream when they are not needed.

[0078] Step 3: Conduct adaptation development for the Camera Server.

[0079] 1. Adapt the frame acquisition framework of the Evs Manager. Implement the IEvsCameraStream interface in the Camera Server to receive video frames transmitted asynchronously. This interface is responsible for handling the video data stream from the camera. IEvsCameraStream is a standard interface for receiving video frame data and can support the requirements of asynchronous stream transmission.

[0080] 2. Define the interaction interfaces between the Camera Server and the Camera App for adapting interfaces such as opening the camera, starting preview, and stopping preview. These interfaces will be used to control operations such as starting, stopping, recording, and taking pictures of the camera.

[0081] 3. Preview and display process. The frame data obtained by the Camera Server needs to be pushed into the Surface (such as SurfaceTexture) of the Camera App for real-time video display. In some embodiments, the callback of SurfaceTexture.OnFrameAvailableListener can be used to monitor the availability of each frame, and the display content can be refreshed through the GLSurfaceView.

[0082] Step 4: Develop the functions of the Camera App.

[0083] Implement the functions of taking pictures and recording videos in the Camera App. Since the EVS system focuses on quick startup and image preview and does not provide complete interfaces for taking pictures and recording videos, the functions of taking pictures and recording videos are implemented through OpenGL. The textures created by OpenGL are passed to the Camera Server for display or recording. The callback of SurfaceTexture.OnFrameAvailableListener is used to continuously update the texture data.

[0084] OpenGL Texture Display and Rendering. In the Camera App, textures can be generated through OpenGL and passed to the CameraServer for display. The specific operations are as follows: Use SurfaceTexture in the GLSurfaceView to receive the frame data transmitted by the camera. Use the onFrameAvailable() callback method to refresh the texture data in real time. Render the texture data in the onDrawFrame() method to render the camera preview screen.

[0085] The video recording function will be completed by the textures received from the IEvsCameraStream. The recording process usually saves the image data of each frame as a video file or processes and stores it in other ways.

[0086] Function extensions can also be made based on the preview image in the Camera App, such as stickers, filters, etc. Among them, the sticker function allows users to overlay predefined images (such as emojis, icons, etc.) on the camera preview screen, and can be moved, scaled, and rotated according to user gestures. Specifically, in the OpenGL rendering loop, load the sticker image (usually in PNG, JPEG, or other image formats) and draw it at the specified position on the camera preview frame. The position and size of the sticker can be dynamically adjusted through touch events, allowing users to drag, scale, and rotate the sticker. The filter function allows users to perform real-time image processing on the camera screen, such as increasing contrast, saturation, blur, black and white, etc. effects. The filter effect can be achieved through the Fragment Shader. Each frame of camera preview data passes through the OpenGL image processing pipeline, and the filter shader is applied to modify its color, contrast, saturation and other characteristics. A set of preset filters can be provided in the Camera App for users to choose. The filter effect can also be applied to the preview image at the same time, and the sticker can be overlaid on the processed image.

[0087] The above design steps include: integrating the IMS and DMS cameras into the EVS system, and implementing the interaction with the CameraServer, the processing of video streams, and the development of the photo-taking and video-recording functions in the Camera App. The whole process involves the adaptation and modification of multiple components, including work in various aspects such as configuration files, hardware enumeration, data interfaces, and image display. After the design of the above steps, users can achieve functions such as video recording, photo-taking, image sticking, and filtering based on the EVS architecture through the Camera App.

[0088] Based on the above-designed system architecture for camera data processing, the method for camera data processing will be introduced below.

[0089] Figure 2A schematic flowchart of an embodiment of a method for processing camera data provided by this application, as shown in Figure 2 shown, the method includes the following steps:

[0090] S101. In response to a user's photo-taking operation or video-recording operation start, the camera application sends a request instruction to the camera service process. The request instruction includes the camera identifier of the camera.

[0091] The user starts the video-recording function through the Camera App, and the user can trigger it by clicking the video-recording button or the photo-taking button in the UI. In the Camera App, the user's click action will be converted into a request instruction. This request can be sent to the underlying system components through the application layer interface (such as a button click event).

[0092] The Camera App will send a request instruction for starting video recording or photo taking to the Camera Server through an appropriate interface (such as a Java API or NDK interface). The request instruction contains the camera identifier selected by the user (such as the ID of the IMS and / or DMS camera), indicating that the video data of this camera needs to be displayed in the Camera App.

[0093] S102. After receiving the request instruction, the camera service process sends an initialization signal to the Evs hardware enumeration component of the EVS native architecture.

[0094] After receiving the start instruction from the Camera App, the Camera Server first needs to perform some preparatory work, such as configuring the camera (such as selecting the resolution, frame rate, etc.) and initializing the video stream. The Camera Server sends an initialization signal to the Evs hardware enumeration component (EvsEnumerator), requesting to enumerate and start the camera stream. The purpose of this initialization signal is to let the Evs hardware enumeration component be ready to interact with the specified camera to ensure that the camera data stream can be obtained subsequently.

[0095] S103. The Evs hardware enumeration component enumerates the camera according to the initialization signal to determine the target camera corresponding to the camera identifier.

[0096] After receiving the initialization signal, the Evs hardware enumeration component starts to enumerate the cameras in the system. The Evs hardware enumeration component determines the target camera that matches the incoming camera identifier by viewing the configuration file and hardware information. Confirm the target camera device corresponding to the camera identifier in the request and register this device with the EVS Manager.

[0097] S104. The Evs hardware enumeration component obtains the video stream of the target camera and sends the video stream to the EVS manager.

[0098] Once the target camera is identified and initialized, the Evs Hardware Enumeration component (EvsEnumerator) will start obtaining the video stream from the target camera and transmit it to the EVS Manager for subsequent processing. Specifically, the Evs Hardware Enumeration component interacts with the camera hardware interface (such as V4L2, etc.) to start obtaining the real-time video stream data of the target camera, and the video stream will be encoded or transmitted to the EVS Manager in the original format.

[0099] S105. The EVS Manager transmits the video stream to the camera service process through an asynchronous mechanism, and the asynchronous mechanism means that the EVS Manager can perform other tasks during the process of transmitting the video stream.

[0100] The processed video stream data will be transmitted from the EVS Manager to the Camera Server through an asynchronous mechanism.

[0101] The asynchronous mechanism allows the EVS Manager to continue performing other operations while transmitting the video stream without waiting for the transmission of the video stream to complete. For example, the EVS Manager may be able to process other tasks while transmitting the video stream, such as encoding video data, frame processing, initialization of other devices, etc. This is different from the synchronous mechanism, which will start the next task only after one task is completed, resulting in some tasks being blocked and unable to be processed in parallel.

[0102] In some embodiments, during asynchronous transmission, video stream data is usually first stored in a buffer or queue. The EVS Manager is responsible for continuously pushing data into the queue, while the Camera Server will asynchronously retrieve the video stream from the queue at preset intervals or based on events for processing. This buffering mechanism enables the video stream transmission and reception to be carried out in an efficient manner, preventing frame loss or stuttering due to certain instantaneous delays. Specifically, the IEvsCameraStream interface is used to define how to receive data from the video stream, and the Camera Server implements the IEvsCameraStream interface. The EVS Manager processes the video data and places each video frame into a buffer or queue. This buffer can store multiple video frames waiting for the Camera Server to retrieve. The Camera Server uses the IEvsCameraStream interface to extract video frames from the buffer regularly or based on events. When the EVS Manager places a video frame into the buffer, it triggers the corresponding callback mechanism to notify the Camera Server that the current frame is available. This process ensures that the Camera Server can timely extract video frames from the buffer. For the callback mechanism, when each new video frame is ready, the EVS Manager checks if there is a registered callback (i.e., onFrameAvailable() in the IEvsCameraStream interface). If so, the EVS Manager asynchronously transmits the video frame to the CameraServer through the callback mechanism.

[0103] S106. The camera service process transmits the video stream to the camera application.

[0104] The camera service process renders the video stream into the display buffer Surface provided by the camera application for display and processing on the user device. Specifically, the video frame data obtained through the IEvsCameraStream interface exists in the form of a buffer, and the buffer stores the pixel data of the video frame. The Camera App provides a Surface, and the Camera Server pushes the buffer data (i.e., frame data) received from the IEvsCameraStream into this Surface to display the picture.

[0105] S107. The camera application displays the rendered video stream on the screen and processes the rendered video stream, and the processing includes storing the current video frame.

[0106] In a camera application, the video stream is rendered onto the screen, allowing the user to see a real-time video preview. If the user selects to take a photo or start recording a video, the application performs additional processing while rendering: for example, when taking a photo, the current video frame is saved as a static image, or when recording a video, the video frames are stored as a video file.

[0107] In a specific implementation, the Camera App uses a graphics rendering library (OpenGL) to create a texture object, which will be used to receive the video stream from the camera. The OpenGL texture is managed through SurfaceTexture, and this texture is passed to the CameraServer to receive the preview frame data of the camera. The Surface is used to encapsulate the SurfaceTexture into an interface that can be passed to the CameraServer. The CameraServer will render the real-time preview data of the camera onto this Surface.

[0108] Set an OnFrameAvailableListener callback for SurfaceTexture. When a new frame of camera data arrives, this callback will be triggered. Through the triggering of the callback, the update of the camera data can be obtained. In the callback function, OpenGL needs to refresh the texture data. Whenever a new frame arrives,

[0109] the texture content is updated via SurfaceTexture.updateTexImage(). When taking a photo, the texture content is converted into image data through OpenGL and saved as a picture file. When recording a video, a video file is generated by continuously obtaining and saving each frame of texture data.

[0110] By rendering the texture through OpenGL and coordinating with the implementation of the camera preview, photo taking, and video recording functions, efficient real-time video stream processing and image capture capabilities can be provided. This method is commonly used in applications that need to process high-definition video streams, such as real-time preview, video recording, and photo taking functions.

[0111] This embodiment provides a method for processing camera data. The method includes: in response to a user's photo-taking operation or video-recording operation start, the camera application sends a request instruction to the camera service process, and the request instruction includes the camera identifier of the camera; after receiving the request instruction, the camera service process sends an initialization signal to the Evs hardware enumeration component of the Evs native architecture; the Evs hardware enumeration component enumerates the camera according to the initialization signal to determine the target camera corresponding to the camera identifier; the Evs hardware enumeration component obtains the video stream of the target camera and sends the video stream to the Evs manager; the Evs manager transmits the video stream to the camera service process through an asynchronous mechanism, and the asynchronous mechanism means that the Evs manager performs other tasks during the process of transmitting the video stream; the camera service process transmits the video stream to the camera application; the camera application displays the rendered video stream on the screen and processes the rendered video stream, and the processing includes storing the current video frame. Through the above method, the camera application can obtain the video stream in the Evs manager through the Camera Server, and then perform function operations such as photo-taking, video-recording, and image processing. Compared with the native Evs architecture, on the basis of only supporting the preview function, it can support a variety of function operations, providing a better user experience for the driver. Compared with the camera link architecture of the mobile phone side, it can achieve fast and low-latency image processing, optimizing the processing performance of the in-vehicle system for the camera video stream.

[0112] Figure 3 The EVS application logic diagram for providing the camera list in this application is as follows Figure 3 shown. Taking the Evs hardware enumeration component as the execution subject, it includes the following steps:

[0113] S201. Obtain the camera list.

[0114] The Evs hardware enumeration component obtains the list of all configured cameras to prepare for subsequent camera selection and initialization. Specifically, the Evs hardware enumeration component calls getCameraList to obtain the list of all configured available cameras. Through this method, the cameraId of each camera can be obtained, and these IDs represent all devices that can be used to capture video or images.

[0115] S202. Request a notification from the vehicle hardware abstraction layer (HAL).

[0116] The Evs hardware enumeration component requests a notification from the vehicle's hardware abstraction layer (HAL). The vehicle's HAL usually notifies the system of some hardware events, such as the available status of the camera, error reports, or hardware resource updates, etc. Ensuring that the system can timely learn about changes in the hardware status and can adjust the behavior of the camera or image stream according to the notification.

[0117] S203. Start the message loop.

[0118] This step starts a polling loop for messages to poll and process the captured video stream.

[0119] S204. Check if there is an IMS.

[0120] The EVS hardware enumeration component checks if an IMS camera is polled.

[0121] S205. If the IMS exists, select the camera related to the IMS.

[0122] S206. If the IMS does not exist, check if there is a DMS.

[0123] S207. If the DMS exists, select the camera related to the DMS.

[0124] S208. If the DMS does not exist, stop the image stream.

[0125] If neither the IMS nor the DMS exists, it means the camera to be used is not started, and the system will stop the current image stream to save resources.

[0126] S209. Turn off the display.

[0127] Perform this operation when there is no camera video stream. Turning off the display can save energy and avoid showing blank content on the screen.

[0128] S210. Start the image stream.

[0129] When a suitable camera is selected and ready to display the image, the system starts the image stream. This step makes the camera start capturing images or videos and transmits them to the display device.

[0130] S211. Set the image stream to be displayed in the next frame.

[0131] Push and display the started image stream as the next frame image of the video stream.

[0132] This embodiment introduces dynamically detecting the status of the camera during the actual use of the vehicle, pushing and displaying the video stream when the camera is in a normal state, and stopping the image stream and turning off the display to save resources when the camera is in an abnormal state.

[0133] Figure 4 It is a schematic structural diagram of the electronic device provided by this application. The electronic device can be a controller in a vehicle. As Figure 4As shown in the figure, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. Among them, the processor 501, the memory 502, and the communication component 503 are connected through a bus 504.

[0134] In the specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above-mentioned method.

[0135] For the specific implementation process of the processor 501, reference can be made to the above method embodiment. The implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0136] In the above embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0137] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0138] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0139] This application also provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0140] The present application also provides a vehicle, which includes: a vehicle body, a controller, and a camera and a display device disposed in the vehicle body, and the controller is configured to execute the above method.

[0141] The present application also provides a computer program product, including a computer program, which when executed by a processor, implements the above method.

[0142] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the above method is implemented.

[0143] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disc. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0144] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0145] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in an electrical, mechanical or other form.

[0146] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, in each embodiment of the present application, each functional unit may be integrated into one processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit.

[0148] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0149] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program may be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROMs, RAMs, magnetic disks, or optical discs that can store program codes.

[0150] Finally, it should be noted that: after considering the specification and practicing the invention disclosed herein, those skilled in the art will readily think of other implementation manners of the present application. The present application is intended to cover any variations, uses, or adaptive changes of the present application. These variations, uses, or adaptive changes follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. It is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A system architecture for camera data processing, characterized in that, The system architecture includes: the EVS native architecture of the Android system, a camera service process, and a camera application. The EVS native architecture includes an EVS service process and an EVS manager; The EVS manager interacts with the camera service process through a first data interface, and the EVS manager interacts with the EVS service process through a second data interface. The business logics of the camera service process and the EVS service process are independent of each other; The camera service process is used to receive the video stream asynchronously sent by the EVS manager and send the video stream to the camera application for rendering; The camera application is used to process the rendered video stream, and the processing includes storing the current video frame.

2. The system architecture according to claim 1, characterized in that, The video stream includes video data collected by a camera of a driver monitoring system (DMS) and / or video data collected by a camera of a cockpit monitoring system (IMS).

3. The system architecture according to claim 2, wherein The configuration file of the EVS native architecture includes the camera information of the IMS and / or the camera information of the DMS.

4. The system architecture according to any one of claims 1-3, characterized in that, The first data interface is the IEvsCameraStream interface.

5. A method for processing camera data, characterized in that, Applied to the system architecture according to any one of claims 1-4, the system architecture includes: the EVS native architecture of the Android system, a camera service process, and a camera application. The EVS native architecture includes an EVS service process and an EVS manager. The method includes: In response to a user's photo-taking operation or video-recording operation start, the camera application sends a request instruction to the camera service process, and the request instruction includes a camera identifier; After receiving the request instruction, the camera service process sends an initialization signal to the Evs hardware enumeration component of the EVS native architecture; The Evs hardware enumeration component enumerates the cameras according to the initialization signal to determine the target camera corresponding to the camera identifier; The Evs hardware enumeration component acquires the video stream of the target camera and sends the video stream to the EVS manager; The EVS manager transmits the video stream to the camera service process through an asynchronous mechanism, and the asynchronous mechanism means that the EVS manager performs other tasks during the process of transmitting the video stream; The camera service process transmits the video stream to the camera application; The camera application displays the rendered video stream on the screen and processes the rendered video stream, and the processing includes storing the current video frame.

6. The method according to claim 5, wherein The EVS manager transmits the video stream to the camera service process through an asynchronous mechanism, including: The EVS manager puts the video frame data of the video stream into a buffer through an asynchronous mechanism; The camera service process uses the IEvsCameraStream interface to extract the video frame data from the buffer at preset time intervals or based on events.

7. The method according to claim 5 or 6, characterized in that, The camera application processes the rendered video stream, including: The camera application acquires the current rendering texture of the video stream through a graphics rendering library and saves the rendering texture as an image or video file.

8. The method according to claim 5 or 6, characterized in that, The camera service process transmits the video stream to the camera application, including: The camera service process renders the video stream into the display buffer Surface provided by the camera application.

9. A controller, characterized in that, Including: A memory, a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 5-8.

10. A vehicle, characterized in that, Including a vehicle body, a controller, and a camera and a display device provided in the vehicle body, and the controller is used to execute the method according to any one of claims 5-8.

11. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 5-8.

12. A computer program product, characterized in that, Including a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 5-8.

Images