Camera data stream acquisition method, camera data stream acquisition system and vehicle-mounted terminal

Abstract

The present application provides a camera data stream acquisition method, a camera data stream acquisition system and a vehicle-mounted terminal, wherein the method includes: obtaining the stream configuration parameters of the target camera on the vehicle, the stream configuration parameters include monitoring events and direct memory access addresses, and the monitoring events include frame ready events; constructing a stream link for the target camera according to the stream configuration parameters and the direct memory access interface; sending a stream instruction to the driver service, and controlling the target camera based on the stream link to write the collected data frames into the corresponding memory according to the direct memory access address, and feedback the frame ready event; when the frame ready event from the driver service is monitored, the data frame in the memory is read. By directly constructing the stream link through the stream configuration parameters and the direct memory access interface, the application as a client can directly read the data frames written by the camera from the shared memory between processes, reducing the overhead of data transmission and copying.

Description

Technical Field

[0001] The present application relates to the field of camera data stream acquisition technology, and in particular to a camera data stream acquisition method, a camera data stream acquisition system and a vehicle-mounted terminal. Background Art

[0002] With the development of smart car technology, the intelligence level of vehicle cockpits is increasing, and camera-based applications have become one of the indispensable functions. For example, panoramic images, reversing images, parking assistance, driving recorders, sentry mode, parking photography and other functions are widely used in vehicles, greatly improving driving safety and convenience. In the in-vehicle Android platform, Ais (Automotive Imaging System) consists of an Ais_server (driver service) process and multiple Ais_client (client) processes. Among them, the Ais_server process is responsible for interacting with the kernel to drive the camera, operate the hardware camera to open, configure, stream, release, etc., and communicate with multiple Ais_clients through socket ipc (multi-process communication mechanism) to receive Ais_client operation instructions.

[0003] At present, based on the camera framework of the in-vehicle Android platform, driver-related services exist in the vendor partition, and applications cannot access them directly. The methods for applications to collect camera data streams are generally based on the native camera interface of the Android system. To access the camera driver, the application needs to go through the Camera1 or Camera2 interface -> CameraServer (camera service) process -> Camera Provider (camera provider) -> Ais_Server process -> KernelDriver (camera driver) -> Camera (camera hardware), where Camera Provider is Ais_Client. CameraProvider belongs to the service of the vendor partition, so it is necessary to use the Camera Server process as middleware in the system partition. The application accesses the Camera Server through AIDL (Android Interface Definition Language), and the Camera Server accesses the Camera Provider through HIDL (HAL interface definition language, hardware abstraction layer interface definition language), and the Camera Provider sends instructions to Ais_Server.

[0004] The above method of obtaining camera data stream requires the support of the system's Camera Server process and Camera Provider process. In addition to the camera application layer and driver layer, the native camera interface link has two more intermediate component processes, Camera Provider and CameraServer. These two components have some framework layer adaptation logic, format conversion and other processing inside, which occupies system resources and increases business complexity. In addition, multi-level inter-process data transmission also brings additional overhead. When a large number of cameras are opened, more resources will be occupied, which is easy to cause system freezes. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present application discloses a camera data stream acquisition method, a camera data stream acquisition system and a vehicle-mounted terminal, which are used to solve the technical problem of complex camera data stream acquisition links in the prior art.

[0006] In a first aspect, the present application provides a camera data stream acquisition method, the method comprising: acquiring stream acquisition configuration parameters of a target camera on a vehicle, the stream acquisition configuration parameters comprising a monitoring event and a direct memory access address, the monitoring event comprising a frame ready event; constructing a stream acquisition link for the target camera according to the stream acquisition configuration parameters and the direct memory access interface; sending a stream acquisition instruction to a driving service, controlling the target camera based on the stream acquisition link to write the collected data frame into a corresponding memory according to the direct memory access address, and feeding back the frame ready event; when the frame ready event from the driving service is monitored, reading the data frame in the memory.

[0007] In one embodiment of the present application, the stream acquisition configuration parameters also include a callback function address, and the monitoring event also includes an input signal change event and an error event; the configuration of the stream acquisition configuration parameters includes: obtaining the camera handle of the target camera; calling a first parameter setting function based on the camera handle, setting the monitoring event, and defining a callback function for processing the monitoring event to obtain the callback function address; setting the monitoring event and the callback function address to the camera handle to complete the configuration of the stream acquisition configuration parameters.

[0008] In one embodiment of the present application, the configuration of the stream configuration parameters also includes: obtaining the camera handle of the target camera; calling the second parameter setting function based on the camera handle, creating a memory space corresponding to the target camera, and obtaining the direct memory access address, wherein the memory space is used to store the data frame; creating a structure and setting the direct memory access address to the structure; setting the structure to the camera handle to complete the configuration of the stream configuration parameters.

[0009] In one embodiment of the present application, setting the direct memory access address to the structure includes: setting the number of buffers for storing the data frame in the structure, wherein the number of buffers is at least one; dividing the buffer addresses corresponding to each buffer according to the number of buffers and the direct memory access address; and setting each of the buffer addresses to the structure.

[0010] In one embodiment of the present application, before setting the structure to the camera handle, it also includes: obtaining the frame rate, resolution, step size and image format of the target camera; configuring the number of planes of the image data represented by the data frame according to the image format; determining the size of the image data represented by the data frame according to the resolution and the step size, wherein the size includes width, height and total length of the data; and setting the frame rate, the number of planes, the step size and the size to the structure.

[0011] In one embodiment of the present application, the direct memory access address includes at least one buffer address; the target camera is controlled based on the stream acquisition link to write the collected data frames into the corresponding memory according to the direct memory access address, and the frame ready event is fed back, including: sending the stream acquisition instruction to the target camera, controlling the target camera to write the data frames into the buffer corresponding to the buffer address in sequence; assigning a serial number to each data frame according to the data frame acquisition order, and monitoring the completion status of each data frame; if the target frame corresponding to the first serial number monitored is in a completed state, an index is created based on the target buffer address where the target frame is located; the index is set to a callback function, and the callback function is executed to complete the feedback of the frame ready event.

[0012] In one embodiment of the present application, reading the data frame in the memory includes: obtaining the index in response to the frame ready event; accessing the target buffer corresponding to the target buffer address according to the index, and reading the target frame in the target buffer.

[0013] In one embodiment of the present application, after reading the data frame in the memory, it also includes: sending a stop flow instruction to the driving service so that the driving service controls the target camera to stop the writing operation of the data frame; releasing the memory and closing the flow link.

[0014] In the second aspect, the present application provides a camera data stream acquisition system, which includes: an acquisition module, used to acquire the stream configuration parameters of a target camera on a vehicle, the stream configuration parameters including a monitoring event and a direct memory access address, and the monitoring event including a frame ready event; a construction module, used to build a stream link for the target camera according to the stream configuration parameters and the direct memory access interface; a sending module, used to send a stream instruction to a driving service, and based on the stream link, control the target camera to write the collected data frame into a corresponding memory according to the direct memory access address, and feedback the frame ready event; a reading module, used to read the data frame in the memory when the frame ready event from the driving service is monitored.

[0015] In a third aspect, the present application provides a vehicle-mounted terminal, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the vehicle-mounted terminal implements the camera data stream acquisition method described in the first aspect.

[0016] As described above, the camera data stream acquisition method, camera data stream acquisition system and vehicle-mounted terminal provided by the embodiments of the present application have the following beneficial effects:

[0017] By obtaining the stream configuration parameters of the target camera on the vehicle, the stream configuration parameters include monitoring events and direct memory access addresses, and the monitoring events include frame ready events. Then, a stream link is constructed for the target camera according to the stream configuration parameters and the direct memory access interface. When the camera data stream needs to be obtained, a stream instruction is sent to the driver service. The driver service controls the target camera based on the stream link to write the collected data frames into the corresponding memory according to the direct memory access address. When the frame ready event from the driver service is monitored, the data frame in the memory is read. In the camera framework, the camera service and the camera provider are bypassed. The application is used as a client to communicate directly with the driver service to obtain the stream. According to the characteristic that the client can access the shared memory between processes, the memory with direct memory access is used as the camera data stream carrier. The stream link is directly constructed through the stream configuration parameters and the direct memory access interface, so that the application as a client can directly read the data frame written by the camera from the shared memory between processes according to the stream link, thereby reducing the complexity of the application stream acquisition business, avoiding the overhead of data transmission and copying between multiple levels and the conversion of image formats, and improving the system performance.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 is a schematic diagram of an implementation environment of a camera data stream acquisition system shown in an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a camera data stream acquisition method shown in an exemplary embodiment of the present application;

[0022] Figure 3 This is a flow chart of a flow pre-processing work shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a flowchart of reading a data frame shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is a flow chart of a post-flow work shown in an exemplary embodiment of the present application;

[0025] Figure 6 is a block diagram of a camera data stream acquisition system shown in an exemplary embodiment of the present application;

[0026] Figure 7 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0028] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0029] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0030] In the in-vehicle Android platform, the methods for applications to collect camera data streams are generally based on the native camera interface implementation of the Android system. To access the camera driver, the application needs to go through the Camera1 or Camera2 interface -> Camera Server process -> Camera Provider -> Ais_Server process -> Kernel Driver -> Camera. The application accesses the Camera Server through AIDL, the Camera Server accesses the CameraProvider through HIDL, and the Camera Provider then issues instructions to the Ais_Server. However, the inventors of this application have found that the above-mentioned method of obtaining the camera data stream requires the support of the system's Camera Server process and Camera Provider process. In addition to the camera application layer and driver layer, the native camera interface link has two more intermediate component processes, Camera Provider and Camera Server. These two components will have some framework layer adaptation logic, format conversion and other processing, which occupies system resources and increases business complexity. In addition, the multi-level inter-process data transmission also brings additional overhead. When the number of cameras opened is large, more resources will be occupied, which is easy to cause system jams.

[0031] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a camera data stream acquisition system shown in an exemplary embodiment of the present application. Figure 1As shown, the implementation environment includes a vehicle 110 and a camera data stream acquisition system 120, wherein the camera data stream acquisition system 120 is embedded in the vehicle 110, and is used to realize the acquisition of the camera data stream by the application in the vehicle 110. The camera data stream acquisition system 120 includes but is not limited to the vehicle system, the vehicle-mounted computer, etc. In the camera framework, by bypassing the two components of the camera service and the camera provider, the application is used as a client to directly communicate with the driver service to obtain the stream, and then according to the characteristic that the client can access the shared memory between processes, the memory with direct memory access is used as the camera data stream carrier, and the stream acquisition link is directly constructed through the stream acquisition configuration parameters and the direct memory access interface, so that the application as a client can directly read the data frame written by the camera from the shared memory between processes according to the stream acquisition link, thereby reducing the complexity of the application stream acquisition business, avoiding the overhead of data transmission and copying between multiple levels and the conversion of image formats, and improving the system performance.

[0032] See also Figure 2 , Figure 2 is a flowchart of a camera data stream acquisition method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the method may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0033] like Figure 2 As shown, in an exemplary embodiment, the camera data stream acquisition method includes at least steps S210 to S240, which are described in detail as follows:

[0034] Step S210, obtaining the stream configuration parameters of the target camera on the vehicle, the stream configuration parameters including the monitoring event and the direct memory access address, the monitoring event including the frame ready event.

[0035] The target camera refers to a specific camera that the application is using or plans to use, and is the data source of the data stream that the application wants to obtain; the stream configuration parameters are the parameters configured by the application for the target camera for obtaining the data stream.

[0036] In this embodiment, the stream configuration parameters also include a callback function address, and the monitoring events include QCARCAM_EVENT_FRAME_READY (frame ready event) as well as QCARCAM_EVENT_INPUT_SIGNAL (input signal change event) and QCARCAM_EVENT_ERROR (error event).

[0037] It should also be noted that the application, as a client, communicates directly with the driver service through the camera framework interface. The driver service can control the hardware camera's opening, configuration, streaming, and release services based on the instructions sent by the application. Before the application obtains the streaming configuration parameters, the camera framework interface is initialized first, and the communication resources between the application and the driver service are configured. After the communication resources between the application and the driver service are configured, the application can call the camera framework interface to obtain the camera's specification information through a multi-process communication mechanism. The specification information includes but is not limited to the camera ID, frame rate, resolution, step size, and image format. It should be noted that the application can obtain the specification information of all cameras on the vehicle from the driver service by calling the camera framework interface, and match the specification information belonging to the target camera from all the obtained specification information based on the camera ID of the target camera.

[0038] In one embodiment, the configuration of the stream configuration parameters includes: obtaining the camera handle of the target camera; calling the first parameter setting function based on the camera handle, setting a monitoring event, and defining a callback function for processing the monitoring event, and obtaining the callback function address; setting the monitoring event and the callback function address to the camera handle to complete the configuration of the stream configuration parameters.

[0039] In this embodiment, the application opens the corresponding camera channel according to the camera identifier of the target camera. Each camera corresponds to a unique camera channel. The camera handle of the target camera is obtained based on the camera channel to configure the target camera according to the camera handle of the target camera.

[0040] In this embodiment, the first parameter setting function corresponding to the target camera can be called through the camera handle, the listening event and the callback function address can be set, and the listening event and the callback function address can be set in the camera handle to notify the listening event and the callback function address to the driver service. In this way, after the set listening event occurs, the driver service can execute the callback function through the callback function address to provide feedback to the application.

[0041] It should also be noted that the frame ready event refers to the event triggered when the data frame is written and can be read by the application; the input signal change event refers to the event triggered when the signal or state related to the data stream changes; the error event refers to the event triggered when hardware failure, network problems, etc. may cause data flow interruption or data corruption. After the set monitoring event occurs, the driver service executes the callback function based on the callback function address to feedback to the application.

[0042] In one embodiment, the configuration of the stream configuration parameters also includes: obtaining a camera handle of a target camera; calling a second parameter setting function based on the camera handle, creating a memory space corresponding to the target camera, and obtaining a direct memory access address, wherein the memory space is used to store data frames; creating a structure, and setting the direct memory access address to the structure; setting the structure to the camera handle to complete the configuration of the stream configuration parameters.

[0043] In this embodiment, the application opens a corresponding camera channel according to the camera identifier of the target camera, and obtains the camera handle of the target camera based on the camera channel.

[0044] In this embodiment, the second parameter setting function corresponding to the target camera can be called through the camera handle to set the memory handle corresponding to the current stream fetching operation. Specifically, the second parameter setting function is called to create a data stream storage space corresponding to the target camera in the shared memory between processes, that is, DMA (Direct Memory Access) memory, and the DMA memory address corresponding to the target camera is described in the form of a constructed structure and set to the camera handle, so as to notify the driver service of the DMA memory address. In this way, the driver service can control the target camera to write the data frame into the memory corresponding to the DMA memory address so that the application can read it directly.

[0045] In one embodiment, setting a direct memory access address to a structure includes: setting the number of buffers for storing data frames in the structure, wherein the number of buffers is at least one; dividing a buffer address corresponding to each buffer according to the number of buffers and the direct memory access address; and setting each buffer address to the structure.

[0046] In this embodiment, considering the problem that after the data frame is written, the application fails to read the data frame in time, causing the current data frame to be overwritten by the next data frame, the DMA memory corresponding to the target camera is divided into multiple buffers, each buffer corresponds to its own buffer address, so that the previous data frame will not be discarded if it is not read, ensuring that the application can read a continuous and complete camera data stream.

[0047] In one embodiment, before setting the structure to the camera handle, it also includes: obtaining the frame rate, resolution, step size and image format of the target camera; configuring the number of planes of the image data represented by the data frame according to the image format; determining the size of the image data represented by the data frame according to the resolution and step size, wherein the size includes width, height and total length of the data; and setting the frame rate, number of planes, step size and size to the structure.

[0048] In this embodiment, the frame rate of the target camera, the number of planes representing the image data of the acquired data frame, the step size and the size of the image data are set to a structure, and then the structure is set to the camera handle. In this way, the driver service can control the target camera to write the data frame according to the format information in the memory handle.

[0049] See also Figure 3 , Figure 3 FIG. 1 is a flow chart showing a flow pre-processing operation according to an exemplary embodiment of the present application. Figure 3 As shown in the figure, the pre-streaming work is described as follows: first, the camera framework interface is initialized, that is, the application calls the camera framework interface, forwards the initialization instruction to the driver service through the multi-process communication mechanism, configures the communication resources between the application and the driver service, and then the driver service feedbacks the initialization status, that is, the initialization success or failure; after the initialization is successful, the application can call the camera framework interface, send the acquisition instruction to the driver service through the multi-process communication mechanism, query the specification information of all cameras, and then the driver service feedbacks the specification information of the camera. After obtaining the specification information of all cameras, the application can match the specification information corresponding to the target camera based on the camera identifier of the target camera; after the specification information is queried, the application calls the camera framework interface, passes in the camera identifier of the target camera, and passes in the multi-process communication mechanism. The multi-process communication mechanism forwards the camera opening instruction to the driver service to open the camera channel corresponding to the target camera, and then the driver service feedbacks the opening status, that is, the opening is successful or abnormal; if the camera channel is opened normally, the application sets the stream configuration parameters, that is, sets the monitoring event, sets the callback function, sets the DMA memory access address and information for cross-process data transmission, and forwards the setting instruction to the driver service through the multi-process communication mechanism. The driver service and the target camera are configured according to the stream configuration parameters, and then the driver service feedbacks the setting status, whether the setting is successful or failed; after the setting is successful, the application calls the camera framework interface, and forwards the stream instruction to the driver service through the multi-process communication mechanism, notifying the target camera to start uploading the data stream, and then the driver service can feedback the incoming result of the data stream, whether it is successful or failed.

[0050] In the Qualcomm Automotive Android Platform Camera Framework, the camera framework interface may be libais_hidl_client (Qualcomm Automotive Android Platform Camera Framework Interface). In this camera framework, the libais_hidl_client under the system partition allows the application to directly connect to the driver service. The application calls qcarcam_initialize (initialization function) through libais_hidl_client, communicates with the driver service through socket ipc (multi-process communication mechanism), and configures the communication resources between the application and the driver service; after successful initialization, the application calls qcarcam_query_inputs (input original information query function) through libais_hidl_client, communicates with the driver service through socket ipc, and queries the specification information of all cameras; after obtaining the specification information of the target camera, the application calls qcarcam_open (open function) through libais_hidl_client, communicates with the driver service through socket ipc, and opens the camera channel corresponding to the target camera; after the camera channel is opened normally, the application calls qcarcam_s_param (parameter configuration function) through libais_hidl_client, sets the monitoring event and callback function, communicates with the driver service through socket ipc, sets the monitoring event and callback function to the driver service, and can also call qcarcam_s_buffers (parameter configuration function) through libais_hidl_client to set the DMA memory access address and information, and communicates with the driver service through socket ipc. ipc communicates with the driver service to set the DMA memory access address and information to the target camera; after the stream configuration parameters are configured, the application calls qcarcam_start (stream function) through libais_hidl_client, communicates with the driver service through socket ipc, and notifies the target camera to start uploading the data stream.

[0051] Step S220: construct a stream acquisition link for the target camera according to the stream acquisition configuration parameters and the direct memory access interface.

[0052] Specifically, the stream acquisition link is that the target camera directly writes the collected data frames into the DMA memory, and the application directly reads the data frames from the DMA memory based on the direct memory access interface.

[0053] Step S230, sending a stream acquisition instruction to the driver service, controlling the target camera based on the stream acquisition link to write the collected data frame into the corresponding memory according to the direct memory access address, and feeding back a frame ready event.

[0054] Please continue to see Figure 3After the stream configuration parameters are given to the driver service, the driver service and the target camera are set according to the stream configuration parameters. After the setting is successful, the application sends a stream instruction to the driver service to control the target camera to start uploading the data stream, that is, writing the collected data frames into the DMA memory according to the set DMA memory address and information.

[0055] In one embodiment, the direct memory access address includes at least one buffer address; based on the stream acquisition link, the target camera is controlled to write the collected data frames into the corresponding memory according to the direct memory access address, and a frame ready event is fed back, including: sending a stream acquisition instruction to the target camera, controlling the target camera to write the data frames into the buffer corresponding to the buffer address in sequence; assigning a serial number to each data frame according to the data frame acquisition order, and monitoring the completion status of each data frame; if the target frame corresponding to the first serial number monitored is in a completed state, an index is created based on the target buffer address where the target frame is located; the index is set to the callback function, and the callback function is executed to complete the feedback of the frame ready event.

[0056] In this embodiment, the driver service controls the target camera to write the data frames into the buffer corresponding to the buffer address in sequence, and assigns a serial number to each data frame according to the data frame acquisition order. The serial number can be a digital number or a timestamp, etc., to represent the order of the data frames acquired by the target camera. During the data frame writing process, the driver service monitors the completion status of each data frame according to the serial number. When the target frame corresponding to the first serial number is monitored to be in the completion state, the target buffer address where the target frame is located is created as an index, and the index is set to the callback function to complete the feedback of the frame ready event, and notify the application that it can read the target frame under the index, so that the application can obtain a continuous and complete camera data stream, ensuring the reliability of the camera data stream acquisition.

[0057] Step S240: when a frame ready event from the driver service is monitored, the data frame in the memory is read.

[0058] After the application issues a stream fetch instruction, it enters the frame ready event listening state. After receiving the frame ready event fed back by the driver service, it can call the DMA memory interface and read the data frame from the DMA memory based on the DMA memory address.

[0059] The entire process of an application as a client obtaining the camera data stream is as follows: the application calls the camera framework interface to send a stream acquisition instruction to the driver service and listens for the frame ready event. After receiving the stream acquisition instruction, the driver service controls the target camera to write the data frame. The target camera writes the data frame to the DMA memory corresponding to the DMA memory address. During the writing process, the driver service monitors the writing status of the data frame. When the writing of the data frame is completed, the frame ready event is fed back through the callback function address. After listening to the frame ready event from the driver service, the application directly reads the data frame from the DMA memory corresponding to the DMA memory address.

[0060] It should also be noted that, for example, if the frame rate of the target camera is 25fps, the driver service will report 25 frame ready events within 1 second.

[0061] In addition, after the application obtains the data frame, it can consume the data frame, that is, perform image recognition, display or encoding and other processing.

[0062] In one embodiment, reading a data frame in a memory includes: obtaining an index in response to a frame ready event; accessing a target buffer corresponding to a target buffer address according to the index, and reading a target frame in the target buffer.

[0063] In this embodiment, since multiple memory buffers are set in the DMA memory, it is necessary to obtain an index based on the frame ready event feedback from the driver service, and access the target buffer address corresponding to the target frame through the index to realize the reading of the target frame, thereby avoiding the situation where the application reads other data frames that have not been written or data frames that are discontinuous with the previous data frame.

[0064] See also Figure 4 , Figure 4 FIG. 1 is a flowchart of reading a data frame according to an exemplary embodiment of the present application. Figure 4 As shown, the driver service reports the frame ready event to the application through the camera framework interface. The application obtains the reported data frame information based on the camera framework interface, that is, the memory address corresponding to the target frame to be obtained, and copies the image data from the address, that is, reads the data frame in the memory. After reading the data frame, the memory status is reset, and the completion status of the data frame writing is reset to receive subsequent image data.

[0065] In the Qualcomm car Android platform camera framework, the driver service reports the frame ready event to the application through libais_hidl_client. Based on libais_hidl_client, the application calls qcarcam_get_frame (data frame information acquisition function) to obtain the memory address corresponding to the reported target frame, and calls memcpy (memory copy function) to read the data frame from the address. After reading, it calls qcarcam_release_frame (memory release function) to reset the memory status.

[0066] In one embodiment, after reading the data frame in the memory, it also includes: sending a stop instruction for obtaining the stream to the driving service, so that the driving service controls the target camera to stop the writing operation of the data frame; releasing the memory and closing the stream obtaining link.

[0067] In this embodiment, when the application does not need the target camera to continue uploading data frames, it sends a stop instruction to the driver service to control the target camera to stop writing data frames, release memory and close the streaming link. In this way, unnecessary memory usage is reduced and memory leaks are avoided, thereby ensuring system performance.

[0068] See also Figure 5 , Figure 5 FIG. 1 is a flowchart of a post-processing operation of a stream acquisition according to an exemplary embodiment of the present application. Figure 5 As shown, the application calls the camera framework interface and sends a stop command to the driver service through the multi-process communication mechanism to notify the target camera to stop reporting the data stream. The driver service then feeds back the operation result, that is, whether the target camera stops writing data frames. Then the application calls the camera framework interface and sends a camera close command to the driver service through the multi-process communication mechanism to control the camera channel to close. Then the operation result is fed back, that is, whether the camera channel is closed. Then the application calls the camera framework interface and sends a release command to the driver service through the multi-process communication mechanism to release DMA memory, camera framework interface, communication resources, and disconnect the stream acquisition link.

[0069] In the Qualcomm car Android platform camera framework, the application calls qcarcam_stop (stop streaming function) through libais_hidl_client, communicates with the driver service through socket ipc, and notifies the target camera to stop reporting the data stream; then the application calls qcarcam_close (camera closing function) through libais_hidl_client, communicates with the driver service through socket ipc, and controls the camera channel to close; then the application calls qcarcam_uninitialize (release function) through libais_hidl_client, communicates with the driver service through socket ipc, releases DMA memory, camera framework interface, communication resources, disconnects the streaming link, etc.

[0070] The above-mentioned camera data stream acquisition method obtains the stream configuration parameters of the target camera on the vehicle, and the stream configuration parameters include monitoring events and direct memory access addresses, and the monitoring events include frame ready events. Then, a stream acquisition link is constructed for the target camera according to the stream configuration parameters and the direct memory access interface. When the camera data stream needs to be acquired, a stream acquisition instruction is sent to the driver service. The driver service controls the target camera based on the stream acquisition link to write the collected data frame into the corresponding memory according to the direct memory access address. When the frame ready event from the driver service is monitored, the data frame in the memory is read. In the camera framework, the two components of the camera service and the camera provider are bypassed, and the application is used as a client to communicate directly with the driver service to acquire the stream. Then, according to the characteristic that the client can access the shared memory between processes, the memory with direct memory access is used as the camera data stream carrier. The stream acquisition link is directly constructed through the stream configuration parameters and the direct memory access interface, so that the application as a client can directly read the data frame written by the camera from the shared memory between processes according to the stream acquisition link, thereby reducing the complexity of the application stream acquisition business, avoiding the overhead of data transmission and copying between multiple levels and the conversion of image formats, and improving the system performance.

[0071] It should be noted that in addition to Qualcomm's in-vehicle Android platform, common in-vehicle platforms in vehicles include: NVIDIADRIVE (a platform launched by VIDIA specifically for the development of autonomous vehicles), Intel AutomotiveSolutions (based on Intel's in-vehicle solutions), Linux in-vehicle platform, QNX (a real-time operating system developed by BlackBerry), AUTOSAR (a standardized platform developed in global cooperation), Apple CarPlay (a car system released by Apple), Android Auto (an Android platform designed specifically for cars launched by Google), and Tesla's own platform, etc. In the embodiment of the present application, the camera framework interface can be the camera framework interface of the above-mentioned in-vehicle platform, and the camera data stream acquisition method can be applicable to the above-mentioned various in-vehicle platforms.

[0072] In vehicle terminals, such as panoramic images, driving recorders and other camera applications, applications can directly obtain data frames through the above-mentioned camera data stream acquisition method for fusion display, encoding, etc. For example, when a 4-channel 1280x800, 25fps camera uses the traditional Camera2 / Camera1 interface, its CPU (Central Processing Unit) occupancy is as follows: Camera Provider process: 29%, Camera Server process: 10%, App process: 16%, the total is 55%; in the embodiment of the present application, its CPU occupancy only involves the App process, which is about 10%, and only accounts for about 20% of the traditional Camera interface, and the performance is significantly improved.

[0073] See also Figure 6 , Figure 6 is a block diagram of a camera data stream acquisition system shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the system may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the system is applicable.

[0074] like Figure 6 As shown, in an exemplary embodiment, the camera data stream acquisition system 600 includes at least an acquisition module 610, a construction module 620, a sending module 630 and a reading module 640, which are described in detail as follows:

[0075] The acquisition module 610 is used to acquire the stream configuration parameters of the target camera on the vehicle, the stream configuration parameters include monitoring events and direct memory access addresses, and the monitoring events include frame ready events;

[0076] A construction module 620 is used to construct a stream acquisition link for a target camera according to the stream acquisition configuration parameters and the direct memory access interface;

[0077] The sending module 630 is used to send a stream acquisition instruction to the driving service, control the target camera based on the stream acquisition link to write the collected data frame into the corresponding memory according to the direct memory access address, and feedback the frame ready event;

[0078] The reading module 640 is used to read the data frame in the memory when monitoring the frame ready event from the driver service.

[0079] It should be noted that the camera data stream acquisition system provided in the above embodiment and the camera data stream acquisition method provided in the above embodiment belong to the same concept, wherein the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0080] See also Figure 7 , Figure 7 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. Figure 7 The structure diagram of the computer system of the vehicle terminal suitable for implementing the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system 700 of the vehicle-mounted terminal shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0081] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 703. The CPU 701, the ROM 702 and the RAM 703 are connected to each other through the bus 704. The I / O (Input / Output) interface 705 is also connected to the bus 704.

[0082] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0084] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0086] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A camera data stream acquisition method, characterized in that: The method comprises: Acquire a stream configuration parameter of a target camera on a vehicle, wherein the stream configuration parameter includes a monitoring event and a direct memory access address, wherein the monitoring event includes a frame ready event, and the direct memory access address includes at least one buffer address; Building a stream acquisition link for the target camera according to the stream acquisition configuration parameters and the direct memory access interface; Sending a stream acquisition instruction to the driving service, controlling the target camera based on the stream acquisition link to write the collected data frame into the corresponding memory according to the direct memory access address, and feeding back the frame ready event, wherein the sending of the stream acquisition instruction is realized by calling the camera framework interface to directly send it to the driving service; When the frame ready event from the driver service is monitored, reading the data frame in the memory; The controlling the target camera based on the stream acquisition link to write the collected data frame into the corresponding memory according to the direct memory access address and feeding back the frame ready event includes: Sending the stream acquisition instruction to the target camera, controlling the target camera to write the data frames into the buffer corresponding to the buffer address in sequence; Assign a sequence number to each data frame according to the data frame acquisition order, and monitor the completion status of each data frame; If the target frame corresponding to the first sequence number monitored is in the completed state, an index is created based on the target buffer address where the target frame is located; The index is set to the callback function, and the callback function is executed to complete the feedback of the frame ready event.

2. The camera data stream acquisition method according to claim 1, characterized in that: The stream acquisition configuration parameters also include a callback function address, and the monitoring events also include an input signal change event and an error event; The configuration of the flow configuration parameters includes: Get the camera handle of the target camera; Based on the camera handle, a first parameter setting function is called to set the monitoring event, and a callback function for processing the monitoring event is defined to obtain the callback function address; The monitoring event and the callback function address are set to the camera handle to complete the configuration of the stream acquisition configuration parameters.

3. The camera data stream acquisition method according to claim 1, characterized in that: The configuration of the flow configuration parameters also includes: Get the camera handle of the target camera; Calling a second parameter setting function based on the camera handle, creating a memory space corresponding to the target camera, and obtaining the direct memory access address, wherein the memory space is used to store the data frame; Creating a structure, and setting the direct memory access address to the structure; The structure is set to the camera handle to complete the configuration of the stream configuration parameters.

4. The camera data stream acquisition method according to claim 3, characterized in that: The step of setting the direct memory access address to the structure includes: The number of buffers for storing the data frame is set in the structure, wherein the number of buffers is at least one; According to the number of buffers and the direct memory access address, a buffer address corresponding to each buffer is divided; Set each of the buffer addresses to the structure.

5. The camera data stream acquisition method according to claim 3, characterized in that: Before setting the structure to the camera handle, the method further includes: Obtain the frame rate, resolution, step size and image format of the target camera; Configuring the number of planes of image data represented by the data frame according to the image format; Determine the size of the image data represented by the data frame according to the resolution and the step size, wherein the size includes width, height and total length of the data; Set the frame rate, the number of planes, the step size, and the size to the structure.

6. The camera data stream acquisition method according to claim 1, characterized in that: The reading of the data frame in the memory includes: In response to the frame ready event, obtaining the index; The target buffer corresponding to the target buffer address is accessed according to the index, and the target frame in the target buffer is read.

7. The camera data stream acquisition method according to any one of claims 1 to 6, characterized in that: After reading the data frame in the memory, the method further includes: Sending a stop instruction of fetching stream to the driving service, so that the driving service controls the target camera to stop the writing operation of the data frame; The memory is released and the stream fetching link is closed.

8. A camera data stream acquisition system, characterized in that: The system comprises: An acquisition module, used for acquiring a stream configuration parameter of a target camera on a vehicle, wherein the stream configuration parameter includes a monitoring event and a direct memory access address, wherein the monitoring event includes a frame ready event, and the direct memory access address includes at least one buffer address; A construction module, used for constructing a stream acquisition link for the target camera according to the stream acquisition configuration parameters and a direct memory access interface; A sending module, used for sending a stream acquisition instruction to a driving service, controlling the target camera based on the stream acquisition link to write the collected data frame into a corresponding memory according to the direct memory access address, and feeding back the frame ready event, wherein the sending of the stream acquisition instruction is realized by directly sending it to the driving service by calling a camera framework interface; A reading module, configured to read the data frame in the memory when monitoring the frame ready event from the driver service; The sending module is specifically used to: send the stream acquisition instruction to the target camera, control the target camera to write the data frames into the buffer corresponding to the buffer address in sequence; assign a serial number to each data frame according to the data frame acquisition order, and monitor the completion status of each data frame; if the target frame corresponding to the first serial number monitored is in the completion status, create an index based on the target buffer address where the target frame is located; set the index to the callback function, and execute the callback function to complete the feedback of the frame ready event.

9. A vehicle-mounted terminal, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the vehicle-mounted terminal to implement the camera data stream acquisition method as described in any one of claims 1 to 7.

Images