Image data processing method, system and equipment, medium and vehicle

Abstract

The invention relates to the field of vehicle-mounted cameras, in particular to an image data processing method, system and device, a medium and a vehicle. According to the embodiment of the invention, the image processing task which is sent by the visual application and contains the data type and the task requirement is received, and the related target data is accurately acquired from the shared continuous physical memory which stores the image data and the video stream parameters acquired by the plurality of different camera devices connected with the vehicle-mounted system by using the data type; performance consumption caused by unnecessary data search, transmission and the like is avoided, then algorithm resources corresponding to task requirements are called to perform targeted processing on target data, a processing result is efficiently obtained, finally the processing result is transmitted to a visual application, the whole process is subjected to accurate data acquisition and adaptive algorithm processing, and the processing efficiency is improved. And the performance loss caused by redundant operation is reduced.

Description

Technical Field

[0001] The present invention relates to a field, and in particular to a method, system, device, medium and vehicle for processing image data. Background Art

[0002] In the context of the continuous development of automotive technology, in-vehicle DMS (Driver Monitoring System) and IMS (Interior Monitoring System) as in-vehicle camera-related technologies have attracted much attention. They use sensors, cameras and related algorithms to monitor the driver's status and the internal conditions of the vehicle to improve driving safety, passenger protection and comfort. Among them, DMS relies on cameras and deep learning algorithms to monitor the driver's posture, eyes and facial expressions to determine their concentration, fatigue or distraction, and can cooperate with safety systems such as automatic emergency braking to ensure safety; IMS uses cameras and image processing algorithms to pay attention to the passengers, seat usage and dangerous items in the car, and can also realize photo and video recording functions. These technologies rely on advanced image processing, computer vision and machine learning algorithms to analyze images or video streams to extract information and make decisions. At the same time, the vehicle needs to be equipped with appropriate cameras, sensors and computing resources and integrate corresponding algorithms.

[0003] However, in the development of in-vehicle DMS and IMS cameras based on the Android system, the current high-level framework (Android camerav2 architecture) for controlling camera devices in the Android system, the complexity and advanced functions of the Android camera application programming interface (Android camera v2) increase device power consumption and generate high CPU, memory and memory bandwidth performance losses. The EVS architecture of the Android Automotive system focuses on the fast startup and low latency of the car's exterior view system (such as reversing images and 360 panoramic images), requiring developers to build their own input management and image drawing, and cannot be directly used in the development of in-vehicle DMS and IMS cameras to solve the performance problems of the existing architecture. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, system, device, medium and vehicle for processing image data to solve the problem that in the development of vehicle-mounted DMS and IMS cameras based on the Android system, the currently used Android camera application programming interface has performance loss, and the EVS architecture of the Android vehicle-mounted system cannot directly solve this performance problem due to its application focus and the need for developers to build some functions themselves.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing image data, the method comprising:

[0006] Receiving an image processing task sent by a visual application, wherein the image processing task includes a data type and a task requirement;

[0007] Acquire target data related to the image processing task from a shared continuous physical memory using the data type, wherein the shared continuous physical memory includes image data and video stream parameters collected by a plurality of different camera devices connected to the vehicle-mounted system;

[0008] The algorithm resources corresponding to the task requirements are called to process the target data, obtain processing results, and transmit the processing results to the visual application.

[0009] Furthermore, the method further comprises:

[0010] Acquiring raw image data collected by different camera devices, wherein the different camera devices are connected to the vehicle-mounted system;

[0011] Processing the original image data according to a pre-configured processing strategy to obtain target image data;

[0012] The video stream parameters corresponding to the target image data are read, and the target image data and the video stream parameters are stored in a shared continuous physical memory.

[0013] Furthermore, the processing of the original image data according to a pre-configured processing strategy to obtain target image data includes:

[0014] Obtaining the processing requirements of image usage scenarios or visual applications through the data processing layer;

[0015] Acquire a first processing strategy corresponding to the image usage scenario or a second processing strategy corresponding to the processing requirement of the visual application;

[0016] The original image data is processed according to the first processing strategy or the second processing strategy to obtain target image data.

[0017] Furthermore, the obtaining target data related to the image processing task from the shared continuous physical memory using the data type includes:

[0018] If the image processing task is an image rendering task, then reading video stream parameters from a shared continuous physical memory according to the data type carried by the image rendering task, and using the video stream parameters as the target data;

[0019] If the image processing task is an image reasoning task, the target image data is read from the shared continuous physical memory according to the data type carried by the image rendering task, and the target image data is used as the target data.

[0020] Further, the calling of the algorithm resources corresponding to the task requirement to process the target data to obtain a processing result includes:

[0021] If the task requirement is a rendering requirement, then obtaining the interface object delivered by the visual application, and extracting color mode parameters, resolution, and frame rate from the video stream parameters;

[0022] Reading original color data of the interface object, and processing the original color data according to the color mode parameters to obtain a processed interface object;

[0023] Calculate the rendering size of the interface object according to the resolution parameter and the original size data of the interface object, and calculate the rendering time corresponding to the interface object according to the frame rate;

[0024] The rendering algorithm is called to render the processed interface object according to the rendering time and the rendering size to obtain the visual interface, and the visual interface is used as the processing result.

[0025] Further, the calling of the algorithm resources corresponding to the task requirement to process the target data to obtain a processing result includes:

[0026] If the task requirement is a reasoning requirement, extracting driver monitoring image data and in-vehicle image data from the target image data;

[0027] Configure the image reasoning algorithm according to the reasoning requirements to obtain the configured image reasoning algorithm;

[0028] The configured image inference algorithm is called to infer the driver monitoring image data and the in-vehicle image data to obtain an inference result, and the inference result is used as the processing result.

[0029] Further, taking the inference result as the processing result includes:

[0030] Get the performance metrics of the configured image inference algorithm;

[0031] Detecting the quality of the driver monitoring image data and the in-vehicle image data;

[0032] Determining a credibility score of the inference result based on the performance indicator and the quality condition;

[0033] If the credibility score is higher than a preset score, the inference result is used as the processing result.

[0034] Furthermore, the transmitting the processing result to the visual application includes:

[0035] Obtaining a preset transmission rule of the cross-process communication interface, and encapsulating the inference result according to the preset transmission rule to obtain an encapsulated inference result;

[0036] Query the target process where the visual application that has registered callbacks and is waiting to receive data is located;

[0037] The encapsulated reasoning result is transmitted to the target process through the cross-process communication interface.

[0038] In a second aspect, an embodiment of the present invention provides an image data processing system, the system comprising an image display service layer and a visual application layer, wherein the visual application layer comprises at least one visual application;

[0039] The image display service layer is used to receive an image processing task sent by any visual application in the visual application layer, wherein the image processing task includes a data type and a task requirement; obtain target data related to the image processing task from a shared continuous physical memory using the data type, wherein the shared continuous physical memory includes image data and video stream parameters collected by multiple different camera devices connected to the vehicle-mounted system; call the algorithm resources corresponding to the task requirement to process the target data, obtain a processing result, and feed the processing result back to the visual application layer;

[0040] The visual application is used to display the visualization interface carried by the processing result.

[0041] Furthermore, the system also includes a driver layer and a data processing layer;

[0042] The driver layer is used to obtain raw image data collected by different camera devices and transmit the raw image data to the data processing layer of the vehicle-mounted system, wherein the different camera devices are connected to the vehicle-mounted system;

[0043] The data processing layer is used to process the original image data according to a pre-configured processing strategy to obtain target image data, read the video stream parameters corresponding to the target image data, and store the target image data and the video stream parameters in a shared continuous physical memory.

[0044] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0046] In a fifth aspect, an embodiment of the present invention provides a vehicle, comprising: an image data processing system and a controller; wherein the controller stores instructions for controlling the operation of the image data processing system, and the image data processing system is used to execute the above-mentioned image data processing method according to the instructions of the controller.

[0047] The embodiment of the present application receives an image processing task containing a data type and task requirements sent by a visual application, and uses the data type to accurately obtain relevant target data from a shared continuous physical memory that stores image data and video stream parameters collected by multiple different camera devices connected to the vehicle-mounted system, thereby avoiding performance consumption caused by unnecessary data search and transmission, and then calls the algorithm resources corresponding to the task requirements to perform targeted processing on the target data, efficiently obtain the processing results, and finally pass them to the visual application. The entire process reduces performance loss caused by redundant operations through precise data acquisition and adaptive algorithm processing.

[0048] The embodiment of the present application can fully integrate multi-source image information by acquiring raw image data collected by different camera devices connected to the vehicle-mounted system, providing rich materials for subsequent diversified processing needs; the raw image data is processed according to a pre-configured processing strategy to obtain target image data, wherein the processing requirements of the image usage scenario or visual application are first obtained from the data processing layer, and then the first or second processing strategy is correspondingly obtained and processed accordingly, which can ensure that the processing process is highly consistent with the actual application scenario and specific needs, so that the processed target image data is more targeted, and can accurately meet different functional requirements such as driver monitoring and in-vehicle environment viewing, avoid invalid or redundant processing operations, and improve processing efficiency; and reading the video stream parameters corresponding to the target image data, and storing the target image data and these parameters in a shared continuous physical memory, not only facilitates the subsequent rapid and accurate call of data to carry out further operations such as image rendering and reasoning, but also optimizes data management, realizes efficient storage and sharing of data, reduces resource consumption caused by repeated acquisition of data, and overall improves the accuracy, efficiency and rationality of resource utilization of the vehicle-mounted system in image data processing.

[0049] The embodiment of the present application reads the original color data of the interface object and processes it according to the color mode parameters, which can optimize the color performance, make the final visual effect more in line with expectations and make the color more accurate and beautiful; calculate the rendering size according to the resolution parameter and the original size data, and calculate the rendering time according to the frame rate, which can ensure that the interface object is accurately adapted to the screen and other display areas under different display requirements, while ensuring that the picture rendering rhythm is reasonable and smooth, avoiding problems such as freezes or inharmonious pictures; call the rendering algorithm to render the interface object after processing according to the above time and size to obtain a visual interface and use it as the processing result, thereby realizing high-quality and standardized interface display, effectively improving the user's visual experience, and meeting the diverse display application needs.

[0050] The embodiment of the present application reads and extracts the corresponding image data from the shared physical continuous memory, which can quickly and accurately locate the required data and improve the efficiency of data acquisition; the image reasoning algorithm is configured according to the reasoning requirements, which can make the algorithm highly adaptable to the specific task and enhance the pertinence and accuracy of the reasoning; the configured algorithm is called to infer the relevant image data to obtain the result, which helps to gain a deep insight into the driver's status and the situation inside the car, etc. And further, by obtaining the algorithm performance indicators and detecting the image data quality, the credibility score of the reasoning result is determined, and the reasoning result is used as the processing result only when the credibility score is higher than the preset score. In this way, the quality of the results can be strictly controlled, and the erroneous judgment caused by poor algorithms or data problems can be minimized, ensuring that the final output processing results are reliable and valuable, and providing a strong basis for subsequent related decisions and operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 is a flowchart of a method for processing image data according to some embodiments of the present invention;

[0053] Figure 2 is a flowchart of another method for processing image data according to some embodiments of the present invention;

[0054] Figure 3 is a schematic diagram of a processing flow of an image display service layer according to an embodiment of the present invention;

[0055] Figure 4 is a structural block diagram of an image data processing system according to an embodiment of the present invention;

[0056] Figure 5It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] According to an embodiment of the present invention, an embodiment of an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a data computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a method, system, device, medium and vehicle for processing image data are provided. Figure 1 is a flowchart of a method for processing image data according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101, receiving an image processing task sent by a visual application, wherein the image processing task includes a data type and task requirements.

[0061] In the embodiment of the present application, when the Android system is started and the DMS and IMS camera hardware devices are ready, if an image processing task (such as image rendering, image reasoning, etc.) sent by a visual application is received, the corresponding processing will be carried out according to the data type and task requirements contained in the task. First, the camera is opened according to the process to obtain image stream data, and then the camera function module service processes the image data according to the specific requirements of these tasks using its capabilities such as DMS and IMS related algorithms, and finally the processing results are fed back to the upper application layer, etc., to support the display, photo and video recording functions of the upper visual application and the execution of related business logic of the upper service. At the end, the camera function is closed according to a process similar to opening the camera.

[0062] Step S102, using the data type to obtain target data related to the image processing task from the shared continuous physical memory, wherein the shared continuous physical memory includes image data and video stream parameters collected by multiple different camera devices connected to the vehicle-mounted system.

[0063] In the embodiment of the present application, the method further includes the following steps A1-A3:

[0064] Step A1, obtaining original image data collected by different camera devices.

[0065] Specifically, first, make sure that different camera devices (DMS cameras for monitoring the driver's status, IMS cameras for viewing the overall situation in the car, etc.) are firmly connected to the vehicle system through appropriate interfaces (such as standard vehicle communication interfaces, USB interfaces, etc.). After the connection is completed, the vehicle system will automatically identify and load the driver corresponding to each camera device. The driver initializes the camera device and starts the acquisition work. Subsequently, the data acquisition component in the vehicle system will actively initiate data acquisition requests to each camera device based on the established communication protocol and data transmission rules. After receiving the request, the camera device will transmit the collected raw image data in the specified data format according to its own set image acquisition parameters (such as resolution, frame rate, etc.).

[0066] Step A2: Process the original image data according to a pre-configured processing strategy to obtain target image data.

[0067] Specifically, the original image data is processed according to a preconfigured processing strategy to obtain target image data, including: obtaining the processing requirements of the image usage scenario or the visual application through the data processing layer; obtaining a first processing strategy corresponding to the image usage scenario or a second processing strategy corresponding to the processing requirements of the visual application; and processing the original image data according to the first processing strategy or the second processing strategy to obtain the target image data.

[0068] During the operation of the vehicle system, the image usage scenario or the specific processing requirements proposed by the visual application layer are first obtained. For the image usage scenario, for example, if it is used for driver monitoring, the usage scenario is to accurately analyze the driver's status, such as judging whether the driver is tired or distracted, etc., then the image data needs to be able to clearly present the driver's facial expression, eye state, head posture and other key information. For another example, in the scenario of in-vehicle environment monitoring, the usage scenario requires that the image can accurately reflect the distribution of passengers in the car, the use of seats, and whether there are dangerous items.

[0069] The processing requirements of visual applications are set by upper-level applications based on specific functions. For example, if an in-vehicle application needs to take photos inside the car and achieve real-time beauty effects, then its processing requirement is to perform beauty processing on the original image data collected by the IMS camera; or if there is an application that performs driver identification based on DMS camera images, its processing requirement is to extract key information such as facial features that can be used for identification by processing the original image data. These usage scenarios and processing requirements information will be passed to the data processing layer through the corresponding system modules or application interfaces to determine the specific processing strategy later.

[0070] Based on the image usage scenarios or processing requirements of visual applications obtained earlier, the data processing layer will look for the corresponding processing strategies. If it is a driver monitoring scenario, the first processing strategy that matches it includes algorithm operations in multiple links. First, the image noise reduction algorithm is used to remove the noise generated in the original image data due to factors such as changes in the light in the car and the noise of the camera equipment itself to ensure the clarity of the image, because a clear image is essential for accurately observing the facial details of the driver. Next, the image enhancement algorithm is used to appropriately increase the contrast and brightness of the image, focusing on the driver's face, eyes and other key parts, so that subsequent feature extraction is more accurate. Then, a special facial feature extraction algorithm is used to accurately locate and extract feature data that can reflect the driver's status, such as the degree of opening and closing of the eyes, the direction of sight, and the shape of the lips, thereby forming a complete first processing strategy.

[0071] If the requirement is the aforementioned in-car photo beautification, the corresponding second processing strategy is to first detect the skin area and distinguish the skin part from other background parts in the image through the image segmentation algorithm. After that, the skin resurfacing algorithm is used to smooth the detected skin area to reduce the blemishes on the skin while retaining the key contour features of the face to make it look natural. Then, color adjustment algorithms such as whitening and ruddy are used to optimize the skin tone to achieve the beautification effect. If the requirement is driver identification, the second processing strategy will focus on extracting more recognizable facial features, such as the relative position of the facial features, unique facial texture, etc. The facial image may be standardized first through a high-precision image alignment algorithm, and then the feature extraction algorithm is used to obtain those stable and unique feature data as the basis for identity identification. This is the corresponding second processing strategy under different requirements.

[0072] After the corresponding processing strategy (first processing strategy or second processing strategy) is determined, the data processing layer will process the original image data according to the specific algorithm steps and sequence in the strategy. If the first processing strategy corresponding to the driver monitoring scenario is executed, the data processing layer first calls the image denoising algorithm module, inputs the original image data into it, and the algorithm module outputs the image data after removing the noise. Then, this intermediate result image data is input into the image enhancement algorithm module to further improve the image quality and highlight the key parts. Finally, the enhanced image data is sent to the facial feature extraction algorithm module to obtain the final data containing the driver's key facial features.

[0073] If the second processing strategy corresponding to the in-car photography beauty requirement is used, the original image data is first processed using the image segmentation algorithm to separate the image data of the skin area, and then it is passed to the skin grinding algorithm module for skin grinding to obtain the skin area image data after preliminary beauty. This data is then input into the color adjustment algorithm module, and after color optimization processing such as whitening and redness, it is recombined with the non-skin area in the original image data to finally generate the target image data that meets the beauty requirement. Similarly, for the processing of the driver identity recognition requirement, the original image data is processed in sequence according to the order of the algorithms in the corresponding second processing strategy, and finally the target image data that can be used to accurately identify the driver's identity is obtained.

[0074] Step A3, reading the video stream parameters corresponding to the target image data, and storing the target image data and the video stream parameters in a shared continuous physical memory.

[0075] Specifically, when the target image data is ready, the vehicle system will start a special data reading module. This module accurately reads the corresponding video stream parameters from the target image data based on the established video stream parameter parsing rules. Key parameters such as frame rate used to determine the playback speed of the image, resolution related to the clarity of the image, and color mode determine the color presentation of the image will be accurately extracted. Subsequently, the vehicle system will locate the available area of ​​the shared continuous physical memory through the memory management mechanism, and then store the target image data and the read video stream parameters in sequence according to the prescribed data storage format and storage order. During the storage process, the continuity and relevance of the data storage will be ensured, such as associating the target image data and the corresponding video stream parameters through specific indexes or tags, so that other subsequent functional modules can quickly and accurately obtain these data from the shared continuous physical memory for further operations such as image rendering and image reasoning.

[0076] Step S103, calling the algorithm resources corresponding to the task requirements to process the target data, obtain the processing results, and transmit the processing results to the visual application.

[0077] In an embodiment of the present application, target data related to an image processing task is obtained from a shared continuous physical memory using a data type, including: if the image processing task is an image rendering task, video stream parameters are read from the shared continuous physical memory according to the data type carried by the image rendering task, and the video stream parameters are used as target data.

[0078] Specifically, when the image processing task is determined to be an image rendering task, the data type carried by the task (such as image format, category, and other key information) will be used, with the help of the access mechanism set by the on-board system for shared continuous physical memory, to accurately locate and read the corresponding image data from it. The read image data is then used as the target data required for subsequent operations to carry out image rendering related work.

[0079] In the embodiment of the present application, calling the algorithm resources corresponding to the task requirements to process the target data and obtain the processing results includes the following steps B1-B4:

[0080] Step B1, if the task requirement is a rendering requirement, then obtain the interface object delivered by the visual application, and extract the color mode parameters, resolution and frame rate from the video stream parameters.

[0081] Specifically, when the task requirement is determined to be a rendering requirement, the system begins to carry out related operations. First, the image display service layer (DMS Camera Service) will establish a stable communication link with the upper-level visual application and wait for the visual application to pass the interface object (here is the Surface object). Once the Surface object passed by the visual application is received, it means that the basic elements that can carry the video display related functions have been obtained.

[0082] Next, key parameter information will be extracted from the video stream parameters being collected or acquired. For the extraction of color mode parameters, specific color modes such as RGB and YUV will be identified through the corresponding parsing algorithm based on the image encoding standard and color space setting followed by the video stream parameters. Relevant identifiers will be found from the header or metadata part of the video stream parameters to determine the number of horizontal and vertical pixels, for example, 1920×1080 resolution. The frame rate is determined by analyzing the time interval information of the image frames in the video stream parameters.

[0083] Step B2, reading the original color data of the interface object, and processing the original color data according to the color mode parameters to obtain a processed interface object.

[0084] Specifically, after obtaining the interface object (Surface object), the system will start a special data reading module to read the original color data of the interface object. This reading process is based on the data storage structure inside the Surface object and the corresponding access interface, and can accurately obtain the original color value information corresponding to each pixel. Then, the original color data is processed according to the color mode parameters previously extracted from the video stream parameters. For example, if the color mode parameter is RGB mode, and the original color data may have color deviation, inaccurate color gamut, etc. due to reasons such as acquisition equipment or transmission process, then it will be adjusted through the color correction algorithm. Specifically, the red, green, and blue color component values ​​of each pixel may be proportionally adjusted, brightness compensated, etc. according to the preset standard RGB color gamut range to make the color more accurate and bright; if it is YUV color mode, it will first be converted into a color space that is easy to process, and then processed according to the corresponding adjustment rules, such as adjusting the brightness (Y component), chroma (U and V components) and other parameters to ensure that the color meets the expected display effect.

[0085] Step B3, calculating the rendering size of the interface object according to the resolution parameter and the original size data of the interface object, and calculating the rendering time corresponding to the interface object according to the frame rate.

[0086] Specifically, the rendering size of the interface object is calculated based on the extracted resolution parameters and the original size data of the interface object. The original size data of the interface object reflects its initial size setting, while the resolution parameter is the pixel specification that should be followed during actual display. For example: determine how to scale the interface object to adapt to the display requirements by comparing the proportional relationship between the original size data and the resolution parameter. If the original size is large and the resolution is relatively low, the interface object needs to be reduced according to a certain scaling ratio. The calculation method may be to divide the length and width of the original size by the length and width corresponding to the resolution to obtain the corresponding scaling factor, and then multiply the original size by the scaling factor to determine the final rendering size (the specific number of pixels of length and width).

[0087] At the same time, the rendering time corresponding to the interface object is calculated according to the frame rate obtained previously. The frame rate indicates the number of image frames that need to be displayed per second, so the rendering time corresponding to each frame of the image can be obtained by dividing 1 by the frame rate. For example, when the frame rate is 30 frames per second, the rendering time corresponding to each frame of the image is 1 / 30 second. This rendering time will serve as an important basis for the subsequent rendering algorithm to control the rendering rhythm of each frame, ensuring that the interface rendering can be performed at an appropriate speed and rhythm to achieve a smooth video display effect.

[0088] Step B4, calling the rendering algorithm to render the processed interface object according to the rendering time and rendering size, obtaining a visual interface, and taking the visual interface as the processing result.

[0089] Specifically, an algorithm suitable for current needs is selected from the existing rendering algorithm library, such as a rendering algorithm based on OpenGLES, which can efficiently render the processed interface objects. According to the rendering time and rendering size calculated previously, the rendering algorithm will accurately control the rendering process of each frame of the image. Taking the rendering time as an example, a timer or timestamp mechanism will be used to ensure that each frame of the image is rendered at a calculated time interval to ensure smooth switching of the screen and avoid freezes or too fast or too slow display problems. For the rendering size, the rendering algorithm will scale, layout, and other operations on the processed interface objects according to the calculated length and width pixel numbers to ensure that the image can be presented completely and clearly in the specified display area. After the rendering algorithm renders the processed interface objects frame by frame according to these parameters, a visual interface will be generated, which contains the video screen that has been color processed, size-adapted, and displayed at a suitable frame rate.

[0090] Figure 2 is a flowchart of a method for processing image data according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0091] Step S201, receiving an image processing task sent by a visual application, wherein the image processing task includes a data type and task requirements.

[0092] In an embodiment of the present application, after the Android system is started and the DMS and IMS camera hardware devices are powered on and enter a working state, when an image processing task containing data types and task requirements sent by a visual application is received, the upper-level application first issues an instruction to open the camera according to the process, which is passed to the EVSHAL layer through the upper-level camera function module service, management interface (EVS Manager), etc. to call the driver interface to make the camera work and register the callback interface. After the camera generates image data and uploads it to the EVSHAL layer, the image stream data is obtained through the callback interface. Subsequently, the camera function module service uses DMS, IMS related algorithms to process the image data according to the data type and task requirements in the task, and feeds back the processed algorithm results to the upper-level application layer for display and other related applications.

[0093] Step S202, using the data type to obtain target data related to the image processing task from the shared continuous physical memory, wherein the shared continuous physical memory includes image data and video stream parameters collected by multiple different camera devices connected to the vehicle-mounted system.

[0094] In an embodiment of the present application, target data related to an image processing task is obtained from a shared continuous physical memory using a data type, including: if the image processing task is an image reasoning task, the target image data is read from the shared continuous physical memory according to the data type carried by the image rendering task, and the target image data is used as the target data.

[0095] Specifically, if it is determined that the image processing task belongs to an image rendering task, the corresponding image data will be accurately extracted from the shared continuous physical memory according to the data type carried by the image rendering task (such as image format, resolution range, application scenario category, etc.), using the system's preset memory reading mechanism, and then the read image data will be determined as the target data to be used in subsequent operations.

[0096] Step S203, calling the algorithm resources corresponding to the task requirements to process the target data, obtain the processing results, and transmit the processing results to the visual application.

[0097] In the embodiment of the present application, calling the algorithm resources corresponding to the task requirements to process the target data and obtain the processing results includes the following steps C1-C3:

[0098] Step C1: if the task requirement is a reasoning requirement, extract driver monitoring image data and in-vehicle image data from the target image data.

[0099] Specifically, through the stable data connection channel established with the shared physical continuous memory, according to the pre-set memory access rules and data index mechanism, the corresponding memory area storing the target image data is accurately located, and then the corresponding read function or data acquisition interface is used to read the target image data from the area completely and orderly. After the target image data is obtained, since it contains multiple types of image information, it is necessary to further extract the driver monitoring image data and the in-vehicle image data from it. This extraction process will be judged and screened based on the identification information of the image data (for example, whether the image source tag is from the DMS camera or the IMS camera), the characteristic attributes of the image (such as the main content of the picture presents the driver's face and posture, which meets the characteristics of driver monitoring, or the in-vehicle environment conditions such as passengers and objects in the car), etc., and the image data that meets the driver monitoring characteristics will be classified as driver monitoring image data, and the image data that reflects the overall environment in the car will be classified as in-vehicle image data, thereby completing the corresponding extraction work and preparing for subsequent reasoning.

[0100] Step C2, configuring the image reasoning algorithm according to the reasoning requirements to obtain the configured image reasoning algorithm.

[0101] Specifically, the content of the reasoning requirements is obtained. For example, if the reasoning is performed on the driver monitoring image data, the requirement may be to determine whether the driver is tired or distracted, etc., then it is necessary to select relevant algorithm modules suitable for detecting facial expressions, eye states, and head posture changes; for the reasoning requirements of the in-vehicle image data, if it is necessary to determine whether there are dangerous items left in the car or abnormal behavior of personnel, etc., it is necessary to select algorithm modules that can perform object recognition and behavior analysis. Then, according to these specific requirements, various basic algorithms in the existing image reasoning algorithm library (such as convolutional neural network related algorithms in deep learning for feature extraction, classification algorithms for category judgment, etc.) are combined and parameter settings are adjusted. For example, for driver fatigue detection, the convolution kernel size, step size and other parameters of the convolutional neural network are adjusted to better extract key facial features, and a suitable classification threshold is set to accurately judge whether fatigue occurs. Through such a series of adjustments and combination operations, an image reasoning algorithm that is optimized and configured for this reasoning requirement is obtained, so that it can accurately adapt to the current task scenario.

[0102] Step C3, calling the configured image inference algorithm to infer the driver monitoring image data and the in-vehicle image data, obtaining an inference result, and using the inference result as the processing result.

[0103] Specifically, after the configured image inference algorithm is obtained, the previously extracted driver monitoring image data and in-car image data are input into the algorithm respectively. The algorithm will process the driver monitoring image data according to the logic and operation flow set inside it. For example, it will first extract the feature information of each key part of the face, analyze the degree of eye opening and closing, blinking frequency, sight direction, mouth shape, head posture changes and other aspects, and make a comprehensive judgment based on the set fatigue, distraction and other state judgment criteria to obtain the inference result about the driver's state; for the in-car image data, the algorithm will identify and locate each object in the image, analyze the behavior of the person, etc., and judge whether there is any abnormality in the car based on the preset dangerous goods feature library and normal behavior patterns and other standards, so as to obtain the corresponding inference result.

[0104] In an embodiment of the present application, the inference result is used as the processing result, including: obtaining the performance indicators of the configured image inference algorithm; detecting the quality of the driver monitoring image data and the in-vehicle image data; determining the credibility score of the inference result based on the performance indicators and the quality; if the credibility score is higher than the preset score, the inference result is used as the processing result.

[0105] Specifically, we first need to obtain the performance indicators of the configured image inference algorithm, and then use the labeled test set containing various types of driver monitoring images and in-car images to input the configured algorithm for inference, and determine the accuracy by counting the percentage of correct inference results. We also calculate the proportion of samples of a specific category that can be accurately detected as the recall rate, and record the running time of the algorithm to process the image data to measure the computational efficiency. At the same time, we add interference factors such as light changes and image blur to the test images to check their robustness. Then we test the quality of the driver monitoring image data and the in-car image data. For the driver monitoring images, we use gradient operators to measure edge sharpness and contrast from the perspective of clarity, use grayscale histograms to count the uniformity of illumination, and use data verification to confirm the image integrity. For in-car images, we also consider clarity, field of view coverage integrity, and color accuracy.

[0106] Afterwards, based on the above performance indicators and quality conditions, the weights of each algorithm performance indicator determined based on past experience or experiments, combined with the weights assigned to each image data quality parameter according to their importance in inference, are substituted into a comprehensive evaluation model such as weighted summation to determine the credibility score of the inference result. Finally, a credibility score threshold that meets the application scenario requirements and the expectation of result accuracy is pre-set. If the calculated credibility score is higher than the preset score, it means that the current inference result is reliable and can be output as a processing result for subsequent operations. Otherwise, corresponding measures such as optimizing image data acquisition and improving algorithm configuration are required to improve credibility.

[0107] In an embodiment of the present application, delivering the processing results to the visual application includes: obtaining a preset transmission rule of the cross-process communication interface, and encapsulating the inference result according to the preset transmission rule to obtain the encapsulated inference result; querying the target process where the visual application that has registered a callback and is waiting to receive data is located; and delivering the encapsulated inference result to the target process through the cross-process communication interface.

[0108] Specifically, since the image display service layer can accept the registered callback of the upper-layer application with the help of the AIDL interface (which can use Java or C++ to realize cross-process data transmission and ensure cross-process communication between Native services and Java applications), it is necessary to obtain the preset transmission rules of the cross-process communication interface (i.e., the AIDL interface), which covers the requirements of data format, transmission order, data verification, etc. Then, when the image display service layer (DMS Camera Service) completes the AI ​​reasoning for the acquired DMS and IMS image data to obtain the reasoning results, it strictly follows these preset transmission rules to perform corresponding format conversion, add necessary verification information and other encapsulation operations on the reasoning results, so as to obtain the encapsulated reasoning results that meet the cross-process transmission specifications. Then, through the corresponding query function module in the system, according to the identification information of the registered callback, the target process where the visual application is waiting to receive data is found among many processes. Finally, relying on this cross-process communication interface and following its transmission mechanism, the encapsulated reasoning results are smoothly and accurately delivered to the found target process, so that the upper-level application can receive the reasoning results and then carry out subsequent functional integration and other related tasks. At the same time, it also maintains a good architecture with clear separation of business and data levels.

[0109] As an example, Figure 3 As shown in the figure, the driver starts the reversing camera application of the vehicle system. At this time, the task requirement is the rendering requirement of the camera application. The image display service layer receives the interface object transmitted from the visual application (reversing camera application), such as the interface element containing the image behind the vehicle. Then, the image display service layer extracts the color mode parameters (such as RGB mode), resolution (assuming 1280x720) and frame rate (such as 30 frames / second) from the video stream parameters. Next, read the original color data of the interface object and process the original color data according to the RGB color mode parameters, such as adjusting the color saturation and contrast. After that, calculate the rendering size of the interface object according to the extracted resolution parameters and the original size data of the interface object to adapt to the screen display. Calculate the rendering time corresponding to the interface object according to the frame rate, for example, the rendering time of each frame is 1 / 30 second. Finally, call the rendering algorithm to render the processed interface object according to the calculated rendering time and rendering size to obtain a visual reversing interface, and display it on the vehicle screen as the processing result.

[0110] When the vehicle is driving, the task requirement is the reasoning requirement of the image reasoning application. The image display service layer reads the target image data from the shared physical continuous memory, and extracts the driver monitoring image data (such as the driver's facial image) and the in-vehicle image data (the overall environment image in the vehicle) from it. Configure the image reasoning algorithm according to the reasoning requirements, such as setting it to detect the driver's fatigue state. Call the configured image reasoning algorithm to reason about the driver monitoring image data and the in-vehicle image data. If the driver is detected to have features such as frequent blinking and yawning, it is judged that the driver may be in a fatigued state, and the reasoning result is "driver fatigue". This result is used as the processing result, and the driver can be reminded to take a rest through voice prompts or displayed on the car screen to improve driving safety.

[0111] In this embodiment, a system for processing image data is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0112] This embodiment provides a system for processing image data. Figure 4 As shown, the system includes an image display service layer and a visual application layer, wherein the visual application layer includes at least one visual application;

[0113] The image display service layer is used to receive an image processing task sent by any visual application in the visual application layer, wherein the image processing task includes a data type and a task requirement; obtain target data related to the image processing task from a shared continuous physical memory using the data type, wherein the shared continuous physical memory includes image data and video stream parameters collected by multiple different camera devices connected to the vehicle system; call the algorithm resources corresponding to the task requirement to process the target data, obtain the processing result, and feed the processing result back to the visual application layer;

[0114] Visual application, used to display the visual interface carried by the processing results.

[0115] In the embodiment of the present application, the system further includes a driver layer and a data processing layer;

[0116] A driver layer, which is used to obtain raw image data collected by different camera devices and transmit the raw image data to a data processing layer of an onboard system, wherein different camera devices are connected to the onboard system;

[0117] The data processing layer is used to process the original image data according to the pre-configured processing strategy to obtain the target image data, read the video stream parameters corresponding to the target image data, and store the target image data and the video stream parameters in the shared continuous physical memory;

[0118] The image display service layer (DMS Camera Service) and the data processing layer (Camera EVS HAL) achieve the ability to share image data across processes by means of Android ION buffer shared memory. ION is responsible for managing one or more memory pools. Some memory pools will be reserved from the overall DDR memory at startup. The purpose is to deal with memory fragmentation problems or meet the needs of hardware modules such as GPU, display controller, and camera applications that have special memory requirements (continuous physical memory is required for data processing and sharing). ION represents these memory pools as ION heaps, and will configure corresponding types of ION heaps for different Android devices based on their respective memory requirements. The image display service layer (DMS Camera Service) and the data processing layer (Camera EVS HAL) allocate and share a piece of ION memory with continuous physical memory. Such continuous physical memory has the advantages of increasing memory access speed and reducing memory bandwidth, which helps to improve the overall data processing and sharing efficiency.

[0119] In an embodiment of the present application, the data processing layer is used to obtain the image usage scenario or the processing requirements of the visual application layer through the data processing layer; obtain the first processing strategy corresponding to the image usage scenario or the second processing strategy corresponding to the processing requirements of the visual application layer; and process the original image data according to the first processing strategy or the second processing strategy to obtain the target image data.

[0120] In an embodiment of the present application, the image display service layer is used to read video stream parameters from a shared continuous physical memory according to the data type carried by the image rendering task if the image processing task is an image rendering task, and use the video stream parameters as target data; if the image processing task is an image inference task, read target image data from a shared continuous physical memory according to the data type carried by the image rendering task, and use the target image data as target data.

[0121] In an embodiment of the present application, the image display service layer is used to obtain the interface object delivered by the visual application if the task requirement is a rendering requirement, and extract the color mode parameters, resolution and frame rate from the video stream parameters; read the original color data of the interface object, and process the original color data according to the color mode parameters to obtain the processed interface object; calculate the rendering size of the interface object according to the resolution parameters and the original size data of the interface object, and calculate the rendering time corresponding to the interface object according to the frame rate; call the rendering algorithm to render the processed interface object according to the rendering time and rendering size to obtain a visual interface, and use the visual interface as the processing result.

[0122] In an embodiment of the present application, the image display service layer is used to extract driver monitoring image data and in-vehicle image data from the target image data if the task requirement is a reasoning requirement; configure the image reasoning algorithm according to the reasoning requirement to obtain the configured image reasoning algorithm; call the configured image reasoning algorithm to reason about the driver monitoring image data and the in-vehicle image data to obtain the reasoning result, and use the reasoning result as the processing result.

[0123] In an embodiment of the present application, the image display service layer is used to obtain the performance indicators of the configured image reasoning algorithm; detect the quality of the driver monitoring image data and the in-vehicle image data; determine the credibility score of the reasoning result based on the performance indicators and the quality; if the credibility score is higher than the preset score, the reasoning result is used as the processing result.

[0124] In an embodiment of the present application, the image display service layer is used to obtain the preset transmission rules of the cross-process communication interface, and encapsulate the inference results according to the preset transmission rules to obtain the encapsulated inference results; query the target process where the visual application that has registered the callback and is waiting to receive data is located; and pass the encapsulated inference results to the target process through the cross-process communication interface.

[0125] It should be noted that the image display service layer (DMS Camera Service) receives the registration callback of the upper-level application in the form of AIDL, and the cross-process communication interface (AIDL interface) can use java or c++ to implement cross-process data transmission, so that Native services and Java applications can perform cross-process data transmission. The image display service layer (DMS Camera Service) performs AI reasoning on the acquired DMS and IMS image data, and feeds back the reasoning results through the cross-process transmission method of AIDL. As a data provider, it strips the business content to achieve a clear separation of business and data levels. In this process, first obtain the preset transmission rules of the cross-process communication interface, encapsulate the reasoning results according to the rules to obtain the encapsulated reasoning results, and then query the target process where the visual application that has registered the callback and is waiting to receive data is located. Finally, the encapsulated reasoning results are passed to the target process through the cross-process communication interface, so that the upper-level application can obtain the AI ​​reasoning results and perform subsequent functional integration tasks.

[0126] An embodiment of the present invention also provides a vehicle, comprising: an image data processing system and a controller; wherein the controller stores instructions for controlling the operation of the image data processing system, and the image data processing system is used to execute the image data processing method described in the above embodiment according to the instructions of the controller.

[0127] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0128] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0129] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0130] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0132] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0133] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0134] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing image data, characterized in that: The method comprises: Receiving an image processing task sent by a visual application, wherein the image processing task includes a data type and a task requirement; Acquire target data related to the image processing task from a shared continuous physical memory using the data type, wherein the shared continuous physical memory includes image data and video stream parameters collected by a plurality of different camera devices connected to the vehicle-mounted system; The algorithm resources corresponding to the task requirements are called to process the target data, obtain processing results, and transmit the processing results to the visual application.

2. The method according to claim 1, characterized in that The method further comprises: Get the raw image data collected by different camera devices; Processing the original image data according to a pre-configured processing strategy to obtain target image data; The video stream parameters corresponding to the target image data are read, and the target image data and the video stream parameters are stored in a shared continuous physical memory.

3. The method according to claim 2, characterized in that The processing of the original image data according to the pre-configured processing strategy to obtain the target image data includes: Obtain the processing requirements of image usage scenarios or visual applications through the data processing layer; Acquire a first processing strategy corresponding to the image usage scenario or a second processing strategy corresponding to the processing requirement of the visual application; The original image data is processed according to the first processing strategy or the second processing strategy to obtain target image data.

4. The method according to claim 1, characterized in that: The obtaining target data related to the image processing task from the shared continuous physical memory by using the data type includes: If the image processing task is an image rendering task, then reading video stream parameters from a shared continuous physical memory according to the data type carried by the image rendering task, and using the video stream parameters as the target data; If the image processing task is an image reasoning task, the target image data is read from the shared continuous physical memory according to the data type carried by the image rendering task, and the target image data is used as the target data.

5. The method according to claim 1 or 4, characterized in that: The calling of the algorithm resources corresponding to the task requirement to process the target data to obtain a processing result includes: If the task requirement is a rendering requirement, then obtaining the interface object delivered by the visual application, and extracting color mode parameters, resolution, and frame rate from the video stream parameters; Reading original color data of the interface object, and processing the original color data according to the color mode parameters to obtain a processed interface object; Calculate the rendering size of the interface object according to the resolution and the original size data of the interface object, and calculate the rendering time corresponding to the interface object according to the frame rate; The rendering algorithm is called to render the processed interface object according to the rendering time and the rendering size to obtain a visual interface, and the visual interface is used as the processing result.

6. The method according to claim 4, characterized in that The calling of the algorithm resources corresponding to the task requirement to process the target data to obtain a processing result includes: If the task requirement is a reasoning requirement, extracting driver monitoring image data and in-vehicle image data from the target image data; Configure the image reasoning algorithm according to the reasoning requirements to obtain the configured image reasoning algorithm; The configured image inference algorithm is called to infer the driver monitoring image data and the in-vehicle image data to obtain an inference result, and the inference result is used as the processing result.

7. The method according to claim 6, characterized in that The taking the inference result as the processing result includes: Get the performance metrics of the configured image inference algorithm; Detecting the quality of the driver monitoring image data and the in-vehicle image data; Determining a credibility score of the inference result based on the performance indicator and the quality condition; If the credibility score is higher than a preset score, the inference result is used as the processing result.

8. The method according to claim 6, characterized in that The transmitting the processing result to the visual application comprises: Obtaining a preset transmission rule of the cross-process communication interface, and encapsulating the inference result according to the preset transmission rule to obtain an encapsulated inference result; Query the target process where the visual application that has registered callbacks and is waiting to receive data is located; The encapsulated reasoning result is transmitted to the target process through the cross-process communication interface.

9. A system for processing image data, characterized in that: The system comprises an image display service layer and a visual application layer, wherein the visual application layer comprises at least one visual application; The image display service layer is used to receive an image processing task sent by any visual application in the visual application layer, wherein the image processing task includes a data type and a task requirement; obtain target data related to the image processing task from a shared continuous physical memory using the data type, wherein the shared continuous physical memory includes image data and video stream parameters collected by multiple different camera devices connected to the vehicle-mounted system; call the algorithm resources corresponding to the task requirement to process the target data, obtain a processing result, and feed the processing result back to the visual application layer; The visual application is used to display the visualization interface carried by the processing result.

10. The system according to claim 9, characterized in that The system also includes a driver layer and a data processing layer; The driver layer is used to obtain raw image data collected by different camera devices and transmit the raw image data to the data processing layer of the vehicle-mounted system, wherein the different camera devices are connected to the vehicle-mounted system; The data processing layer is used to process the original image data according to a pre-configured processing strategy to obtain target image data, read the video stream parameters corresponding to the target image data, and store the target image data and the video stream parameters in a shared continuous physical memory.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 8 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 8.

13. A vehicle, characterized in that: The vehicle comprises: an image data processing system and a controller; wherein the controller stores instructions for controlling the operation of the image data processing system, and the image data processing system is used to execute the image data processing method described in any one of claims 1 to 8 according to the instructions of the controller.

Images