Infrared image data processing method and system, device and storage medium

Abstract

Embodiments of the present application provide an infrared image data processing method and system, device and medium, the method comprising: obtaining target format image data obtained by signal conversion of infrared image original data collected by an infrared detector in response to an operation request of a camera application; returning the target format image data to a camera service subsystem through a predefined camera device interface supporting target format data transmission; returning the target format image data to the camera application through a camera hardware abstraction layer according to a preset data channel corresponding to a data format type of the target format image data by the camera service subsystem; and displaying the target format image data by the camera application.

Description

Technical Field

[0001] This application relates to the field of infrared imaging technology, and in particular to an infrared image data processing method and system, an infrared thermal imaging device, and a computer-readable storage medium. Background Technology

[0002] Currently, the infrared image data processing system of thermal imagers mainly consists of an infrared core, a System on a Chip (SOC), and a display screen. The infrared core comprises an infrared detector, an image processing FPGA (programmable gate array) board, etc. The SOC receives data from the infrared core, processes it according to the business needs of various industries, and then sends it for display. This type of infrared image data processing system has low system integration difficulty, low requirements for the SOC, and a wide range of applications, playing an irreplaceable role in fields such as epidemic prevention and temperature measurement, power line inspection, and fire protection.

[0003] However, the integration of such an infrared image data processing system requires the separate development of different data transmission and processing adapter boards, such as parallel data, serial data, and USB (Universal Serial Bus) data access boards, which is difficult to integrate, results in poor overall stability, and high power consumption. Summary of the Invention

[0004] To address the existing technical problems, this application provides an infrared image data processing method and system, an infrared thermal imaging device, and a computer-readable storage medium that can support direct connection with an infrared module to access target format image data for imaging processing, thereby achieving integrated infrared data access and processing and improving image data processing efficiency.

[0005] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an infrared image data processing method, including:

[0007] The target format image data is obtained by converting the raw infrared image data acquired by the infrared detector in response to the operation request of the camera application;

[0008] The target format image data is returned to the camera service subsystem through a predefined camera device interface that supports target format data transmission;

[0009] The camera service subsystem returns the target format image data to the camera application through the camera hardware abstraction layer, according to the preset data channel corresponding to the data format type of the target format image data.

[0010] The camera application displays the target format image data.

[0011] Secondly, embodiments of this application provide an infrared image data processing system, comprising:

[0012] The driver layer is used to obtain target format image data after the infrared detector acquires the raw infrared image data in response to the operation request of the camera application, performs signal conversion, and returns the target format image data to the camera service subsystem through a predefined camera device interface that supports target format data transmission.

[0013] The hardware abstraction layer is used by the camera service subsystem to return the target format image data to the camera application through the camera hardware abstraction layer, according to the preset data channel corresponding to the data format type of the target format image data.

[0014] The application layer is used by the camera application to display the target format image data.

[0015] Thirdly, embodiments of this application provide an infrared thermal imaging device, including a memory, a processor, and an infrared module connected to the processor; the infrared module is used to acquire raw infrared image data in response to an operation request from a camera application, and convert the raw infrared image data into target format image data of MIPI RAW14 data format type; the memory stores a computer program, and when the computer program is executed by the processor, it implements the infrared image data processing method as described in any embodiment of this application.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by the processor, implements the infrared image data processing method as described in any embodiment of this application.

[0017] In the above embodiments, the infrared image data processing method converts the raw infrared image data collected by the infrared detector into target format image data. The target format image data is then returned to the camera service subsystem via a predefined camera device interface that supports target format data transmission. The camera service subsystem then returns the target format image data to the camera application for display via the camera hardware abstraction layer, according to a preset data channel corresponding to the data format type of the target format image data. This allows the high-performance mobile processor to support access to data formats that are not limited to those under a specific standard protocol, such as those conforming to MIPI. The CSI-2 protocol supports RAW8 and RAW10 formats, as well as parallel data formats conforming to LVCMOS level standards. This enables a direct-drive system that connects the infrared module directly to a high-performance mobile processor, integrating and intelligently managing infrared data access and processing. For example, for infrared modules outputting RAW14 data format infrared image data, the same direct-drive system can be implemented by directly connecting the infrared module to the high-performance mobile processor. After the high-performance mobile processor receives the imaging data, it no longer needs to perform data conversion and stitching, which would affect the image data processing efficiency and imaging frame rate. This not only reduces system power consumption but also effectively improves image data processing efficiency.

[0018] In the above embodiments, the infrared image data processing system, infrared thermal imaging device, and computer-readable storage medium are based on the same concept as the corresponding infrared image data processing method embodiments, and thus have the same technical effects as the corresponding infrared image data processing method embodiments, which will not be repeated here. Attached Figure Description

[0019] Figure 1 This is an architecture diagram of an optional application scenario for an infrared image data processing method in one embodiment;

[0020] Figure 2 This is a flowchart of an infrared image data processing method in one embodiment;

[0021] Figure 3 This is a schematic diagram of the camera software framework in one embodiment;

[0022] Figure 4 This is a schematic diagram of the structure of an infrared image data processing system in one embodiment;

[0023] Figure 5 This is a schematic diagram of the architecture and principle of an infrared image data processing system, which is an optional specific example.

[0024] Figure 6 This is a schematic diagram of the architecture and principles of the driver layer in an optional specific example.

[0025] Figure 7 This is a schematic diagram of the architecture and principles of the HAL layer in an optional specific example.

[0026] Figure 8 This is a schematic diagram illustrating the working principle of a camera application in one embodiment;

[0027] Figure 9 This is a schematic diagram of the structure of an infrared thermal imaging device in one embodiment. Detailed Implementation

[0028] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0031] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] To address the shortcomings of current infrared image data processing systems in infrared thermal imaging devices, such as thermal imagers, which require the separate development of different data transmission and processing adapter boards, leading to high integration difficulty, poor overall stability, and high power consumption, the inventors of this application, through dedicated research, propose a direct-drive system on a high-performance processor, such as a mobile processor platform running the Android system, to improve the overall parameters of infrared image data processing systems in infrared thermal imaging devices, including system integration, image quality, and reduced system power consumption. This system achieves integrated and intelligent infrared data access and processing. Furthermore, the high-performance mobile processor can support data formats that are not limited to those under a specific standard protocol, such as those conforming to MIPI. Data formats such as RAW8 and RAW10 of the CSI-2 protocol, and parallel data formats conforming to the LVCMOS level standard, such as infrared modules for outputting infrared image data in RAW14 format, also support direct connection with high-performance mobile processors to achieve direct drive systems. After the high-performance mobile processor receives the imaging data, there is no need to convert and stitch the data, which would affect the image data processing efficiency and imaging frame rate. The system has low power consumption and high scalability, and can be better applied to different fields, especially improving the image processing effect of infrared image data in multi-light fusion application scenarios.

[0033] Please see Figure 1 This diagram illustrates an optional application scenario of the infrared image data processing method provided in this application embodiment. The infrared module is connected to a high-performance processor, such as the RAM (Random Access Memory) platform of a high-performance mobile processor running Android (an operating system based on the Linux kernel), to process and image the infrared image data. The infrared module receives the raw infrared image data from the infrared detector, which is then processed by a signal conversion module to output target format image data. In an optional specific example, the target format image data includes, but is not limited to, infrared image data in formats such as MIPI RAW14 and LVCMOS (Low Voltage Complementary Metal-Oxide-Semiconductor) level parallel data. The high-performance mobile processor's operating system adds a Camera driver and a Camera HAL (Hardware Abstraction Layer), and the RAM platform's operating system development supports the access, transmission, and parsing of infrared image data of the corresponding target data format type. This allows direct acquisition of target format image data for imaging processing, eliminating data stitching calculations, reducing image data latency, and improving image data processing efficiency and imaging frame rate. The Camera application can acquire the image data from the infrared module and process it for imaging.

[0034] Please see Figure 2The infrared image data processing method provided in one embodiment of this application can be applied to... Figure 1 The illustrated RAM platform uses a high-performance mobile processor running the Android system. The infrared image data processing method includes the following steps:

[0035] S101, Obtain target format image data after signal conversion of the raw infrared image data acquired by the infrared detector in response to the operation request of the camera application;

[0036] S103, the target format image data is returned to the camera service subsystem through a predefined camera device interface that supports target format data transmission;

[0037] S105, the camera service subsystem returns the target format image data to the camera application through the camera hardware abstraction layer, according to the preset data channel corresponding to the data format type of the target format image data;

[0038] S107, the camera application displays the target format image data.

[0039] The operating system divides the accessible memory space into two parts: kernel space and user space. User space is the memory area accessible to general applications; the camera application and camera hardware abstraction layer reside in user space. Kernel space is the memory area accessed by the operating system kernel. The camera service subsystem resides in kernel space and uses a unified interface provided by the kernel to applications and video drivers to operate the camera module and acquire data from the camera sensor chip.

[0040] The camera application provides a general interface for camera applications. For example, the camera application can provide a human-computer interaction interface to support human-computer interaction operations, receiving user operation requests for camera use, such as opening the camera module, taking pictures, capturing video images, and previewing images. In this embodiment, the infrared detector can be connected to the high-performance mobile processor in the form of an infrared module. All operations related to the camera and its accessories refer to operations on the infrared module. The infrared module includes an infrared detector and a signal conversion module connected to the infrared detector. The infrared detector collects raw infrared image data in response to user operation requests to the camera application. The signal conversion module converts the raw infrared image data into target format image data. The infrared module directly connects the target format image data to the high-performance mobile processor. The raw infrared image data refers to the electrical signal converted from the collected infrared radiation light signal by the infrared detector. The target format image data is in RAW14 format. Infrared detector data is processed by a signal conversion module, which outputs infrared image data in MIPI RAW14 format. This signal conversion module can use, but is not limited to, processors with data conversion and processing capabilities such as FPGAs and CPLDs. The signal conversion function includes: preprocessing the data output from the infrared detector, including but not limited to signal initialization functions such as data correction to reduce the impact of signal noise; and converting the preprocessed data to MIPI RAW14 format. This signal conversion simplifies the data processing complexity of the infrared detector, enabling high-performance mobile processors to obtain more raw infrared data, perform more complex image processing, reduce the hardware cost and complexity of the infrared module, and improve stability.

[0041] Optionally, the target image data format is not limited to a specific standard protocol; it can also be an LVCMOS parallel format. The signal conversion module performs preprocessing on the data output from the infrared detector, such as data correction and signal initialization, and converts the preprocessed data into parallel data formats.

[0042] The high-performance mobile processor, based on the existing camera software framework, can add a predefined camera device interface supporting target format data transmission to the driver layer, and a preset data channel corresponding to the data format type of the target format image data to the camera hardware abstraction layer. Specifically, the camera service subsystem provides the infrared module with a camera device interface supporting target format data transmission. Through the camera service subsystem using the predefined camera device interface supporting target format data transmission, and the camera hardware abstraction layer using the data channel corresponding to the data format type of the target format image data, a direct-drive system is achieved that directly connects the infrared module and the high-performance mobile processor.

[0043] In the above embodiments, the data format type of the target format image data is not limited to the data format under a certain standard protocol. For example, it is not limited to data format types such as RAW8 and RAW10 conforming to the MIPI CSI-2 protocol, or parallel data format conforming to the LVCMOS level standard. The data format supported by the high-performance mobile processor is not limited to the data format under a certain standard protocol. This enables a direct-drive system that directly connects the infrared module and the high-performance mobile processor, integrating and intelligently processing infrared data. For example, for infrared modules that output infrared image data in RAW14 format, a direct-drive system that directly connects to the high-performance mobile processor can also be used to process the target format image data. After the high-performance mobile processor receives the imaging data, it does not need to perform data conversion and stitching, which would affect the image data processing efficiency and imaging frame rate. This not only reduces system power consumption but also effectively improves image data processing efficiency.

[0044] In some embodiments, before obtaining the target format image data after signal conversion of the raw infrared image data acquired by the infrared detector in response to the operation request of the camera application, S101 includes:

[0045] The camera application creates a corresponding data container through the surface view control based on the received operation request; the operation request contains data format type information for opening image data.

[0046] Call the application programming interface to send a request command to the camera service;

[0047] The camera service calls the camera hardware interface to connect to the camera hardware abstraction layer and opens the corresponding infrared detector by calling the kernel driver.

[0048] A surface view control is a control that serves as the display interface. Based on user operation requests, the camera application uses the surface view control to create a data container (surface) corresponding to the requested camera data. Each operation request corresponds to a set of data results, stored within the corresponding data container. The operation request contains all operation configuration information, which may include at least one of the following: resolution, pixel format, manual sensor, lens, flash control, operation mode, and data format processing control.

[0049] Please refer to Figure 3 The Camera software framework of the Android system on a high-performance mobile processor includes the application layer, the Framework layer, the HAL (Hardware Abstraction Layer) layer, and the Driver layer.

[0050] At the application layer, camera applications can call the interfaces provided by AOSP (Android Open Source Project). AOSP interfaces are the general application interfaces for camera applications provided by Android. These interfaces will operate and transfer data with the camera services in the Framework layer through Binder (inter-process interaction technology).

[0051] The Camera Service, located in the Framework layer, plays a crucial role in connecting the camera application above and the HAL layer below.

[0052] In the HAL layer, Android defines the protocols and interfaces for communication between the Framework layer services and the HAL layer. The HAL layer can define standard interfaces for hardware vendors to implement, allowing Android to avoid considering the implementation of underlying drivers. Through the definition of the HAL layer architecture, cross-process communication with the camera service is handled through the HIDL (Interface Description Language) interface, while the actual operations for the Camera are issued through the standard HAL3 interface.

[0053] In the Driver layer, data is processed from hardware to the driver layer. The driver layer receives data from the HAL layer and transmits sensor data to the HAL layer. Different sensor chips correspond to different drivers.

[0054] The camera application packages the operation request carrying the data container into a request instruction through the application programming interface and passes it to the camera service in the Framework layer. This is then packaged into an instance that interfaces between the Framework layer and the HAL layer, creating a corresponding event and handing it over to the HAL layer for processing. Based on the event, the HAL layer calls the kernel driver in the Driver layer to open the corresponding infrared detector.

[0055] The above embodiments provide optional specific examples of camera software frameworks that implement infrared image data processing methods within high-performance mobile processors. By relying on standard camera software frameworks to achieve direct-drive design of infrared modules with new data formats by high-performance mobile processors, after the high-performance mobile processor receives imaging data, there is no need to convert and stitch the data, which would affect the image data processing efficiency and imaging frame rate. This not only reduces system power consumption but also effectively improves image data processing efficiency.

[0056] In some embodiments, S103, the target format image data is returned to the camera service subsystem through a predefined camera device interface that supports target format data transmission, including:

[0057] The camera service subsystem connects to the kernel driver through the kernel driver interface and to the infrared detector through a predefined camera device interface that supports target format data transmission.

[0058] The target format image data is returned to the camera service subsystem through the camera device interface.

[0059] Camera drivers can be divided into camera host drivers and camera device drivers. Camera devices include internal components such as the camera's sensor and video AD chip, while the camera host driver can be the camera's module controller. Each camera device driver can manage multiple camera devices. The camera subsystem provides a unified interface between the camera host driver and the camera sensor driver, allowing the same camera sensor driver to be ported to multiple camera hosts without significant modifications. The camera subsystem provides an interface to the kernel driver for connection to kernel drivers and connects to infrared detectors via a predefined camera device interface that supports target format data transmission. In this embodiment, by adding a predefined camera device interface that supports target format data transmission to the camera subsystem, and through the camera service subsystem managed by the kernel driver in the Driver layer, the camera device is operated through the corresponding camera device interface to obtain image data output by the infrared module. The camera device interface development supports data parsing and transmission of SENSOR_14_BIT_DIRECT corresponding to the MIPI RAW14 data format. By relying on the standard camera software framework, the development of a new data format in the camera kernel space is realized.

[0060] In the above embodiments, by relying on the standard camera software framework, a predefined camera device interface supporting target format data transmission is added to the downward connection of the camera subsystem. This completes the direct drive design for the system driver layer to directly access new data formats, such as MIPI RAW14 data format. This achieves integrated and intelligent access and processing of infrared image data supporting non-standard RAW14 data format, improves the low power consumption and high scalability of high-performance mobile processors, and enables better application in different fields.

[0061] In some embodiments, S105, the camera service subsystem returns the target format image data to the camera application through the camera hardware abstraction layer, according to a preset data channel corresponding to the data format type of the target format image data, including:

[0062] The camera service subsystem transmits the target format image data to the camera hardware abstraction layer;

[0063] The camera hardware abstraction layer returns the target format image data to the camera service through a data channel that supports the data format type;

[0064] The camera service caches the received target format image data into a data container corresponding to the operation request of the camera application.

[0065] The camera service in the Framework layer packages the request carrying the data container into an instance that interfaces between the Framework layer and the HAL layer based on the operation request of the camera application. It then creates a corresponding event and hands it over to the HAL layer for processing. The HAL layer, based on the event, calls the corresponding data channel through its internal camera interface, according to the data format type of the request and the camera ID. The camera channel converts and parses the image data returned from the kernel space based on the interface layer and implementation layer. By developing data channels that support the data format type corresponding to the target format image data (e.g., developing a data channel that supports the HAL_PIXEL_FORMAT_RAW14 data format corresponding to SENSOR_14_BIT_DIRECT), the camera service returns the image data to the camera service. In this embodiment, the camera hardware abstraction layer, by developing data channels that support the data format type of the target format image data, returns the target format image data to the camera service. This, by relying on the standard camera software framework, expands the data conversion and transmission of new data formats in the HAL layer.

[0066] In the above embodiments, by relying on the standard camera software framework, data channels corresponding to the data format type of the target format image data are added to complete the data conversion and transmission of the new data format by the HAL layer, such as supporting the HAL_PIXEL_FORMAT_RAW14 data conversion and transmission corresponding to MIPI RAW14. This realizes the direct-drive design for high-performance mobile processors to directly access the MIPI RAW14 data format, and achieves the integration and intelligence of infrared image data access and processing in the RAW14 data format. This improves the low power consumption and high scalability of the high-performance mobile processor, enabling it to be better applied to different fields.

[0067] In some embodiments, caching the received target format image data into a data container corresponding to the operation request of the camera application via the camera service includes:

[0068] If the infrared detector is successfully turned on, the camera service establishes a loop thread corresponding to the request command;

[0069] The camera service delivers the request instruction to the camera hardware abstraction layer, receives the response data returned by the camera hardware abstraction layer based on the request instruction through the loop thread, and extracts the target format image data from the response data and caches it into the corresponding data container.

[0070] Optionally, S105, the camera application displays the target format image data, including:

[0071] The camera application retrieves the target format image data from the corresponding data container and displays it based on the received image display operation.

[0072] A data container (surface) can be understood as a drawing buffer in memory used to cache the pixel data of the current window. After the infrared detector (open camera) successfully opens, a callback is sent from the camera service to the camera application. The camera service initializes a loop thread corresponding to the request instruction to wait for the response data for the operation request. Essentially, the data container refers to both the user and the encapsulator of image data. When the data container corresponding to the operation request is full, the caching mechanism notifies the camera application, at which point the view controls in the camera application will retrieve the content from their respective data containers and consume it.

[0073] In an optional specific example, taking the view preview as an operation request, after successfully opening the camera, a callback is sent from the camera service to the camera application (APP). In the `onOpenedCamera` callback, an operation similar to `startPreview` is called. At this time, a `CameraCaptureSession` thread is created. During the creation process, a `ConfigureStream` operation is called to the camera service. The parameters of `ConfigureStream` contain references to the data container in the view control. Essentially, the App provides the data container to the camera service. The camera service wraps this data container as a stream and passes it to the HAL layer via HIDL (Interface Description Language, an interface description language between HAL and the user). Then, the HAL also performs a `configureStream` operation. After `configureStream` succeeds, the camera service sends a callback to the APP to notify that `configureStream` succeeded. Next, during the initialization of the camera service, a loop thread is started to wait for receiving requests. The camera service submits the request to the HAL layer for processing. After receiving the HAL layer's processing result, it extracts the image data from the request's processing result and fills it into the data container provided by the app. If the operation request is a preview, it is handed over to the Preview control's data container; if the operation request is a capture (e.g., taking a picture), it is handed over to the ImageReader's data container. The content in the Preview control's data container is then provided to the SurfaceFlinger through a view for compositing and final display, i.e., the preview. When the ImageReader's data container is full, the app will retrieve it, save it as an image file, and consume it.

[0074] In the above embodiments, the camera application, camera service, and camera hardware abstraction layer establish a data container corresponding to the operation request. The target format image data received based on the direct drive design is parsed and transmitted as response data corresponding to the operation request and cached in the corresponding data container for the camera application to extract and display. This enables the camera application developed with a high-performance mobile processor to obtain image data in a new data format by relying on the standard camera software framework, and then extract and display the image.

[0075] Please see Figure 4 In another aspect, this application also provides an infrared image data processing system, including a driver layer 21, used to acquire target format image data obtained by signal conversion of raw infrared image data collected by an infrared detector in response to an operation request from a camera application, and return the target format image data to a camera service subsystem through a predefined camera device interface that supports target format data transmission; a hardware abstraction layer 22, used by the camera service subsystem to return the target format image data to the camera application through the camera hardware abstraction layer according to a preset data channel corresponding to the data format type of the target format image data; and an application layer 23, used by the camera application to display the target format image data.

[0076] In some embodiments, the infrared image data processing system further includes an infrared module, which includes an infrared detector and a signal conversion module connected to the infrared detector; the infrared detector is used to acquire raw infrared image data in response to an operation request from a camera application; the signal conversion module is specifically used to convert the raw infrared image data into the target format image data with a data format type of MIPI RAW14.

[0077] In some embodiments, the driver layer 21 includes a kernel driver and a camera service subsystem; the camera service subsystem is connected to the kernel driver via a kernel driver interface; the camera service subsystem is connected to the camera control module and the camera device via a camera interface, wherein the camera service subsystem includes a camera device interface that supports target format data transmission and is connected to the infrared detector.

[0078] In some embodiments, the infrared image data processing system further includes a framework service layer 24, which includes a camera service. The camera application operates and transmits data with the camera service through a general interface, and the hardware abstraction layer connects to the camera service through standard communication protocols and interfaces.

[0079] In some embodiments, the hardware abstraction layer 22 includes an interface layer connected to the camera service subsystem and a data channel connected to the interface layer that supports data transmission of the target format image data. The hardware abstraction layer obtains the target format image data returned by the kernel space through the interface layer. The data channel converts and parses the target format image data and returns it to the camera service. The camera service then caches the received target format image data in a data container corresponding to the operation request of the camera application.

[0080] In the above embodiments, the infrared image data processing system relies on a standard camera software framework. The driver layer develops predefined camera device interfaces that support target format data transmission, allowing downward connections to the camera subsystem. Through the camera service subsystem managed by the kernel driver in the driver layer, the system operates the camera device via the corresponding device interface to acquire image data output from the infrared module. The predefined camera device interface supports target data format types, such as the SENSOR_14_BIT_DIRECT data parsing and transmission corresponding to the MIPIRAW14 data format, thus realizing the development of new data formats in the camera kernel space. Furthermore, the camera hardware abstraction layer develops data channels that support target format image data types, returning the target format image data to the camera service, thereby expanding the data conversion and transmission of new data formats in the hardware abstraction layer. Thus, for infrared modules outputting non-standard RAW14 format infrared image data, direct connection between the infrared module and a high-performance mobile processor is also supported to achieve a direct-drive system. After the high-performance mobile processor receives the imaging data, there is no need for data conversion and stitching, which would affect the image data processing efficiency and imaging frame rate. This not only reduces system power consumption but also effectively improves image data processing efficiency.

[0081] To gain a more comprehensive understanding of the architecture and processing methods of the infrared image data processing system provided in this application, please refer to the following: Figures 5 to 8 Taking MIPI RAW14 as the target format image data as an example, this paper describes the infrared image data processing system and the infrared image data processing method for direct-drive imaging processing of MIPI RAW14 format infrared data.

[0082] like Figure 5As shown, the Android system camera software framework on a high-performance mobile processor includes: a camera application, camera interfaces (camera Java and camera C++ interfaces), camera services, camera hardware interfaces (camera Google HAL interface), a camera hardware abstraction layer (camera HAL instance), and a driver layer (V4L2 driver). The infrared image data processing system relies on this camera software framework to implement MIPI RAW14 data format access, conversion, and parsing. The camera call process based on the camera software framework mainly includes: the camera application opens the specified camera; based on the data format type and camera ID specified by the camera application, it calls the camera hardware Google HAL interface through the camera C++ interface and the corresponding bound camera service; this interface function calls the camera HAL instance, which is determined by whether the data format type specified when the camera application opens the camera is supported, and uses the supported data format type to determine the corresponding data channel to call the V4L2 driver; the V4L2 driver opens the corresponding camera sensor based on the data format type of the called data channel and the camera ID.

[0083] The camera hardware abstraction layer (HAL) serves as the link between the upper-layer camera interface and the lower-layer Linux camera driver. Development has enhanced HAL instance support for HAL_PIXEL_FORMAT_RAW14 data format type conversion corresponding to MIPI RAW14. The V4L2 driver development has added parsing and transmission capabilities for the SENSOR_14_BIT_DIRECT data format type corresponding to MIPI RAW14. After correctly parsing the infrared image data of the corresponding data format type, the MIPI RAW14 image data acquired and output by the infrared module can be obtained and fed back upwards. The camera service returns the data feedback to the camera application via a Surface view for image display.

[0084] like Figure 6As shown, the driver layer architecture includes: camera driver, camera host driver, and camera sensor driver; file system, storage management, V4L2 (Video For Linux Two, a unified interface provided by the kernel for applications to access audio and video drivers) driver, camera service subsystem (soc-camera subsystem, providing a unified interface between camera host driver and camera sensor driver), soc camera host (a controller embedded in the system chip), soc camera device (a type of I2C device), camera hardware (an I2C device), and camera sensor (an I2C device). The working principle of the camera driver based on the driver layer architecture is as follows: The file system and storage management are memory areas in the kernel space that store and manage camera image data. Depending on the data format type, the camera host can perform operations on the camera service subsystem via the V4L2 driver, and can also operate the camera host and the soc camera device through platform-specific interfaces. The camera service subsystem calls the camera device interface to operate the soc camera device to obtain data collected by the infrared detector. The camera device interface obtains the data type that the infrared detector can output based on the data format type. The camera device interface development supports the data parsing and transmission of SENSOR_14_BIT_DIRECT corresponding to MIPI RAW14, thereby realizing the data format development in the kernel space.

[0085] like Figure 7As shown, the HAL layer architecture includes: a camera device management layer, an interface conversion layer and a data stream parsing layer, an interface layer, and an implementation layer. The camera HAL layer connects the application programming interface (API) of the camera service in the framework service layer to the underlying camera driver and hardware. The implementation process of the HAL layer mainly includes: the camera device management layer, through the camera interface within the HAL layer, calls the corresponding data channel in the interface conversion layer and the data stream parsing layer according to the data format type and camera ID number specified by the camera application. The data channel converts and parses the target format image data returned from the kernel space by the interface layer and the implementation layer. Specifically, the interface conversion layer and the data stream parsing layer support the data channel of the HAL_PIXEL_FORMAT_RAW14 data format type corresponding to SENSOR_14_BIT_DIRECT, returning the image data to the camera service, thus enabling the HAL layer to support HAL_PIXEL_FORMAT_RAW14 data conversion and transmission corresponding to MIPI RAW14.

[0086] like Figure 8 As shown, the camera application, based on the camera software framework, acquires target format image data output by the infrared module and displays it through the camera application.

[0087] Please see Figure 9 In another aspect, this application provides an infrared thermal imaging device, including a processor 211, an infrared module 213 connected to the processor 211, and a memory 212. The infrared module 213 is used to acquire raw infrared image data in response to an operation request from a camera application, and convert the raw infrared image data into target format image data of MIPI RAW14 data format type. The memory 212 stores a computer program, which, when executed by the processor 211, implements the infrared image data processing method described in this application. The processor can be a high-performance mobile processor SOC.

[0088] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the infrared image data processing method embodiments described above, achieving the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be, for example, a read-only memory (ROM) or a random access memory (RAM).

[0089] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a smart terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing infrared image data, characterized in that, include: The target format image data is obtained by converting the raw infrared image data acquired by the infrared detector in response to the operation request of the camera application; The target format image data is returned to the camera service subsystem through a predefined camera device interface that supports target format data transmission; The camera service subsystem returns the target format image data to the camera application through the camera hardware abstraction layer, according to the preset data channel corresponding to the data format type of the target format image data. The camera application displays the target format image data; The step of the camera service subsystem returning the target format image data to the camera application via the camera hardware abstraction layer, according to a preset data channel corresponding to the data format type of the target format image data, includes: The camera service subsystem transmits the target format image data to the camera hardware abstraction layer; the camera hardware abstraction layer returns the target format image data to the camera service through a data channel that supports the data format type; the camera service caches the received target format image data in a data container corresponding to the operation request of the camera application.

2. The infrared image data processing method as described in claim 1, characterized in that, Before acquiring the target format image data obtained by signal conversion of the raw infrared image data acquired by the infrared detector in response to the operation request of the camera application, the process includes: The camera application creates a corresponding data container through the surface view control based on the received operation request; the operation request contains data format type information for opening image data. Call the application programming interface (API) to send a request command to the camera service; The camera service calls the camera hardware interface to connect to the camera hardware abstraction layer and opens the corresponding infrared detector by calling the kernel driver.

3. The infrared image data processing method as described in claim 2, characterized in that, The step of returning the target format image data to the camera service subsystem through a predefined camera device interface that supports target format data transmission includes: The camera service subsystem connects to the kernel driver through the kernel driver interface and to the infrared detector through a predefined camera device interface that supports target format data transmission. The target format image data is returned to the camera service subsystem through the camera device interface.

4. The infrared image data processing method as described in claim 3, characterized in that, The step of caching the received target format image data into a data container corresponding to the operation request of the camera application via the camera service includes: If the infrared detector is successfully turned on, the camera service establishes a loop thread corresponding to the request command; The camera service delivers the request instruction to the camera hardware abstraction layer, receives the response data returned by the camera hardware abstraction layer based on the request instruction through the loop thread, and extracts the target format image data from the response data and caches it into the corresponding data container.

5. The infrared image data processing method as described in claim 4, characterized in that, The camera application displays the target format image data, including: The camera application retrieves the target format image data from the corresponding data container and displays it based on the received image display operation.

6. An infrared image data processing system, characterized in that, include: The driver layer is used to obtain the target format image data after the infrared detector acquires the raw infrared image data in response to the operation request of the camera application, performs signal conversion, and returns the target format image data to the camera service subsystem through a predefined camera device interface that supports target format data transmission. The hardware abstraction layer is used by the camera service subsystem to return the target format image data to the camera application through the camera hardware abstraction layer, according to a preset data channel corresponding to the data format type of the target format image data; the hardware abstraction layer includes an interface layer connected to the camera service subsystem and a data channel connected to the interface layer that supports the data transmission of the data format type of the target format image data; Specifically, the hardware abstraction layer obtains the target format image data returned by the kernel space through the interface layer; The data channel converts and parses the target format image data, returns the target format image data to the camera service, and the camera service caches the received target format image data in the data container corresponding to the operation request of the camera application. The application layer is used by the camera application to display the target format image data.

7. The infrared image data processing system as described in claim 6, characterized in that, It also includes an infrared module, which includes an infrared detector and a signal conversion module connected to the infrared detector; The infrared detector is used to acquire raw infrared image data in response to an operation request from the camera application; the signal conversion module is specifically used to convert the raw infrared image data into the target format image data with a data format type of MIPI RAW14.

8. The infrared image data processing system as described in claim 6, characterized in that, The driver layer includes kernel drivers and a camera service subsystem; The camera service subsystem is connected to the kernel driver via the kernel driver interface. The camera service subsystem connects to the camera control module and camera device via a camera interface. The camera service subsystem includes a camera device interface that connects to the infrared detector via a predefined interface that supports target format data transmission.

9. The infrared image data processing system as described in claim 6, characterized in that, It also includes a framework service layer, which includes a camera service. The camera application operates and transmits data with the camera service through a general interface. The hardware abstraction layer connects to the camera service through standard communication protocols and interfaces.

10. An infrared thermal imaging device, characterized in that, Includes a memory, a processor, and an infrared module connected to the processor; The infrared module is used to acquire raw infrared image data in response to an operation request from a camera application, and to convert the raw infrared image data into target format image data with a data format type of MIPI RAW14. The memory stores a computer program, which, when executed by the processor, implements the infrared image data processing method as described in any one of claims 1 to 5.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the infrared image data processing method as described in any one of claims 1 to 5.

Images