Method and apparatus for processing image data from multi-channel camera

Abstract

Disclosed are a method and apparatus for processing image data from a multi-channel camera. The method comprises: respectively storing a plurality of pieces of raw image data captured by a multi-channel camera in a plurality of different storage spaces; respectively encoding and compressing the plurality of pieces of raw image data in the plurality of storage spaces to obtain a plurality of pieces of compressed image data; and transmitting the plurality of pieces of compressed image data to an upper-layer application.

Description

Method and device for processing multi-camera image data

[0001] REFERENCE TO RELATED APPLICATION

[0002] The present disclosure claims the full benefit of the Chinese Patent Application No. 202410895720.2, filed on July 4, 2024, entitled “Method and device for processing multi-camera image data”, and incorporates by reference the entire content of the Chinese Patent Application.

[0003] TECHNICAL FIELD

[0004] The present disclosure generally relates to the field of image processing, and more particularly to a method and device for processing multi-camera image data.

[0005] BACKGROUND

[0006] With the development of image technology, more and more technical fields use multi-camera to take pictures to obtain multiple image data of different directions.

[0007] In some possible scenarios, after the multi-camera takes multiple image data, it is necessary to transmit these image data to an upper application for further processing or use, and finally show the result of processing multiple image data to the user in the upper application.

[0008] However, with the improvement of the shooting quality of image data, the memory space occupied by these image data is getting larger and larger, and the computing power required to process these image data is getting higher and higher. The transmission and processing of multiple image data obtained by the multi-camera require a large amount of memory space and computing power resources, resulting in a shortage of memory and computing power resources.

[0009] SUMMARY

[0010] In a first aspect, the present disclosure provides a method for processing multi-camera image data, comprising:

[0011] storing multiple original image data taken by a multi-camera in multiple different storage spaces respectively;

[0012] encoding and compressing multiple original image data in multiple storage spaces respectively to obtain multiple compressed image data; and

[0013] transmitting multiple compressed image data to an upper application.

[0014] In some embodiments, storing multiple original image data taken by a multi-camera in multiple different storage spaces respectively comprises:

[0015] According to the identity of the original image data and a preset storage rule, the original image data is stored in a corresponding storage space; wherein the identity of the original image data is a sensor ID of the camera shooting the original image data, and the storage rule is used to describe the corresponding relationship between the identity and the storage space.

[0016] In some embodiments, the original image data in the plurality of storage spaces is encoded and compressed respectively to obtain a plurality of compressed image data, comprising:

[0017] Obtaining a compression setting; the compression setting comprises a compression size and a compression encoding format of the compressed image data;

[0018] According to the compression encoding format, the original image data is encoded and compressed into the compressed image data according to the compression size.

[0019] In some embodiments, the compression setting is obtained, comprising:

[0020] The original size of the original image data is sent to the upper layer application;

[0021] The compression size and the compression encoding format returned by the upper layer application are obtained; wherein the compression size is calculated by the upper layer application according to a set compression ratio and the original size.

[0022] In some embodiments, the compression encoding format is H264 format or H265 format.

[0023] In some embodiments, after the original image data in the plurality of storage spaces is encoded and compressed respectively to obtain a plurality of compressed image data, the method further comprises:

[0024] Obtaining an effect parameter list; the effect parameter list includes a plurality of first effect parameters and a plurality of second effect parameters, the first effect parameter is an effect parameter of the camera, and the second effect parameter is an effect parameter supported by the upper layer application;

[0025] Selecting the optimal effect parameter from the effect parameter list; and

[0026] According to the optimal effect parameter, the plurality of compressed image data is processed.

[0027] In some embodiments, the plurality of compressed image data is transmitted to the upper layer application, comprising:

[0028] All the compressed image data is packaged into a compressed image library file; and

[0029] transmit the compressed image data to the upper layer application.

[0030] In a second aspect, the present disclosure provides a processing device for multi-camera image data, comprising:

[0031] a storage module configured to store a plurality of raw image data captured by a plurality of cameras in a plurality of different storage spaces respectively;

[0032] a compression module configured to encode and compress a plurality of the raw image data in a plurality of the storage spaces respectively to obtain a plurality of compressed image data; and

[0033] a transmission module configured to transmit the plurality of compressed image data to an upper layer application.

[0034] In a third aspect, the present disclosure provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0035] the memory is configured to store a computer program;

[0036] the processor is configured to execute the program stored on the memory to implement the processing method for multi-camera image data according to the present disclosure.

[0037] In a fourth aspect, the present disclosure provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the processing method for multi-camera image data according to the present disclosure.

[0038] In some embodiments, storing the raw image data in different storage spaces can reduce the use of memory space, and transmitting the compressed image data to the upper layer application for processing can relieve the computing power pressure of the upper layer application when processing data. In some embodiments, the technical solutions provided by the present disclosure can help to reduce the requirements for memory and computing power resources when transmitting and processing a plurality of image data obtained by a plurality of cameras.

[0039] BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required by the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0042] FIG. 1 is a flow diagram of a method for processing multi-camera image data according to an embodiment of the present disclosure;

[0043] FIG. 2 is a flow diagram of a method for processing multi-camera image data according to an embodiment of the present disclosure;

[0044] FIG. 3 is a structural diagram of a device for processing multi-camera image data according to an embodiment of the present disclosure; and

[0045] FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present disclosure.

[0046] DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are a part rather than all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0048] Before the technical solutions provided by the present disclosure are described in detail, some possible application scenarios of the technical solutions provided by the present disclosure are described first.

[0049] The technical solutions provided by the present disclosure are used for processing original image data captured by multi-cameras. In some possible application scenarios, after the multi-cameras capture a plurality of original image data, the original image data is transmitted to a corresponding processing device, the processing device processes the plurality of original image data, thereby obtaining processed image data, the processed image data integrates the plurality of original image data, and the processed image data is displayed to a user in an upper-layer application of the processing device.

[0050] For example, in the field of automobiles, some automobiles are equipped with a 360-degree panoramic view function, which can display a 360-degree panoramic view of the automobile to a user in real time on a car machine of the automobile. The function can be realized by using multi-cameras. A plurality of cameras are arranged at different positions of the automobile body. When the user starts the 360-degree panoramic view function, the car machine system of the automobile calls image data captured by the cameras, processes the image data, thereby obtaining a 360-degree panoramic view of the automobile, and displays the 360-degree panoramic view to the user on a display device.

[0051] It can be understood that in the related art, a processing device directly calls raw image data captured by a multi-lens camera and processes the raw image data. With the improvement of shooting technology, the quality of the raw image data is higher and higher, and when multiple high-quality raw image data are processed at the same time, huge requirements are put on memory resources and computing resources.

[0052] An embodiment of the present disclosure provides a processing method for multi-lens camera image data. FIG. 1 is a flowchart of a processing method for multi-lens camera image data provided by an embodiment of the present disclosure. Referring to FIG. 1, the processing method for multi-lens camera image data provided by the present disclosure specifically includes the following steps:

[0053] S1: storing multiple raw image data captured by a multi-lens camera in multiple different storage spaces, respectively;

[0054] S2: encoding and compressing the multiple raw image data in the multiple storage spaces, respectively, to obtain multiple compressed image data; and

[0055] S3: transmitting the multiple compressed image data to an upper-layer application.

[0056] In some embodiments, when the multi-lens camera is working, multiple raw image data captured by the multi-lens camera are stored separately in multiple different storage spaces. In the storage space, the raw image data is stored in the form of YUV data.

[0057] In some embodiments, storing the multiple raw image data captured by the multi-lens camera in the multiple different storage spaces, respectively, includes:

[0058] According to the identity of the raw image data and a preset storage rule, the raw image data is stored in a corresponding storage space; wherein the identity of the raw image data is a sensor ID of a camera that captures the raw image data, and the storage rule is used to describe the correspondence between the identity and the storage space.

[0059] In some embodiments, when storing the multiple raw image data, the storing is based on the identity of the raw image data, and the identity of the raw image data is a sensor ID (Sensor ID) of a camera that captures the raw image data. For any one camera in the multi-lens camera, it has a unique corresponding sensor ID, and any one camera can be uniquely located by the sensor ID. Taking the sensor ID as the identity of the raw image data captured by the camera, the raw image data can be uniquely identified and distinguished from the multiple raw image data.

[0060] The storage rule is used to describe the correspondence between the identity and the storage space. In some embodiments, the storage rule can be a rule pre-set by a technician.

[0061] In some embodiments, the storage rule can also be a rule generated in real time according to the current storage state of the storage space. In some embodiments, the current storage state of each storage space can be determined by detecting the entire storage space before storing the plurality of original image data. The camera parameters of each camera can be obtained. The data size of each original image data can be estimated by giving the camera parameters of each camera. The storage rule can be generated according to the data size of each original image data and the current storage state of each storage space.

[0062] In some embodiments, the storage rule can be in the form of a data table, which at least includes the identity of the original image data and the storage address of the storage space, and the identity of one original image corresponds to the storage address of only one storage space. When the plurality of original image data captured by the plurality of cameras is obtained, the identity of the original image data is searched in the data table to obtain the storage address of the storage space corresponding to the identity of the original image data, and the original image data is stored in the corresponding storage space according to the obtained storage address.

[0063] In order to speed up the processing speed of the original image data, in some embodiments, the storage of the plurality of original image data is performed concurrently.

[0064] In some embodiments, the original image data is encoded and compressed in each storage space respectively, and the data amount of the original image data is reduced by encoding and compressing the original image data to convert the original image data into compressed image data.

[0065] In some embodiments, the plurality of original image data in the plurality of storage spaces is encoded and compressed to obtain a plurality of compressed image data, including:

[0066] obtaining a compression setting; the compression setting includes a compression size and a compression encoding format of the compressed image data; and

[0067] According to the compression encoding format, the original image data is encoded and compressed into compressed image data according to the compression size.

[0068] In some embodiments, the compression setting is used to specify the way in which the original image data is encoded and compressed, and the compression setting at least includes the compression size of the compressed image data obtained after encoding and compression and the compression encoding format of the current encoding and compression of the original image data. The original image data is encoded and compressed into a compression size according to the compression encoding format, so as to complete the encoding and compression of the original image data and obtain the compressed image data.

[0069] In some embodiments, the encoding compression format is H264 format or H265 format.

[0070] The performance gap between H264 format and H265 format is small when encoding compression is performed. The compression ratio of H265 format is 1:200, and the compression ratio of H264 format is 1:100. In theory, H265 format takes 10ms more time than H264 format to compress a frame of original image data. Therefore, in order to speed up the encoding compression speed of the original image data, in some embodiments, H264 format is selected to encode and compress the original image data.

[0071] In some embodiments, the original image data is first encoded and compressed, and then the compressed image data obtained after encoding and compression is transmitted to the upper application for processing. However, in some embodiments, the user has multiple upper applications that can process the compressed image data. The user can select a specific upper application to process the compressed image data according to the current demand to obtain the expected processing result. At this time, if the multiple original image data is encoded and compressed by a fixed compression setting, the compressed image data obtained after encoding and compression may not match the current upper application.

[0072] In some embodiments, the compression setting is obtained, including the following steps:

[0073] The original size of the original image data is transmitted to the upper application; and

[0074] The compression size and the compression encoding format returned by the upper application are obtained; wherein the compression size is calculated by the upper application according to the set compression ratio and the original size.

[0075] In some embodiments, the compression setting is determined according to the setting of the upper application on the compression encoding format in which the compressed image data is stored. The original size of the original image data is transmitted to the upper application by calling the interface. The upper application calculates the compression size according to the compression ratio and the original size of the original image data, and transmits the compression size and the compression encoding format back again by calling the interface, so as to determine the compression setting.

[0076] The compression encoding format and the compression ratio in the upper application are parameters set by the user according to the current demand. The calling interface can be Nv21Scale interface.

[0077] Through the above-mentioned compression setting acquisition method, the compression setting obtained can better meet the user's demand, and for different upper applications, the compression setting consistent with the upper application currently processing the compressed image data can be obtained.

[0078] In order to enable the upper-layer application to display the processing result obtained by processing the compressed image data to the user with better effect, in some embodiments, the compressed image data is processed with effect.

[0079] In some embodiments, after the multiple original image data in the multiple storage spaces are respectively encoded and compressed to obtain the multiple compressed image data, the method further includes the following steps:

[0080] obtaining an effect parameter list, the effect parameter list including multiple first effect parameters and multiple second effect parameters, the first effect parameters being effect parameters of the cameras, and the second effect parameters being effect parameters supported by the upper-layer application;

[0081] selecting an optimal effect parameter from the effect parameter list;

[0082] processing the multiple compressed image data with the optimal effect parameter.

[0083] In some embodiments, the effect parameter list includes multiple effect parameters, which specifically include multiple first effect parameters and multiple second effect parameters, the first effect parameters being original effect parameters of the cameras, and the second effect parameters being effect parameters supported by the upper-layer application.

[0084] The first effect parameters and the second effect parameters are both used to process the compressed image data with effect, such as sharpening processing, reverse color processing, blurring processing, etc. of the compressed image data. By processing the compressed image data with a suitable effect parameter, the display effect of the compressed image data can be effectively improved, so that the processing result obtained by the upper-layer application processing the compressed image data also has a better display effect.

[0085] An effect parameter is selected from the multiple first effect parameters and the multiple second effect parameters included in the effect parameter list as the optimal effect parameter, and the multiple compressed image data is processed with the optimal effect parameter.

[0086] The optimal effect parameter is the effect parameter that is most suitable for the compressed image data at present. The compressed image data processed with the optimal effect parameter can have a better display effect. Meanwhile, selecting one unique optimal effect parameter to process all the compressed image data can also make the results of processing the compressed image data with effect the same, which is convenient for the upper-layer application to process the compressed image data.

[0087] The optimal effect parameter can be selected by the user in the upper-layer application, or can be automatically selected based on the comparison result of the comparison between the multiple first effect parameters and the multiple second effect parameters.

[0088] In some embodiments, the plurality of compressed image data is transmitted to the upper layer application by calling an interface, and the upper layer application further processes or utilizes the compressed image data after receiving the plurality of compressed image data.

[0089] In some embodiments, transmitting the plurality of compressed image data to the upper layer application comprises:

[0090] encapsulating all the compressed image data into a compressed image library file; and

[0091] transmitting the compressed image library file to the upper layer application.

[0092] FIG. 2 is a flowchart of a method for processing multi-lens camera image data according to the present disclosure, which shows the execution process of the technical solution provided by the present disclosure in some actual implementations.

[0093] When the multi-lens camera accesses the Android system, the technical solution provided by the present disclosure can be implemented by modifying the native driver framework of the multi-lens camera. The native driver framework of the multi-lens camera can be modified in the driver code of hardware / qcom / camera / QCamera2, the compression settings are obtained, and a list of effect parameters is added. After the driver framework obtains the raw image data captured by the multi-lens camera, the raw image data can be encoded and compressed and effect processed to obtain compressed image data.

[0094] Referring to FIG. 2, the upper layer application transmits the compression size to the driver framework through the Nv21Scale interface. After the multi-lens camera is successfully opened and the raw image data is obtained, the main CPU reads the SensorID of each camera through I2C and stores each raw image data in different storage spaces. The get_h264_callback interface is encapsulated at the bottom layer to implement H264 encoding, and the raw image data is encoded and compressed. The calling interface is encapsulated at the Framework layer to set the compression size of the compressed raw image data. The calling interface is encapsulated at the Framework layer to set the effect parameters of the compressed raw image data. The raw image data is H264 encoded and compressed according to the compression size and is effect processed according to the effect parameters to obtain compressed image data. The upper layer application obtains the compressed image data, and the main CPU further processes or utilizes the compressed image data.

[0095] In some embodiments, storing the original image data in different storage spaces can reduce the use of memory space; transmitting the compressed compressed image data to the upper application for processing can alleviate the algorithmic pressure of the upper application when processing data. In some embodiments, the technical solutions provided by the present disclosure can help reduce the requirements for memory and algorithmic resources when multiple image data obtained by the multi-camera is transmitted and processed.

[0096] Referring to FIG. 3, the present embodiment also provides a processing device for multi-camera image data corresponding to the above method embodiment, which comprises:

[0097] A storage module 301 configured to store a plurality of original image data captured by a multi-camera in a plurality of different storage spaces respectively;

[0098] A compression module 302 configured to encode and compress the plurality of original image data in the plurality of storage spaces respectively to obtain a plurality of compressed image data; and

[0099] A transmission module 303 configured to transmit the plurality of compressed image data to an upper application.

[0100] In some embodiments, the storage module 301 comprises:

[0101] A storage unit configured to store the original image data in the corresponding storage space according to the identity of the original image data and the preset storage rule; wherein the identity of the original image data is the sensor ID of the camera that captures the original image data, and the storage rule is used to describe the correspondence between the identity and the storage space.

[0102] In some embodiments, the compression module 302 comprises:

[0103] An acquisition unit configured to acquire a compression setting; the compression setting comprises a compression size and a compression encoding format of the compressed image data;

[0104] A compression unit configured to encode and compress the original image data into the compressed image data according to the compression encoding format and the compression size.

[0105] In some embodiments, the acquisition unit comprises:

[0106] A sending subunit configured to send the original size of the original image data to the upper application; and

[0107] An acquisition subunit configured to acquire the compression size and the compression encoding format returned by the upper application; wherein the compression size is calculated by the upper application according to the set compression ratio and the original size.

[0108] In some embodiments, the compression encoding format is H264 format or H265 format.

[0109] In some embodiments, the apparatus further comprises:

[0110] an acquisition module configured to acquire an effect parameter list, the effect parameter list including a plurality of first effect parameters and a plurality of second effect parameters, the first effect parameters being effect parameters of the camera, and the second effect parameters being effect parameters supported by the upper-layer application;

[0111] a selection module configured to select optimal effect parameters from the effect parameter list; and

[0112] a processing module configured to perform effect processing on the plurality of compressed image data according to the optimal effect parameters.

[0113] In some embodiments, the compression module 303 comprises:

[0114] an encapsulation unit configured to encapsulate all the compressed image data into a compressed image library file; and

[0115] a transmission unit configured to transmit the compressed image library file to the upper-layer application.

[0116] As shown in FIG. 4, an embodiment of the present disclosure provides an electronic device, which includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404,

[0117] the memory 403 is configured to store a computer program;

[0118] the processor 401 is configured to execute the program stored in the memory 403, and implement a multi-path camera image data processing method, which includes:

[0119] storing a plurality of original image data captured by a plurality of cameras in a plurality of different storage spaces respectively;

[0120] encoding and compressing the plurality of original image data in the plurality of storage spaces respectively to obtain a plurality of compressed image data; and

[0121] transmitting the plurality of compressed image data to an upper-layer application.

[0122] In some embodiments, the electronic device can be a module capable of realizing a communication function or a terminal device containing the module, etc., and the terminal device can be a mobile terminal or a smart terminal. In some embodiments, the mobile terminal can be at least one of a mobile phone, a tablet computer, a notebook computer, etc. In some embodiments, the smart terminal can be a smart car, a smart watch, a shared bicycle, a smart cabinet, etc. containing a wireless communication module. In some embodiments, the module can be a wireless communication module, such as any one of a 2G communication module, a 3G communication module, a 4G communication module, a 5G communication module, an NB-IOT communication module, etc.

[0123] An embodiment of the present disclosure further provides a computer readable storage medium, which has stored thereon a computer program, and the computer program is executed by a processor to implement the processing method of multi-lens camera image data.

[0124] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the stated elements.

[0125] The above description is merely one specific implementation of the present disclosure, and persons skilled in the art can understand or implement the present disclosure based on the above description. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing multi-camera image data, comprising: storing a plurality of raw image data captured by a plurality of cameras in a plurality of different storage spaces respectively; encoding and compressing the plurality of raw image data in the plurality of storage spaces respectively to obtain a plurality of compressed image data; and transmitting the plurality of compressed image data to an upper application. 2.The method for processing multi-camera image data according to claim 1, wherein storing the plurality of raw image data captured by the plurality of cameras in the plurality of different storage spaces respectively comprises: storing the raw image data in the corresponding storage space according to the identity of the raw image data and a preset storage rule, wherein the identity of the raw image data is a sensor ID of the camera that captures the raw image data, and the storage rule is used to describe the correspondence between the identity and the storage space. 3.The method for processing multi-camera image data according to claim 1 or 2, wherein encoding and compressing the plurality of raw image data in the plurality of storage spaces respectively to obtain the plurality of compressed image data comprises: obtaining a compression setting; the compression setting comprises a compression size and a compression encoding format of the compressed image data; and encoding and compressing the raw image data into the compressed image data according to the compression encoding format and the compression size. 4.The method for processing multi-camera image data according to claim 3, wherein obtaining the compression setting comprises: sending an original size of the raw image data to the upper application; and obtaining the compression size and the compression encoding format returned by the upper application, wherein the compression size is calculated by the upper application according to a preset compression ratio and the original size. 5.The method for processing multi-camera image data according to claim 3 or 4, wherein the compression encoding format is H264 format or H265 format. 6.The method for processing multi-camera image data according to any one of claims 1 to 5, wherein after encoding and compressing the plurality of raw image data in the plurality of storage spaces respectively to obtain the plurality of compressed image data, the method further comprises: obtaining an effect parameter list; the effect parameter list comprises a plurality of first effect parameters and a plurality of second effect parameters, the first effect parameters are effect parameters of the camera, and the second effect parameters are effect parameters supported by the upper application; selecting optimal effect parameters from the effect parameter list; and performing effect processing on the plurality of compressed image data according to the optimal effect parameters. 7.The method for processing multi-camera image data according to claim 1, wherein transmitting the plurality of compressed image data to the upper application comprises: packaging all the compressed image data into a compressed image library file; and transmitting the compressed image library file to the upper application. 8.An apparatus for processing multi-camera image data, comprising: a storage module configured to store a plurality of raw image data captured by a plurality of cameras in a plurality of different storage spaces respectively; ​ ​ ​ ​ The compression module is configured to encode and compress the original image data in the storage spaces respectively to obtain compressed image data. And The transmission module is configured to transmit the compressed image data to an upper layer application.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein, The processor, the communication interface, and the memory complete mutual communication through a communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored on the memory to implement the multi-camera image data processing method in any one of claims 1 to 7.

10. A computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-camera image data processing method in any one of claims 1 to 7.

Images