Data acquisition system and method and storage medium

Abstract

The invention discloses a data acquisition system and method and a storage medium, the data acquisition system comprises a first processing subsystem and a second processing subsystem, and the first processing subsystem and the second processing subsystem carry out cross-core communication; wherein the first processing subsystem is used for collecting non-image sensor data and sending the non-image sensor data to the second processing subsystem; and the second processing subsystem is used for receiving the non-image sensor data and storing the non-image sensor data to the corresponding buffer queue according to the sensor type. Thus, cross-core communication is carried out through the first processing subsystem and the second processing subsystem, the stability and the real-time performance of NCS data acquisition can be ensured, high-frequency NCS data acquisition can be realized, and the overall performance of camera services is further improved.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to a data acquisition system, method and storage medium. Background Art

[0002] In the process of taking pictures with a camera, in addition to the direct reliance on the image sensor, detection sensors such as angular velocity, acceleration, color temperature, and light frequency are also required, namely non-image sensors (NCS).

[0003] Figure 1 Figure 1 is a block diagram of NCS data acquisition based on a homogeneous platform. Figure 1 As shown, the box Fig.10 It can include an application processor subsystem (APSS) and non-image sensors. Among them, the APSS subsystem includes components such as an image signal processor (ISP), an application central processing unit (ACPU), and a driver module (Sensor Driver). The camera hardware abstraction layer (Camera HAL) on the ACPU side periodically obtains the collected data of non-image sensors through the Sensor Driver module to adapt to business needs. However, because Camera HAL needs to handle complex and heavy Camera business, the stability of collecting NCS data is easily disturbed by other businesses, kernel interactions, etc., resulting in poor stability of NCS data collection, and even inability to achieve high-frequency NCS data collection. Summary of the invention

[0004] The present application proposes a data acquisition system, method and storage medium, which can ensure the stability and real-time performance of collecting NCS data, and can also realize high-frequency NCS data acquisition, thereby improving the overall performance of camera services.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a data acquisition system, which includes a first processing subsystem and a second processing subsystem, and the first processing subsystem and the second processing subsystem perform cross-core communication; wherein:

[0007] a first processing subsystem, configured to collect non-image sensor data and send the non-image sensor data to a second processing subsystem;

[0008] The second processing subsystem is used to receive non-image sensor data and store the non-image sensor data into a corresponding buffer queue according to the sensor type; wherein the non-image sensor data carries a timestamp.

[0009] In some embodiments, the first processing subsystem includes a sensor management module, which is used to control the non-image sensor to collect data at a first frequency, obtain non-image sensor data carrying a timestamp, and send the non-image sensor data to the second processing subsystem at a second frequency; wherein the second frequency is less than or equal to the first frequency.

[0010] In some embodiments, the type of non-image sensor includes at least one of the following: an angular velocity sensor, an acceleration sensor, a color temperature sensor, and a light frequency sensor.

[0011] In some embodiments, different types of non-image sensors have different corresponding first frequencies.

[0012] In some embodiments, the second processing subsystem includes a kernel layer, a user layer and a camera hardware abstraction layer, wherein: the kernel layer includes a first driver module, which is used to receive non-image sensor data according to a first driver program, and forward the non-image sensor data to the camera hardware abstraction layer through a message monitoring thread of the user layer; the camera hardware abstraction layer includes a buffer module, which is used to store the non-image sensor data in a corresponding buffer queue according to the sensor type in the buffer module; wherein the buffer module includes multiple buffer queues, and different sensor types correspond to different buffer queues.

[0013] In some embodiments, the camera hardware abstraction layer further includes a first consumer module, configured to obtain accessed non-image sensor data from the buffer module according to a service access requirement.

[0014] In some embodiments, the data acquisition system also includes a third processing subsystem, which communicates with the first processing subsystem and the second processing subsystem across cores respectively; wherein: the third processing subsystem is used to obtain target image data during the first exposure period; and receive non-image sensor data sent by the first processing subsystem, and filter out target non-image sensor data associated with the first exposure period from the non-image sensor data, bind the target non-image sensor data with the target image data, obtain first metadata, and send the first metadata to the second processing subsystem.

[0015] In some embodiments, the third processing subsystem includes an image signal processing module and a micro-processing module, wherein: the image signal processing module is used to generate target image data during a first exposure period based on a preset frame rate, and determine an exposure timestamp corresponding to the first exposure period; the micro-processing module is used to receive non-image sensor data sent by the first processing subsystem at a third frequency, and select target non-image sensor data associated with the first exposure period from the non-image sensor data according to the timestamp and exposure timestamp carried by the non-image sensor data; bind the target non-image sensor data with the target image data to obtain first metadata, and send the first metadata to the second processing subsystem.

[0016] In some embodiments, in the second processing subsystem, wherein: the kernel layer also includes a second driver module, which is used to receive the first metadata according to the second driver program, and forward the first metadata to the camera hardware abstraction layer through the message monitoring thread of the user layer; the camera hardware abstraction layer includes a metadata module, which is used to parse the first metadata in the metadata module to obtain target non-image sensor data, and store the target non-image sensor data in a preset storage area.

[0017] In some embodiments, the camera hardware abstraction layer further includes a second consumer module for acquiring target non-image sensor data from a preset storage area and performing image processing according to the target non-image sensor data.

[0018] In a second aspect, an embodiment of the present application provides a data collection method, the method comprising:

[0019] collecting non-image sensor data via a first processing subsystem;

[0020] The non-image sensor data sent by the first processing subsystem is received by the second processing subsystem, and the non-image sensor data is stored in a buffer module according to the sensor type; wherein the non-image sensor data carries a timestamp.

[0021] In some embodiments, the method further includes: receiving a business access requirement of a first consumer module; obtaining accessed non-image sensor data from a buffer module according to the business access requirement; wherein the buffer module includes a plurality of buffer queues, and different sensor types correspond to different buffer queues.

[0022] In some embodiments, the method further includes: receiving non-image sensor data sent by the first processing subsystem through a third processing subsystem, and filtering out target non-image sensor data associated with the first exposure period from the non-image sensor data; and acquiring target image data during the first exposure period through the third processing subsystem, binding the target non-image sensor data with the target image data to obtain first metadata, and sending the first metadata to the second processing subsystem.

[0023] In some embodiments, the method further includes: after sending the first metadata to the metadata module in the second processing subsystem, parsing the first metadata by the metadata module to obtain target non-image sensor data, and storing the target non-image sensor data in a preset storage area.

[0024] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the method described in any one of the aspects.

[0025] The embodiment of the present application provides a data acquisition system, method and storage medium, the data acquisition system includes a first processing subsystem and a second processing subsystem, the first processing subsystem and the second processing subsystem perform cross-core communication; wherein the first processing subsystem is used to collect non-image sensor data and send the non-image sensor data to the second processing subsystem; the second processing subsystem is used to receive the non-image sensor data and store the non-image sensor data in a corresponding buffer queue according to the sensor type. In this way, through the cross-core communication between the first processing subsystem and the second processing subsystem, not only can the first processing subsystem realize high-frequency NCS data acquisition, but also can improve the problem that the NCS data is easily disturbed by other services, kernel interactions and other aspects in the second processing subsystem when collecting NCS data, thereby improving the stability and real-time performance of NCS data. At the same time, since the NCS data is sent to the second processing subsystem for storage on demand, system resources can be saved, thereby improving the overall performance of the camera service. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of a NCS data acquisition block diagram based on a homogeneous platform;

[0027] Figure 2 A schematic diagram of the structure of a data acquisition system provided in an embodiment of the present application Figure 1 ;

[0028] Figure 3 A schematic diagram of real-time transmission of NCS data provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of a data acquisition system provided in an embodiment of the present application Figure 2 ;

[0030] Figure 5 A schematic diagram of the structure of a data acquisition system provided in an embodiment of the present application Figure 3 ;

[0031] Figure 6 A schematic diagram of the association binding of NCS data during exposure provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the structure of a data acquisition system provided in an embodiment of the present application Figure 4 ;

[0033] Figure 8 A schematic diagram of the framework of an NCS acquisition system based on a heterogeneous platform in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the software architecture of an NCS acquisition system based on a heterogeneous platform in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of a buffer control flow of NCS data provided in an embodiment of the present application;

[0036] Fig.11 A schematic diagram of a data collection method provided in an embodiment of the present application Figure 1 ;

[0037] Fig.12 A schematic diagram of a data collection method provided in an embodiment of the present application Figure 2 ;

[0038] Fig.13 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are described first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0044] Camera;

[0045] Application Processor Sub System (APSS);

[0046] Camera Sub System (CSS);

[0047] Smart Engine Processor (SME);

[0048] Hub;

[0049] Gyroscope (Gyroscope, Gyro);

[0050] Accelerometer (ACC);

[0051] Non-Camera Sensor (NCS);

[0052] Microprocessor (Micro Control Unit, MCU);

[0053] Image Signal Processor (ISP);

[0054] Application Central Processing Unit (ACPU);

[0055] Hardware layer (Hardware);

[0056] Firmware layer (Firmware);

[0057] Kernel layer;

[0058] User layer (User);

[0059] Hardware Abstract Layer (HAL);

[0060] Metadata;

[0061] Inter Integrated Circuit (I2C);

[0062] Improved Inter Integrated Circuit (I3C);

[0063] Serial Peripheral Interface (SPI);

[0064] Inter Process Communication (IPC).

[0065] With the continuous development of computer technology, the use of electronic devices such as mobile phones, tablet computers, laptops, PDAs, personal digital assistants, wearable devices, etc. is becoming more and more popular. Most users use images taken by electronic devices to record their study, work, and life. Among them, in the process of taking pictures with mobile phone cameras, in addition to the direct reliance on image sensors, detection sensors such as angular velocity, acceleration, color temperature, and light frequency, that is, non-image sensors, are also required.

[0066] See also Figure 1 , which shows an exemplary data acquisition block diagram of Camera NCS based on a homogeneous platform and a traditional mobile phone chip. Fig.10 It can include an APSS subsystem and non-image sensors. The APSS subsystem includes an ISP module, an ACPU module, a driver module (Sensor Driver) and other components.

[0067] based on Figure 1The Camera HAL on the ACPU side periodically obtains the data collected by non-image sensors through the Sensor Driver module on the Kernel side to adapt to business needs. However, because the Camera HAL needs to handle complex and heavy Camera business, the stability of collecting NCS data is easily disturbed by other businesses and kernel interactions, and the regularity and timeliness of the data are insufficient. In addition, if high-frequency sampling such as 500 Hz, 1KHz and above is required, it may not be achieved, or the data stability is not good.

[0068] In addition, during image exposure, external jitter, color temperature, and light strobe will affect the output effect of the image. The NCS data during this exposure period needs to be associated and bound with the image data and metadata of the corresponding exposure period. Due to the instability of the sampling interval of non-image sensors during acquisition, individual data is usually sampled and selected, and data alignment cannot be effectively achieved. It may also cause "pollution" of NCS data in non-related time periods, affecting the accuracy of subsequent control, algorithm and other processing.

[0069] Based on this, an embodiment of the present application provides a data acquisition system, which performs cross-core communication between a first processing subsystem and a second processing subsystem. Not only can the first processing subsystem realize high-frequency NCS data acquisition, but also the problem that the NCS data is easily disturbed by other services, core interactions, etc. in the second processing subsystem when acquiring the NCS data can be improved, thereby improving the stability and real-time performance of the NCS data. In addition, by performing cross-core communication with the first processing subsystem and the second processing subsystem respectively through the third processing subsystem, the image data during the image exposure period can be bound to the associated NCS data, thereby correcting the influence of external jitter, color temperature, light strobe and other factors on the output effect of the image, avoiding the "pollution" of non-related NCS data, solving the correlation problem of the NCS data, and thus improving the overall performance of the camera service.

[0070] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0071] In one embodiment of the present application, see Figure 2 , which shows a schematic diagram of the composition structure of a data acquisition system provided by an embodiment of the present application. Figure 2 As shown, the data acquisition system 20 may include a first processing subsystem 21 and a second processing subsystem 22, and the first processing subsystem 21 and the second processing subsystem 22 perform cross-core communication; wherein:

[0072] The first processing subsystem 21 is used to collect non-image sensor data and send the non-image sensor data to the second processing subsystem 22;

[0073] The second processing subsystem 22 is used to receive non-image sensor data and store the non-image sensor data into a corresponding buffer queue according to the sensor type; wherein the non-image sensor data carries a timestamp.

[0074] It should be noted that in the embodiment of the present application, the data acquisition system 20 may be a non-image sensor acquisition system in a Camera system based on a heterogeneous platform, wherein the first processing subsystem 21 may be an SME subsystem, the second processing subsystem may be an APSS subsystem, and data exchange may be performed between the SME subsystem and the APSS subsystem.

[0075] It should also be noted that, in the embodiment of the present application, the first processing subsystem 21 collects non-image sensor data in real time, and then periodically sends the non-image sensor data to the buffer module in the second processing subsystem 22 for storage, thereby solving the real-time problem of the non-image sensor data.

[0076] In some embodiments, see Figure 2 The data acquisition system 20 may further include a non-image sensor 23, and the first processing subsystem 21 may include a sensor management module 211, wherein:

[0077] The sensor management module 211 is used to control the non-image sensor 23 to collect data at a first frequency, obtain non-image sensor data carrying a timestamp, and send the non-image sensor data to the second processing subsystem 22 at a second frequency; wherein the second frequency is less than or equal to the first frequency.

[0078] It should be noted that in the embodiment of the present application, the sensor management module 211 is a hardware-based software control module, which can be called a Sensor Hub. In the first processing subsystem 21, through the Sensor Hub, the non-image sensor 23 can be periodically collected to obtain non-image sensor data, and then periodically sent to the second processing subsystem 22.

[0079] It should also be noted that, in the embodiment of the present application, the first frequency can be represented by f0, and for example, the value of f0 is set to 240 Hz; the second frequency can be represented by f1, and for example, the value of f1 is set to 120 Hz. Here, the values ​​of f0 and f1 are specifically set according to actual conditions and are not limited thereto.

[0080] It should also be noted that in the embodiment of the present application, the non-image sensor 23 is periodically collected by the Sensor Hub, and a timestamp is required for each collection, wherein the timestamp can be a number that is continuously accumulated based on a clock system.

[0081] Furthermore, considering the overhead of cross-core communication between the first processing subsystem 21 and the second processing subsystem 22, when collecting non-image sensors 23, the collected multiple NCS collection data can be packaged into a "cluster" as the non-image sensor data described in the embodiment of the present application, and then actively sent to the second processing subsystem 22 at a frequency f1 not higher than f0.

[0082] In some embodiments, the type of non-image sensor may include at least one of the following: an angular velocity sensor, an acceleration sensor, a color temperature sensor, and a light frequency sensor.

[0083] It should be noted that, in the embodiment of the present application, the type of the non-image sensor may be an angular velocity sensor—Gyro, an acceleration sensor—ACC, a color temperature sensor—Color Sensor, a light frequency sensor—FlickerSensor, and the like.

[0084] It should also be noted that in the embodiment of the present application, the sampling frequencies (i.e., f0) of these non-image sensors can be set to be the same, or they can also be set to be different. In order to save power consumption, different types of non-image sensors can have different first frequencies set accordingly.

[0085] That is to say, when collecting data from non-image sensors, different sampling frequencies can be set for different types of non-image sensors to save power. In real-time transmission processing, in order to facilitate the subsequent common transmission frequency, interpolation and other methods can also be used to supplement the problem of fewer data points at low sampling frequencies.

[0086] For example, Figure 3 A schematic diagram of real-time transmission of NCS data provided in an embodiment of the present application. Figure 3 As shown, NCS data sampling, such as Gyro, ACC, light frequency, color temperature and other NCS data, under the control of the Sensor Hub in the SME subsystem, periodically (sampling frequency is f0, for example, 240Hz) outputs non-image sensor data, and each sampling is accompanied by a timestamp (for example, 0, 1, 2, 3, 4, ..., 14, etc.). When NCS data is transmitted, considering the overhead of cross-core communication, the collected multiple NCS data are packaged into a "cluster", and then the non-image sensor data is "actively dispatched" to the APSS subsystem at an upload frequency of f1 (for example, 120Hz).

[0087] In some embodiments, Figure 2 Based on the data acquisition system 20 shown, see Figure 4 , the second processing subsystem 22 may include a kernel layer, a user layer and a camera hardware abstraction layer, wherein:

[0088] The kernel layer may include a first driver module 221 for receiving non-image sensor data according to a first driver program and forwarding the non-image sensor data to the camera hardware abstraction layer through a message monitoring thread 222 of the user layer;

[0089] The camera hardware abstraction layer may include a buffer module 223, which is used to store non-image sensor data in a corresponding buffer queue according to the sensor type in the buffer module 223; wherein the buffer module 223 includes multiple buffer queues, and different sensor types correspond to different buffer queues.

[0090] It should be noted that, in the embodiment of the present application, the first driver module 221 may be an SMEdriver module of the Kernel layer, and the buffer module 223 may be an NCS module of the Camera HAL layer.

[0091] Specifically, in an embodiment of the present application, when the first processing subsystem 21 sends the non-image sensor data to the second processing subsystem 22, the SME driver module of the Kernel layer can be interactively received. At the Kernel layer, according to the driver of the SME driver module, under the drive of the NCS data reception event, the message monitoring thread (Message Manager) of the User layer is sent, and then according to the periodic message event, the non-image sensor data is sent to the NCS module of the Camera HAL layer. For the NCS module, the received non-image sensor data can be stored in the corresponding buffer queue according to the sensor type. Exemplarily, the NCS module may include a first buffer queue, a second buffer queue, a third buffer queue, and a fourth buffer queue. Among them, the non-image sensor data collected by the angular velocity sensor can be stored in the first buffer queue, the non-image sensor data collected by the acceleration sensor can be stored in the second buffer queue, the non-image sensor data collected by the color temperature sensor can be stored in the third buffer queue, the non-image sensor data collected by the light frequency sensor can be stored in the fourth buffer queue, and so on. In addition, each cluster of data here carries a timestamp. In this way, the "active dispatch" and cache of non-image sensor data can be realized.

[0092] In some embodiments, in the second processing subsystem 22, the camera hardware abstraction layer may further include a first consumer module, wherein the first consumer module is used to obtain the accessed non-image sensor data from the buffer module 223 according to the service access requirement.

[0093] It should also be noted that in the embodiment of the present application, non-image sensor data can be distributed not only actively but also passively. The first consumer module is Consumer-B, for example. The Consumer-B module can acquire non-image sensor data by calling the buffer module 223 according to business needs. In this way, through consumer events or periodic access requirements, passive distribution can provide NCS data sets based on timestamp queries or specified numbers.

[0094] Simply put, in the embodiment of the present application, the SME subsystem collects NCS data, sends it to the APSS subsystem side at a high frequency, and "actively distributes" it to the NCS module of the Camera HAL for storage as needed; it relies on consumer events or periodic access for "passive distribution" to provide a continuous or specified number of NCS data sets, thereby solving the real-time problem of NCS data.

[0095] Furthermore, during image exposure, external jitter, color temperature, and light strobe will affect the output effect of the image. The NCS data during this exposure period needs to be associated and bound with the image data and metadata of the corresponding exposure period. However, due to the instability of the sampling interval of the non-image sensor during acquisition, individual data is usually sampled and selected, and data alignment cannot be effectively achieved. It may also cause "contamination" of the NCS data in non-related time periods, affecting the accuracy of subsequent control, algorithm and other processing.

[0096] In some embodiments, Figure 2 Based on the data acquisition system 20 shown, see Figure 5 The data acquisition system 20 may further include a third processing subsystem 24, which performs cross-core communication with the first processing subsystem 21 and the second processing subsystem 22 respectively; wherein:

[0097] The third processing subsystem 24 is used to obtain the target image data during the first exposure period; and receive the non-image sensor data sent by the first processing subsystem 21, and filter out the target non-image sensor data associated with the first exposure period from the non-image sensor data, bind the target non-image sensor data with the target image data, obtain the first metadata, and send the first metadata to the second processing subsystem 22.

[0098] It should be noted that in the embodiment of the present application, the third processing subsystem 24 can be a CSS subsystem, which can be used to control the generation, processing and storage of image data, bind the received NCS data with periodic image data, and realize interaction and data transmission with the SME subsystem and the APSS subsystem.

[0099] It should also be noted that, in the embodiment of the present application, the exposure time may refer to the short period of time between the opening and closing of the shutter, which may also be referred to as the shutter time. "Binding" specifically associates the image data within this short period of time with the corresponding NCS data, and packages them into the front-end (FE) stage metadata to obtain the first metadata, and then sends the first metadata to the APSS subsystem side through cross-core communication.

[0100] In some embodiments, see Figure 5 , the third processing subsystem 24 may include an image signal processing module 241 and a micro-processing module 242, wherein:

[0101] An image signal processing module 241, configured to generate target image data during a first exposure period based on a preset frame rate, and determine an exposure timestamp corresponding to the first exposure period;

[0102] The micro-processing module 242 is used to receive the non-image sensor data sent by the first processing subsystem 21 at a third frequency, and select the target non-image sensor data associated with the first exposure period from the non-image sensor data according to the timestamp and exposure timestamp carried by the non-image sensor data; bind the target non-image sensor data with the target image data to obtain the first metadata, and send the first metadata to the second processing subsystem 22.

[0103] It should be noted that, in the embodiment of the present application, the image signal processing module 241 may be an image signal processor (ISP), and the micro-processing module 242 may be a microprocessor (MCU).

[0104] It should also be noted that, in the embodiment of the present application, the third frequency can be represented by f2, and for example, the value of f2 is set to 60 Hz; the preset frame rate can be represented by f3, and for example, the value of f3 is set to 30 frames per second (fps). Here, the values ​​of f2 and f3 are specifically set according to actual conditions and are not limited thereto.

[0105] It should also be noted that in the embodiment of the present application, as the preset frame rate increases, for example, the value of f3 increases from 30fps to 60fps, 120fps, 240fps, etc., the frequency f2 of interaction between the CSS subsystem and the SME subsystem does not necessarily have to increase accordingly, but can also increase with the frequency of CSS packaging multi-frame data upload to reduce the number of cross-core communications.

[0106] For example, Figure 6 A schematic diagram of the association binding of NCS data during exposure provided by an embodiment of the present application. Figure 6As shown in the figure, under the control of the Sensor Hub of the SME subsystem, non-image sensor data is periodically (sampling frequency is f0, for example, 240Hz) collected and cached in the SME subsystem. The image sensor runs at a certain frame rate (frame rate is f3, for example, 30fps) and generates image data during exposure, as shown in the figure. Figure 6 The grid-filled part in is used to indicate the exposure time period. During the image exposure period, external jitter, color temperature, and light strobe light will affect the output effect of the image, so the corresponding NCS data generated during this period is needed to correct the impact of the above factors.

[0107] In addition, Figure 6 In the figure, for the image sensor, sof represents the start of frame, that is, the opening time of the shutter; eof represents the end of frame, that is, the closing time of the shutter. 7 cycles (i.e., 0, 1, 2, 3, 4, 5, 6) are schematically provided here, and each cycle can be divided into an exposure time period and a blank time period. Assuming that the ratio of exposure time: blank time is r (for example, r = 3:1), the corresponding NCS data amount in the exposure time of each frame image is n, and the specific calculation formula is as follows:

[0108]

[0109] Among them, if the value of f0 is 240 Hz, the value of f3 is 30 fps, r=3:1, then it can be obtained that n=6.

[0110] That is to say, in the embodiment of the present application, the SME subsystem can periodically (frequency is f2, such as 120Hz or 60Hz, which needs to be greater than f3) send the non-image sensor data to the microprocessor module 242 in the CSS subsystem. The microprocessor module 242 caches the non-image sensor data, selects the NCS data during the exposure period (the number is n, such as 6) according to the exposure timestamp and the sampling timestamp carried by the NCS data; then binds it with the image data related to the periodic (frame rate is f3, such as 30fps) shutter opening and closing (i.e., exposure time) of the image sensor, packages it into the metadata of the FE stage, and sends the obtained first metadata to the APSS subsystem through cross-core communication.

[0111] In some embodiments, Figure 4 Based on the data acquisition system 20 shown, see Figure 7 , for the second processing subsystem 22, wherein:

[0112] The kernel layer also includes a second driver module 224, which is used to receive the first metadata according to the second driver program, and forward the first metadata to the camera hardware abstraction layer through the message monitoring thread 222 of the user layer;

[0113] The camera hardware abstraction layer also includes a metadata module 225, which is used to parse the first metadata in the metadata module 225 to obtain the target non-image sensor data, and store the target non-image sensor data in a preset storage area.

[0114] It should be noted that, in the embodiment of the present application, the second driver module 224 may be a kernel mode driver (KMD) module of the kernel layer, and the metadata module 225 may be a MetaNode module of the Camera HAL layer. In addition, the preset storage area may be a storage area for storing metadata of each frame of the image, that is, a DataTable bound to the image frame.

[0115] Specifically, in an embodiment of the present application, when the first processing subsystem 21 sends non-image sensor data to the third processing subsystem 24, it can be sent to the MCU on the CSS side periodically (frequency is f2, for example, 60Hz) through cross-core communication. Among them, the MCU caches the non-image sensor data, selects the target non-image sensor data during exposure, and binds the target image data related to the periodic (frame rate is f3, for example, 30fps) shutter opening and closing (ie, exposure time) of the image sensor, and packages it into the metadata of the FE stage, and sends the obtained first metadata to the Kernel layer on the APSS side through cross-core communication. At the Kernel layer, according to the driver of the KMD module, driven by the NCS data reception event, the message monitoring thread (Message Manager) of the User layer is sent, and then the meta buffer is sent to the pipeline (Pilelines) according to the periodic transmission of the metadata. Based on the pipeline (Pilelines), the first metadata in the meta buffer is sent to the MetaNode module. In the MetaNode module, the first metadata in the meta buffer is parsed to obtain the target non-image sensor data, and is stored in the DataTable bound to the image frame according to the storage structure of the non-image sensor data, and then passed to related business modules, such as the Consumer-A module.

[0116] Furthermore, in some embodiments, in the second processing subsystem 22, the camera hardware abstraction layer may further include a second consumer module, wherein the second consumer module is configured to obtain target non-image sensor data from a preset storage area and perform image processing according to the target non-image sensor data.

[0117] It should also be noted that in the embodiment of the present application, taking Consumer-A as an example, the second consumer module can obtain target non-image sensor data from the DataTable, and then further use it for algorithm fusion, image processing, etc.

[0118] Simply put, in an embodiment of the present application, the SME subsystem collects NCS data and passes it to the CSS subsystem. The MCU therein packages the NCS data and image data associated during image exposure together to obtain the first metadata, which is passed to the MetaNode module on the Camera HAL side for parsing, and actively distributed to other business modules for use, thereby solving the correlation problem of the NCS data.

[0119] The embodiment of the present application provides a data acquisition system, which includes a first processing subsystem, a second processing subsystem and a third processing subsystem. Among them, by cross-core communication between the first processing subsystem and the second processing subsystem, not only can the first processing subsystem realize high-frequency NCS data acquisition, but also can improve the problem that the NCS data is easily disturbed by other services, kernel interactions and other aspects in the second processing subsystem when acquiring NCS data, thereby improving the stability and real-time performance of NCS data; in addition, by cross-core communication with the first processing subsystem and the second processing subsystem respectively through the third processing subsystem, the image data during the image exposure period can also be bound with the associated NCS data, so as to correct the influence of external jitter, color temperature, light strobe light and other factors on the output effect of the image, avoid "pollution" of non-related NCS data, solve the correlation problem of NCS data, and thus improve the overall performance of the camera business.

[0120] In another embodiment of the present application, as the complexity of chip hardware increases, in addition to the ACPU, the data acquisition system based on the heterogeneous platform also adds MCU to control the ISP and SME subsystems to maintain a constant open state or event detection, etc. Therefore, the high-complexity mobile phone camera business needs drive the integration of cross-system solutions. Based on this, the embodiment of the present application provides a technical solution for non-image sensor data acquisition and distribution that can meet the real-time and relevance of the business.

[0121] In some embodiments, Figure 8 FIG. 1 is a schematic diagram of a NCS acquisition system framework based on a heterogeneous platform in an embodiment of the present application. Figure 8As shown, in addition to non-image sensors and camera modules, the framework can also include SME subsystems, APSS subsystems, and CSS subsystems. This framework is a block diagram of the Camera system NCS data acquisition system based on a heterogeneous platform in this technical solution, which mainly solves the requirements of the Camera business for the real-time and correlation of NCS data.

[0122] exist Figure 8 In the case of Camera system NCS data collection, distribution and use based on heterogeneous platforms, the three main subsystems involved are as follows:

[0123] (1) The SME subsystem collects non-image sensor data such as angular velocity (Gyro), acceleration (ACC), color temperature (ColorSensor), and light frequency (FlikerSensor) in real time through a hardware-based software control module (i.e., the Sensor Hub module). The sampling frequency of each non-image sensor can be configured to be the same or different.

[0124] (2) The CSS subsystem, which includes the MCU module and ISP module, can be used to control the generation, processing, and storage of image data, bind the received NCS data with the periodic image data, and realize the interaction and data transmission with the SME subsystem and APSS subsystem.

[0125] (3) The APSS subsystem includes the ACPU module, which runs the camera core control center (i.e., Camera HAL) and can interact with the SME subsystem and the CSS subsystem through the forwarding module (i.e., Mailbox).

[0126] like Figure 8 As shown, for data path 1, the NCS data is collected periodically (frequency is f0, for example, 240 Hz) by the Sensor Hub module in the SME subsystem, and is periodically (frequency is f1, for example, 120 Hz) sent to the ACPU module on the APSS subsystem side.

[0127] like Figure 8 As shown, for data path 2-1, the Sensor Hub module in the SME subsystem periodically (frequency is f0, for example, 240Hz) collects NCS data and periodically (frequency is f2, for example, 60Hz) sends it to the MCU module on the CSS subsystem side. The MCU module caches the NCS data and binds it to the periodic (frame rate is f3, for example, 30fps) shutter opening and closing (i.e., exposure time) of the image sensor.

[0128] like Figure 8As shown, for data path 2-2, the optical image stabilizer (OIS) driver in the camera module can directly read the gyro information for use in the optical image stabilization algorithm.

[0129] That is to say, the embodiment of the present application proposes a Camera system NCS data collection and distribution solution based on a heterogeneous platform, which mainly includes: (1) The SME subsystem continuously collects NCS data, sends it to the APCU module on the APSS subsystem side at a high frequency, and "actively distributes" it to the NCS module in the Camera HAL for storage on demand, relying on consumer event or periodic access "passive distribution", providing continuous or specified number of NCS data sets, thereby solving the real-time problem of NCS data. (2) The SME subsystem periodically collects NCS data and passes it to the CSS subsystem. The MCU module in it packages the NCS data associated with the image exposure period into the metadata of the FE stage, and then passes it to the MetaNode module in the Camera HAL for parsing, and actively distributes it to other services for use, thereby solving the problem of NCS data relevance.

[0130] In some embodiments, Fig. 9 Schematic diagram of the software architecture of an NCS acquisition system based on a heterogeneous platform in the embodiment of the present application. Fig. 9 As shown in the figure, a camera system NCS acquisition software architecture block diagram based on a heterogeneous platform is proposed here, which is described in detail from the following aspects.

[0131] (1) NCS data “real-time” processing channel.

[0132] like Fig. 9 As shown, for NCS data, the data path can be Module0->Module1>Module7->Module8->Module9>Module6->Module10 to solve the real-time problem of non-image sensor data acquisition.

[0133] [Module0] Hardware layer, non-image sensors (such as Gyro, ACC, light flash frequency, temperature, etc.) are controlled by the Sensor Hub module of the SME subsystem, and periodically (frequency is f0, such as 240Hz) output NCS data and transmit it to the SensorHub module through hardware buses such as SPI and I2C.

[0134] [Module1] Firmware layer, SME FW program periodically (frequency is f1, for example, 120Hz) sends data to the APSS side through IPC cross-core communication, and the Kernel layer SME driver driver receives the data interactively.

[0135]

Module7

[0136] [Module8] User layer, the message monitoring thread of the User space, "actively dispatches" NCS data to the NCS module of the Camera HAL layer based on periodic message events.

[0137]

Module9

[0138] [Module6] Camera HAL layer, other modules obtain NCS data stored in the buffer area by calling.

[0139]

Module10

[0140] (2) NCS data “correlation” processing pathway.

[0141] like Fig. 9 As shown, for NCS data, the data path follows Module0->Module2->Module3->Module4->Module5->Module6->Module11 to solve the correlation problem of non-image sensor data acquisition.

[0142] [Module0] Hardware layer, non-image sensors (such as Gyro, ACC, light flash frequency, temperature, etc.) are controlled by the Sensor Hub module of the SME subsystem, and periodically (frequency is f0, such as 240Hz) output NCS data, and transmit it to the SensorHub module through hardware buses such as SPI, I2C, etc. At this time, a certain amount of data is cached in the Sensor Hub module.

[0143] [Module1] Firmware layer, the SME FW program is sent to the MCU module on the CSS side periodically (frequency is f1, for example, 120 Hz) through IPC cross-core communication.

[0144]

Module2

[0145]

Module3

[0146]

Module4

[0147] [Module5] Camera HAL layer, pipelines (Pilelines) send meta buffers to related processing modules, such as MetaNode modules.

[0148]

Module11

[0149]

Module12

[0150] (3) NCS real-time transmission solution based on high-frequency “active dispatch” of SME subsystem.

[0151] like Figure 3As shown, NCS data sampling, such as Gyro, ACC, light flash frequency, temperature and other NCS data, is controlled by the Sensor Hub module of the SME subsystem, and periodically (frequency is f0, for example, 240Hz) outputs NCS data, and each sampling is accompanied by a timestamp (a number that is continuously accumulated based on the clock system). For the sampling frequency f0, it can be set according to different sensor types, or it can be adjusted according to the requirements of the upper-layer application. In addition, considering the overhead of IPC cross-core communication, the SME FW module can package multiple NCS data into a "cluster" and "actively dispatch" the NCS data to the APSS side at an upload frequency f1 (for example, 120Hz or 60Hz) lower than f0.

[0152] (4) An associative transmission scheme based on NCS data binding during image exposure.

[0153] like Figure 6 As shown, under the control of the Sensor Hub module of the SME subsystem, the NCS data is collected periodically (frequency is f0, for example, 240Hz) and the first-level cache of the NCS data is prepared. The image sensor runs at a certain frame rate f3 (for example, 30fps) and generates image data during the exposure period, as shown in FIG. Figure 6 The grid-filled part in the middle is used to indicate the exposure time period. During the image exposure period, external jitter, color temperature, and light strobe light will affect the output effect of the image, so the corresponding NCS data generated during this period is needed to correct the impact of the above factors.

[0154] The image sensor generates images at a frame rate of f3 (for example, 30fps), and each cycle can be divided into an exposure period and a blank period. Assuming that the ratio of exposure time: blank time is r (for example, r = 3:1), the amount of NCS data corresponding to the exposure time of each frame of image is n, and the specific calculation formula is as follows:

[0155]

[0156] If the value of f0 is 240 Hz, the value of f3 is 30 fps, and r=3:1, then we can get n=6.

[0157] That is to say, the SME FW program periodically (frequency is f2, such as 120Hz or 60Hz, which needs to be greater than f3) sends the data to the MCU module on the CSS side through IPC cross-core communication. The MCU module caches the NCS data, selects the NCS data during the exposure period (the number is n, such as 6) according to the exposure timestamp and the sampling timestamp carried by the NCS data, and then binds the image data related to the periodic (frame rate is f3, such as 30fps) shutter opening and closing (i.e., exposure time) of the image sensor, packages it into the metadata of the FE stage, and sends the meta buffer to the APSS side through IPC communication.

[0158] (5) Buffer control scheme for NCS data.

[0159] See also Fig.10 , which provides a schematic diagram of the buffer control process of NCS data, which can further explain the following contents, such as Fig.10 As shown:

[0160]

Module21

[0161]

Module22

[0162]

Module23

[0163] [Module24] The CSS subsystem interacts with the ACPU module in the APSS subsystem, and transmits the NCS sensor data and image-related information to the Camera HAL on the APCU side through the Meta data Buffer.

[0164] Furthermore, in an embodiment of the present application, during data association processing, as the image frame rate increases, for example, from 30fps to 60fps, 120fps, 240fps, etc., the frequency of interaction between the CSS subsystem and the SME subsystem does not necessarily have to increase, but may increase with the frequency of packaging multi-frame meta uploads by the CSS subsystem, thereby reducing the number of IPC cross-core communications.

[0165] Furthermore, in the embodiment of the present application, when the SME subsystem performs NCS data sampling, different sampling rates can be set for different types of non-image sensors to save power consumption. In real-time transmission processing, in order to facilitate the subsequent common transmission frequency, interpolation and other methods can also be used to supplement the problem of few data points at low sampling frequencies.

[0166] In summary, in the embodiment of the present application, for real-time acquisition of NCS data, the Sensor Hub module of the SME subsystem periodically collects NCS data, interacts with the SMEService in the ACPU module on the APSS side with the help of the IPC cross-core communication mechanism, and "actively dispatches" the NCS module on the Camera HAL layer as needed. The NCS module provides a data buffer pool for each type of non-image sensor. With the help of "passive dispatch" of access by other services, a time-continuous or specified number of NCS data sets can be provided. In other words, the SME subsystem continuously collects NCS data, sends it to the ACPU module at a high frequency, and "actively dispatches" it to the NCS module of the Camera HAL layer for storage as needed; relying on consumer event or periodic access "passive dispatch", providing a continuous or specified number of NCS data sets, can solve the problem of real-time NCS data. For the acquisition of NCS data by association, the Sensor Hub module of the SME subsystem periodically collects NCS data, interacts with the MCU module on the CSS side with the help of the IPC cross-core communication mechanism, and the MCU module caches NCS data, corresponding to the periodic shutter opening and closing (i.e., exposure time) of the image sensor. The associated NCS data and image data are packaged in the FE stage, communicated with the CSS Service in the ACPU module through the IPC cross-core communication mechanism, and passed to the MetaNode module of the Camera HAL layer for analysis, and actively distributed to other subsequent businesses, such as image processing algorithms such as Auto White Balance (AWB) and Auto Focus Sensor (ATF). In other words, SME periodically collects NCS data and passes it to the CSS subsystem. Its MCU module binds the NCS data associated with the image data during image exposure, passes it to the MetaNode module of the Camera HAL layer for analysis, and actively distributes it to other businesses, which can solve the problem of NCS data relevance.

[0167] The embodiment of the present application provides a data acquisition system, and the specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that through the technical solution of the aforementioned embodiment, for the NCS sensor required by the imaging technology, the SME subsystem is used to send the NCS data to the ACPU side at a high frequency and "actively distribute" it to the NCS module storage of the Camera HAL layer as needed, thereby providing a continuous or specified number of NCS data sets. By "passively distributing" to consumers, on-demand distribution is achieved to save system resources. In addition, during image exposure, external jitter, color temperature, and light strobes will affect the output effect of the image. Through the correlation of NCS data, the corresponding NCS data generated during the exposure period is bound to the image data, which can directly correct the impact of the above factors, and at the same time, it can also avoid the "pollution" of irrelevant NCS data; thereby improving the overall performance of the Camera system.

[0168] In another embodiment of the present application, based on the data acquisition system described in the above embodiment, see Fig.11 , which shows a flow chart of a data collection method provided by an embodiment of the present application. Fig.11 As shown, the method may include:

[0169] S1101: Collect non-image sensor data through a first processing subsystem.

[0170] S1102: Receive non-image sensor data sent by the first processing subsystem through the second processing subsystem, and store the non-image sensor data into a buffer module according to sensor type; wherein the non-image sensor data carries a timestamp.

[0171] In an embodiment of the present application, the method is applied to the data acquisition system described in the aforementioned embodiment. The data acquisition system may include a first processing subsystem, a second processing subsystem, and a third processing subsystem. Here, the first processing subsystem may be an SME subsystem, the second processing subsystem may be an APSS subsystem, and the third processing subsystem may be a CSS subsystem.

[0172] In some embodiments, the first processing subsystem includes a sensor management module, and the method further includes:

[0173] The sensor management module controls the non-image sensor to collect data at a first frequency to obtain non-image sensor data carrying a timestamp, and sends the non-image sensor data to the second processing subsystem at a second frequency; wherein the second frequency is less than or equal to the first frequency.

[0174] In the embodiment of the present application, the type of non-image sensor may include at least one of the following: an angular velocity sensor, an acceleration sensor, a color temperature sensor, and a light frequency sensor. The first frequencies corresponding to different types of non-image sensors may be the same or different.

[0175] In some embodiments, the second processing subsystem may include a kernel layer, a user layer, and a camera hardware abstraction layer. Accordingly, receiving non-image sensor data through the second processing subsystem and storing the non-image sensor data in a buffer module according to the sensor type may include:

[0176] Based on the first driver module in the kernel layer, receiving non-image sensor data according to the first driver program, and forwarding the non-image sensor data to the camera hardware abstraction layer through the message monitoring thread of the user layer;

[0177] Based on the buffer module in the camera hardware abstraction layer, the non-image sensor data is stored in the corresponding buffer queue according to the sensor type.

[0178] In the embodiment of the present application, the first driver module may be an SME driver module of the Kernel layer, and the buffer module may be an NCS module of the Camera HAL layer, wherein the buffer module includes a plurality of buffer queues, and different sensor types correspond to different buffer queues.

[0179] That is to say, when sending the non-image sensor data to the second processing subsystem, at the Kernel layer, according to the driver of the SME driver module, driven by the NCS data receiving event, the message monitoring thread (Message Manager) of the User layer is sent, and then according to the periodic message event, the non-image sensor data is sent to the NCS module of the Camera HAL layer. For the NCS module, the received non-image sensor data can be stored in the corresponding buffer queue according to the sensor type to realize the "active dispatch" and caching of the non-image sensor data.

[0180] In some embodiments, the method further includes: receiving a service access requirement of the first consumer module; and acquiring the accessed non-image sensor data from the buffer module according to the service access requirement.

[0181] In the embodiment of the present application, the first consumer module is taken as Consumer-B, and the Consumer-B module can acquire the non-image sensor data according to business needs by calling the buffer module 223. In this way, the passive distribution of non-image sensor data is achieved through consumer events or periodic access requirements.

[0182] In some embodiments, see Fig.12 , after step S1101, the method further includes:

[0183] S1103: receiving the non-image sensor data sent by the first processing subsystem through the third processing subsystem, and filtering out the target non-image sensor data associated with the first exposure period from the non-image sensor data; and acquiring the target image data during the first exposure period through the third processing subsystem, binding the target non-image sensor data with the target image data to obtain first metadata, and sending the first metadata to the second processing subsystem.

[0184] In the embodiment of the present application, the third processing subsystem includes an image signal processing module and a microprocessing module. Accordingly, for step S1103, it can specifically include: generating target image data during the first exposure period based on a preset frame rate through the image signal processing module, and determining the exposure timestamp corresponding to the first exposure period; receiving non-image sensor data sent by the first processing subsystem at a third frequency through the microprocessing module, and selecting target non-image sensor data associated with the first exposure period from the non-image sensor data according to the timestamp and exposure timestamp carried by the non-image sensor data; binding the target non-image sensor data with the target image data to obtain first metadata, and sending the first metadata to the second processing subsystem.

[0185] In this way, during the image exposure, external jitter, color temperature, and light strobe will affect the output effect of the image. Through the correlation of NCS data, the corresponding NCS data generated during the exposure period is bound to the image data, which can directly correct the impact of the above factors and avoid the "pollution" of irrelevant NCS data.

[0186] In some embodiments, the method further includes: after sending the first metadata to the metadata module in the second processing subsystem, parsing the first metadata by the metadata module to obtain target non-image sensor data, and storing the target non-image sensor data in a preset storage area.

[0187] In an embodiment of the present application, in the second processing subsystem, a second driver module based on the kernel layer receives the first metadata according to the second driver program, and forwards the first metadata to the camera hardware abstraction layer through the message monitoring thread of the user layer; a metadata module based on the camera hardware abstraction layer parses the first metadata to obtain the target non-image sensor data, and stores the target non-image sensor data in a preset storage area. Among them, the second driver module can be a kernel mode driver (KMD) module of the kernel layer, and the metadata module can be a MetaNode module of the camera HAL layer. In addition, the preset storage area can be a DataTable bound to the image frame.

[0188] In this way, when the obtained first metadata is sent to the APSS side, at the Kernel layer, according to the driver of the KMD module, driven by the NCS data receiving event, the message monitoring thread of the User layer is sent, and then the meta buffer is sent to the pipeline (Pilelines) according to the periodic transmission of the metadata, and then the first metadata in the meta buffer is sent to the MetaNode module based on the pipeline. In the MetaNode module, the first metadata in the meta buffer is parsed to obtain the target non-image sensor data, and is stored in the DataTable bound to the image frame according to the storage structure of the non-image sensor data, and then passed to the relevant business module, such as the Consumer-A module.

[0189] In some embodiments, the camera hardware abstraction layer further includes a second consumer module. Accordingly, the method further includes: acquiring the target non-image sensor data from a preset storage area, and sending the target non-image sensor data to the second consumer module.

[0190] In the embodiment of the present application, taking Consumer-A as an example, the Consumer-A module can obtain the target non-image sensor data from the DataTable, and then further use it for algorithm fusion, image processing, etc.

[0191] That is to say, the SME subsystem collects NCS data and passes it to the CSS subsystem. The MCU inside it packages the NCS data and image data associated during image exposure together to obtain the first metadata, passes it to the MetaNode module on the Camera HAL side for parsing, and actively distributes it to other business modules for use, thereby solving the correlation problem of NCS data.

[0192] The embodiment of the present application provides a data acquisition method, which is applied to the data acquisition system based on a heterogeneous platform in the aforementioned embodiment. On the one hand, the NCS data is sent to the ACPU side at a high frequency with the help of the SME subsystem and "actively distributed" to the NCS module storage of the Camera HAL layer as needed, thereby providing a continuous or specified number of NCS data sets. By "passively distributing" to consumers, on-demand distribution is achieved to save system resources. On the other hand, during image exposure, external jitter, color temperature, and light strobes will affect the output effect of the image. Through the correlation of NCS data, the corresponding NCS data generated during the exposure period is bound to the image data, which can directly correct the impact of the above factors, and at the same time avoid the "pollution" of irrelevant NCS data; thereby improving the overall performance of the Camera system.

[0193] In another embodiment of the present application, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0194] It should be noted that the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment.

[0195] It should also be noted that the storage medium here may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0196] Based on the above computer readable storage medium, see Fig.13 , which shows a schematic diagram of the structure of a terminal device 130 provided in an embodiment of the present application. Fig.13 As shown, the terminal device 130 may include: a communication interface 1301, a memory 1302 and a processor 1303; each component is coupled together via a bus system 1304. It is understood that the bus system 1304 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.13 In FIG. 1 , various buses are labeled as bus system 1304. Fig.13The processing units for the SME subsystem, CSS subsystem, and APSS subsystem can be collectively referred to as processor 1303. Among them:

[0197] The communication interface 1301 is used to receive and send signals during the process of sending and receiving information with other external network elements;

[0198] A memory 1302, used for storing a computer program that can be run on the processor 1303;

[0199] The processor 1303 is configured to execute, when running the computer program:

[0200] collecting non-image sensor data via a first processing subsystem;

[0201] The non-image sensor data sent by the first processing subsystem is received by the second processing subsystem, and the non-image sensor data is stored in a buffer module according to the sensor type; wherein the non-image sensor data carries a timestamp.

[0202] It can be understood that the memory 1302 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0203] The processor 1303 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1303. The above-mentioned processor 1303 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1302, and the processor 1303 reads the information in the memory 1302 and completes the steps of the above method in combination with its hardware.

[0204] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0205] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0206] Optionally, as another embodiment, the processor 1303 is further configured to execute the steps of the method described in any one of the aforementioned embodiments when running the computer program.

[0207] It should be noted that in the embodiments of the present application, the terminal device 130 can be implemented in various forms. For example, the terminal device described in the embodiments of the present application can be a mobile phone, a tablet computer, a laptop computer, a PDA, a navigation device, a wearable device, etc., and the embodiments of the present application do not specifically limit it.

[0208] It should also be noted that in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0209] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0210] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0211] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0212] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0213] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data acquisition system, characterized in that: The data acquisition system includes a first processing subsystem and a second processing subsystem, wherein the first processing subsystem and the second processing subsystem perform cross-core communication; wherein: The first processing subsystem is used to collect non-image sensor data and send the non-image sensor data to the second processing subsystem; The second processing subsystem is used to receive the non-image sensor data and store the non-image sensor data into a corresponding buffer queue according to the sensor type; wherein the non-image sensor data carries a timestamp.

2. The data acquisition system according to claim 1, characterized in that: The first processing subsystem includes a sensor management module, wherein: The sensor management module is used to control the non-image sensor to collect data at a first frequency, obtain non-image sensor data carrying a timestamp, and send the non-image sensor data to the second processing subsystem at a second frequency; wherein the second frequency is less than or equal to the first frequency.

3. The data acquisition system according to claim 2, characterized in that: The type of the non-image sensor includes at least one of the following: Angular velocity sensor, acceleration sensor, color temperature sensor and light frequency sensor.

4. The data acquisition system according to claim 3, characterized in that: Different types of the non-image sensors have different corresponding first frequencies.

5. The data acquisition system according to any one of claims 1 to 4, characterized in that: The second processing subsystem includes a kernel layer, a user layer and a camera hardware abstraction layer, wherein: The kernel layer includes a first driver module, which is used to receive the non-image sensor data according to the first driver program, and forward the non-image sensor data to the camera hardware abstraction layer through the message monitoring thread of the user layer; The camera hardware abstraction layer includes a buffer module, which is used to store the non-image sensor data in a corresponding buffer queue according to the sensor type in the buffer module; wherein the buffer module includes multiple buffer queues, and different sensor types correspond to different buffer queues.

6. The data acquisition system according to claim 5, characterized in that: The camera hardware abstraction layer also includes a first consumer module, wherein: The first consumer module is used to obtain the accessed non-image sensor data from the buffer module according to business access requirements.

7. The data acquisition system according to claim 5, characterized in that: The data acquisition system further includes a third processing subsystem, and the third processing subsystem performs cross-core communication with the first processing subsystem and the second processing subsystem respectively; wherein: The third processing subsystem is used to obtain target image data during the first exposure period; and receive non-image sensor data sent by the first processing subsystem, and filter out target non-image sensor data associated with the first exposure period from the non-image sensor data, bind the target non-image sensor data with the target image data to obtain first metadata, and send the first metadata to the second processing subsystem.

8. The data acquisition system according to claim 7, characterized in that: The third processing subsystem includes an image signal processing module and a micro-processing module, wherein: The image signal processing module is used to generate target image data during the first exposure period based on a preset frame rate, and determine an exposure timestamp corresponding to the first exposure period; The micro-processing module is used to receive the non-image sensor data sent by the first processing subsystem at a third frequency, and select the target non-image sensor data associated with the first exposure period from the non-image sensor data according to the timestamp carried by the non-image sensor data and the exposure timestamp; bind the target non-image sensor data with the target image data to obtain the first metadata, and send the first metadata to the second processing subsystem.

9. The data acquisition system according to claim 7, characterized in that: In the second processing subsystem, wherein: The kernel layer further includes a second driver module, configured to receive the first metadata according to a second driver program, and forward the first metadata to the camera hardware abstraction layer through a message monitoring thread of the user layer; The camera hardware abstraction layer includes a metadata module, which is used to parse the first metadata in the metadata module to obtain the target non-image sensor data, and store the target non-image sensor data in a preset storage area.

10. The data acquisition system according to claim 9, characterized in that: The camera hardware abstraction layer also includes a second consumer module, wherein: The second consumer module is used to obtain the target non-image sensor data from the preset storage area and perform image processing according to the target non-image sensor data.

11. A data collection method, characterized in that: The method comprises: collecting non-image sensor data via a first processing subsystem; The non-image sensor data sent by the first processing subsystem is received by the second processing subsystem, and the non-image sensor data is stored in a buffer module according to the sensor type; wherein the non-image sensor data carries a timestamp.

12. The method according to claim 11, characterized in that The method further comprises: Receive business access requirements of the first consumer module; According to the service access requirement, the accessed non-image sensor data is obtained from the buffer module; wherein the buffer module includes a plurality of buffer queues, and different sensor types correspond to different buffer queues.

13. The method according to claim 11, characterized in that The method further comprises: receiving, by a third processing subsystem, the non-image sensor data sent by the first processing subsystem, and filtering out target non-image sensor data associated with the first exposure period from the non-image sensor data; and The target image data during the first exposure period is acquired through the third processing subsystem, the target non-image sensor data is bound to the target image data to obtain first metadata, and the first metadata is sent to the second processing subsystem.

14. The method according to claim 13, characterized in that The method further comprises: After the first metadata is sent to the metadata module in the second processing subsystem, the first metadata is parsed by the metadata module to obtain the target non-image sensor data, and the target non-image sensor data is stored in a preset storage area.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 11 to 14 is implemented.

Images