A multi-channel data combination method, system and intelligent wearable device
The multi-channel data combination method addresses the challenges of integrating image and motion data in smart devices by merging and timestamping data channels, reducing circuit area and power consumption, and improving processing efficiency.
Patent Information
- Application Number
- CN202010693593.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-17
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-07-17
AI Technical Summary
Existing technologies face challenges in efficiently addressing the processing of multi-channel data in smart devices, particularly in the integration of image data and motion data, leading to increased circuit area, power consumption, and susceptibility to electromagnetic interference.
A multi-channel data combination method that merges and reorganizes multiple channels of data into one channel, adding timestamps to unify image and sensor data in time, reducing circuit area and power consumption, and enhancing anti-interference capabilities.
The method effectively reduces circuit area and power consumption while improving algorithm processing by unifying image and sensor data, enhancing the system's resistance to interference.
Smart Images

Figure CN113949807B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent wearable devices, and in particular to a multi-channel data combination method, system and intelligent wearable device. Background Art
[0002] As smart wearable devices gradually develop towards miniaturization, lightness, low power consumption and intelligence, higher requirements are placed on the software and hardware design of smart wearable devices. For example, the implementation of the SLAM (simultaneous localization and mapping) function of smart glasses requires the simultaneous collection of two channels of image data and one channel of motion data. Specifically, two cameras collect and transmit two channels of image data respectively, and a motion sensor collects and transmits one channel of motion data. After that, the processor performs algorithm processing on these data to realize specific applications.
[0003] At present, in order to facilitate the processor to process multi-channel image data, multi-channel image data transmission often uses a parallel interface, which will inevitably increase the circuit area and power consumption, and is not conducive to the circuit's resistance to electromagnetic interference, and can easily lead to image confusion; moreover, image data and motion data are often transmitted separately, which requires additional transmission lines, and is not conducive to the temporal unification of image data and motion data, affecting the subsequent algorithm processing effect.
[0004] Therefore, how to provide a solution to the above technical problems is a problem that technicians in this field need to solve at present. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-channel data combination method, system and intelligent wearable device, which can combine and reorganize multiple channels of data to obtain one channel of data for subsequent calculation and processing, thereby reducing the circuit area, reducing system power consumption, and improving the system's anti-interference ability; moreover, the present application adds timestamps to both image data and sensor data, so that the image data and sensor data are unified in time, which helps to improve the subsequent algorithm processing effect.
[0006] In order to solve the above technical problems, the present invention provides a multi-channel data combination method, which is applied to a device including multiple cameras and sensor chips, comprising:
[0007] After the multiple cameras and the sensor chips enter the specified working mode, each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chips are respectively obtained;
[0008] When acquiring each frame of image data, an image timestamp representing the generation time of the current frame of image data is generated, and when acquiring each frame of sensor data, a sensor timestamp representing the generation time of the current frame of sensor data is generated;
[0009] The current frame image data output by each of the multiple cameras are merged to obtain image merged data, and then the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data to finally obtain the data to be calculated.
[0010] Preferably, the process of merging the current frame image data output by each of the multiple cameras to obtain the image merged data includes:
[0011] When acquiring the frame image data output by any camera, the frame image data is converted into image parallel data, and the image parallel data is written into the first FIFO queue corresponding to the camera;
[0012] When the first FIFO queues corresponding to the multiple cameras are all non-empty queues, poll and read a pixel data in each first FIFO queue, and write it into the total FIFO queue;
[0013] Pixel data is read from the total FIFO queue, and when the amount of pixel data read reaches the amount of data after the current frame image data output by each of the multiple cameras is merged, the image merged data is obtained.
[0014] Preferably, for the plurality of first FIFO queues, the read clock frequency=the total number of paths of image data×the write clock frequency.
[0015] Preferably, the process of embedding the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to finally obtain the data to be calculated includes:
[0016] When acquiring each frame of sensor data, each frame of sensor data and its corresponding sensor timestamp are written into the second FIFO queue;
[0017] When the image merged data is obtained, the sensor timestamp and sensor data corresponding to the image merged data are read from the second FIFO queue, and the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data to obtain the data to be calculated.
[0018] Preferably, the process of embedding the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to obtain the data to be calculated includes:
[0019] The image timestamp, sensor timestamp and sensor data corresponding to the image merged data replace the edge position data of the image merged data to obtain data to be calculated.
[0020] Preferably, after the multiple cameras and the sensor chip enter a specified working mode, the process of respectively acquiring each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chip comprises:
[0021] Determine in advance the total frame value of the embedded sensor data corresponding to each frame of image data according to the frame rates of the output data of the multiple cameras and the frame rates of the output data of the sensor chip, and determine the target frame value of the sensor data corresponding to the output of a frame of image data based on the condition that a frame of image data can just carry the same sensor data as the total frame value;
[0022] After the multiple cameras and sensor chips enter the specified working mode, each frame of sensor data output by the sensor chip is obtained, and frame values of the obtained sensor data are accumulated;
[0023] When the frame value is equal to the target frame value, each camera is triggered to output a frame of image data, and single-frame image data output by each of the multiple cameras is obtained respectively;
[0024] When the frame value is equal to the total frame value, the accumulated frame value is cleared to re-accumulate the frame value of the acquired sensor data;
[0025] Correspondingly, the process of replacing the edge position data of the image merged data with the image timestamp, sensor timestamp and sensor data corresponding to the image merged data to obtain the data to be calculated includes:
[0026] When the frame value is equal to the total frame value, the edge position data of the image merged data is replaced by the image timestamp, sensor timestamp and sensor data corresponding to the image merged data to obtain data to be calculated.
[0027] Preferably, the multi-channel data combination method further includes:
[0028] After the data to be calculated is obtained, the data to be calculated is converted into standard MIPI format data for subsequent calculation and processing.
[0029] In order to solve the above technical problems, the present invention also provides a multi-channel data combination system, comprising:
[0030] A data acquisition module, used to respectively acquire each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chip after the multiple cameras and the sensor chip enter a specified working mode;
[0031] A timestamp generation module, used to generate an image timestamp representing the generation time of the current frame of image data when acquiring each frame of image data, and to generate a sensor timestamp representing the generation time of the current frame of sensor data when acquiring each frame of sensor data;
[0032] The data combination module is used to merge the current frame image data output by multiple cameras to obtain image merged data, and then embed the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to finally obtain the data to be calculated.
[0033] In order to solve the above technical problems, the present invention further provides an intelligent wearable device, comprising a plurality of cameras and a sensor chip, and further comprising:
[0034] The processors connected to the multiple cameras and sensor chips respectively are used to implement the steps of any of the above-mentioned multiplexed data combination methods when executing the computer program stored in the processors.
[0035] Preferably, the processor is an FPGA chip.
[0036] The present invention provides a multi-channel data combination method. After multiple cameras and sensor chips enter a specified working mode, each frame of image data output by multiple cameras and each frame of sensor data output by the sensor chip are respectively obtained; when each frame of image data is obtained, an image timestamp representing the generation time of the current frame of image data is generated, and when each frame of sensor data is obtained, a sensor timestamp representing the generation time of the current frame of sensor data is generated; the current frame of image data output by each of the multiple cameras is merged to obtain image merged data, and then the corresponding image timestamp, sensor timestamp and sensor data are embedded in the image merged data to finally obtain the data to be calculated. It can be seen that the present application obtains one channel of data for subsequent calculation and processing after merging and reorganizing multiple channels of data, thereby reducing the circuit area, reducing the system power consumption, and improving the system's anti-interference ability; moreover, the present application adds timestamps to both the image data and the sensor data, so that the image data and the sensor data are unified in time, which helps to improve the subsequent algorithm processing effect.
[0037] The present invention also provides a multi-channel data combination system and an intelligent wearable device, which have the same beneficial effects as the multi-channel data combination method described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0039] Figure 1 A flowchart of a multi-channel data combination method provided by an embodiment of the present invention;
[0040] Figure 2 A schematic diagram of a data format of image data output by a first camera provided in an embodiment of the present invention;
[0041] Figure 3 A schematic diagram of a data format of image data output by a second camera provided in an embodiment of the present invention;
[0042] Figure 4 A schematic diagram of a data format after image data output by a first camera and a second camera are combined according to an embodiment of the present invention;
[0043] Figure 5 A schematic diagram of a data format after image merging data and sensor data are reorganized according to an embodiment of the present invention;
[0044] Figure 6 A schematic diagram of a multi-channel data combination provided by an embodiment of the present invention;
[0045] Figure 7 A schematic diagram of the structure of a smart wearable device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] The core of the present invention is to provide a multi-channel data combination method, system and intelligent wearable device, which merge and reorganize the multiple channels of data to obtain one channel of data for subsequent calculation and processing, thereby reducing the circuit area, reducing system power consumption, and improving the system's anti-interference ability; moreover, the present application adds timestamps to both image data and sensor data, so that the image data and sensor data are unified in time, which helps to improve the subsequent algorithm processing effect.
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Please refer to Figure 1 , Figure 1 A flowchart of a multi-channel data combination method provided by an embodiment of the present invention.
[0049] The multi-channel data combination method is applied to a device including multiple cameras and sensor chips, such as smart wearable devices such as smart glasses, and the multi-channel data combination method includes:
[0050] Step S1: After the multiple cameras and the sensor chips enter a designated working mode, each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chips are respectively obtained.
[0051] Specifically, after the system is powered on, multiple cameras and sensor chips are powered on and activated, and the multiple cameras and sensor chips enter a specified working mode. Specifically, the multiple cameras and sensor chips can be enabled to enter a specified working mode by configuring registers of the multiple cameras and sensor chips.
[0052] After multiple cameras enter a specified working mode (such as specifying the exposure time of the camera, the frame rate of the camera output data, etc.), each frame of image data output by the multiple cameras is obtained according to the frame rate of the multiple camera output data; similarly, after the sensor chip enters a specified working mode (such as specifying the range of the sensor chip, the frame rate of the sensor chip output data, etc.), each frame of sensor data output by the sensor chip is obtained according to the frame rate of the sensor chip output data for subsequent data combination processing.
[0053] More specifically, the data output of multiple cameras and sensor chips can be automatic output or externally triggered output. For example, the data output of multiple cameras are all externally triggered output, and the specific implementation method is: according to the frame rate of the output data of the target camera (referring to any camera), a high pulse trigger signal is output to the target camera, so that the target camera outputs a complete frame of image data when receiving the high pulse trigger signal; the data output of the sensor chip is automatic output, and the specific implementation method is: the sensor chip automatically outputs a complete frame of sensor data at intervals according to the frame rate of its own output data.
[0054] It should be noted that the number of cameras in this application is multiple, that is, the device includes ≥2 cameras; the number of sensor chips can be one or more, and this application does not make any special restrictions here, and it depends on the actual situation.
[0055] Step S2: when acquiring each frame of image data, an image timestamp representing the generation time of the current frame of image data is generated; and when acquiring each frame of sensor data, a sensor timestamp representing the generation time of the current frame of sensor data is generated.
[0056] Specifically, in order to achieve the temporal unification of image data and sensor data, the present application is based on the system clock. When acquiring each frame of image data output by the camera, an image timestamp for representing the generation time of the current frame of image data is generated, that is, the generation time of the current frame of image data can be determined based on the image timestamp corresponding to the current frame of image data; similarly, when acquiring each frame of sensor data output by the sensor chip, a sensor timestamp for representing the generation time of the current frame of sensor data is generated, that is, the generation time of the current frame of sensor data can be determined based on the sensor timestamp corresponding to the current frame of sensor data.
[0057] It should be noted that when multiple cameras output image data, multiple current frame image data output by multiple cameras one by one correspond to one image timestamp, that is, one image timestamp indicates the generation time of multiple current frame image data. For example, in general, the frame rates of multiple cameras output data of a device are set to the same value, and multiple cameras are triggered to output image data at the same time. In this case, an image timestamp is generated while triggering multiple cameras to output image data, as the image timestamp corresponding to the current frame image data output by multiple cameras, that is, the generation position of the image timestamp is the first byte of the first line of each frame image data. Alternatively, the frame rates of multiple cameras output data of a device are set to the same value, but multiple cameras output image data in a sequential order. In this case, an image timestamp is generated while triggering the first camera to output image data, as the image timestamp corresponding to the current frame image data output by multiple cameras, that is, the generation position of the image timestamp is the first byte of the first line of the frame image data output earliest in each camera. Of course, the generation position of the image timestamp can also be set at other positions of the frame image data, and this application does not make any special restrictions here.
[0058] Step S3: Merge the current frame image data output by each of the multiple cameras to obtain image merged data, and then embed the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to finally obtain the data to be calculated.
[0059] Specifically, when acquiring each frame of image data output by the camera, the present application merges the current frame image data output by multiple cameras to obtain merged image data, that is, merges multiple channels of image data into one channel of image data, and then recombines the merged image data with its corresponding image timestamp, sensor timestamp and sensor data to obtain one channel of data to be calculated for subsequent calculation and processing, thereby reducing the circuit area, reducing system power consumption, and improving the system's anti-interference ability.
[0060] The present invention provides a multi-channel data combination method. After multiple cameras and sensor chips enter a specified working mode, each frame of image data output by multiple cameras and each frame of sensor data output by the sensor chip are respectively obtained; when each frame of image data is obtained, an image timestamp representing the generation time of the current frame of image data is generated, and when each frame of sensor data is obtained, a sensor timestamp representing the generation time of the current frame of sensor data is generated; the current frame of image data output by each of the multiple cameras is merged to obtain image merged data, and then the corresponding image timestamp, sensor timestamp and sensor data are embedded in the image merged data to finally obtain the data to be calculated. It can be seen that the present application obtains one channel of data for subsequent calculation and processing after merging and reorganizing multiple channels of data, thereby reducing the circuit area, reducing the system power consumption, and improving the system's anti-interference ability; moreover, the present application adds timestamps to both the image data and the sensor data, so that the image data and the sensor data are unified in time, which helps to improve the subsequent algorithm processing effect.
[0061] Based on the above embodiments:
[0062] As an optional embodiment, the process of merging the current frame image data output by multiple cameras to obtain image merged data includes:
[0063] When acquiring the frame image data output by any camera, the frame image data is converted into image parallel data, and the image parallel data is written into the first FIFO queue corresponding to the camera;
[0064] When the first FIFO queues corresponding to the multiple cameras are all non-empty queues, poll and read a pixel data in each first FIFO queue, and write it into the total FIFO queue;
[0065] Pixel data is read from the total FIFO queue, and when the amount of pixel data read reaches the amount of data after the current frame image data output by each of the multiple cameras is merged, the image merged data is obtained.
[0066] Specifically, the present application configures a FIFO (First Input First Output) queue (called the first FIFO queue) for each of the multiple cameras in advance for subsequent writing of image parallel data. When acquiring the frame image data output by any camera, the frame image data is converted into image parallel data, and then the image parallel data is written into the first FIFO queue corresponding to this camera. It should be noted that according to the data format specified by the MIPI (Mobile Industry Processor Interface) protocol, the write clock frequency of the first FIFO queue is 1 / 8 or 1 / 16 of the single-channel image data transmission frequency.
[0067] When the first FIFO queues corresponding to multiple cameras are all non-empty queues, that is, when the first FIFO queues corresponding to multiple cameras are all written with image parallel data, it means that the multi-channel frame image data meets the alignment condition, and then the frame image data in each first FIFO queue is read in a polling manner, specifically, one pixel data is read out from a first FIFO queue each time, and then written into the total FIFO queue to realize image data merging. It can be understood that starting from reading the first pixel data corresponding to the current frame image data output by multiple cameras from the total FIFO queue, when the amount of pixel data read reaches the amount of data after the current frame image data output by each of the multiple cameras is merged, the image merged data after the current frame image data output by each of the multiple cameras is merged is obtained.
[0068] For example, the device has two cameras, the first camera and the second camera. Figure 2 As shown, the image frame size output by the first camera is 640*480; Figure 3 As shown, the image frame size output by the second camera is also 640*480, that is, each frame of image data has 480 lines, each line has 640 pixels, and each pixel is represented by one byte. Figure 2 and Figure 3 In the figure, left or right is used to represent pixels, so as to facilitate observation and distinction of the positions of the original pixels after synthesis. Based on this, the image data output by the first camera and the second camera are merged by using the above-mentioned image merging method, and the following can be obtained: Figure 4 The images shown merge data from Figure 4 As can be seen in FIG, the two-way image data is merged in a pixel polling reading mode, and the merged image merged data size is 1280*480. Of course, the image frame size output by each camera in the device can also be other values, and this application does not make any special limitation here.
[0069] As an optional embodiment, for multiple first FIFO queues, the read clock frequency=the total number of paths of image data×the write clock frequency.
[0070] Specifically, for multiple first FIFO queues, the read clock frequency corresponding to polling and reading pixel data in multiple first FIFO queues = the total number of image data paths × the write clock frequency corresponding to writing image parallel data into the first FIFO queues, so that shortly after the latest frame image data output from multiple cameras is written into the corresponding first FIFO queue, the overall frame image data will be written into the total FIFO queue.
[0071] It should be noted that multiple channels of image data are merged into one channel of image data. Although the total amount of data is large, it will be read and used very quickly, so the capacity of the first FIFO queue does not need to be too large, and can be 1024*8.
[0072] As an optional embodiment, a process of embedding corresponding image timestamps, sensor timestamps, and sensor data into image merged data to finally obtain data to be calculated includes:
[0073] When acquiring each frame of sensor data, each frame of sensor data and its corresponding sensor timestamp are written into the second FIFO queue;
[0074] When the image merged data is obtained, the sensor timestamp and sensor data corresponding to the image merged data are read from the second FIFO queue, and the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data to obtain the data to be calculated.
[0075] Specifically, the present application configures a FIFO queue (called the second FIFO queue) for the sensor chip in advance for subsequent writing of sensor data and its corresponding sensor timestamp. When acquiring each frame of sensor data output by the sensor chip, each frame of sensor data and its corresponding sensor timestamp are written into the second FIFO queue. Specifically, the sensor timestamp corresponding to a frame of sensor data can be written into the second FIFO queue first, and then the frame of sensor data (including the frame of sensor data with a parity check code) is written into the second FIFO queue. The sensor timestamp corresponding to a frame of sensor data and the frame of sensor data can be collectively referred to as a frame of sensor combined data.
[0076] Based on this, when obtaining the image merged data after merging the current frame image data output by multiple cameras, the present application reads the sensor timestamp and sensor data corresponding to the image merged data from the second FIFO queue, and embeds the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to obtain the data to be calculated.
[0077] As an optional embodiment, a process of embedding corresponding image timestamps, sensor timestamps and sensor data into image merged data to obtain data to be calculated includes:
[0078] The image timestamp, sensor timestamp and sensor data corresponding to the image merged data are used to replace the edge position data of the image merged data to obtain the data to be calculated.
[0079] Specifically, reorganizing the image merged data with its corresponding image timestamp, sensor timestamp and sensor data actually means replacing part of the data in the original image merged data with the image timestamp, sensor timestamp and sensor data corresponding to the image merged data. Considering that this data replacement method will cause the image timestamp, sensor timestamp and sensor data corresponding to the image merged data to occupy the original image data, the application needs to reasonably arrange the placement of the image timestamp, sensor timestamp and sensor data to avoid these data from having a significant impact on the original image. Specifically, the image timestamp, sensor timestamp and sensor data corresponding to the image merged data can replace the edge position data of the image merged data, and the impact of these data on the original image can be ignored.
[0080] As an optional embodiment, after the multiple cameras and the sensor chips enter a specified working mode, the process of respectively acquiring each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chips includes:
[0081] Determine in advance the total frame value of the embedded sensor data corresponding to each frame of image data based on the frame rates of the output data of the multiple cameras and the frame rates of the output data of the sensor chip, and determine the target frame value of the sensor data corresponding to the output of a frame of image data based on the condition that a frame of image data can just carry the same sensor data as the total frame value;
[0082] After the multiple cameras and sensor chips enter the specified working mode, each frame of sensor data output by the sensor chip is obtained, and the frame values of the obtained sensor data are accumulated;
[0083] When the frame value is equal to the target frame value, each camera is triggered to output a frame of image data, and the single frame image data output by each of the multiple cameras is obtained respectively;
[0084] When the frame value is equal to the total frame value, the accumulated frame value is cleared to re-accumulate the frame value of the acquired sensor data;
[0085] Accordingly, the process of replacing the edge position data of the image merged data with the image timestamp, sensor timestamp and sensor data corresponding to the image merged data to obtain the data to be calculated includes:
[0086] When the frame value is equal to the total frame value, the edge position data of the image merged data is replaced by the image timestamp, sensor timestamp and sensor data corresponding to the image merged data to obtain the data to be calculated.
[0087] Specifically, in general, the frame rate of camera output data is slower, while the frame rate of sensor chip output data is faster, so multiple frames of sensor data will be embedded in one frame of image data. The specific method for determining the total frame value of embedded sensor data is as follows (the frame rates of multiple camera output data are the same): divide the frame rate of sensor chip output data by the frame rate of camera output data, and the integer part of the calculated result is the total frame value of sensor data embedded in each frame of image data. For example, if the frame rate of camera output data is 40fps and the frame rate of sensor chip output data is 400fps, then each frame of image data must be embedded with 10 frames of sensor data, that is, 10 frames of sensor combination data.
[0088] At the same time, considering that the camera does not output frame image data continuously, there will be some time intervals between adjacent frame image data. Therefore, in order to ensure that a frame image data can just carry the same sensor data as the total frame value, this application needs to determine in advance the target frame value of the sensor data corresponding to when a frame image data is just output, based on the condition that a frame image data can just carry the same sensor data as the total frame value, so as to provide a reference for subsequent triggering of image data output.
[0089] Based on this, after multiple cameras and sensor chips enter the specified working mode, each frame of sensor data output by the sensor chip is obtained according to the frame rate of the sensor chip output data, and the frame value of the obtained sensor data is accumulated. When the frame value is equal to the target frame value, it means that triggering the camera to output frame image data at this time can ensure that one frame of image data can just carry the same sensor data as the total frame value, then trigger each camera to output one frame of image data, and obtain the single frame of image data output by each of the multiple cameras. When the frame value is equal to the total frame value, it means that the multi-frame sensor combination data (sensor timestamp + sensor data) to be embedded in one frame of image data has been collected, then the image timestamp and multi-frame sensor combination data corresponding to the image merged data replace the edge position data of the image merged data to obtain the data to be calculated. In addition, it can be understood that when the frame value is equal to the total frame value, the accumulated frame value needs to be cleared to lay the foundation for triggering each camera to output the next frame of image data, thereby ensuring that each frame of image data can just carry the same sensor data as the total frame value.
[0090] For example, if each frame of image data is to be embedded with 10 frames of sensor data, and according to actual experience, when the target frame value is equal to 3 (the specific value can be flexibly adjusted according to the actual application), it can be ensured that one frame of image data can carry exactly 10 frames of sensor data. Then the multi-channel data output and reorganization process is as follows: after multiple cameras and sensor chips enter the specified working mode, obtain each frame of sensor data output by the sensor chip, and accumulate the frame values of the obtained sensor data. When the frame value is equal to 3, trigger each camera to output a frame of image data, and obtain the single frame image data output by each of the multiple cameras. When the frame value is equal to 10, clear the accumulated frame value, and replace the edge position data of the image merged data with the image timestamp corresponding to the image merged data and the 10-frame sensor combination data to obtain the data to be calculated.
[0091] More specifically, the present application may replace the last row of pixel data of the image merged data with the image timestamp, sensor timestamp and sensor data corresponding to the image merged data, such as Figure 5 As shown, the image timestamp is placed in the first 4 bytes of the last line of the same frame of image data, and the sensor timestamp and sensor data are placed after the image timestamp. Of course, the image timestamp, sensor timestamp and sensor data can also be placed in other positions of the image merged data, and this application does not make any special restrictions here.
[0092] As an optional embodiment, the multi-channel data combination method further includes:
[0093] After the data to be calculated is obtained, the data to be calculated is converted into standard MIPI format data for subsequent calculation and processing.
[0094] Furthermore, after obtaining the data to be calculated, the present application can convert the data to be calculated into standard MIPI format data for subsequent calculation and processing. The reason for converting the data to be calculated into standard MIPI format data is that the power consumption of this data conversion is the smallest and the circuit is the simplest. Of course, the present application can also convert the data to be calculated into data in other formats, and the present application is not particularly limited here.
[0095] In summary, the above-mentioned multi-channel data combination method can be implemented by programming the processor of the device. The prior art devices usually use MCU (Micro controller Unit) chips as processors, but the time accuracy of MCU chips is low and the algorithm adjustment is not flexible enough. Therefore, the device of this application uses FPGA (Field-Programmable Gate Array) chips with high time accuracy and flexible algorithm adjustment as processors. Its functional principle is as follows: Figure 6As shown: the camera power-on and register configuration module is used to complete the power-on and register configuration of multiple cameras; multiple MIPI parsing modules are used to complete the parsing of the image data output by multiple cameras one by one, that is, to convert the input MIPI serial image data into image parallel data for subsequent image merging operations; multiple sensor chip power-on and register configuration modules are used to complete the power-on and register configuration of multiple sensor chips one by one, and to read sensor data from the sensor chip at a regular interval; FIFO_1 to FIFO_N and FIFO_N+1 to FIFO_N+M are used to cache image parallel data and sensor data; the multi-channel data merging module is used to merge multiple channels of image data into one channel of image data to obtain image merged data; the image timestamp and sensor data adding module is used to embed the image timestamp and sensor combination data into the image merged data at an appropriate position to achieve data reorganization; the MIPI sending module is used to convert the reorganized parallel data into MIPI format output for subsequent calculation processing. More specifically, the FPGA chip can be respectively connected through multiple groups of different I 2 C bus to configure registers for multiple camera and sensor chips.
[0096] It can be seen that the circuit structure of the present application is simple, only using the MIPI interface and the IIC interface, with high reliability, and provides convenience for programming to a certain extent. In addition, the functions of the circuit of the present application can be flexibly configured and modified, for example, the working parameters of the external camera and sensor chip can be flexibly modified according to actual needs, and the data reorganization method can also be flexibly modified according to the needs of the back-end algorithm.
[0097] The present application also provides a multi-channel data combination system, comprising:
[0098] A data acquisition module, used to respectively acquire each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chip after the multiple cameras and the sensor chip enter a specified working mode;
[0099] A timestamp generation module, used to generate an image timestamp representing the generation time of the current frame of image data when acquiring each frame of image data, and to generate a sensor timestamp representing the generation time of the current frame of sensor data when acquiring each frame of sensor data;
[0100] The data combination module is used to merge the current frame image data output by multiple cameras to obtain image merged data, and then embed the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to finally obtain the data to be calculated.
[0101] For an introduction to the multi-channel data combination system provided in the present application, please refer to the above-mentioned embodiment of the multi-channel data combination method, and the present application will not go into details here.
[0102] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a smart wearable device provided in an embodiment of the present invention.
[0103] The smart wearable device includes a plurality of cameras 1 and a sensor chip 2, and further includes:
[0104] The processor 3 connected to the plurality of cameras 1 and the sensor chips 2 respectively is used to implement the steps of any one of the above-mentioned multiplexed data combination methods when executing the computer program stored in the processor 3.
[0105] As an optional embodiment, the processor 3 is a FPGA chip.
[0106] For an introduction to the smart wearable device provided in this application, please refer to the above-mentioned embodiment of the multi-channel data combination method, and this application will not go into details here.
[0107] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0108] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for combining multiple data channels, It is characterized in that Applies to devices containing multiple camera and sensor chips, including: After the multiple cameras and the sensor chips enter the specified working mode, each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chips are respectively obtained; When acquiring each frame of image data, an image timestamp representing the generation time of the current frame of image data is generated, and when acquiring each frame of sensor data, a sensor timestamp representing the generation time of the current frame of sensor data is generated; The current frame image data outputted by the multiple cameras are merged to obtain the image merged data, and then the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data, so as to finally obtain a channel of data to be calculated for subsequent calculation and processing; After obtaining the data to be calculated, converting the data to be calculated into standard MIPI format data, and outputting the data to be calculated after conversion for subsequent calculation processing; After the multiple cameras and the sensor chips enter the specified working mode, the process of respectively acquiring each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chips includes: Determine in advance the total frame value of the embedded sensor data corresponding to each frame of image data according to the frame rates of the output data of the multiple cameras and the frame rates of the output data of the sensor chip, and determine the target frame value of the sensor data corresponding to the output of a frame of image data based on the condition that a frame of image data can just carry the same sensor data as the total frame value; After the multiple cameras and sensor chips enter the specified working mode, each frame of sensor data output by the sensor chip is obtained, and frame values of the obtained sensor data are accumulated; When the frame value is equal to the target frame value, each camera is triggered to output a frame of image data, and single-frame image data output by each of the multiple cameras is obtained respectively; When the frame value is equal to the total frame value, the accumulated frame value is cleared to re-accumulate the frame value of the acquired sensor data; Correspondingly, the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data, and finally a process of obtaining a path of data to be calculated for subsequent calculation processing includes: When the frame value is equal to the total frame value, the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data, and finally a channel of data to be calculated for subsequent calculation processing is obtained.
2. The multi-channel data combination method as claimed in claim 1, It is characterized in that The process of merging the current frame image data output by multiple cameras to obtain image merged data includes: When acquiring the frame image data output by any camera, the frame image data is converted into image parallel data, and the image parallel data is written into the first FIFO queue corresponding to the camera; When the first FIFO queues corresponding to the multiple cameras are all non-empty queues, poll and read a pixel data in each first FIFO queue, and write it into the total FIFO queue; Pixel data is read from the total FIFO queue, and when the amount of pixel data read reaches the amount of data after the current frame image data output by each of the multiple cameras is merged, the image merged data is obtained.
3. The multi-channel data combination method as claimed in claim 2, It is characterized in that For multiple first FIFO queues, the read clock frequency = the total number of image data channels × the write clock frequency.
4. The multi-channel data combination method as claimed in claim 1, It is characterized in that The process of embedding the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to finally obtain the data to be calculated includes: When acquiring each frame of sensor data, each frame of sensor data and its corresponding sensor timestamp are written into the second FIFO queue; When the image merged data is obtained, the sensor timestamp and sensor data corresponding to the image merged data are read from the second FIFO queue, and the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data to obtain the data to be calculated.
5. The multi-channel data combination method as claimed in claim 4, It is characterized in that The process of embedding the corresponding image timestamp, sensor timestamp and sensor data into the image merged data to obtain the data to be calculated includes: The image timestamp, sensor timestamp and sensor data corresponding to the image merged data replace the edge position data of the image merged data to obtain data to be calculated.
6. A multi-channel data combination system, It is characterized in that include: A data acquisition module, used to respectively acquire each frame of image data output by the multiple cameras and each frame of sensor data output by the sensor chip after the multiple cameras and the sensor chip enter a specified working mode; A timestamp generation module, used to generate an image timestamp representing the generation time of the current frame of image data when acquiring each frame of image data, and to generate a sensor timestamp representing the generation time of the current frame of sensor data when acquiring each frame of sensor data; A data combination module is used to combine the current frame image data output by multiple cameras to obtain image combined data, and then embed the corresponding image timestamp, sensor timestamp and sensor data into the image combined data, and finally obtain a channel of data to be calculated for subsequent calculation processing; The MIPI sending module is used to convert the data to be calculated into standard MIPI format data after obtaining the data to be calculated, and output the data to be calculated after conversion for subsequent calculation processing; The data acquisition module is further used for: Determine in advance the total frame value of the embedded sensor data corresponding to each frame of image data according to the frame rates of the output data of the multiple cameras and the frame rates of the output data of the sensor chip, and determine the target frame value of the sensor data corresponding to the output of a frame of image data based on the condition that a frame of image data can just carry the same sensor data as the total frame value; After the multiple cameras and sensor chips enter the specified working mode, each frame of sensor data output by the sensor chip is obtained, and frame values of the obtained sensor data are accumulated; When the frame value is equal to the target frame value, each camera is triggered to output a frame of image data, and single-frame image data output by each of the multiple cameras is obtained respectively; When the frame value is equal to the total frame value, the accumulated frame value is cleared to re-accumulate the frame value of the acquired sensor data; Correspondingly, the data combination module is also used for: When the frame value is equal to the total frame value, the corresponding image timestamp, sensor timestamp and sensor data are embedded into the image merged data, and finally a channel of data to be calculated for subsequent calculation processing is obtained.
7. A smart wearable device, It is characterized in that Includes multiple camera and sensor chips, as well as: A processor connected to the multiple cameras and sensor chips respectively, used to implement the steps of the multi-channel data combination method as described in any one of claims 1-5 when executing the computer program stored in the processor.
8. The smart wearable device according to claim 7, It is characterized in that The processor is an FPGA chip.
Citation Information
Patent Citations
Wearable equipment and method for determining user displacement in wearable equipment
CN106546238A
Data processing method applied to robot, device, equipment and medium
CN109275048A
High-speed data collection device and method based on FPGA
CN111343379A
