Multi-device data synchronization method and device of RGB-D camera, terminal and medium

Abstract

The invention provides a multi-device data synchronization method and device for an RGB-D camera, a terminal and a medium. The method comprises the following steps: determining a fixed processing step length of a depth sensor in the RGB-D camera about a depth image data calculation process; adjusting the processing process of the RGB image data in the RGB-D camera according to the fixed processing step length so as to control the assembly line delay of the RGB data processing process and the depth image data calculation process to be equal; and acquiring the state of the external inertial measurement unit according to the field synchronization signal of the image sensor acquired in real time so as to control the RGB-D camera and the external inertial measurement unit to keep synchronous in time. According to the method, the assembly line delays of RGB data processing and depth data calculation are equal, and the state of the external inertial measurement unit is acquired according to the field synchronization signal of the image sensor, so that the time synchronization of the image sensor, the depth sensor and the external inertial measurement unit of the RGB-D camera is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization processing, and in particular to a multi-device data synchronization method, device, terminal and medium for an RGB-D camera. Background Art

[0002] At present, the RGB camera, depth sensor, and external IMU (Inertial Measurement Unit) in an RGB-D (Red Green Blue-Depth) camera usually have different hardware delays, resulting in inconsistencies in the exposure time of the image sensor, the laser emission and reception time of the depth sensor, and the sampling time of the IMU. In other words, these hardware delays will cause the data generated by the sensors to be misaligned in time, and different sensors usually have different sampling rates. For example, RGB cameras usually capture images at a fixed frame rate, while IMUs may collect acceleration and angular velocity data at different frequencies, resulting in asynchronous sampling rates, which causes the data they generate to be unable to fully match in time. In addition, due to measurement errors in the calibration process, inherent uncertainties in the equipment, and approximations in the calibration algorithm, calibration errors may still exist even if these hardware are calibrated in advance.

[0003] In summary, the existing technology usually solves the problem of time misalignment of data generated by multiple sensors based on prior calibration. However, due to hardware delays, different sensor sampling rates, etc., there may still be time differences after calibration. Therefore, how to ensure that the data generated by the RGB camera, depth sensor, and external IMU device of the RGB-D camera can be synchronized in time is a problem to be solved by those skilled in the art. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a multi-device data synchronization method, apparatus, terminal and medium of an RGB-D camera are provided, aiming to solve the problem of time misalignment of data generated by the RGB camera, depth sensor, and external IMU device of the RGB-D camera in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows: A multi-device data synchronization method for an RGB-D camera, wherein the method comprises: Determine a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; Adjusting the processing process of the image signal processor in the RGB-D camera regarding the RGB image data according to the fixed processing step size corresponding to the calculation process of the depth image data, so as to control the pipeline delay corresponding to the processing process of the RGB image data to be equal to the pipeline delay corresponding to the calculation process of the depth image data; During the processing of the RGB image data and the calculation of the depth image data, a field synchronization signal corresponding to the image sensor in the RGB-D camera is acquired in real time; The state of an external inertial measurement unit is collected according to the field synchronization signal corresponding to the image sensor to control the RGB-D camera to maintain time synchronization with the external inertial measurement unit.

[0006] In one implementation, the RGB image data is processed based on a binocular stereo matching algorithm to calculate the depth image data, including: Acquire the RGB image data of the left and right sides of the RGB-D camera to obtain the left RGB image data and the right RGB image data; Comparing each pixel of the left RGB image data and the right RGB image data, and calculating a matching cost between a pixel in the left RGB image data and a corresponding pixel in the right RGB image data; A corresponding disparity value is calculated based on the matching cost, and depth data of each pixel is calculated according to the acquired camera parameters and the disparity value.

[0007] In one implementation, the calculating the corresponding disparity value based on the matching cost includes: A minimum matching cost is determined, and a corresponding disparity value is calculated according to the minimum matching cost.

[0008] In one implementation, calculating the depth data of each pixel according to the acquired camera parameters and the disparity value includes: The acquired camera parameters and the disparity value are calculated using a preset triangulation method to determine the depth data of each pixel.

[0009] In one implementation, during the processing of the RGB image data, the image signal processor in the RGB-D camera performs processing operations on the raw image signal captured from the image sensor including: black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction, and high dynamic processing.

[0010] In one implementation, the multi-device data synchronization method of the RGB-D camera further includes: Real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes; When it is detected that the field synchronization signal corresponding to the image sensor changes, a new field scan is triggered.

[0011] In one implementation, the real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes includes: Real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image; When the field synchronization signal corresponding to the image sensor is detected to change, a new field scan is triggered, including: When it is detected that the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image, a new field scan is triggered.

[0012] The present invention also discloses a multi-device data synchronization device for an RGB-D camera, wherein the device comprises: A step length determination module is used to determine a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; a pipeline adjustment module, configured to adjust a processing process of the image signal processor in the RGB-D camera with respect to the RGB image data according to the fixed processing step size corresponding to the calculation process of the depth image data, so as to control a pipeline delay corresponding to the processing process of the RGB image data to be equal to a pipeline delay corresponding to the calculation process of the depth image data; A signal acquisition module, used for acquiring, in real time, a field synchronization signal corresponding to the image sensor in the RGB-D camera during the processing of the RGB image data and the calculation of the depth image data; The state acquisition module is used to acquire the state of the external inertial measurement unit according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera to keep synchronization with the external inertial measurement unit in time.

[0013] The present invention also discloses a terminal, which includes: a memory, a processor, and a multi-device data synchronization program of an RGB-D camera stored in the memory and executable on the processor. When the multi-device data synchronization program of the RGB-D camera is executed by the processor, the steps of the multi-device data synchronization method of the RGB-D camera as described above are implemented.

[0014] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the multi-device data synchronization method of the RGB-D camera as described above.

[0015] The present invention provides a multi-device data synchronization method, apparatus, terminal and medium for an RGB-D camera. The multi-device data synchronization method for an RGB-D camera comprises: determining a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; adjusting the processing process of the image signal processor in the RGB-D camera on the RGB image data according to the fixed processing step length corresponding to the calculation process of the depth image data, so as to control the pipeline delay corresponding to the processing process of the RGB image data to be equal to the pipeline delay corresponding to the calculation process of the depth image data; acquiring a field synchronization signal corresponding to the image sensor in the RGB-D camera in real time during the processing process of the RGB image data and the calculation process of the depth image data; and acquiring a state of an external inertial measurement unit according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera to be synchronized with the external inertial measurement unit in time. It can be seen from this that the present invention achieves pixel-level information synchronization of RGB image data and depth image data in an RGB-D camera by making the pipeline delay of RGB image data processing equal to the pipeline delay of calculating depth image data, thereby obtaining spatially aligned RGB image data and depth image data, and acquires the state of an external inertial measurement unit according to a field synchronization signal of the image sensor, thereby achieving synchronization of the RGB-D camera and the external inertial measurement unit, thereby obtaining temporally aligned RGB-D data and IMU data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a preferred embodiment of the multi-device data synchronization method of the RGB-D camera of the present invention; Figure 2 is a schematic diagram of pipeline delay in the present invention; Figure 3 It is a schematic diagram of IMU synchronization in the present invention; Figure 4 It is a functional principle block diagram of a preferred embodiment of the multi-device data synchronization device for RGB-D cameras in the present invention; Figure 5 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] See also Figure 1 , Figure 1 : is a flow chart of the multi-device data synchronization method of the RGB-D camera in the present invention. Figure 1 As shown, the multi-device data synchronization method of the RGB-D camera according to the embodiment of the present invention includes: Step S11, determining a fixed processing step size of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data, and an image signal processor in the RGB-D camera processes an original image signal captured from the image sensor to obtain RGB image data.

[0019] It can be understood that the depth information (that is, the depth image data) is calculated based on the binocular stereo matching algorithm, that is, the RGB image data is processed based on the binocular stereo matching algorithm to calculate the depth image data, and the image signal processor in the RGB-D camera processes the original image signal captured from the image sensor to obtain the RGB image data, and then the result of the processing by the image signal processor, that is, the RGB image data, is used as the input of the binocular stereo matching algorithm to calculate the depth image data, and the depth image data calculation process is a fixed processing step.

[0020] It should be pointed out that the RAW format data of the Camera can be processed into RGB format data through the Image Signal Processor (ISP), and the ISP is a key component in the field of digital image processing. It is responsible for performing various processing on the original image signal captured from the image sensor to produce the final high-quality image. That is to say, the Image Signal Processor in the RGB-D camera processes the original image signal captured from the image sensor to obtain RGB image data.

[0021] In the process of processing the RGB image data based on the binocular stereo matching algorithm to calculate the depth image data, it can specifically include: acquiring the RGB image data of the left and right sides of the RGB-D camera to obtain the left RGB image data and the right RGB image data; comparing each pixel of the left RGB image data and the right RGB image data, and calculating the matching cost between the pixel in the left RGB image data and the corresponding pixel in the right RGB image data; calculating the corresponding disparity value based on the matching cost, and calculating the depth data of each pixel according to the acquired camera parameters and the disparity value.

[0022] And the calculation of the disparity value based on the matching cost, and the calculation of the corresponding disparity value based on the matching cost, and then calculating the depth data of each pixel according to the acquired camera parameters and the disparity value can specifically include: determining the minimum matching cost, and calculating the corresponding disparity value according to the minimum matching cost, and then calculating the acquired camera parameters and the disparity value using a preset triangulation method to determine the depth data of each pixel.

[0023] It should be pointed out that common binocular stereo matching algorithms include but are not limited to disparity map algorithms, dynamic programming algorithms, BF (Belief Propagation) algorithms, etc., and the disparity map algorithm can also be SAD (Sum of Absolute Differences), SSD (Single Shot MultiBox Detector), SGM (Semi-Global Matching), NCC (Normalized Cross Correlation). These algorithms may differ in matching cost calculation and aggregation steps, but the basic idea is similar, that is, by comparing each pixel in the left and right cameras, and calculating the matching cost between the left image and the pixel in the right image in the corresponding row, after the matching cost is calculated, a minimum matching cost is selected for each pixel to calculate the corresponding disparity value, and then the triangulation method is used to calculate the depth information of each pixel based on the disparity value and the known camera parameters.

[0024] Step S12: adjusting the processing process of the image signal processor in the RGB-D camera regarding the RGB image data according to the fixed processing step size corresponding to the calculation process of the depth image data, so as to control the pipeline delay corresponding to the processing process of the RGB image data to be equal to the pipeline delay corresponding to the calculation process of the depth image data.

[0025] In this embodiment, after determining the fixed processing step size of the depth sensor in the RGB-D camera for the depth image data calculation process, the image signal processor in the RGB-D camera adjusts the processing process of the RGB image data according to the fixed processing step size. It can be understood that the result processed by the image signal processor is used as the input of the binocular stereo matching algorithm. Therefore, in order to achieve pixel-level information synchronization between the RGB image data and the depth image data in the RGB-D camera, it is necessary to add an additional pipeline equivalent to the binocular stereo matching algorithm process in the processing of the RGB image data. In other words, the RGB data of the image sensor can also be synchronized with the depth image data after adding a fixed delay. The fixed delay represents the fixed pipeline clock cycle of stereo matching and depth calculation. See Figure 2 As shown, the processing process of the image signal processor in the RGB-D camera on the RGB image data is adjusted according to the fixed processing step size of the depth image data calculation process, so that the pipeline delay of the RGB image data processing and the pipeline delay of the depth image data calculation are equal, thereby realizing the synchronization of the pixel-level information of the RGB image data and the depth image data in the RGB-D camera, thereby obtaining the spatially aligned RGB image data and the depth image data, wherein, in the Figure 2 In the figure, T1 represents the sum of the pipeline processing cycles of each processing unit in image processing, T2 represents the sum of the pipeline processing cycles of binocular stereo matching and depth calculation, and T2 (RGB-Delay) is the cycle number of only transmitting RGB data (without processing).

[0026] It should be pointed out that pipeline delay includes the processing time of each stage and the transfer time between stages, where processing time refers to the number of clock cycles required for each stage to complete its specific task, and transfer time refers to the time required to transfer the result from one stage to the next.

[0027] In the process of processing the RGB image data, the image signal processor in the RGB-D camera performs processing operations on the original image signal captured from the image sensor, including: black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction and high dynamic processing. That is to say, the main image processing functions of the image sensor are: black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction, and high dynamic processing, wherein each part of the operation is a fixed pipeline delay.

[0028] Step S13: During the processing of the RGB image data and the calculation of the depth image data, a field synchronization signal corresponding to the image sensor in the RGB-D camera is acquired in real time.

[0029] It should be pointed out that in the field of image sensors, the field synchronization signal is a signal used to synchronize the acquisition timing of the image sensor. The field synchronization signal can ensure that the image sensor starts to collect pixel data at a specific position in the image frame, thereby coordinating image acquisition between multiple sensors or devices. The field synchronization signal is usually generated by the electronic circuit inside the image sensor or camera system. The field synchronization signal marks the beginning of an image field (frame), usually all the rows contained in a complete image frame.

[0030] In this embodiment, during the processing of RGB image data and the calculation of depth image data, a field synchronization signal corresponding to the image sensor in the RGB-D camera is acquired in real time.

[0031] It is also possible to monitor in real time whether the field synchronization signal corresponding to the image sensor changes; when the field synchronization signal corresponding to the image sensor changes, a new field scan is triggered. Specifically, it is possible to monitor in real time whether the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image; when the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image, a new field scan is triggered. It is understandable that the field synchronization signal is also called the vertical synchronization signal (VSYNC signal), see Figure 3 As shown, the VSYNC signal changes in the vertical direction of the image, indicating the beginning of a field or a frame. When the sensor scans to the bottom of the image, the VSYNC signal changes, telling the image acquisition system to start a new field scan.

[0032] Step S14: collecting the state of an external inertial measurement unit according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera to maintain time synchronization with the external inertial measurement unit.

[0033] It should be pointed out that after the field synchronization signal corresponding to the image sensor is valid, it means that a frame of the image is about to come. After the row synchronization indicates that the image data is valid, image processing begins, such as the black level correction mentioned above, and then binocular stereo matching is performed and the depth image data is calculated through the disparity value. At this time, the field synchronization signal when the depth image data is transmitted is composed of the field synchronization signal of the image sensor plus a fixed delay, and the fixed delay is the fixed processing cycle number of the pipeline of the processing unit, and the data acquisition frequency of the inertial measurement unit is usually expressed in Hertz (Hz), indicating the number of data points collected per second. The inertial measurement unit usually measures acceleration and angular velocity, and the data acquisition frequency set when measuring acceleration and angular velocity is usually several hundred Hz or several thousand Hz, or a higher data acquisition frequency is set, so the field synchronization signal of the depth camera is the ACK signal of the external inertial measurement unit.

[0034] In this embodiment, during the processing of RGB image data and the calculation of depth image data, a field synchronization signal corresponding to the image sensor in the RGB-D camera is obtained in real time, and the state of the external inertial measurement unit is acquired according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera and the external inertial measurement unit to keep synchronization in time, and achieve synchronization between the RGB-D camera and the external inertial measurement unit, so as to obtain time-aligned RGB-D data and IMU data, that is, the information synchronization of RGB (image sensor), Depth (depth sensor), and IMU of the RGB-D camera is achieved. Among them, keeping the RGB-D camera synchronized with the external inertial measurement unit is a common task in the fields of computer vision and robotics. The synchronization is to ensure that the data generated by the two devices are consistent in time, so as to obtain time-aligned RGB-D data and IMU data, so as to facilitate data fusion and joint processing.

[0035] It should also be noted that in addition to achieving synchronization between the RGB-D camera and the external inertial measurement unit through the technical solution of the present application, it is also possible to introduce an event-based synchronization mechanism, that is, trigger data collection when a specific event occurs, thereby avoiding relying solely on fixed time synchronization, or adopt a unified time synchronization protocol so that the RGB camera, depth sensor and external IMU device can all participate in and maintain global time synchronization.

[0036] It can be seen that in the embodiment of the present invention, by making the pipeline delay of RGB image data processing and the pipeline delay of calculating depth image data equal, the pixel-level information synchronization of RGB image data and depth image data in the RGB-D camera is achieved, so as to obtain spatially aligned RGB image data and depth image data, and the state of the external inertial measurement unit is collected according to the field synchronization signal of the image sensor, so as to achieve synchronization between the RGB-D camera and the external inertial measurement unit, so as to obtain temporally aligned RGB-D data and IMU data.

[0037] In one embodiment, if Figure 4 As shown, based on the above-mentioned multi-device data synchronization method of RGB-D camera, the present invention also provides a multi-device data synchronization device of RGB-D camera, including: The step length determination module 100 is used to determine a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; A pipeline adjustment module 200, configured to adjust the processing process of the image signal processor in the RGB-D camera with respect to the RGB image data according to the fixed processing step corresponding to the calculation process of the depth image data, so as to control the pipeline delay corresponding to the processing process of the RGB image data to be equal to the pipeline delay corresponding to the calculation process of the depth image data; A signal acquisition module 300, used for acquiring in real time a field synchronization signal corresponding to the image sensor in the RGB-D camera during the processing of the RGB image data and the calculation of the depth image data; The state acquisition module 400 is used to acquire the state of the external inertial measurement unit according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera to keep synchronization with the external inertial measurement unit in time.

[0038] Figure 5 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include: A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0039] When the processor 502 executes the program, the multi-device data synchronization method of the RGB-D camera provided in the above embodiment is implemented.

[0040] Furthermore, the terminal further includes: The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0041] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0042] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0043] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0044] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0045] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0046] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the multi-device data synchronization method of the RGB-D camera as described above is implemented.

[0047] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0048] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0049] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0050] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0051] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0052] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0053] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0054] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0055] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-device data synchronization method for an RGB-D camera, characterized in that: The method comprises: Determine a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; Adjusting the processing process of the image signal processor in the RGB-D camera regarding the RGB image data according to the fixed processing step size corresponding to the calculation process of the depth image data, so as to control the pipeline delay corresponding to the processing process of the RGB image data to be equal to the pipeline delay corresponding to the calculation process of the depth image data; During the processing of the RGB image data and the calculation of the depth image data, a field synchronization signal corresponding to the image sensor in the RGB-D camera is acquired in real time; The state of an external inertial measurement unit is collected according to the field synchronization signal corresponding to the image sensor to control the RGB-D camera to maintain time synchronization with the external inertial measurement unit.

2. The multi-device data synchronization method of the RGB-D camera according to claim 1, characterized in that: The RGB image data is processed based on the binocular stereo matching algorithm to calculate the depth image data, including: Acquire the RGB image data of the left and right sides of the RGB-D camera to obtain the left RGB image data and the right RGB image data; Comparing each pixel of the left RGB image data and the right RGB image data, and calculating a matching cost between a pixel in the left RGB image data and a corresponding pixel in the right RGB image data; A corresponding disparity value is calculated based on the matching cost, and depth data of each pixel is calculated according to the acquired camera parameters and the disparity value.

3. The multi-device data synchronization method of the RGB-D camera according to claim 2, characterized in that: The calculating the corresponding disparity value based on the matching cost includes: A minimum matching cost is determined, and a corresponding disparity value is calculated according to the minimum matching cost.

4. The multi-device data synchronization method of the RGB-D camera according to claim 2, characterized in that: The calculating the depth data of each pixel according to the acquired camera parameters and the disparity value includes: The acquired camera parameters and the disparity value are calculated using a preset triangulation method to determine the depth data of each pixel.

5. The multi-device data synchronization method of the RGB-D camera according to claim 1, characterized in that: During the processing of the RGB image data, the image signal processor in the RGB-D camera performs processing operations on the raw image signal captured from the image sensor including: black level correction, white balance, demosaicing, color correction, denoising, sharpening, exposure control, autofocus, epipolar correction, lens shading correction, saturation adjustment, tone mapping, gamma correction, and high dynamic processing.

6. The multi-device data synchronization method for RGB-D cameras according to any one of claims 1 to 5, characterized in that: Also includes: Real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes; When it is detected that the field synchronization signal corresponding to the image sensor changes, a new field scan is triggered.

7. The multi-device data synchronization method of the RGB-D camera according to claim 6, characterized in that: The real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes includes: Real-time monitoring of whether the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image; When the field synchronization signal corresponding to the image sensor is detected to change, a new field scan is triggered, including: When it is detected that the field synchronization signal corresponding to the image sensor changes in the vertical direction of the image, a new field scan is triggered.

8. A multi-device data synchronization device for an RGB-D camera, characterized in that: The device comprises: A step length determination module is used to determine a fixed processing step length of a process in which a depth sensor in an RGB-D camera processes RGB image data based on a binocular stereo matching algorithm to calculate depth image data; the RGB image data is image data obtained by an image signal processor in the RGB-D camera processing an original image signal captured from an image sensor; a pipeline adjustment module, configured to adjust a processing process of the image signal processor in the RGB-D camera with respect to the RGB image data according to the fixed processing step size corresponding to the calculation process of the depth image data, so as to control a pipeline delay corresponding to the processing process of the RGB image data to be equal to a pipeline delay corresponding to the calculation process of the depth image data; A signal acquisition module, used for acquiring, in real time, a field synchronization signal corresponding to the image sensor in the RGB-D camera during the processing of the RGB image data and the calculation of the depth image data; The state acquisition module is used to acquire the state of the external inertial measurement unit according to the field synchronization signal corresponding to the image sensor, so as to control the RGB-D camera to keep synchronization with the external inertial measurement unit in time.

9. A terminal, characterized in that: include: A memory, a processor, and a multi-device data synchronization program for an RGB-D camera stored in the memory and executable on the processor, wherein when the multi-device data synchronization program for the RGB-D camera is executed by the processor, the steps of the multi-device data synchronization method for the RGB-D camera as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the multi-device data synchronization method for an RGB-D camera as described in any one of claims 1 to 7.

Images