Data synchronization processing method and device, medium and equipment
By calibrating the timestamps of infrared and visible light sensors in electrical appliances, determining the clock offset parameters and synchronization time points, and dynamically adjusting the buffer capacity, the latency problem in multi-sensor data synchronization acquisition is solved, and accurate data alignment and synchronous display are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO FOTILE KITCHEN WARE CO LTD
- Filing Date
- 2026-01-07
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, data misalignment issues arise from time delays during synchronous acquisition of multi-sensor data. This is especially true when integrating infrared sensors and visible light cameras into electrical appliances, where delays in data acquisition, transmission, and processing affect the accuracy of multi-sensor data fusion and synchronous display.
By acquiring visible light and infrared data frame sequences from a dynamic buffer, calibrating their timestamps, determining clock offset parameters for time offset compensation, and determining a reference synchronization time point and dynamic time window based on the clock offset parameters, the time-synchronized data frames are selected for display processing. The buffer capacity is dynamically adjusted to cope with network fluctuations and changes in processing resources.
It effectively eliminated the systematic deviation of dual-channel data, avoided data misalignment, and achieved precise alignment of visible light data and infrared data in terms of time and data, providing a reliable data foundation for subsequent synchronous display.
Smart Images
Figure CN122027752A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, specifically to data synchronization processing methods, apparatus, media, and equipment. Background Technology
[0002] To better demonstrate the operating status of electrical appliances, infrared sensors and visible light cameras can be integrated inside the appliances. During operation, infrared and visible light data are collected in real time and converted into images for synchronous display. However, delays occur at each stage of data acquisition, transmission, and processing, significantly impacting the fusion or separate synchronous display of multi-sensor data.
[0003] In related technologies, the synchronous acquisition of multiple sensor sides is achieved only through hardware pulses, and multi-channel data is simply aligned by timestamps, resulting in significant data misalignment. Summary of the Invention
[0004] To achieve more accurate alignment of multi-channel data, this application provides a data synchronization processing method, apparatus, medium, and device. The technical solution is as follows: Firstly, this application provides a data synchronization processing method, the method comprising: Visible light data frame sequences and infrared data frame sequences are obtained from a dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status. The timestamps of the infrared data frame sequence and the visible light data frame sequence are calibrated to determine the clock offset parameters; Based on the clock offset parameter, the infrared data frame sequence is subjected to time offset compensation processing to obtain the target infrared data frame sequence; Based on the timestamps of the target infrared data frame sequence, multiple reference synchronization time points are determined; Determine the dynamic time window corresponding to each of the plurality of reference synchronization time points; Based on the dynamic time window corresponding to each reference synchronization time point, a first target data frame is selected from the visible light data frame sequence and a second target data frame is selected from the target infrared data frame sequence, which are used as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
[0005] Optionally, determining the dynamic time window corresponding to each of the plurality of reference synchronization time points includes: A reference synchronization period is determined based on the first acquisition frequency of the visible light data frame sequence and the second acquisition frequency of the infrared data frame sequence. Based on the transmit and receive timestamps of the target infrared data frames corresponding to each reference synchronization time point in the target infrared data frame sequence, the current network transmission delay is determined. Based on each reference synchronization time point, the reference synchronization period, and the current network transmission delay, a dynamic time window corresponding to each reference synchronization time point is determined.
[0006] Optionally, the method further includes: When there is no visible light data frame in the dynamic time window corresponding to the current reference synchronization time point in the visible light data frame sequence, the first target data frame determined based on the dynamic time window corresponding to the previous reference synchronization time point is used for interpolation to obtain the first target data frame in the dynamic time window corresponding to the current reference synchronization time point. The current reference synchronization time point is any one of the plurality of reference synchronization time points.
[0007] Optionally, the method further includes: If the data display processing status indicates that the current data display processing delay exceeds the reference synchronization period, the current reference synchronization time point is discarded, where the current reference synchronization time point is any one of the plurality of reference synchronization time points.
[0008] Optionally, calibrating the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameter includes: Multiple timestamp pairs are constructed based on the timestamps of each infrared data frame in the infrared data frame sequence and the timestamps of each visible light data frame in the visible light data frame sequence; The clock offset parameter is determined by minimizing the sum of squared residuals corresponding to the plurality of timestamp pairs. The clock offset parameter includes a clock frequency ratio and a time offset. The clock frequency ratio indicates the ratio of the operating frequencies between the clock source of the first device acquiring the visible light data frame sequence and the clock source of the second device acquiring the infrared data frame sequence.
[0009] Optionally, the method further includes: Based on the plurality of timestamp pairs, the clock frequency ratio, and the time offset, determine the fitting difference corresponding to each of the plurality of timestamp pairs; If the fitting difference for any timestamp pair is greater than a preset threshold, the timestamps of the infrared data frame sequence and the visible light data frame sequence are recalibrated.
[0010] Optionally, the method further includes: Obtain the network fluctuation weight coefficient, the speed difference weight coefficient, and the basic capacity; the network fluctuation weight coefficient indicates the sensitivity of data synchronization processing to network transmission fluctuations, and the speed difference weight coefficient indicates the importance of the difference between the data transmission arrival speed and the data synchronization processing speed; Determine the network bandwidth volatility and average network bandwidth in the previous time period: the network bandwidth volatility indicates the transmission stability of the network bandwidth, and the average network bandwidth indicates the average amount of data transmitted by the network in the previous time period; Determine the data transmission arrival speed and data synchronization processing speed in the preceding time period; The capacity of the dynamic buffer within the current time period is determined based on the network fluctuation weight coefficient, the speed difference weight coefficient, the basic capacity, the network bandwidth fluctuation rate, the average network bandwidth, the data transmission arrival speed, and the data synchronization processing speed.
[0011] Optionally, the method further includes: Obtain the capacity occupancy ratio of the first transmission buffer of the first device used to acquire the visible light data frame sequence or the capacity occupancy ratio of the second transmission buffer of the second device used to acquire the infrared data frame sequence; If the capacity occupancy ratio of the first transmission buffer is greater than the first ratio threshold and the number of visible light data to be transmitted in the first transmission buffer is increasing, or if the capacity occupancy ratio of the second transmission buffer is greater than the second ratio threshold and the number of infrared data to be transmitted in the second transmission buffer is increasing, then the capacity increment is determined, and the capacity of the dynamic buffer is updated based on the capacity increment. If the capacity occupancy ratio of the first transmission buffer is less than the third ratio threshold and the amount of visible light data to be transmitted in the first transmission buffer is decreasing, or if the capacity occupancy ratio of the second transmission buffer is less than the fourth ratio threshold and the amount of infrared data to be transmitted in the second transmission buffer is decreasing, then capacity reduction is determined, and the capacity of the dynamic buffer is updated based on the capacity reduction.
[0012] Secondly, this application provides a data synchronization processing apparatus, the apparatus comprising: The acquisition module is used to acquire visible light data frame sequences and infrared data frame sequences from a dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status. The calibration module is used to calibrate the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameters; The time offset compensation module is used to perform time offset compensation processing on the infrared data frame sequence based on the clock offset parameter to obtain the target infrared data frame sequence; The reference synchronization time point determination module is used to determine multiple reference synchronization time points based on the timestamps of the target infrared data frame sequence. The dynamic time window determination module is used to determine the dynamic time window corresponding to each of the plurality of reference synchronization time points; The data frame alignment module is used to select a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence based on the dynamic time window corresponding to each reference synchronization time point, as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
[0013] Thirdly, this application provides a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the data synchronization processing method as described in the first aspect.
[0014] Fourthly, this application provides a computer device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data synchronization processing method as described in the first aspect.
[0015] Fifthly, this application provides a computer program product, the computer program product including computer instructions, which, when executed by a processor, implement the data synchronization processing method as described in the first aspect.
[0016] The data synchronization processing method, apparatus, medium, and equipment provided in this application have the following technical advantages: The solution provided in this application obtains visible light data frame sequences and infrared data frame sequences from a dynamic buffer, the capacity of which is determined by the network data transmission status and data display processing status. It calibrates the timestamps of the infrared and visible light data frame sequences to determine clock offset parameters. Based on these clock offset parameters, it performs time offset compensation processing on the infrared data frame sequences to obtain the target infrared data frame sequence. Based on the timestamps of the target infrared data frame sequences, it determines multiple reference synchronization time points. It then determines a dynamic time window corresponding to each of these reference synchronization time points. Based on the dynamic time window corresponding to each reference synchronization time point, it selects a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing. This effectively eliminates systemic bias in dual-channel data, avoids data misalignment caused by data overflow due to network transmission and display processing, and achieves precise alignment of visible light and infrared data in terms of time and data, providing a reliable data foundation for subsequent synchronized display.
[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0018] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic diagram of the implementation environment of a data synchronization processing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a data synchronization processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a data synchronization processing device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of a device for implementing a data synchronization processing method provided in an embodiment of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0022] Please see Figure 1 This is a schematic diagram illustrating the implementation environment of a data synchronization processing method provided in this application embodiment, such as... Figure 1As shown, the implementation environment may include at least a visible light camera 110, an infrared sensor 120, a transmission network 130, a central control terminal 140, and a display terminal 150. The visible light camera 110 is used to collect visible light data in real time. The visible light camera 110 transmits the visible light data as visible light data frames through the transmission network 130 to the dynamic buffer of the central control terminal 140. The infrared sensor 120 is used to collect infrared data in real time and transmits the infrared data as infrared data frames through the transmission network 130 to the dynamic buffer of the central control terminal 140. The central control unit 140 acquires visible light data frame sequences and infrared data frame sequences from a dynamic buffer, the capacity of which is determined by the network data transmission status and data display processing status. It calibrates the timestamps of the infrared and visible light data frame sequences to determine clock offset parameters. Based on these clock offset parameters, it performs time offset compensation processing on the infrared data frame sequences to obtain the target infrared data frame sequence. Based on the timestamps of the target infrared data frame sequences, it determines multiple reference synchronization time points. It then determines a dynamic time window corresponding to each of these reference synchronization time points. Based on the dynamic time window corresponding to each reference synchronization time point, it selects a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence, using these as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing. The data display processing may include: the central control unit 140 decoding the first target data frame into an RGB (red, green, blue) image; and the central control unit 140 converting the second target data frame into a pseudo-color image. The RGB and pseudo-color images can then be displayed separately or merged on the display unit 150 to demonstrate the operating status of the electrical appliance.
[0023] The following describes a data synchronization processing method provided in this application. Figure 2 This is a flowchart illustrating a data synchronization processing method provided in an embodiment of this application. This application provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Please refer to... Figure 2 The data synchronization processing method provided in this application embodiment may include the following steps: S210: Obtain the visible light data frame sequence and the infrared data frame sequence from the dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status.
[0024] In one embodiment of this application, a visible light camera and an infrared sensor can be integrated inside the electrical device. The visible light camera can collect visible light data in real time at a certain frequency when the electrical device is working, and transmit it to the dynamic buffer of the central control terminal in the form of visible light data frames via a wireless local area network. The central control terminal can decode the visible light data frames into RGB (red, green, and blue three-channel) images. The infrared sensor can collect infrared data in real time at a certain frequency when the electrical device is working, and transmit it to the dynamic buffer of the central control terminal in the form of infrared data frames via a wireless local area network. The central control terminal can convert the infrared data frames into pseudo-color images through interpolation processing, so that the RGB images and pseudo-color images can be displayed separately or merged on the display terminal to show the working status of the electrical device.
[0025] In one embodiment of this application, the capacity of the buffer used to temporarily store the visible light data frame sequence and infrared data frame sequence acquired in real time over a period of time is dynamically variable, and the size of the buffer is related to the network data transmission status and the data display processing status, such as being affected by network fluctuations when the wireless local area network transmits visible light data frames and infrared data frames, and the speed difference between extracting visible light data frames and infrared data frames for display processing, etc. Specifically, the capacity of the dynamic buffer can be determined by the following steps: S201: Obtain network fluctuation weight coefficient, speed difference weight coefficient, and basic capacity.
[0026] Among them, the network fluctuation weight coefficient indicates the sensitivity of data synchronization processing to network transmission fluctuations. This coefficient can be an empirical value based on business requirements, hardware performance, etc. The speed difference weight coefficient indicates the importance, or degree of influence, of the difference between the data transmission arrival speed and the data synchronization processing speed. This difference is also the difference between the speed at which visible light data frames arrive at the buffer and the speed at which visible light data is extracted for data synchronization processing. This speed difference weight coefficient can also be an empirical value determined based on business requirements, hardware performance, etc. The base capacity can be the minimum buffer capacity or a fixed value determined based on business requirements, hardware performance, etc.
[0027] S202: Determine the network bandwidth volatility and average network bandwidth in the previous time period: Network bandwidth volatility indicates the stability of network bandwidth transmission, and average network bandwidth indicates the average amount of data transmitted over the network in the previous time period.
[0028] The duration of the previous time period can be a dynamically adjusted time interval of the buffer, or it can be a certain multiple of the aforementioned time interval.
[0029] Network bandwidth volatility refers to the change in network bandwidth over time. Network bandwidth refers to the amount of data a network can transmit within a certain period, and network bandwidth volatility reflects the stability of network bandwidth transmission. Changes in network bandwidth can be caused by factors such as the addition or removal of devices in the network, device failures, and changes in application bandwidth usage. Feasibly, multiple sampling measurements can be performed on visible light or infrared data frames transmitted in the previous time period to obtain a set of bandwidth data, and the standard deviation of this set of bandwidth data can be calculated to measure the network bandwidth volatility in the previous time period.
[0030] The average network bandwidth indicates the average amount of data transmitted over the network in the previous time period, and can also measure the data transmission rate of the network in the previous time period. Feasibly, the average network bandwidth can be determined based on the total amount of data transmitted in the previous time period and the duration of the previous time period.
[0031] S203: Determine the data transmission arrival speed and data synchronization processing speed in the previous time period.
[0032] S204: Determine the capacity of the dynamic buffer within the current time period based on the network fluctuation weight coefficient, speed difference weight coefficient, basic capacity, network bandwidth fluctuation rate, average network bandwidth, data transmission arrival speed, and data synchronization processing speed.
[0033] For example, based on network fluctuation weight coefficients Network bandwidth volatility Average network bandwidth Speed difference weighting coefficient Data transmission arrival speed Data synchronization processing speed Basic capacity The capacity of the dynamic buffer within the current time period is calculated. Specifically, it can be shown in formula (1):
[0034] In the above embodiments, compared with using a fixed-capacity buffer, the embodiments of this application adopt a method of dynamically adjusting the buffer size based on the network data transmission status and data display processing status. This can avoid buffer data overflow caused by network fluctuations and insufficient processing resources, thereby preventing serious misalignment between visible light data and infrared data, and providing a reliable data foundation for achieving synchronous display of visible light data and infrared data.
[0035] In one embodiment of this application, the above method may further include: S205: Obtain the capacity occupancy ratio of the first transmission buffer of the first device used for acquiring visible light data frame sequences or the capacity occupancy ratio of the second transmission buffer of the second device used for acquiring infrared data frame sequences.
[0036] The first device can be a visible light camera, and the first transmission buffer is used to store the visible light data collected by the first device in real time; the second device can be an infrared sensor, and the second transmission buffer is used to store the infrared data collected by the second device in real time.
[0037] S206: If the capacity occupancy ratio of the first transmission buffer is greater than the first ratio threshold and the number of visible light data to be transmitted in the first transmission buffer is increasing, or if the capacity occupancy ratio of the second transmission buffer is greater than the second ratio threshold and the number of infrared data to be transmitted in the second transmission buffer is increasing, determine the capacity increment and update the capacity of the dynamic buffer based on the capacity increment.
[0038] It is feasible that the amount of visible light data to be transmitted in the first transmission buffer is increasing, that is, the capacity occupancy ratio of the first transmission buffer is increasing, or the amount of data in the first transmission buffer is increasing, and the same applies to the second transmission buffer.
[0039] Feasibly, the first proportional threshold can be 80%, and the capacity increment can be a preset proportion of the previously calculated capacity of the dynamic buffer, such as 20%. The second proportional threshold can be the same as or different from the first proportional threshold, depending on the specific performance of the device. This application embodiment does not limit this.
[0040] S207: If the capacity occupancy ratio of the first transmission buffer is less than the third ratio threshold and the number of visible light data to be transmitted in the first transmission buffer is decreasing, or if the capacity occupancy ratio of the second transmission buffer is less than the fourth ratio threshold and the number of infrared data to be transmitted in the second transmission buffer is decreasing, determine the capacity reduction and update the capacity of the dynamic buffer based on the capacity reduction.
[0041] Feasibly, the third proportional threshold can be 30%, and the capacity reduction can be a preset proportion of the previously calculated dynamic buffer capacity, such as 10%. The fourth proportional threshold can be the same as or different from the third proportional threshold, and can be determined according to the specific needs of the device performance and application. This application embodiment does not limit it.
[0042] In one feasible implementation, the capacity of the updated dynamic buffer cannot be lower than the lower capacity threshold.
[0043] In one feasible implementation, the capacity of the updated dynamic buffer cannot exceed the upper capacity threshold.
[0044] In the above embodiments, by combining the capacity occupancy ratio of the transmission buffer on the acquisition device side, the data transmission arrival speed and data transmission volume are estimated, and the capacity of the central control buffer is dynamically adjusted. This allows for more precise control of the buffer capacity, improving the utilization rate of buffer resources while avoiding data overflow.
[0045] S220: Calibrate the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameters.
[0046] The acquisition and transmission of dual-channel data inevitably involve systematic deviations, resulting in a certain degree of difference in timestamps between the infrared data frame sequence and the visible light data frame sequence, that is, a certain degree of misalignment in the dual-channel data.
[0047] In one embodiment of this application, the timestamp of the visible light data frame sequence can be used as a reference time axis, and the timestamp of the infrared data frame sequence can be used as the time sampling to be synchronized. Through fitting processing, the error or offset between the timestamps of the infrared data frame sequence and the visible light data frame sequence can be minimized. Specifically, step S220 may include the following steps: S221 constructs multiple timestamp pairs based on the timestamps of each infrared data frame in the infrared data frame sequence and the timestamps of each visible light data frame in the visible light data frame sequence.
[0048] S222: Determine the clock offset parameters, which include clock frequency ratio and time offset, while minimizing the sum of squared residuals for multiple timestamp pairs. The clock frequency ratio indicates the ratio of the operating frequencies between the clock source of the first device acquiring the visible light data frame sequence and the clock source of the second device acquiring the infrared data frame sequence.
[0049] For example, construct n pairs of timestamps ( ),in It is the timestamp corresponding to the visible light data frame in the i-th timestamp pair. It is the timestamp corresponding to the infrared data frame in the i-th timestamp pair. (This relates to a linear relationship.) A fitting is performed, where k is the clock frequency ratio. To calculate the time offset, we calculate the sum of squared residuals of n pairs of timestamps. When the sum of squared residuals is minimized (as shown in formula (2)), we solve for the clock frequency ratio and the time offset to obtain the clock offset parameters.
[0050]
[0051] In the above embodiments, the linear relationship between the two is fitted by minimizing the squared residuals of the timestamps of the visible light data frame sequence and the infrared data frame sequence, thereby eliminating the systematic bias of the dual channels.
[0052] In one embodiment of this application, the above method may further include: S223: Based on multiple timestamp pairs, clock frequency ratio, and time offset, determine the fitting difference corresponding to each timestamp pair among the multiple timestamp pairs.
[0053] That is, based on eliminating the systematic bias of the dual channels by using the fitted linear relationship, we further measure the degree of temporal misalignment between the visible light data frame sequence and the infrared data frame sequence.
[0054] For example, the fit difference can be expressed as .
[0055] S224: If the fitting difference for any timestamp pair is greater than a preset threshold, recalibrate the timestamps of the infrared data frame sequence and the visible light data frame sequence.
[0056] It is feasible to change the relationship and fitting algorithm to be fitted during recalibration, but this application does not limit this.
[0057] Alternatively, the average of the fitting difference for each timestamp can be taken, the average value can be compared with the preset threshold, and a determination can be made based on the comparison results to determine whether recalibration is required.
[0058] S230: Based on the clock offset parameter, perform time offset compensation processing on the infrared data frame sequence to obtain the target infrared data frame sequence.
[0059] In one embodiment of this application, based on the linear relationship between the infrared data frame sequence and the visible light data frame sequence in terms of timestamps indicated by the clock offset parameter, the timestamps of the infrared data frame sequence are aligned with the visible light data frame sequence using the timestamps of the visible light data frame sequence as a reference time axis. That is, the timestamps corresponding to each infrared data frame are updated by the clock offset parameter to obtain the target infrared data frame sequence.
[0060] S240: Determine multiple reference synchronization time points based on the timestamps of the target infrared data frame sequence.
[0061] Considering that the acquisition frequencies of the visible light data frame sequence and the infrared data frame sequence are different, there will still be visual inconsistencies when the time-calibrated target infrared data frame sequence and visible light data frame sequence are displayed synchronously (either separately or fused). Therefore, this embodiment of the application also aligns the data dimensions of the target infrared data frame sequence and the visible light data frame sequence.
[0062] Feasibly, taking the target infrared data frame sequence as a reference, the timestamp of each target infrared data frame in the target infrared data frame sequence is used as a reference synchronization time point, and a set of data frames corresponding to the reference synchronization time point in the dual-channel data is determined. The set of data frames includes one visible light data frame and one target infrared data frame.
[0063] S250: Determine the dynamic time window corresponding to each of the multiple reference synchronization time points.
[0064] In one embodiment of this application, considering that the acquisition frequencies of the visible light data frame sequence and the infrared data frame sequence are different, step S250 may include the following steps: S251: Determine the reference synchronization period based on the first acquisition frequency of the visible light data frame sequence and the second acquisition frequency of the infrared data frame sequence.
[0065] Among them, based on the first sampling frequency The first acquisition period of the visible light data frame sequence can be determined. Based on the second sampling frequency The second acquisition period of the infrared data frame sequence can be determined. The reference synchronization period can be the smaller of the first and second acquisition periods, that is... .
[0066] S252: Determine the current network transmission delay based on the transmit and receive timestamps of the target infrared data frames corresponding to each reference synchronization time point in the target infrared data frame sequence.
[0067] Among them, the current network transmission latency It can be the difference between the received timestamp and the transmitted timestamp of the corresponding target infrared data frame.
[0068] S253: Determine the dynamic time window corresponding to each reference synchronization time point based on each reference synchronization time point, reference synchronization period, and current network transmission delay.
[0069] For example, a reference synchronization time point can be represented as The corresponding dynamic time window can then be represented as [ T / 2, T / 2].
[0070] In one embodiment of this application, in the method of determining the dynamic time window corresponding to each reference synchronization time point, if the data display processing status indicates that the current data display processing delay exceeds the reference synchronization period, it may be caused by a sudden high CPU utilization. In this case, the current reference synchronization time point is discarded, and the current reference synchronization time point is any one of the multiple reference synchronization time points.
[0071] S260: Based on the dynamic time window corresponding to each reference synchronization time point, select the first target data frame from the visible light data frame sequence and the second target data frame from the target infrared data frame sequence, as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
[0072] In one embodiment of this application, for each reference synchronization time point, the first target data frame in the corresponding set of time-synchronized data frames is the visible light data frame in the visible light data frame sequence that is closest to the corresponding reference synchronization time point, and the second target data frame in the corresponding set of time-synchronized data frames is the visible light data frame in the infrared data frame sequence that is closest to the corresponding reference synchronization time point.
[0073] In one embodiment of this application, for each reference synchronization time point, the first target data frame and the second target data frame in the corresponding set of time-synchronized data frames are most similar in timestamp.
[0074] In one embodiment of this application, during the process of determining multiple reference synchronization time points based on the timestamps of the target infrared data frame sequence, the target infrared data frame corresponding to the reference synchronization time point may be used as the second target data frame.
[0075] In one embodiment of this application, when there is no visible light data frame in the visible light data frame sequence within the dynamic time window corresponding to the current reference synchronization time point, interpolation is performed using the first target data frame determined based on the dynamic time window corresponding to the previous reference synchronization time point to obtain the first target data frame within the dynamic time window corresponding to the current reference synchronization time point; the current reference synchronization time point can be any one of multiple reference synchronization time points. The processing of infrared data frame sequences is similar.
[0076] In one embodiment of this application, step S240 can be replaced by determining multiple reference synchronization time points based on the timestamps of the visible light data frame sequence. Subsequent processing can refer to steps S250 and S260, which will not be repeated here.
[0077] As can be seen from the above embodiments, the data synchronization processing method provided in this application obtains visible light data frame sequences and infrared data frame sequences from a dynamic buffer, wherein the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status; calibrates the timestamps of the infrared data frame sequences and the visible light data frame sequences to determine clock offset parameters; performs time offset compensation processing on the infrared data frame sequences based on the clock offset parameters to obtain a target infrared data frame sequence; determines multiple reference synchronization time points based on the timestamps of the target infrared data frame sequences; determines a dynamic time window corresponding to each of the multiple reference synchronization time points; and selects a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence based on the dynamic time window corresponding to each reference synchronization time point, as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing. This effectively eliminates the system deviation of dual-channel data, avoids data misalignment caused by data overflow due to network transmission and display processing, and achieves precise alignment of visible light data and infrared data in terms of time and data, providing a reliable data foundation for subsequent synchronized display.
[0078] This application embodiment also provides a data synchronization processing device 300, such as... Figure 3 As shown, the device may include: The acquisition module 310 is used to acquire visible light data frame sequences and infrared data frame sequences from a dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status. The calibration module 320 is used to calibrate the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameters. The time offset compensation module 330 is used to perform time offset compensation processing on the infrared data frame sequence based on the clock offset parameter to obtain the target infrared data frame sequence. The reference synchronization time point determination module 340 is used to determine multiple reference synchronization time points based on the timestamps of the target infrared data frame sequence. The dynamic time window determination module 350 is used to determine the dynamic time window corresponding to each of the plurality of reference synchronization time points; The data frame alignment module 360 is used to select a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence based on the dynamic time window corresponding to each reference synchronization time point, as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
[0079] In one embodiment of this application, the dynamic time window determination module 350 may include: A reference synchronization period is determined based on the first acquisition frequency of the visible light data frame sequence and the second acquisition frequency of the infrared data frame sequence. Based on the transmit and receive timestamps of the target infrared data frames corresponding to each reference synchronization time point in the target infrared data frame sequence, the current network transmission delay is determined. Based on each reference synchronization time point, the reference synchronization period, and the current network transmission delay, a dynamic time window corresponding to each reference synchronization time point is determined.
[0080] In one embodiment of this application, the device 300 may further include: When there is no visible light data frame in the dynamic time window corresponding to the current reference synchronization time point in the visible light data frame sequence, the first target data frame determined based on the dynamic time window corresponding to the previous reference synchronization time point is used for interpolation to obtain the first target data frame in the dynamic time window corresponding to the current reference synchronization time point. The current reference synchronization time point is any one of the plurality of reference synchronization time points.
[0081] In one embodiment of this application, the device 300 may further include: If the data display processing status indicates that the current data display processing delay exceeds the reference synchronization period, the current reference synchronization time point is discarded, where the current reference synchronization time point is any one of the plurality of reference synchronization time points.
[0082] In one embodiment of this application, the calibration module 320 may include: Multiple timestamp pairs are constructed based on the timestamps of each infrared data frame in the infrared data frame sequence and the timestamps of each visible light data frame in the visible light data frame sequence; The clock offset parameter is determined by minimizing the sum of squared residuals corresponding to the plurality of timestamp pairs. The clock offset parameter includes a clock frequency ratio and a time offset. The clock frequency ratio indicates the ratio of the operating frequencies between the clock source of the first device acquiring the visible light data frame sequence and the clock source of the second device acquiring the infrared data frame sequence.
[0083] In one embodiment of this application, the device 300 may further include: Based on the plurality of timestamp pairs, the clock frequency ratio, and the time offset, determine the fitting difference corresponding to each of the plurality of timestamp pairs; If the fitting difference for any timestamp pair is greater than a preset threshold, the timestamps of the infrared data frame sequence and the visible light data frame sequence are recalibrated.
[0084] In one embodiment of this application, the device 300 may further include: Obtain the network fluctuation weight coefficient, the speed difference weight coefficient, and the basic capacity; the network fluctuation weight coefficient indicates the sensitivity of data synchronization processing to network transmission fluctuations, and the speed difference weight coefficient indicates the importance of the difference between the data transmission arrival speed and the data synchronization processing speed; Determine the network bandwidth volatility and average network bandwidth in the previous time period: the network bandwidth volatility indicates the transmission stability of the network bandwidth, and the average network bandwidth indicates the average amount of data transmitted by the network in the previous time period; Determine the data transmission arrival speed and data synchronization processing speed in the preceding time period; The capacity of the dynamic buffer within the current time period is determined based on the network fluctuation weight coefficient, the speed difference weight coefficient, the basic capacity, the network bandwidth fluctuation rate, the average network bandwidth, the data transmission arrival speed, and the data synchronization processing speed.
[0085] In one embodiment of this application, the device 300 may further include: Obtain the capacity occupancy ratio of the first transmission buffer of the first device used to acquire the visible light data frame sequence or the capacity occupancy ratio of the second transmission buffer of the second device used to acquire the infrared data frame sequence; If the capacity occupancy ratio of the first transmission buffer is greater than the first ratio threshold and the number of visible light data to be transmitted in the first transmission buffer is increasing, or if the capacity occupancy ratio of the second transmission buffer is greater than the second ratio threshold and the number of infrared data to be transmitted in the second transmission buffer is increasing, then the capacity increment is determined, and the capacity of the dynamic buffer is updated based on the capacity increment. If the capacity occupancy ratio of the first transmission buffer is less than the third ratio threshold and the amount of visible light data to be transmitted in the first transmission buffer is decreasing, or if the capacity occupancy ratio of the second transmission buffer is less than the fourth ratio threshold and the amount of infrared data to be transmitted in the second transmission buffer is decreasing, then capacity reduction is determined, and the capacity of the dynamic buffer is updated based on the capacity reduction.
[0086] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0087] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0088] This application provides a computer device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement a data synchronization processing method as provided in the above method embodiments.
[0089] Figure 4 A schematic diagram of the hardware structure of a device for implementing a data synchronization processing method provided in an embodiment of this application is shown. This device may constitute or include the apparatus or system provided in the embodiment of this application. Figure 4 As shown, device 10 may include one or more processors 1002 (shown as 1002a, 1002b, ..., 1002n in the figure) 1002 (processor 1002 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, device 10 may also include a... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.
[0090] It should be noted that the aforementioned one or more processors 1002 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within device 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0091] The memory 1004 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of this application. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, thereby realizing the above-mentioned data synchronization processing method. The memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include memory remotely located relative to the processor 1002, and these remote memories can be connected to the device 10 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0092] The transmission device 1006 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of device 10. In one example, the transmission device 1006 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 1006 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0093] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of device 10 (or mobile device).
[0094] This application also provides a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a data synchronization processing method in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the data synchronization processing method provided in the above method embodiment.
[0095] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0096] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data synchronization processing method provided in the various optional embodiments described above.
[0097] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
[0098] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0099] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0100] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A data synchronization processing method, characterized in that, The method includes: Visible light data frame sequences and infrared data frame sequences are obtained from a dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status. The timestamps of the infrared data frame sequence and the visible light data frame sequence are calibrated to determine the clock offset parameters; Based on the clock offset parameter, the infrared data frame sequence is subjected to time offset compensation processing to obtain the target infrared data frame sequence; Based on the timestamps of the target infrared data frame sequence, multiple reference synchronization time points are determined; Determine the dynamic time window corresponding to each of the plurality of reference synchronization time points; Based on the dynamic time window corresponding to each reference synchronization time point, a first target data frame is selected from the visible light data frame sequence and a second target data frame is selected from the target infrared data frame sequence, which are used as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
2. The method according to claim 1, characterized in that, Determining the dynamic time window corresponding to each of the plurality of reference synchronization time points includes: A reference synchronization period is determined based on the first acquisition frequency of the visible light data frame sequence and the second acquisition frequency of the infrared data frame sequence. Based on the transmit and receive timestamps of the target infrared data frames corresponding to each reference synchronization time point in the target infrared data frame sequence, the current network transmission delay is determined. Based on each reference synchronization time point, the reference synchronization period, and the current network transmission delay, a dynamic time window corresponding to each reference synchronization time point is determined.
3. The method according to claim 1, characterized in that, The method further includes: When there is no visible light data frame in the dynamic time window corresponding to the current reference synchronization time point in the visible light data frame sequence, the first target data frame determined based on the dynamic time window corresponding to the previous reference synchronization time point is used for interpolation to obtain the first target data frame in the dynamic time window corresponding to the current reference synchronization time point. The current reference synchronization time point is any one of the plurality of reference synchronization time points.
4. The method according to claim 2, characterized in that, The method further includes: If the data display processing status indicates that the current data display processing delay exceeds the reference synchronization period, the current reference synchronization time point is discarded, where the current reference synchronization time point is any one of the plurality of reference synchronization time points.
5. The method according to claim 1, characterized in that, The calibration of the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameters includes: Multiple timestamp pairs are constructed based on the timestamps of each infrared data frame in the infrared data frame sequence and the timestamps of each visible light data frame in the visible light data frame sequence; The clock offset parameter is determined by minimizing the sum of squared residuals corresponding to the plurality of timestamp pairs. The clock offset parameter includes a clock frequency ratio and a time offset. The clock frequency ratio indicates the ratio of the operating frequencies between the clock source of the first device acquiring the visible light data frame sequence and the clock source of the second device acquiring the infrared data frame sequence.
6. The method according to claim 5, characterized in that, The method further includes: Based on the plurality of timestamp pairs, the clock frequency ratio, and the time offset, determine the fitting difference corresponding to each of the plurality of timestamp pairs; If the fitting difference for any timestamp pair is greater than a preset threshold, the timestamps of the infrared data frame sequence and the visible light data frame sequence are recalibrated.
7. The method according to claim 1, characterized in that, The method further includes: Obtain the network fluctuation weight coefficient, the speed difference weight coefficient, and the basic capacity; the network fluctuation weight coefficient indicates the sensitivity of data synchronization processing to network transmission fluctuations, and the speed difference weight coefficient indicates the importance of the difference between the data transmission arrival speed and the data synchronization processing speed; Determine the network bandwidth volatility and average network bandwidth in the previous time period: the network bandwidth volatility indicates the transmission stability of the network bandwidth, and the average network bandwidth indicates the average amount of data transmitted by the network in the previous time period; Determine the data transmission arrival speed and data synchronization processing speed in the preceding time period; The capacity of the dynamic buffer within the current time period is determined based on the network fluctuation weight coefficient, the speed difference weight coefficient, the basic capacity, the network bandwidth fluctuation rate, the average network bandwidth, the data transmission arrival speed, and the data synchronization processing speed.
8. The method according to claim 7, characterized in that, The method further includes: Obtain the capacity occupancy ratio of the first transmission buffer of the first device used to acquire the visible light data frame sequence or the capacity occupancy ratio of the second transmission buffer of the second device used to acquire the infrared data frame sequence; If the capacity occupancy ratio of the first transmission buffer is greater than the first ratio threshold and the number of visible light data to be transmitted in the first transmission buffer is increasing, or if the capacity occupancy ratio of the second transmission buffer is greater than the second ratio threshold and the number of infrared data to be transmitted in the second transmission buffer is increasing, then the capacity increment is determined, and the capacity of the dynamic buffer is updated based on the capacity increment. If the capacity occupancy ratio of the first transmission buffer is less than the third ratio threshold and the amount of visible light data to be transmitted in the first transmission buffer is decreasing, or if the capacity occupancy ratio of the second transmission buffer is less than the fourth ratio threshold and the amount of infrared data to be transmitted in the second transmission buffer is decreasing, then capacity reduction is determined, and the capacity of the dynamic buffer is updated based on the capacity reduction.
9. A data synchronization processing device, characterized in that, The device includes: The acquisition module is used to acquire visible light data frame sequences and infrared data frame sequences from a dynamic buffer; the capacity of the dynamic buffer is determined by the network data transmission status and the data display processing status. The calibration module is used to calibrate the timestamps of the infrared data frame sequence and the visible light data frame sequence to determine the clock offset parameters; The time offset compensation module is used to perform time offset compensation processing on the infrared data frame sequence based on the clock offset parameter to obtain the target infrared data frame sequence; The reference synchronization time point determination module is used to determine multiple reference synchronization time points based on the timestamps of the target infrared data frame sequence. The dynamic time window determination module is used to determine the dynamic time window corresponding to each of the plurality of reference synchronization time points; The data frame alignment module is used to select a first target data frame from the visible light data frame sequence and a second target data frame from the target infrared data frame sequence based on the dynamic time window corresponding to each reference synchronization time point, as a set of time-synchronized data frames corresponding to each reference synchronization time point for data display processing.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the data synchronization processing method as described in any one of claims 1 to 8.
11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data synchronization processing method as described in any one of claims 1 to 8.