Frame synchronization cooperative control method based on dual-machine linkage and its control system

By obtaining the performance parameters of the camera module and historical clock drift data in real time, calculating the time offset dynamically, and adjusting the frame synchronization signal using neural network model, solving the problem of insufficient frame synchronization control accuracy and stability in the existing technology, achieving high accuracy and flexible frame synchronization and coordinated control.

CN118450062BActive Publication Date: 2024-11-19SICHUAN NATIONAL INNOVATION VISION UHD VIDEO TECHNOLOGY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410709598.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-03
Publication Date
2024-11-19
Estimated Expiration
2044-06-03

AI Technical Summary

Technical Problem

The prior art has limitations in achieving high-precision and high-efficiency frame synchronization collaborative control, especially due to the degradation of synchronization accuracy due to equipment performance differences and clock drift.

Method used

By obtaining the performance parameters of the camera module in real time, and dynamically compute the time offset with historical clock drift data, a neural network-based model is used to predict the dynamic time offset, and adjust the transmission time of the frame synchronization signal to realize dual-machine or multi-machine Image frame synchronization in the linkage system.

Benefits of technology

It significantly improves the accuracy and stability of frame synchronization, effectively avoids synchronization errors caused by device performance differences and clock drift, and has flexibility and scalability, adapts to different application scenarios and device performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118450062B_ABST
    Figure CN118450062B_ABST
Patent Text Reader

Abstract

The present invention provides a frame synchronization collaborative control method based on dual-machine linkage and a control system thereof, which belongs to the field of image processing technology. By acquiring the performance parameters of a camera module in real time and combining historical clock drift data, the time offset can be accurately calculated and dynamically adjusted to ensure that image frames in a dual-machine or multi-machine linkage system remain highly synchronized in time, thereby improving the accuracy and stability of frame synchronization and effectively avoiding synchronization errors caused by equipment performance differences and clock drift. By calculating the image content overlap rate and comparing it with a preset threshold, the synchronization effect can be evaluated in real time, and the time offset can be adaptively adjusted according to the evaluation result, thereby achieving continuous optimization of synchronization control and being able to adapt to different application scenarios and equipment performance. The present invention acquires the performance parameters of a camera module in real time and combines historical clock drift data to dynamically calculate the time offset, thereby achieving accurate control of a frame synchronization signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a frame synchronization collaborative control method based on dual-machine linkage and a control system thereof. Background Art

[0002] With the rapid development of intelligent device technology, dual-machine linkage systems have been widely used in many fields due to their unique advantages. Especially in scenarios that require high-precision synchronous control, such as security monitoring, autonomous driving, aerospace, etc., the frame synchronization collaborative control of dual-machine or multi-machine linkage systems is particularly important. However, due to factors such as device performance differences, clock drift, and network delays, how to achieve high-precision and high-efficiency frame synchronization collaborative control has always been a difficult point in industry research.

[0003] Traditional frame synchronization control methods mainly rely on hardware synchronization signals or software timestamp calibration, but these methods often have limitations. Although hardware synchronization signals have high accuracy, they are costly and complex to implement; although software timestamp calibration is flexible, it is easily affected by network delays and device performance fluctuations, resulting in reduced synchronization accuracy.

[0004] Therefore, it is necessary to provide a frame synchronization collaborative control method based on dual-machine linkage and its control system to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a frame synchronization collaborative control method based on dual-machine linkage and its control system, which achieves precise control of the frame synchronization signal by acquiring the performance parameters of the camera module in real time and combining historical clock drift data to dynamically calculate the time offset.

[0006] The present invention provides a frame synchronization collaborative control method based on dual-machine linkage, comprising a main control module and a first camera module and a second camera module interacting with the main control module. The control method comprises the following steps:

[0007] S1: When the first camera module captures a first image frame, issuing a synchronization instruction;

[0008] S2: the main control module obtains the performance parameters of the first camera module and the second camera module when shooting the first image frame in response to the synchronization instruction, and determines the dynamic time offset in combination with the known historical clock drift data between the first camera module and the second camera module;

[0009] S3: determining a sending time of a frame synchronization signal based on a dynamic time offset when shooting the first image frame, and sending the frame synchronization signal to the second camera module within the sending time to control the second camera module to shoot a second image frame;

[0010] S4: acquiring and calculating, by the main control module, an image content overlap ratio between the first image frame and the second image frame, and comparing the image content overlap ratio with a preset threshold;

[0011] S5: updating the dynamic time offset of the next frame based on the comparison result, and performing the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

[0012] Preferably, in step S2, the performance parameters include the current load rate and the current operating temperature of the respective processors of the first camera module and the second camera module.

[0013] Preferably, step S2 specifically includes the following steps:

[0014] S201: the main control module reads the current load rate and current operating temperature of the processors of the first camera module and the second camera module, and obtains known historical clock drift data from a database;

[0015] S202: Building a neural network-based offset determination model based on historical performance parameters and known historical clock drift data;

[0016] S203: Input the current load rate, the current operating temperature and the known historical clock drift data into the offset determination model for forward propagation, and output the dynamic time offset.

[0017] Preferably, in step S203:

[0018] The offset determination model includes an input layer, a hidden layer and an output layer, wherein the input layer is used to receive the current load rate, the current operating temperature and known historical clock drift data, the hidden layer is used to use an activation function to learn the nonlinear relationship between the current load rate, the current operating temperature and known historical clock drift data and the dynamic time offset, and the output layer is used to output the predicted dynamic time offset.

[0019] Preferably, step S3 specifically includes the following steps:

[0020] S301: Calculating the time point at which the second camera module sends a frame synchronization signal based on the dynamic time offset;

[0021] S302: At the calculated frame synchronization signal sending time point, the main control module sends a frame synchronization signal to the second camera module to trigger a shooting operation of the second camera module;

[0022] S303: After receiving the frame synchronization signal, the second camera module captures a second image frame according to the frame synchronization signal to achieve frame synchronization with the first camera module.

[0023] Preferably, step S4 specifically includes the following steps:

[0024] S401: Acquire, by the main control module, first image data of a first image frame and second image data of a second image frame;

[0025] S402: extracting a first feature set of the first image data and a second feature set of the second image data and matching them, and calculating an image content overlap rate based on the matching result, wherein the image content overlap rate is calculated as follows:

[0026]

[0027] Where |M| is the number of matching feature pairs in the matching results, |F1| and |F2| are the sizes of the first feature set and the second feature set, respectively, and K is the image content overlap rate;

[0028] S403: Compare the image content overlap rate with a preset threshold value to obtain a comparison result, wherein the comparison result includes that the image content overlap rate is less than, equal to, or greater than the threshold value.

[0029] Preferably, step S5 is specifically:

[0030] When the image content overlap rate is greater than a threshold, the current dynamic time offset is not adjusted.

[0031] When the image content overlap rate is less than or equal to the threshold, the current dynamic time offset is adjusted, and finally the frame synchronization signal is controlled again according to the unadjusted dynamic time offset or the adjusted dynamic time offset.

[0032] The present invention also provides a frame synchronization cooperative control system based on dual-machine linkage, comprising a main control module and a first camera module and a second camera module interacting with the main control module, wherein the main control module comprises:

[0033] An instruction sending submodule, used for sending a synchronization instruction when the first camera module captures the first image frame;

[0034] A dynamic time offset determination submodule, used for the main control module to respond to the synchronization instruction, obtain the performance parameters of the first camera module and the second camera module when shooting the first image frame, and determine the dynamic time offset in combination with the known historical clock drift data between the first camera module and the second camera module;

[0035] A synchronous shooting submodule, used for determining a sending time of a frame synchronization signal based on a dynamic time offset when shooting a first image frame, and sending the frame synchronization signal to a second camera module within the sending time to control the second camera module to shoot a second image frame;

[0036] A comparison submodule, configured to obtain and calculate an image content overlap ratio between the first image frame and the second image frame through the main control module, and compare the image content overlap ratio with a preset threshold;

[0037] The updating submodule is used to update the dynamic time offset of the next frame based on the comparison result, and to perform the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

[0038] Compared with the related art, the frame synchronization cooperative control method based on dual-machine linkage and the control system thereof provided by the present invention have the following beneficial effects:

[0039] The present invention can accurately calculate and dynamically adjust the time offset by acquiring the performance parameters of the camera module in real time and combining them with historical clock drift data, thereby ensuring that the image frames in a dual-machine or multi-machine linkage system remain highly synchronized in time, significantly improving the accuracy and stability of frame synchronization, and effectively avoiding synchronization errors caused by equipment performance differences and clock drift. At the same time, by calculating the image content overlap rate and comparing it with a preset threshold, the synchronization effect can be evaluated in real time, and the time offset can be adaptively adjusted according to the evaluation result, thereby achieving continuous optimization of synchronization control, being able to adapt to different application scenarios and equipment performance, and having greater flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a frame synchronization cooperative control method based on dual-machine linkage provided by the present invention;

[0041] Figure 2 A schematic diagram of the module structure of a frame synchronization cooperative control system based on dual-machine linkage provided by the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0043] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0044] Embodiment 1

[0045] The present invention provides a frame synchronization collaborative control method based on dual-machine linkage, comprising a main control module and a first camera module and a second camera module interacting with the main control module. The control method comprises the following steps:

[0046] S1: When the first camera module captures a first image frame, a synchronization instruction is issued.

[0047] In this embodiment, a synchronization instruction is issued when the first camera module captures the first image frame to start coordinating the workflows of the first camera module and the second camera module, so as to achieve frame synchronization.

[0048] Specifically, when the main control module detects that the first camera module has started the image capture process, it immediately sends a digital synchronization signal instruction packet containing shooting start information, which is passed to the second camera module via a high-speed bus or wireless communication protocol (such as Wi-Fi, Bluetooth or a custom wireless protocol).

[0049] S2: The main control module obtains performance parameters of the first camera module and the second camera module when shooting the first image frame in response to the synchronization instruction, and determines the dynamic time offset in combination with known historical clock drift data between the first camera module and the second camera module.

[0050] In actual applications, camera systems such as self-propelled devices will face various changing operating environments and conditions, including temperature fluctuations, processor load changes and other factors. These changes will affect the clock frequency of the camera, resulting in time drift. Therefore, static time offset cannot reflect the instant state changes during actual operation. Then, dynamic time offset is adopted, which can take into account the actual situation of each shooting, especially for applications that require high-precision synchronization (such as 3D modeling, motion tracking, multi-view video shooting), which can significantly improve the accuracy of synchronization.

[0051] Specifically, the main control module collects the performance parameters and historical clock drift data of the first camera module and the second camera module, and these data are input into a pre-trained neural network-based offset determination model. The offset determination model predicts the optimal offset based on the performance parameters and historical clock drift data, such as how many microseconds in advance or delayed the second image frame should be captured to achieve synchronization. Ultimately, the synchronization problem caused by hardware performance fluctuations and time drift is solved through dynamic adjustment, thereby improving the synchronization accuracy and adaptability.

[0052] S3: determining a sending time of a frame synchronization signal based on a dynamic time offset when shooting the first image frame, and sending the frame synchronization signal to the second camera module within the sending time to control the second camera module to shoot a second image frame.

[0053] In this embodiment, the shooting timing of the second camera module is precisely controlled to ensure synchronization with the image frame of the first camera module within a predetermined time window.

[0054] Specifically, the main control module determines an accurate frame synchronization signal sending time point according to the calculated dynamic time offset, and sends the frame synchronization signal to the second camera module through the channel. The latter starts the shooting process immediately after receiving the signal, thereby improving the synchronization reliability and shooting efficiency.

[0055] S4: acquiring and calculating the image content overlap ratio between the first image frame and the second image frame through the main control module, and comparing the image content overlap ratio with a preset threshold.

[0056] In this embodiment, the image content overlap ratio is obtained by calculating the overlap ratio of the content between the first image frame and the second image frame to verify whether the synchronization strategy meets expectations and provide a basis for the next adjustment.

[0057] Specifically, the main control module collects data between the first image frame and the second image frame, calculates the image content overlap rate, and then compares the image content overlap rate with a preset threshold. Exemplarily, the threshold is 80%. Then, if the image content overlap rate is greater than 80%, it is considered that the synchronization is good, providing a feedback mechanism for quantifying the synchronization quality to ensure the continuity and quality of the image content.

[0058] S5: updating the dynamic time offset of the next frame based on the comparison result, and performing the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

[0059] In this embodiment, in order to ensure long-term synchronization accuracy, it is necessary to dynamically adjust the strategy according to the comparison result and continuously optimize the synchronization process.

[0060] Specifically, based on the comparison result between the image content overlap rate and the threshold in S4, the main control module decides whether to adjust the dynamic offset of the next frame. If it does not meet the standard, it directly makes corresponding adjustments based on the original dynamic time offset, and then returns to the S2-S4 cycle, thus forming a closed-loop control. It can continuously adjust itself according to real-time feedback, enhance adaptability, and maintain a high level of frame synchronization for a long time, making it suitable for dynamically changing shooting environments.

[0061] Specifically, in step S2, the performance parameters include the current load rate and the current operating temperature of the processors of the first camera module and the second camera module respectively.

[0062] In this embodiment, the load rate of the processor reflects its busyness when executing tasks. A high load rate may cause delays in image processing and data transmission, thereby affecting the punctuality of shooting. By monitoring the load rates of the processors of the first camera module and the second camera module in real time and inputting this information into the offset determination model based on the neural network, the model can learn the relationship between the load and the time offset to adjust the shooting time of the second camera module to ensure synchronization.

[0063] The operating temperature of the processor is another important factor, as high temperatures can affect the performance and clock frequency of the processor, leading to timing uncertainty. High temperatures are often associated with reduced performance and potentially unstable operation. Taking the operating temperature into account in the model can compensate for the resulting capture time deviation by adding an offset when the temperature rise causes performance degradation, and vice versa.

[0064] Finally, by integrating the current load rate and operating temperature, two performance parameters, and combining them with historical clock drift data, the model can dynamically predict a more accurate offset in a real-time changing operating environment, effectively coping with the challenges brought by hardware performance fluctuations and time drift. This not only improves the accuracy of synchronization, but also enhances the adaptability and robustness of the system in a variety of environments, ensuring long-term stable frame synchronization effects, especially in application scenarios with strict requirements on synchronization accuracy, such as 3D modeling, motion tracking, and multi-view video shooting.

[0065] Specifically, step S2 specifically includes the following steps:

[0066] S201: The main control module reads the current load rate and current operating temperature of the processors of the first camera module and the second camera module, and obtains known historical clock drift data from a database.

[0067] In this embodiment, in order to obtain key information of the current state in real time, including the processor load rate and operating temperature of the first camera module and the second camera module, and the historical clock drift data extracted from the database, the necessary input is provided for the subsequent dynamic time offset calculation.

[0068] Specifically, the main control module queries and collects the real-time processor load rate and operating temperature data of the first camera module and the second camera module through an internal communication interface or a network connection. At the same time, it accesses historical records stored in a local or remote database to extract the clock drift data between the first camera module and the second camera module. These data reflect the patterns and trends of past clock inconsistencies.

[0069] S202: Constructing a neural network-based offset determination model based on historical performance parameters and known historical clock drift data.

[0070] In this embodiment, based on the collected real-time performance parameters and historical clock drift data, a neural network-based model is established to predict the dynamic time offset. The model can learn the complex nonlinear relationship between parameters and improve the accuracy of the prediction.

[0071] Specifically, using historical performance parameters and known historical clock drift data, a neural network model consisting of an input layer, a hidden layer, and an output layer is constructed through a machine learning framework. An activation function is applied in the hidden layer to handle nonlinear relationships. During model training, the back propagation algorithm and optimizer are used to minimize the error between the predicted offset and the actual offset.

[0072] S203: Input the current load rate, the current operating temperature and the known historical clock drift data into the offset determination model for forward propagation, and output the dynamic time offset.

[0073] In this embodiment, the constructed neural network model is used to input the current load rate, the current operating temperature and the known historical clock drift data, and the dynamic time offset is calculated through the forward propagation process.

[0074] Specifically, the main control module uses the current load rate, operating temperature and historical clock drift data collected in S201 as input vectors and feeds them into the neural network model. The model performs forward propagation calculations, and the hidden layer performs nonlinear transformations on the inputs. Finally, the output layer generates a value, which is the dynamic time offset required for the next frame.

[0075] The dynamic time offset obtained through forward propagation directly guides the precise timing of sending the frame synchronization signal, ensuring that the shooting of the second camera module is highly synchronized with the first camera module, improving the flexibility and response speed of the synchronization control, especially in application scenarios where environmental conditions change rapidly and hardware status fluctuates greatly, significantly enhancing the stability and synchronization effect of the system.

[0076] Specifically, step S3 specifically includes the following steps:

[0077] S301: Calculate the time point at which the second camera module sends a frame synchronization signal based on the dynamic time offset.

[0078] In this embodiment, the calculated dynamic time offset is used to plan the shooting triggering moment of the second camera module to ensure that it is perfectly aligned with the shooting time window of the first camera module.

[0079] Specifically, the main control module uses the dynamic time offset to calculate a specific time point, that is, the moment when the second camera module should receive the frame synchronization signal and start shooting. The calculation process is to add the offset to the reference time point of the first camera module to ensure the precise matching of the shooting time of the first camera module and the second camera module, ensuring that even under different environmental conditions and hardware performance fluctuations, the second camera module can start shooting at the predetermined optimal time, improving the accuracy of the synchronization operation and the overall response speed of the system.

[0080] S302: At the calculated frame synchronization signal sending time point, the main control module sends a frame synchronization signal to the second camera module to trigger a shooting operation of the second camera module.

[0081] In this embodiment, it is ensured that the main control module can accurately send a control instruction to the second camera module according to the calculated time point to trigger it to capture an image.

[0082] Specific implementation method: The main control module sends a frame synchronization signal to the second camera module. The frame synchronization signal allows the second camera module to understand that this is a command to start shooting. The second camera module can respond to the instruction at the most appropriate time and shoot almost at the same time as the first camera module, which greatly improves the success rate of synchronized shooting and the consistency of image sequences, which is crucial for applications that require high-precision synchronization.

[0083] S303: After receiving the frame synchronization signal, the second camera module captures a second image frame according to the frame synchronization signal to achieve frame synchronization with the first camera module.

[0084] In this embodiment, it is ensured that the second camera module can quickly respond to the command of the main control module, start shooting in time, achieve accurate frame synchronization with the first camera module, and provide a high-quality synchronous data source for subsequent image processing and analysis.

[0085] Specifically, after receiving the frame synchronization signal, the second camera module immediately activates the internal shooting program and starts capturing image frames according to the preset configuration parameters, which directly ensures the synchronization of dual-machine shooting and makes the captured image sequences accurately aligned in time, providing the necessary data foundation for applications such as 3D reconstruction and motion analysis. Through fast and accurate response, it can effectively utilize each frame resource and improve overall shooting efficiency and data quality.

[0086] Specifically, step S4 specifically includes the following steps:

[0087] S401: Acquire, through the main control module, first image data of a first image frame and second image data of a second image frame.

[0088] In this embodiment, by acquiring specific image data of the first image frame and the second image frame, the correlation and synchronization quality between the first image frame and the second image frame can be deeply analyzed.

[0089] Specifically, the main control module receives the first image data and the second image data captured by the first camera module and the second camera module respectively, and the first image data and the second image data include a pixel matrix, color information, and a brightness value.

[0090] S402: extracting a first feature set of the first image data and a second feature set of the second image data and matching them, and calculating an image content overlap rate based on the matching result, wherein the image content overlap rate is calculated as follows:

[0091]

[0092] Among them, |M| is the number of matching feature pairs in the matching results, |F1| and |F2| are the sizes of the first feature set and the second feature set respectively, and K is the image content overlap rate.

[0093] In this embodiment, feature matching is used to quantify the similarity between the contents of the first image frame and the second image frame, so as to evaluate whether the synchronization achieves the expected effect.

[0094] Specifically, a SIFT feature extractor is used to extract a first feature set from the first image data, and a second feature set is similarly extracted from the second image data. The corresponding relationship between the first feature set and the second feature set is then found through the feature matching algorithm KNN. The number of matching feature pairs is counted, and the image content overlap rate is calculated according to the formula. Through feature level comparison, a method for quantitatively evaluating the image synchronization quality is provided. As an intuitive indicator, the image content overlap rate can directly reflect the effectiveness of the synchronization strategy and provide fast feedback for the system.

[0095] S403: Compare the image content overlap rate with a preset threshold value to obtain a comparison result, wherein the comparison result includes that the image content overlap rate is less than, equal to, or greater than the threshold value.

[0096] In this embodiment, whether the synchronization meets the requirement is determined based on the comparison result between the image content overlap rate and the preset threshold, so as to provide a decision basis for the subsequent synchronization strategy adjustment.

[0097] Specifically, the main control module compares the calculated image content overlap rate with the preset threshold to determine whether the overlap rate is less than, equal to or greater than the threshold. Through comparison, it immediately identifies whether the current synchronization strategy is successful or not. If it fails to meet the standard, the adjustment mechanism is triggered. This instant feedback loop ensures adaptability and dynamic optimization capabilities, maintains or improves synchronization quality, which is particularly important in dynamically changing shooting environments.

[0098] Specifically, step S5 is as follows:

[0099] When the image content overlap rate is greater than a threshold, the current dynamic time offset is not adjusted.

[0100] When the image content overlap rate is less than or equal to the threshold, the current dynamic time offset is adjusted, and finally the frame synchronization signal is controlled again according to the unadjusted dynamic time offset or the adjusted dynamic time offset.

[0101] In this embodiment, according to the actual effect of the previous frame synchronization operation (ie, the comparison result of the image content overlap ratio with the preset threshold), the behavior of the system is dynamically adjusted to continuously optimize the accuracy and stability of the frame synchronization.

[0102] Specifically, first, the main control module compares the calculated image content overlap rate with a preset threshold.

[0103] When the image content overlap rate is greater than the preset threshold, it indicates that the current dynamic time offset is good enough to meet the synchronization requirements, so no adjustment is made and the current offset is kept unchanged to maintain a stable synchronization state.

[0104] On the contrary, when the image content overlap rate is less than or equal to the threshold, it indicates that the synchronization effect is not as expected. At this time, the dynamic time offset needs to be adjusted. The adjustment method is based on the gap between the overlap rate and the threshold. The preset rules determine whether to increase or decrease the offset, as well as the adjustment amplitude, to optimize the synchronization performance of the next frame. Regardless of maintaining the existing offset or applying the adjusted offset, the main control module applies this dynamic time offset to the synchronization control of the next frame. By calculating the optimal time to send the frame synchronization signal, it ensures that the second camera module captures the image frame at a more precise time point, forming a closed-loop feedback control.

[0105] Specifically, the preset rule is that when the overlap rate is lower than the threshold by a fixed percentage, the offset increases or decreases according to a fixed value or ratio.

[0106] For example, assuming the threshold is 80%, if the image content overlap rate is 70%, which is 10 percentage points below the threshold, it is preset that the dynamic time offset increases by Δt (for example, 10 microseconds) for every 1% below the threshold. Therefore, the dynamic time offset needs to increase by Δt*10=100 microseconds.

[0107] The present invention provides a frame synchronization collaborative control method based on dual-machine linkage, and the working principle is as follows: first, when the first camera module shoots the first image frame, a synchronization instruction is issued. After receiving the synchronization instruction, the main control module immediately obtains the performance parameters of the first camera module and the second camera module when shooting the first image frame, such as frame rate, exposure time, etc. At the same time, the main control module also combines the known historical clock drift data between the first camera module and the second camera module to calculate the current dynamic time offset.

[0108] Then, based on the dynamic time offset when shooting the first image frame, the main control module determines the sending time of the frame synchronization signal, and sends the frame synchronization signal to the second camera module within the time to control the second camera module to shoot the second image frame. Since the performance difference of the equipment and the influence of clock drift are taken into consideration, it can be ensured that the image frames shot by the first camera module and the second camera module are highly synchronized in time.

[0109] Then, the main control module will obtain and calculate the image content overlap rate between the first image frame and the second image frame, and compare it with the preset threshold. If the overlap rate is lower than the threshold, it means that the synchronization accuracy is insufficient. At this time, the main control module will update the dynamic time offset of the next frame according to the comparison result, and execute the control of the frame synchronization signal again based on the new dynamic time offset. Through continuous iteration and optimization, the accuracy and stability of frame synchronization can be gradually improved. The frame synchronization collaborative control method of the present invention not only has the characteristics of high precision and high efficiency, but also has good adaptability and scalability.

[0110] Embodiment 2

[0111] The present invention also provides a frame synchronization cooperative control system based on dual-machine linkage, referring to Figure 2 As shown, it includes a main control module and a first camera module and a second camera module interacting with the main control module, and the main control module includes:

[0112] The instruction sending submodule is used to send a synchronization instruction when the first camera module captures the first image frame.

[0113] The dynamic time offset determination submodule is used for the main control module to respond to the synchronization instruction, obtain the performance parameters of the first camera module and the second camera module when shooting the first image frame, and determine the dynamic time offset in combination with the known historical clock drift data between the first camera module and the second camera module.

[0114] The synchronous shooting submodule is used to determine the sending time of the frame synchronization signal based on the dynamic time offset when shooting the first image frame, and send the frame synchronization signal to the second camera module within the sending time to control the second camera module to shoot the second image frame.

[0115] The comparison submodule is used to obtain and calculate the image content overlap rate between the first image frame and the second image frame through the main control module, and compare the image content overlap rate with a preset threshold.

[0116] The updating submodule is used to update the dynamic time offset of the next frame based on the comparison result, and to perform the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

[0117] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0119] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

Claims

1. A frame synchronization collaborative control method based on dual-machine linkage, comprising a main control module and a first camera module and a second camera module interacting with the main control module, characterized in that: The control method comprises the following steps: S1: When the first camera module captures a first image frame, issuing a synchronization instruction; S2: the main control module obtains the performance parameters of the first camera module and the second camera module when shooting the first image frame in response to the synchronization instruction, and determines the dynamic time offset in combination with the known historical clock drift data between the first camera module and the second camera module; In step S2, the performance parameters include the current load rate and the current operating temperature of the processors of the first camera module and the second camera module respectively; Wherein step S2 specifically comprises the following steps: S201: the main control module reads the current load rate and current operating temperature of the processors of the first camera module and the second camera module, and obtains known historical clock drift data from a database; S202: Building a neural network-based offset determination model based on historical performance parameters and known historical clock drift data; S203: inputting the current load rate, the current operating temperature and the known historical clock drift data into the offset determination model for forward propagation, and outputting the dynamic time offset; In step S203: The offset determination model includes an input layer, a hidden layer and an output layer, wherein the input layer is used to receive the current load rate, the current operating temperature and known historical clock drift data, the hidden layer is used to learn the nonlinear relationship between the current load rate, the current operating temperature and the known historical clock drift data and the dynamic time offset by using an activation function, and the output layer is used to output the predicted dynamic time offset; S3: determining a sending time of a frame synchronization signal based on a dynamic time offset when shooting the first image frame, and sending the frame synchronization signal to the second camera module within the sending time to control the second camera module to shoot a second image frame; S4: acquiring and calculating, by the main control module, an image content overlap ratio between the first image frame and the second image frame, and comparing the image content overlap ratio with a preset threshold; S5: updating the dynamic time offset of the next frame based on the comparison result, and performing the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

2. According to the frame synchronization cooperative control method based on dual-machine linkage according to claim 1, it is characterized in that: Step S3 specifically includes the following steps: S301: Calculating the time point at which the second camera module sends a frame synchronization signal based on the dynamic time offset; S302: At the calculated frame synchronization signal sending time point, the main control module sends a frame synchronization signal to the second camera module to trigger a shooting operation of the second camera module; S303: After receiving the frame synchronization signal, the second camera module captures a second image frame according to the frame synchronization signal to achieve frame synchronization with the first camera module.

3. A frame synchronization cooperative control method based on dual-machine linkage according to claim 2, characterized in that: Step S4 specifically includes the following steps: S401: Acquire, by the main control module, first image data of a first image frame and second image data of a second image frame; S402: extracting a first feature set of the first image data and a second feature set of the second image data and matching them, and calculating an image content overlap rate based on the matching result, wherein the image content overlap rate is calculated as follows: in, is the number of matching feature pairs in the matching results, and are the sizes of the first feature set and the second feature set, respectively. is the image content overlap ratio; S403: Compare the image content overlap rate with a preset threshold value to obtain a comparison result, wherein the comparison result includes that the image content overlap rate is less than, equal to, or greater than the threshold value.

4. The frame synchronization cooperative control method based on dual-machine linkage according to claim 3 is characterized in that: Step S5 is specifically as follows: When the image content overlap rate is greater than a threshold, the current dynamic time offset is not adjusted. When the image content overlap rate is less than or equal to the threshold, the current dynamic time offset is adjusted, and finally the frame synchronization signal is controlled again according to the unadjusted dynamic time offset or the adjusted dynamic time offset.

5. A frame synchronization cooperative control system based on dual-machine linkage, comprising a main control module and a first camera module and a second camera module interacting with the main control module, for executing a frame synchronization cooperative control method based on dual-machine linkage according to any one of claims 1 to 4, characterized in that: The main control module comprises: An instruction sending submodule, used for sending a synchronization instruction when the first camera module captures the first image frame; A dynamic time offset determination submodule, used for the main control module to respond to the synchronization instruction, obtain the performance parameters of the first camera module and the second camera module when shooting the first image frame, and determine the dynamic time offset in combination with the known historical clock drift data between the first camera module and the second camera module; A synchronous shooting submodule, used for determining a sending time of a frame synchronization signal based on a dynamic time offset when shooting a first image frame, and sending the frame synchronization signal to a second camera module within the sending time to control the second camera module to shoot a second image frame; A comparison submodule, configured to obtain and calculate an image content overlap ratio between the first image frame and the second image frame through the main control module, and compare the image content overlap ratio with a preset threshold; The updating submodule is used to update the dynamic time offset of the next frame based on the comparison result, and to perform the control of the frame synchronization signal again according to the updated dynamic time offset until the stop condition is met.

Citation Information

Patent Citations

  • Compensation method and device for holder drift, holder and unmanned aerial vehicle

    CN106525074A

  • Intelligent traffic control method, intelligent traffic control device, electronic equipment and storage medium

    CN110796864A

  • Visual inertial odometer online time calibration method based on two-stage extended Kalman filtering

    CN116592917A

  • Distributed unit timing synchronization

    US20240098666A1