Frequency coordination-based dual-spectrum image soft synchronization method and system for internet access camera

Abstract

The invention discloses an internet access camera dual-spectrum image soft synchronization method based on frequency coordination. The method comprises the following steps: step 1, measuring an initial time difference delta T0 of a visible light camera and an infrared camera in a dual-camera system; step 2, adjusting an infrared frame rate; and step 3, periodic feedback updating. The frame synchronization of the infrared and visible light images is realized by respectively calculating the image output time of the images in the infrared and visible light image processing modules and dynamically adjusting the frame rate difference of the two paths of flows. The application of the method can avoid greatly modifying a bottom-layer architecture and reducing the system coupling degree, realizes efficient synchronization of the dual-spectrum image without depending on an external synchronization signal, can be suitable for a scene in which a network camera pushes a dual-path image flow, and improves the image fusion quality and the system stability.

Description

Technical Field

[0001] This invention relates to the field of frame synchronization technology for infrared and visible light images, and particularly to a method and system for soft synchronization of dual-spectral images from a network camera based on frequency coordination. Background Technology

[0002] The purpose of this invention is to solve the problem of frame asynchrony between infrared and visible light images in current network cameras, especially when the underlying framework only supports a single stream and visible light images need to be processed by an ISP, while infrared images do not. This results in the infrared image transmission speed being faster than the visible light image, thus causing frame rate inconsistency and synchronization difficulties in dual-spectrum images.

[0003] Therefore, how to achieve efficient synchronization of dual-spectral images without relying on external synchronization signals, without significantly modifying the underlying architecture and reducing system coupling, and how to make it applicable to scenarios where network cameras push dual-channel image streams, thereby improving image fusion quality and system stability, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the present invention provides a method and system for soft synchronization of dual-spectral images of network cameras based on frequency coordination. The purpose is to avoid significant modifications to the underlying architecture and reduce system coupling, and to achieve efficient synchronization of dual-spectral images without relying on external synchronization signals. It can be applied to scenarios where network cameras push dual-channel image streams, thereby improving image fusion quality and system stability.

[0005] To achieve the above objectives, this invention discloses a frequency-coordinated soft synchronization method for dual-spectral images of a network camera; characterized by comprising the following steps:

[0006] Step 1: Measure the initial time difference ΔT0 between the visible light camera and the infrared camera in the dual-camera system;

[0007] Step 2: Adjust the infrared frame rate;

[0008] Step 3: Periodic feedback and updates.

[0009] Preferably, before performing step 1, the dual-camera system is activated and an initial frame rate is set;

[0010] The visible light images captured by the visible light camera are processed by an ISP.

[0011] The infrared images captured by the infrared camera are not processed by an ISP.

[0012] When setting the initial frame rate, the ISP process sets the frame rate of the visible light camera to be consistent with that of the infrared camera.

[0013] Preferably, in step 1, two data streams are simultaneously acquired: a visible light image captured by the visible light camera and an infrared image captured by the infrared camera. After capturing the visible light image and the corresponding infrared image, the timestamp T of the visible light image is recorded. visible,0 and the corresponding timestamp T of the infrared image ir,0 ;

[0014] Then, the initial time difference ΔT0 = T ir,0 -T visible,0

[0015] Preferably, step 2 includes initial frequency reduction and dynamic feedback frequency reduction;

[0016] The initial frequency reduction is as follows: after measuring the initial time difference ΔT0 between the visible light camera and the infrared camera, the infrared frame rate is reduced by inter-frame insertion delay of the infrared image captured by the infrared camera, so that the visible light image frame rate of the visible light camera is close to the infrared frame rate.

[0017] More preferably, the dynamic feedback frequency reduction is specifically as follows:

[0018] Step 2.1: Continuously collect the processing time of each frame of the visible light image at the same time interval, and simultaneously record the image output time of each frame of the visible light image captured by the visible light camera and the infrared image, and record the timestamp T of each visible light image. visible,n and the corresponding timestamp T of the infrared image ir,n ;

[0019] Step 2.2: Dynamically update the infrared frame delay:

[0020] Step 2.2.1: Time difference calculation;

[0021] Based on the timestamp T of each of the visible light images visible,n and the corresponding timestamp T of the infrared image ir,n Obtain the time difference ΔT between the current visible light frame and the infrared frame. n ;

[0022] ΔT n =T ir,n -T visible,n ;

[0023] Step 2.2.2: Adjust the delay of the infrared camera according to the new time difference;

[0024] If the time difference ΔT between the current visible light frame and the infrared frame is nA negative number indicates that the infrared camera is faster than the visible light camera, according to ΔT. n Increase the latency of the infrared camera;

[0025] If the time difference ΔT between the current visible light frame and the infrared frame is n A positive number indicates that the infrared camera is slower than the visible light camera, according to ΔT. n Reduce the latency of the infrared camera.

[0026] Preferably, step 3, periodic feedback update, refers to periodically updating the frame frequency and performing feedback updates at the same time interval; specifically, it involves continuously collecting the processing time of each frame of the visible light image and adjusting the delay value of the corresponding frame of the infrared image, so that each frame of the infrared image is precisely synchronized with the corresponding frame in the visible light image.

[0027] Preferably, the same time intervals in steps 2.1 and 3 are 15 seconds, 30 seconds, 45 seconds, 60 seconds, or 75 seconds.

[0028] Ideally, fault tolerance mechanisms should also be included;

[0029] The fault-tolerance mechanism means that if an external factor is detected that causes the time difference between each frame of the infrared image and the corresponding frame in the visible light image to exceed a set threshold, then step 2 is repeated.

[0030] The present invention also provides a system for executing the above-mentioned frequency-coordinated dual-spectral image soft synchronization method for network cameras, including a visible light camera, an infrared camera, an ISP processing module, a frame synchronization module, a feedback module, and a fault tolerance mechanism module.

[0031] The beneficial effects of this invention are:

[0032] The application of this invention can avoid significant modifications to the underlying architecture and reduce system coupling. It can achieve efficient synchronization of dual-spectral images without relying on external synchronization signals, and can be applied to scenarios where network cameras push dual-channel image streams, thereby improving image fusion quality and system stability.

[0033] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0034] Figure 1 A flowchart of an embodiment of the present invention is shown.

[0035] Figure 2 A schematic diagram of the data flow process in one embodiment of the present invention is shown.

[0036] Figure 3 A structural diagram of a system according to an embodiment of the present invention is shown. Detailed Implementation

[0037] Example

[0038] like Figures 1 to 2 As shown, the frequency-coordinated dual-spectral image soft synchronization method for network cameras includes the following steps:

[0039] Step 1: Measure the initial time difference ΔT0 between the visible light camera and the infrared camera in the dual-camera system;

[0040] Step 2: Adjust the infrared frame rate;

[0041] Step 3: Periodic feedback and updates.

[0042] The invention achieves frame synchronization of infrared and visible light images by calculating the image output time in the infrared and visible light image processing modules respectively and dynamically adjusting the frame rate difference between the two streams.

[0043] In some embodiments, the dual-camera system is activated and an initial frame rate is set before performing step 1;

[0044] Among them, the visible light images captured by the visible light camera are processed by ISP;

[0045] Infrared images captured by infrared cameras are not processed by an ISP.

[0046] When setting the initial frame rate, the visible light camera's frame rate is set to be consistent with that of the infrared camera through ISP processing.

[0047] In some embodiments, in step 1, two data streams are simultaneously acquired: a visible light image captured by a visible light camera and an infrared image captured by an infrared camera. After capturing the visible light image and the corresponding infrared image, the timestamp T of the visible light image is recorded. visible,0 and the corresponding infrared image timestamp T ir,0 ;

[0048] Therefore, the initial time difference ΔT0 = T ir,0 -T visible,0

[0049] This invention determines the synchronization deviation between a visible light camera and an infrared camera by detecting the initial time difference ΔT0. To address the issue of the infrared image frame rate being faster than the visible light frame rate, a waiting method is used to adjust the infrared frame rate. Specifically, each time an infrared image frame is output, a delay time is set to the calculated time difference, thereby synchronizing the infrared frame rate with the visible light frame rate as much as possible.

[0050] In some embodiments, step 2 includes initial frequency reduction and dynamic feedback frequency reduction;

[0051] The initial frequency reduction is as follows: after measuring the initial time difference ΔT0 between the visible light camera and the infrared camera, the infrared frame rate is reduced by inserting a delay between the frames of the infrared images captured by the infrared camera, so that the visible light image frame rate of the visible light camera is close to the infrared frame rate.

[0052] In some embodiments, dynamic feedback frequency reduction is specifically as follows:

[0053] Step 2.1: Continuously acquire the processing time of each frame of the visible light image at the same time interval, and simultaneously record the image output time of each frame of the visible light image and infrared image captured by the visible light camera, and record the timestamp T of each visible light image. visible,n and the corresponding infrared image timestamp T ir,n ;

[0054] Step 2.2: Dynamically update the infrared frame delay:

[0055] Step 2.2.1: Time difference calculation;

[0056] Based on the timestamp T of each visible light image visible,n and the corresponding infrared image timestamp T ir,n Obtain the time difference ΔT between the current visible light frame and the infrared frame. n ;

[0057] ΔT n =T ir,n -T visible,n ;

[0058] Step 2.2.2: Adjust the delay of the infrared camera according to the new time difference;

[0059] If the time difference ΔT between the current visible light frame and the infrared frame is n A negative value indicates that the infrared camera is faster than the visible light camera, according to ΔT. n Improve the latency of infrared cameras;

[0060] If the time difference ΔT between the current visible light frame and the infrared frame is n A positive value indicates that the infrared camera is slower than the visible light camera, according to ΔT. n To reduce latency in infrared cameras, this invention addresses the issue that system performance can degrade due to factors such as increased temperature leading to decreased CPU frequency and increased memory utilization, resulting in latency fluctuations. To improve synchronization accuracy, this invention further introduces a dynamic feedback mechanism to resolve these problems.

[0061] In some embodiments, step 3, periodic feedback update, refers to periodically updating the frame frequency and performing feedback updates at the same time interval; specifically, it involves continuously collecting the processing time of each frame of the visible light image and adjusting the delay value of the corresponding infrared image frame, so that each frame of the infrared image is precisely synchronized with the corresponding frame in the visible light image.

[0062] In practical applications, regularly updating the frame frequency can ensure long-term synchronization between infrared and visible light frames.

[0063] After each feedback update, the delay value of each frame of the infrared image is gradually adjusted according to the time difference between the corresponding visible light frame and the infrared frame to ensure high-precision synchronization with the visible light frame.

[0064] After multiple iterations and updates, the time difference between each frame of the infrared image and the corresponding frame in the visible light image can be controlled to within 8 decimal places, ensuring frame synchronization.

[0065] In some embodiments, the same time interval in steps 2.1 and 3 is 15 seconds, 30 seconds, 45 seconds, 60 seconds, or 75 seconds.

[0066] In some embodiments, a fault-tolerance mechanism is also included;

[0067] The fault-tolerance mechanism means that if an external factor is detected that causes the time difference between each frame of the infrared image and the corresponding frame in the visible light image to exceed a set threshold, then step 2 is repeated.

[0068] In practical applications, frame time may change due to external factors such as network fluctuations or hardware jitter. The system should have a fault tolerance mechanism. The threshold for the fault tolerance mechanism to determine whether to re-execute step 2 is usually 1 millisecond.

[0069] like Figure 3 As shown, the present invention also provides a system for executing the frequency coordination-based dual-spectral image soft synchronization method for network cameras described above, including a visible light camera, an infrared camera, an ISP processing module, a frame synchronization module, a feedback module, and a fault tolerance mechanism module.

[0070] Unlike traditional methods that use fixed time adjustments to achieve frame synchronization, this invention uses a dynamic feedback mechanism to collect the time difference between visible light and infrared frames in real time, and dynamically adjusts the infrared frame rate according to changes in system performance (such as CPU frequency and memory usage), thus achieving a more accurate synchronization effect.

[0071] The frame rate captured by the infrared camera in this invention can be automatically adjusted according to the time difference of each frame captured by the visible light camera. A feedback mechanism is used to achieve periodic acquisition and calibration, thereby greatly improving the frame rate of the infrared camera and the synchronization accuracy of the visible light camera. This mechanism allows the system to maintain frame rate synchronization even after long-term operation, enhancing the system's robustness.

[0072] This invention can also use a hardware timer to automatically collect the time difference between visible light frames and infrared frames at fixed intervals, such as 60 seconds, with high precision. Compared to a software timer, this method provides more accurate time acquisition and avoids acquisition errors caused by system load fluctuations.

[0073] This invention employs an adaptive infrared frame delay mechanism. Each time a new visible light frame time is detected, the time difference is automatically calculated and the infrared frame delay value is updated in real time. In this way, the infrared frame can dynamically adjust its frequency to gradually approach the frame rate of the visible light frame, and the synchronization accuracy is improved through continuous iteration.

[0074] This invention can precisely control the time difference between infrared frames and visible light frames in accordance with the IEEE 754 standard through multiple periodic feedback updates.

[0075] This level of precision is far superior to traditional frame synchronization methods, ensuring that a high degree of frame rate synchronization can still be achieved after long-term operation.

[0076] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for soft synchronization of dual-spectral images from a network camera based on frequency coordination; characterized in that, Includes the following steps: Step 1: Measure the initial time difference ΔT0 between the visible light camera and the infrared camera in the dual-camera system; Step 2: Adjust the infrared frame rate; Step 3: Periodic feedback and updates.

2. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, Before performing step 1, start the dual-camera system and set the initial frame rate; The visible light images captured by the visible light camera are processed by an ISP. The infrared images captured by the infrared camera are not processed by an ISP. When setting the initial frame rate, the ISP process sets the frame rate of the visible light camera to be consistent with that of the infrared camera.

3. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, In step 1, two data streams are simultaneously acquired: a visible light image captured by the visible light camera and an infrared image captured by the infrared camera. After capturing the visible light image and the corresponding infrared image, the timestamp T of the visible light image is recorded. visible,0 and the corresponding timestamp T of the infrared image ir,0 ; Then, the initial time difference ΔT0 = T ir,0 -T visible,0 .

4. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, Step 2 includes initial frequency reduction and dynamic feedback frequency reduction; The initial frequency reduction is as follows: after measuring the initial time difference ΔT0 between the visible light camera and the infrared camera, the infrared frame rate is reduced by inter-frame insertion delay of the infrared image captured by the infrared camera, so that the visible light image frame rate of the visible light camera is close to the infrared frame rate.

5. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 4, characterized in that, The dynamic feedback frequency reduction is described in the following details: Step 2.1: Continuously collect the processing time of each frame of the visible light image at the same time interval, and simultaneously record the image output time of each frame of the visible light image captured by the visible light camera and the infrared image, and record the timestamp T of each visible light image. visible,n and the corresponding timestamp T of the infrared image ir,n ; Step 2.2: Dynamically update the infrared frame delay: Step 2.2.1: Time difference calculation; Based on the timestamp T of each of the visible light images visible,n and the corresponding timestamp T of the infrared image ir,n Obtain the time difference ΔT between the current visible light frame and the infrared frame. n ; ΔT n =T ir,n -T visible,n ; Step 2.2.2: Adjust the delay of the infrared camera according to the new time difference; If the time difference ΔT between the current visible light frame and the infrared frame is n A negative number indicates that the infrared camera is faster than the visible light camera, according to ΔT. n Increase the latency of the infrared camera; If the time difference ΔT between the current visible light frame and the infrared frame is n A positive number indicates that the infrared camera is slower than the visible light camera, according to ΔT. n Reduce the latency of the infrared camera.

6. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, Step 3, periodic feedback update, refers to periodically updating the frame frequency and performing feedback updates at the same time interval. Specifically, it involves continuously collecting the processing time of each frame of the visible light image and adjusting the delay value of the corresponding frame of the infrared image, so that each frame of the infrared image is precisely synchronized with the corresponding frame in the visible light image.

7. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, The same time intervals mentioned in steps 2.1 and 3 are 15 seconds, 30 seconds, 45 seconds, 60 seconds, or 75 seconds.

8. The method for soft synchronization of dual-spectral images of a network camera based on frequency coordination according to claim 1, characterized in that, It also includes fault tolerance mechanisms; The fault-tolerance mechanism refers to the process of repeating step 2 if an external factor is detected that causes the time difference between each frame of the infrared image and the corresponding frame in the visible light image to exceed a set threshold.

9. The system, characterized in that, The method for performing frequency-coordinated dual-spectral image soft synchronization of a network camera as described in any one of claims 1 to 9 includes a visible light camera, an infrared camera, an ISP processing module, a frame synchronization module, a feedback module, and a fault-tolerance mechanism module.

Images