Binocular camera image timing determination method, electronic device and storage medium
By using vibrators to determine the timing relationship of binocular camera images in the photocoagulator, the problem of inconsistent timing of binocular camera acquisition is solved, and the stitched image is consistent with the actual scene, which improves the treatment effect of the photocoagulator.
Patent Information
- Application Number
- CN202410314683.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-03-19
AI Technical Summary
In the photocoagulator, the inconsistent timing of the images acquired by the binocular camera leads to a large difference between the stitched image and the actual scene, affecting the treatment effect.
When shooting at the first camera module and the second camera module, y images are captured at each interval, and the preset vibrating body is controlled to vibrate once. The vibrating body is arranged in the overlap shooting area, and the single vibration time of the vibrating body is smaller than that of the shooting Period, the vibration amplitudes of any two times are different, and the positional relationship of the vibrating body is used to determine the image timing relationship.
It effectively avoids differences caused by stitching of images taken at different moments, ensures that the stitching of images is consistent with the actual scene, and improves the treatment effect of the photocoagulator.
Smart Images

Figure CN118042268B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of photocoagulation instruments, and in particular to a binocular camera image timing determination method, electronic equipment and storage medium. Background Art
[0002] The ophthalmic treatment system, also known as a photocoagulator, is an instrument used to treat the user's eye disease area. Its observation part is similar to MR (mixed reality), and the surgical scene is mainly obtained through the surgical field camera, rather than the eyes directly seeing the surgical scene. The area obtained by a single surgical field camera is limited. In order to increase the visible surgical scene, two surgical field cameras are usually used to obtain two images with overlapping areas, and then stitch them together to obtain a larger surgical scene. Although the two surgical field cameras can be controlled to shoot the corresponding areas at the same time, the time length for different surgical field cameras to process images will not be exactly the same, resulting in differences in the timing of the image sequences output by the two surgical field cameras. If two images taken at different times are stitched together, the stitched image will be quite different from the actual scene. Summary of the invention
[0003] In view of the above technical problems, the technical solution adopted by the present invention is:
[0004] According to a first aspect of the present application, a method for determining a binocular camera image timing is provided, the method comprising:
[0005] S100, when the first camera module and the second camera module are shooting, y images are shot at intervals, and a preset vibrating body is controlled to vibrate once; wherein, the vibrating body is arranged in the overlapping shooting area of the first camera module and the second camera module; the single vibration duration of the vibrating body is less than the shooting cycle of the first camera module and the second camera module; and the vibration amplitudes of any two times of the vibrating body are different.
[0006] S200, acquiring the latest 2y+2 first images output by the first camera module to obtain a third image list SW=(SW1, SW2, ..., SW x , …, D 2y+2 ), x=1, 2, ..., 2y+2; and obtaining the latest 2y+2 second images output by the second camera module to obtain a fourth image list FW=(FW1, FW2, ..., FW x ,…,FW 2y+2 );where SW x The latest x-th first image output by the first camera module, FW x It is the latest x-th second image output by the second camera module; the first image in SW and the second image in FW are arranged in chronological order of shooting time.
[0007] S300, traverse SW, if SW x If the vibrating body is not in the initial position, then SW x It is determined as the middle first image to obtain two middle first images ZH1 and ZH2; wherein ZH1 is arranged before ZH2 in SW.
[0008] S400, traverse FW, if FW x If the vibrating body in the case is not in the initial position, then FW x It is determined as the intermediate second image to obtain two intermediate second images BH1 and BH2; wherein BH1 is arranged before BH2 in FW.
[0009] S500 , determining a time sequence relationship between a first image in SW and a second image in FW according to arrangement positions of ZH1 and ZH2 in SW and arrangement positions of BH1 and BH2 in FW.
[0010] According to another aspect of the present application, a non-transitory computer-readable storage medium is also provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned binocular camera image timing determination method.
[0011] According to another aspect of the present application, an electronic device is provided, including a processor and the above-mentioned non-transitory computer-readable storage medium.
[0012] The present invention has at least the following beneficial effects:
[0013] The binocular camera image timing determination method of the present invention is as follows: when the first camera module and the second camera module are shooting, y images are shot at intervals, and a preset vibrating body is controlled to vibrate once; and the vibrating body is arranged in the overlapping shooting area of the first camera module and the second camera module; the single vibration duration of the vibrating body is less than the shooting cycle of the first camera module and the second camera module; the vibration amplitudes of any two times of the vibrating body are different; thereby, the position of the vibrating body in the first image taken when the vibrating body vibrates and the position of the vibrating body in the second image are the same, and the position of the vibrating body in any two first images is different, and the position of the vibration in any two second images is different; thereby, the timing relationship between the acquired first image and the second image is determined according to the first image and the second image in which the vibrating body exists, so as to avoid splicing two images taken at different times, resulting in a problem that the spliced image is greatly different from the actual scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0015] Figure 1 A flowchart of a method for determining the timing of binocular camera images provided by an embodiment of the present invention.
[0016] Figure 2 A flowchart of a binocular camera synchronization control method provided by an embodiment of the present invention;
[0017] Figure 3 This is a framework diagram of a binocular camera synchronization control system provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0019] It should be noted that, based on the present disclosure, those skilled in the art should understand that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.
[0020] The following will refer to Figure 1 The flowchart of the method for determining the timing of binocular camera images shown in the figure introduces a method for determining the timing of binocular camera images.
[0021] The binocular camera image timing determination method may include the following steps:
[0022] S100, when the first camera module and the second camera module are shooting, y images are shot at intervals, and a preset vibrating body is controlled to vibrate once; wherein, the vibrating body is arranged in the overlapping shooting area of the first camera module and the second camera module; the single vibration duration of the vibrating body is less than the shooting cycle of the first camera module and the second camera module; and the vibration amplitudes of any two times of the vibrating body are different.
[0023] In this embodiment, the vibrator can be a controllable elastic vibrator, for example, an electrically controlled vibration device; after the vibrator vibrates once, it returns to its initial position. Since the vibrator is arranged in the overlapping shooting area of the first camera module and the second camera module, and the vibration amplitudes of the vibrator are different at any two times, the position of the vibrator in the first image taken when the vibrator vibrates is the same as the position of the vibrator in the second image, and the position of the vibrator in any two first images is different, and the position of the vibration in any two second images is different; so as to facilitate the subsequent accurate identification of the timing relationship between the first image and the second image.
[0024] S200, acquiring the latest 2y+2 first images output by the first camera module to obtain a third image list SW=(SW1, SW2, ..., SW x , …, D 2y+2 ), x=1, 2, ..., 2y+2; and obtaining the latest 2y+2 second images output by the second camera module to obtain a fourth image list FW=(FW1, FW2, ..., FW x ,…,FW 2y+2 );where SW x The latest x-th first image output by the first camera module, FW x It is the latest x-th second image output by the second camera module; the first image in SW and the second image in FW are arranged in chronological order of shooting time.
[0025] In this embodiment, when the first camera module and the second camera module are shooting, since y images are taken at every interval, the preset vibrating body is controlled to vibrate once; therefore, the latest 2y+2 first images and the latest 2y+2 first images are obtained, so that SW must include two first images taken when the vibrating body is vibrating, and FW must include two second images taken when the vibrating body is vibrating, thereby providing a basis for subsequent timing determination.
[0026] S300, traverse SW, if SW x If the vibrating body is not in the initial position, then SW x It is determined as the middle first image to obtain two middle first images ZH1 and ZH2; wherein ZH1 is arranged before ZH2 in SW.
[0027] In this embodiment, the fact that the vibrating body is not in the initial position indicates that the vibrating body is in a vibrating state; by traversing SW and identifying the position of the vibrating body in each first image in SW, ZH1 and ZH2 can be determined.
[0028] S400, traverse FW, if FW x If the vibrating body in the case is not in the initial position, then FW xIt is determined as the intermediate second image to obtain two intermediate second images BH1 and BH2; wherein BH1 is arranged before BH2 in FW.
[0029] By traversing SW and identifying the position of the vibrating body in each first image in SW, BH1 and BH2 can be determined.
[0030] S500 , determining a time sequence relationship between a first image in SW and a second image in FW according to arrangement positions of ZH1 and ZH2 in SW and arrangement positions of BH1 and BH2 in FW.
[0031] In this embodiment, determining the timing relationship between the first image in SW and the second image in FW according to the arrangement positions of ZH1 and ZH2 in SW and the arrangement positions of BH1 and BH2 in FW may include the following steps:
[0032] S510, if the position of the vibrator in ZH1 is the same as the position of the vibrator in BH1, and the position of the vibrator in ZH2 is the same as the position of the vibrator in BH2, then obtain the arrangement number N1 of ZH1 in SW and the arrangement number M1 of BH1 in FW.
[0033] This situation indicates that ZH1 in SW and BH1 in FW are taken at the same time, and ZH2 in SW and BH2 in FW are taken at the same time; the arrangement number N1 of ZH1 in SW and the arrangement number M1 of BH1 in FW can be obtained.
[0034] S511, according to N1 and M1, determine the difference ΔNM1=|M1-N1| between the arrangement numbers of ZH1 and BH1.
[0035] It should be noted that the difference ΔNM1 between the arrangement numbers of ZH1 and BH1 may also be determined according to the arrangement number of ZH2 in SW and the arrangement number of BH2 in FW to achieve the same effect.
[0036] S512, if ΔNM1 = 0, determine SW x Shooting time and FW x The shooting time is the same.
[0037] In this embodiment, ΔNM1=0, indicating that the first image and the second image with the same sequence number in SW and FW are taken at the same time; therefore, SW is determined x Shooting time and FW x The shooting time is the same.
[0038] S513, if N1<M1, determine SW a Shooting time and FW a+ΔNM1The shooting time is the same; a=1, 2,…, y-ΔNM1.
[0039] In this embodiment, if N1<M1, it means that the timing of the second image in FW is later than the timing of the corresponding first image in SW by an amount ΔNM1. Therefore, SW is determined a Shooting time and FW a+ΔNM1 The shooting time is the same.
[0040] S514, if N1>M1, determine FW a Shooting time and SW a+ΔNM1 The shooting time is the same.
[0041] In this embodiment, if N1>M1, it means that the timing of the first image in SW is later than the timing of the corresponding second image in FW, and the amount of the later timing is ΔNM1; therefore, FW is determined a Shooting time and SW a+ΔNM1 The shooting time is the same.
[0042] S520, if the position of the vibrator in ZH1 is the same as the position of the vibrator in BH2, and the position of the vibrator in ZH2 is different from the position of the vibrator in BH1, then obtain the arrangement number N2 of ZH1 in SW and the arrangement number M2 of BH2 in FW.
[0043] This situation indicates that ZH1 in SW and BH2 in FW are taken at the same time, and there is only one set of first image and second image of the vibrating body in a vibrating state; the arrangement number N2 of ZH1 in SW and the arrangement number M2 of BH2 in FW can be obtained.
[0044] It is understandable that this situation generally occurs when ZH1 is in the middle position of SW. At this time, BH1 is the corresponding first image and is not in SW.
[0045] S521, according to N2 and M2, determine the difference in arrangement numbers between ZH1 and BH2, ΔNM2 = M2 - N2.
[0046] S522, determine SW b Shooting time and FW b+ΔNM2 The shooting time is the same; where b=1, 2,…, y-ΔNM2.
[0047] In this embodiment, ZH1 in SW and BH2 in FW are taken at the same time, which means that the timing of the second image in FW is later than the timing of the corresponding first image in SW, so N2 must be less than M2; and the amount of later is ΔNM2; therefore, SW is determined b Shooting time and FW b+ΔNM2 The shooting time is the same.
[0048] Further, step S500 may include the following steps:
[0049] S530, if the position of the vibrator in ZH2 is the same as the position of the vibrator in BH1, and the position of the vibrator in ZH1 is different from the position of the vibrator in BH2, then obtain the arrangement number N3 of ZH2 in SW and the arrangement number M3 of BH1 in FW.
[0050] This situation indicates that ZH2 in SW and BH1 in FW are taken at the same time, and there is only one set of first image and second image of the vibrating body in a vibrating state; the arrangement number N3 of ZH2 in SW and the arrangement number M3 of BH1 in FW can be obtained.
[0051] S531, according to N3 and M3, determine the difference in arrangement sequence numbers between ZH2 and BH1, ΔNM3 = N3 - M3.
[0052] S532, determine FW c Shooting time and SW c+ΔNM3 The shooting time is the same; where c = 1, 2, …, y-ΔNM3.
[0053] In this embodiment, ZH2 in SW and BH1 in FW are taken at the same time, which means that the timing of the first image in SW is later than the timing of the corresponding second image in FW, so N3>M3; and the amount of later is ΔNM3; therefore, determine FW c Shooting time and SW c+ΔNM3 The shooting time is the same.
[0054] A method for determining the timing of binocular camera images, when a first camera module and a second camera module are shooting, y images are shot at intervals, and a preset vibrator is controlled to vibrate once; and the vibrator is set in the overlapping shooting area of the first camera module and the second camera module; the single vibration duration of the vibrator is less than the shooting cycle of the first camera module and the second camera module; the vibration amplitudes of any two times of the vibrator are different; thereby, the position of the vibrator in the first image taken when the vibrator vibrates and the position of the vibrator in the second image are the same, and the position of the vibrator in any two first images is different, and the position of the vibration in any two second images is different; thereby, the timing relationship between the acquired first image and the second image is determined according to the first image and the second image in which the vibrator exists, so as to avoid splicing two images taken at different times, resulting in a problem that the spliced image is greatly different from the actual scene.
[0055] Furthermore, the use of a vibrator can avoid the influence of preset color or light on the first image and the second image. No complicated image processing is required for subsequent stitching. The stitching can be performed by simply removing the vibrator, and the stitching efficiency and stitching effect are good.
[0056] In an exemplary embodiment, in order to ensure that the first camera module and the second camera module shoot at the same time, a binocular camera synchronization control method is provided; the binocular camera synchronization control method can be applied to a photocoagulation device having a binocular camera, such as Figure 2 As shown, the binocular camera synchronization control method may include the following steps:
[0057] T100, in response to generating a preset shooting instruction, starting a preset timing T now ; Wherein, the shooting instruction is used to simultaneously start the preset first camera module and second camera module to shoot the corresponding area.
[0058] In this embodiment, the binocular camera synchronization control method can be applied to Figure 3 The binocular camera synchronization control system shown in the figure includes a synchronization control board, a first camera module and a second camera module; wherein the first camera module includes a first camera and a first image processing module, and the second camera module includes a second camera and a second image processing module; the first camera is communicatively connected to the first image processing module, and the picture taken by the first camera is output to the first image processing module, and the first image processing module processes the image output by the first camera and outputs the first image; the second camera is communicatively connected to the second image processing module, and the picture taken by the second camera is output to the second image processing module, and the second image processing module processes the image output by the second camera and outputs the second image.
[0059] The synchronization control board is respectively connected to the first camera, the first image processing module, the second camera and the second image processing module; the synchronization control board can generate a preset shooting instruction, send it to the first camera and the second camera at the same time, and then start the preset timing T now ; At the same time, when the first image processing module outputs the first image, it can generate a first image generation instruction; when the second image processing module outputs the second image, it can generate a second image generation instruction; the synchronous control board can also receive in real time the first image generation instruction sent by the first image processing module and the second image generation instruction sent by the second image processing module.
[0060] T200, if T now≤T0, and a first image generation instruction Z1 corresponding to the first camera module and a second image generation instruction Z2 corresponding to the second camera module are received, then a time interval Δt1 between receiving Z1 and receiving Z2 is determined.
[0061] In this embodiment, within a timing cycle T0, the first image generation instruction sent by the first image processing module and the second image generation instruction sent by the second image processing module can be acquired by the synchronization control board; for example, the synchronization control board now ≤T0, the time when Z1 is received is t1, the time when t2 is received is t2, Δt1=|t2-t1|; T0 can be 20ms, thereby ensuring that the first camera module and the second camera module output 50 images within 1S.
[0062] Further, in order to increase the shooting frequency of the first camera module and the second camera module, T0 may be set to be smaller. However, if T0 is set to be too small, misjudgment may be caused. Based on the above problem, T0 may be determined by the following steps:
[0063] T210, obtaining the initial duration of each first image output by the first camera module and the initial duration of each second image output by the second camera module in the historical time period, so as to obtain an initial duration list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; where, A i is the i-th initial duration obtained, and n is the number of initial durations obtained.
[0064] For the first camera module and the second camera module, due to environmental factors or the influence of their own precision, the initial duration of each first image corresponding to the output of the first camera module in the historical time period and the initial duration of each second image corresponding to the output of the second camera module are not exactly the same. The initial duration of each first image output by the first camera module in the historical time period and the initial duration of each second image output by the second camera module can be obtained to obtain an initial duration list A; it should be noted that the initial duration in A includes the initial duration corresponding to the first camera module and the initial duration corresponding to the second camera module, and the ratio of the corresponding numbers between the two is close to 1:1.
[0065] T220, obtain the initial duration volatility η corresponding to A = 1 / n×∑ n i=1 (A i -(1 / n×∑ n i=1 A i )) 2 .
[0066] T230, if η<QT, then determine T0=α×1 / n×∑ n i=1 A i ; Otherwise, go to step S240; wherein QT is the preset initial duration volatility threshold; α is the preset first weight, α>1.
[0067] In this embodiment, the larger the value of η is, the greater the fluctuation of the initial duration in A is. If η≥QT, it means that the fluctuation of the initial duration in A is too large, and some initial durations in A are abnormal. Therefore, only when η<QT can the initial duration in A be used to determine T0, thereby improving the accuracy and rationality of the determined T0. The setting of α can make the determined T0 slightly larger, so that the redundancy of this method is better.
[0068] T240, arrange the initial durations in A in ascending order to obtain an arranged initial duration list B = (B1, B2, ..., B i , …, B n ), where B i is the i-th initial duration obtained by arranging the initial durations in A in ascending order.
[0069] T250, determine the first m initial durations in B as the intermediate durations, so as to obtain an intermediate duration list C = (C1, C2, ..., C j , …, C m ), j = 1, 2, ..., m; where C j is the jth intermediate duration among the m intermediate durations; β is a preset second weight, 0<β<1; is the default floor function.
[0070] In this embodiment, β may be 0.9 or 0.8, so that some relatively large initial durations can be filtered out, and the first 80% or 90% of the initial durations in B are used to determine T0.
[0071] T260, according to C, determine T0 = α × 1 / m × ∑ m j=1 C j .
[0072] In this embodiment, by determining T0 using the above method, abnormal data in historical data can be filtered out, and only relatively normal data is used to determine T0, so that the determined T0 is more reasonable and accurate, and will not cause T0 to be too small.
[0073] T300, if Δt1≤T', then at T now=T0, generate the preset shooting instruction again and enter step T100; otherwise, stop generating the preset shooting instruction again; wherein Z1 is used to represent the first image corresponding to the output of the first camera module, and Z2 is used to represent the second image corresponding to the output of the second camera module; T0 is the preset timing cycle; T' is the preset time interval threshold.
[0074] In this embodiment, if Δt1≤T', it means that the difference between the time taken for the first camera module to output the first image and the time taken for the second camera module to output the second image is within a preset range. now = T0, the preset shooting instruction is generated again, thereby ensuring that the timing of the first image and the second image is the same or similar; T' may be 5ms.
[0075] In this embodiment, set T now = T0, the preset shooting instruction is generated again, which can ensure that the shooting time of the first camera module and the second camera module is the same each time, and further ensure that the timing of the first image and the second image is the same or similar.
[0076] Furthermore, after step T300, the method may further include the following steps:
[0077] T400, if T now = T0, and Z1 and Z2 are not received completely, then at T now = T0, stop generating the preset shooting instructions again.
[0078] In this embodiment, when a timing cycle ends, if Z1 and Z2 are not all received, for example, only Z1 or Z2 is received, or both Z1 and Z2 are not received, it means that at least one of the first camera module and the second camera module is abnormal. now = T0, stop generating the preset shooting instruction again; and further generate an alarm prompt to remind the user that at least one of the first camera module and the second camera module has an abnormality.
[0079] Furthermore, after step T300, the method may further include the following steps:
[0080] T500, if T now =T0, and Z1 is received but Z2 is not received, it is determined that the second camera module is abnormal.
[0081] T510, if T now =T0, and Z1 is not received but Z2 is received, it is determined that the first camera module is abnormal.
[0082] In this embodiment, at T now= T0, the above method can be used to further determine which camera module is abnormal, so that the abnormal camera module can be maintained in a targeted manner.
[0083] The binocular camera synchronization control method of this embodiment, in response to generating a preset shooting instruction, starts a preset timing T now ; Simultaneously start the first camera module and the second camera module to shoot the corresponding area. If T now ≤T0, and a first image generation instruction Z1 corresponding to the first camera module and a second image generation instruction Z2 corresponding to the second camera module are received, then the time interval Δt1 between receiving Z1 and receiving Z2 is determined; if Δt1≤T', then at T now =T0, the preset shooting instruction is generated again; thereby, the first camera module and the second camera module start shooting at the same time each time, ensuring that the two subsequently spliced images are shot at the same time, and the spliced image is consistent with the actual scene, thereby improving the treatment effect of the photocoagulation device.
[0084] Furthermore, under the premise of Δt1≤T', now =T0, the preset shooting instruction is generated again, so that after the first camera module and the second camera module shoot simultaneously, the time interval between the first camera module generating the corresponding first image and the second camera module generating the corresponding second image is within T0, avoiding the time interval between the first camera and the second camera generating images being too long, resulting in timing disorder of subsequent images; thereby further improving the treatment effect of the photocoagulation device.
[0085] In a typical embodiment, the method in the above embodiment can ensure that the first camera module and the second camera module shoot at the same time, but there will still be a difference in the timing of the first image sequence output by the first camera module and the second image sequence output by the second camera module. When the first image and the second image are spliced later, if the timing of the two spliced images is different, the spliced image obtained by splicing will be different from the actual scene; based on this, when the first camera module and the second camera module shoot, q images are taken each time, and light of a preset color is emitted to the shooting areas corresponding to the first camera module and the second camera module; wherein the preset color is a color other than the colors included in the first image and the second image;
[0086] After step T300, the method may further include the following steps:
[0087] T600: Acquire the latest q+1 first images output by the first camera module to obtain a first image list D=(D1, D2, ..., D p , …, D q+1), p = 1, 2, ..., q + 1; and obtaining the latest q + 1 second images output by the second camera module to obtain a second image list E = (E1, E2, ..., E p , …, E q+1 ), where D p is the latest p-th first image output by the first camera module, E p It is the latest p-th second image output by the second camera module; the first image in D and the second image in E are arranged in chronological order of shooting time.
[0088] T610 , obtaining a first image WA with a preset color in D and a second image WB with a preset color in E.
[0089] In this embodiment, q images are taken each time, and light of preset colors is emitted to the shooting areas corresponding to the first camera module and the second camera module, and the latest q+1 first images and q+1 second images are obtained; therefore, D must include a first image with a preset color, and E must include a second image with a preset color; WA and WB can be determined by traversing D and E.
[0090] T620, determine the arrangement number NUM1 of WA in D according to the arrangement position of WA in D; and determine the arrangement number NUM2 of WB in E according to the arrangement position of WB in E.
[0091] T630, based on NUM1 and NUM2, determine the arrangement sequence number difference ΔNUM=|NUM1-NUM2| corresponding to WA and WB.
[0092] T640, if 0<ΔNUM<QE, and NUM1<NUM2, then D r and E r+ΔNUM Determine the images taken at the same time; r=1, 2, ..., q+1-ΔNUM; QE is a preset first sequence number difference threshold.
[0093] In this embodiment, since the first camera module and the second camera module take images at the same time, the timing difference of the images output by the two will not be too large; is a preset round-down function; thereby ensuring that the difference ΔNUM between the arrangement numbers corresponding to WA and WB is not too large, thus avoiding the occurrence of misjudgment.
[0094] If 0<ΔNUM<QE, and NUM1<NUM2, it means that the second image in E is behind the first image in D by ΔNUM. Therefore, D r and Er+ΔNUM Determine the images to be taken at the same time.
[0095] T650, if 0<ΔNUM<QE, and NUM1>NUM2, then E r and D r+ΔNUM Determine the images to be taken at the same time.
[0096] If 0<ΔNUM<QE, and NUM1>NUM2, it means that the first image in D is behind the second image in E by ΔNUM. Therefore, E r and D r+ΔNUM Determine the images to be taken at the same time.
[0097] Furthermore, after step T650, the method may further include the following steps:
[0098] T660, if ΔNUM ≥ QE, and NUM1 < NUM2, then E r and D r+q+1-ΔNUM Determine the images to be taken at the same time.
[0099] In this embodiment, under normal circumstances, the timing difference between WA and WB is not too large. If ΔNUM ≥ QE, and NUM1 < NUM2, it means that the acquired WA is at the front position in D, and WB is at the back position in E. The two are images taken when the preset color light is emitted twice in a row; therefore, E is r and D r+q+1-ΔNUM Determine the images to be taken at the same time.
[0100] T670, if ΔNUM ≥ QE, and NUM1 > NUM2, then D r and E r+q+1-ΔNUM Determine the images to be taken at the same time.
[0101] If ΔNUM ≥ QE, and NUM1 > NUM2, it means that the acquired WA is at the rear position in D, and WB is at the front position in E, and the two are images taken when the preset color light is emitted twice in a row; therefore, D r and E r+q+1-ΔNUM Determine the images to be taken at the same time.
[0102] T680, if NUM1 = NUM2, then D p and E p Determine the images to be taken at the same time.
[0103] This is the case where the timing of the first image in D is the same as the timing of the second image in E.
[0104] After the time sequence relationship between the first image and the second image is determined, the first image and the second image with the same time sequence may be spliced together to ensure that the spliced image is consistent with the actual scene.
[0105] In an exemplary embodiment, in order to eliminate the influence of the preset color on the image in the above embodiment, when the first camera module and the second camera module are shooting, every time v images are taken, when the next image is taken, the shooting areas corresponding to the first camera module and the second camera module are illuminated at the same time; for example, LED lights are set at the lens of the first camera module and the lens of the second camera module, and the LED lights are used to illuminate the shooting area of the first camera module and the shooting area of the second camera module at the same time; the illumination time is less than or equal to one shooting cycle.
[0106] After step T300, the method further comprises the following steps:
[0107] T700, obtain the latest v+1 first images output by the first camera module to obtain a fifth image list DT=(DT1, DT2, ..., DT u ,…,DT v+1 ), u=1, 2, ..., v+1; and obtain the latest v+1 second images output by the second camera module to obtain a sixth image list ET=(ET1, ET2, ..., ET u ,…,ET v+1 ), where DT u is the latest u-th first image output by the first camera module, ET u It is the latest u-th second image output by the second camera module; the first image in DT and the second image in ET are arranged in the order of shooting time.
[0108] T710, obtaining the first image GA with the highest brightness in DT and the second image GB with the highest brightness in ET.
[0109] In this embodiment, v images are taken each time, and the shooting areas corresponding to the first camera module and the second camera module are illuminated at the same time, and the latest v+1 first images and v+1 second images are obtained; therefore, DT must include a first image with a larger brightness, and ET must include a second image with a larger brightness; GA and GB can be determined by traversing DT and ET.
[0110] T720, determine the arrangement number LA of GA in DT according to the arrangement position of GA in DT; and determine the arrangement number LB of GB in ET according to the arrangement position of GB in ET.
[0111] T730, based on LA and LB, determine the arrangement sequence number difference TR = |LA-LB| corresponding to GA and GB.
[0112] T740, if 0<TR<QM, and LA<LB, then DT e and E.T. e+TR Determined to be images taken at the same time; e=1, 2, ..., v+1-TR; QM is a preset second sequence number difference threshold.
[0113] In this embodiment, since the first camera module and the second camera module take images at the same time, the timing difference of the images output by the two will not be too large; is a preset rounding-down function; thereby ensuring that the difference TR between the arrangement numbers corresponding to GA and GB is not too large, thus avoiding the occurrence of misjudgment.
[0114] If 0<TR<QM, and LA<LB, it means that the second image in ET is later in time sequence than the first image in DT, and the amount of later time is TR; therefore, DT e and E.T. e+TR Determine the images to be taken at the same time.
[0115] T750, if 0<TR<QM, and LA>LB, then ET e and DT e+TR Determine the images to be taken at the same time.
[0116] If 0<TR<QM, and LA>LB, it means that the first image in DT is later in time sequence than the second image in ET, and the amount of later time is TR; therefore, DT e and E.T. e+TR Determine the images to be taken at the same time.
[0117] T760, if TR ≥ QM, and LA < LB, then ET e and DT e+v+1-TR Determine the images to be taken at the same time.
[0118] In this embodiment, under normal circumstances, the timing difference between GA and GB is not too large. If TR ≥ QM and LA < LB, it means that the obtained GA is at the front position in DT and GB is at the back position in ET. The two are images taken during two consecutive illuminations. The first image in DT is at the back position by the amount v+1-TR. Therefore, ET e and DT e+v+1-TR Determine the images to be taken at the same time.
[0119] T770, if TR ≥ QM, and LA>LB, then DT eand E.T. e+v+1-TR Determine the images to be taken at the same time.
[0120] If TR ≥ QM, and LA>LB, it means that the acquired GA is at the rear position in DT, and GB is at the front position in ET, and the two are images taken during two consecutive illuminations; the second image in ET is at the rear in time sequence, and the number is v+1-TR; therefore, DT e and E.T. e+v+1-TR Determine if the images were taken at the same time
[0121] T780, if TR = 0, then DT u and E.T. u Determine the images to be taken at the same time.
[0122] This is the case where the timing of the first image in DT is the same as the timing of the second image in ET.
[0123] In this embodiment, the timing relationship between the first image and the second image is determined by illuminating the shooting area, which can avoid the influence of the preset color on the image. During the later stitching, there is no need to filter out the color, only the brightness of the image with higher brightness needs to be adjusted, and the stitching efficiency is relatively high.
[0124] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0125] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.
[0126] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0127] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0128] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0129] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0130] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.
[0131] The electronic device according to this embodiment of the present application is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0132] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).
[0133] The storage stores program codes, which can be executed by the processor, so that the processor executes the steps in various embodiments described in this specification.
[0134] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).
[0135] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.
[0136] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.
[0137] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0138] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0139] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.
[0140] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.
Claims
1. A method for determining the timing of binocular camera images, characterized in that: The method comprises: S100, when the first camera module and the second camera module are shooting, every time y images are shot, a preset vibrating body is controlled to vibrate once; wherein the vibrating body is arranged in the overlapping shooting area of the first camera module and the second camera module; the single vibration duration of the vibrating body is less than the shooting cycle of the first camera module and the second camera module; and the vibration amplitudes of any two vibrations of the vibrating body are different; S200, obtaining the latest 2y+2 first images output by the first camera module to obtain a third image list SW=(SW1, SW2, ..., SW x , …, D 2y+2 ), x=1, 2, ..., 2y+2; and obtaining the latest 2y+2 second images output by the second camera module to obtain a fourth image list FW=(FW1, FW2, ..., FW x ,…,FW 2y+2 ); Among them, SW x The latest x-th first image output by the first camera module, FW x The latest x-th second image output by the second camera module; the first image in SW and the second image in FW are arranged in the order of shooting time; S300, traverse SW, if SW x If the vibrating body is not in the initial position, then SW x Determine as the middle first image, so as to obtain two middle first images ZH1 and ZH2; wherein ZH1 is arranged before ZH2 in SW; S400, traverse FW, if FW x If the vibrating body in the case is not in the initial position, then FW x Determine as the intermediate second image, so as to obtain two intermediate second images BH1 and BH2; wherein BH1 is arranged before BH2 in FW; S500, determining a time sequence relationship between a first image in SW and a second image in FW according to arrangement positions of ZH1 and ZH2 in SW and arrangement positions of BH1 and BH2 in FW; Step S500 includes the following steps: S510, if the position of the vibrator in ZH1 is the same as the position of the vibrator in BH1, and the position of the vibrator in ZH2 is the same as the position of the vibrator in BH2, then obtain the arrangement number N1 of ZH1 in SW and the arrangement number M1 of BH1 in FW; S511, according to N1 and M1, determine the difference ΔNM1=|M1-N1| between the arrangement numbers of ZH1 and BH1; S512, if ΔNM1=0, determine SW x Shooting time and FW x The shooting time is the same; S513, if N1<M1, determine SW a Shooting time and FW a+ΔNM1 The shooting time is the same; a=1, 2, …, y-ΔNM1; S514, if N1>M1, determine FW a Shooting time and SW a+ΔNM1 The shooting time is the same; S520, if the position of the vibrating body in ZH1 is the same as the position of the vibrating body in BH2, and the position of the vibrating body in ZH2 is different from the position of the vibrating body in BH1, then obtain the arrangement number N2 of ZH1 in SW and the arrangement number M2 of BH2 in FW; S521, according to N2 and M2, determine the difference ΔNM2=M2-N2 between the arrangement numbers of ZH1 and BH2; S522, determine SW b Shooting time and FW b+ΔNM2 The shooting time is the same; where b=1, 2, …, y-ΔNM2; S530, if the position of the vibrating body in ZH2 is the same as the position of the vibrating body in BH1, and the position of the vibrating body in ZH1 is different from the position of the vibrating body in BH2, then obtain the arrangement number N3 of ZH2 in SW and the arrangement number M3 of BH1 in FW; S531, according to N3 and M3, determine the difference ΔNM3=N3-M3 between the arrangement numbers of ZH2 and BH1; S532, determine FW c Shooting time and SW c+ΔNM3 The shooting time is the same; where c=1, 2,…, y-ΔNM3.
2. The method for determining the timing of binocular camera images according to claim 1, characterized in that: The first camera module and the second camera module shoot, and are controlled by the following steps: S110, in response to generating a preset shooting instruction, starting a preset timer T now ; Wherein, the shooting instruction is used to simultaneously start the preset first camera module and the second camera module to shoot the corresponding area; S120, if T now ≤T0, and a first image generation instruction Z1 corresponding to the first camera module and a second image generation instruction Z2 corresponding to the second camera module are received, then a time interval Δt1 between receiving Z1 and receiving Z2 is determined; S130, if Δt1≤T', then at T now =T0, generate the preset shooting instruction again and enter step S100; otherwise, stop generating the preset shooting instruction again; wherein Z1 is used to represent the first image corresponding to the output of the first camera module, and Z2 is used to represent the second image corresponding to the output of the second camera module; T0 is the preset timing cycle; T' is the preset time interval threshold.
3. The method for determining the timing of binocular camera images according to claim 2, characterized in that: T0 is determined by the following steps: S121, obtaining the initial duration of each first image output by the first camera module and the initial duration of each second image output by the second camera module within the historical time period, so as to obtain an initial duration list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; among them, A i is the i-th initial duration obtained, and n is the number of initial durations obtained; S122, obtain the initial duration volatility η corresponding to A = 1 / n×∑ n i=1 (A i -(1 / n×∑ n i=1 A i )) 2 ; S123, if η<QT, then determine T0=α×1 / n×∑ n i=1 A i ; Otherwise, go to step S240; α is the preset first weight, α>1; S124, arranging the initial durations in A in ascending order to obtain an arranged initial duration list B = (B1, B2, ..., B i , …, B n ); where B i The i-th initial duration is obtained by arranging the initial durations in A in ascending order; S125, determining the first m initial durations in B as the intermediate durations, so as to obtain an intermediate duration list C = (C1, C2, ..., C j , …, C m ), j = 1, 2, ..., m; where C j is the jth intermediate duration among the m intermediate durations; ; β is the preset second weight, 0<β<1; is the preset floor function; S126, according to C, determine T0=α×1 / m×∑ m j=1 C j .
4. The method for determining the timing of binocular camera images according to claim 2, characterized in that: After step S130, the method comprises the following steps: S140, if T now =T0, and Z1 and Z2 are not received completely, then at T now =T0, stop generating the preset shooting instructions again.
5. The method for determining the timing of binocular camera images according to claim 2, characterized in that: After step S130, the method comprises the following steps: S150, if T now =T0, and Z1 is received but Z2 is not received, it is determined that the second camera module is abnormal; S160, if T now =T0, and Z1 is not received but Z2 is received, it is determined that the first camera module is abnormal.
6. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the binocular camera image timing determination method as described in any one of claims 1-5.
7. An electronic device, characterized in that: Includes a processor and the non-transitory computer-readable storage medium of claim 6.
Citation Information
Patent Citations
Binocular photogrammetry method of large flexible structure vibration displacement
CN103954221A
Target trajectory measurement method and device under time-varying condition of camera station and vibration measurement method
CN117029761A
