Dual-camera self-calibration method and system

Abstract

The present disclosure relates to a dual-camera self-calibration method and system. The method comprises determining a first image collected by a first shooting module after the focusing module completes focusing, and a second image collected by a second shooting module; determining a target image corresponding to the first image, and determining a matching relationship of the target image in the second image based on the clarity of the target image satisfying a preset first condition; determining a matching coordinate based on the matching relationship satisfying a preset second condition, based on the position coordinate of the target image in the first image and the matching relationship; determining a calibration point based on the matching coordinate and a focusing position corresponding to the focusing module, and updating the current calibration parameter based on the calibration point. Without performing the traditional factory calibration process, the production cost and manufacturing complexity are reduced, the assembly error and long-term use drift are effectively compensated, the calibration parameter accuracy is improved, the explicit user calibration process is avoided, the user experience is greatly improved, the robustness in abnormal scenes is enhanced, and the engineering adaptability is strong.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of photography and image processing technology, and specifically relate to a dual-camera self-calibration method and system. Background Technology

[0002] Dual-camera systems typically consist of a first camera and a second camera. Due to differences in the mounting position, field of view, optical axis direction, focal length, and imaging area of ​​the two cameras within the overall system, the image position of the same target in the first camera image and the second camera image are usually different. For dual-camera structures with partially overlapping fields of view, a target area in the first camera image often corresponds to only one local search area in the second camera image, and the position of this local search area changes with the distance to the target.

[0003] In applications such as dual-camera focusing, ranging, or depth estimation, it is typically necessary to determine the coordinate offset relationship (i.e., parallax or coordinate difference) between the target position in the first camera image and the corresponding position in the second camera image, and to calculate the target distance or generate focus control values ​​based on this relationship. In other words, it is necessary to obtain the mapping relationship between parallax or coordinate offset and target distance.

[0004] In existing dual-camera autofocus technologies, there are schemes that utilize the parallax between the main camera and the auxiliary camera to estimate distance and determine the focus position based on the distance estimation result (e.g., patent US20160295097A1); and there are also schemes that obtain the object distance value and the difference in imaging coordinates of feature points, and combine them with pre-stored parameters and the optical axis angle formula to create a mapping table between the imaging point difference and the object distance value (e.g., patent CN104730802B). In other words, existing technologies already have schemes that utilize dual-camera parallax, imaging point difference, target area coordinates, or the focus mapping relationship between the main and auxiliary cameras to achieve focus or distance estimation. The common feature of these schemes is that in order to calculate the target distance or achieve autofocus based on the dual-camera images, it is usually necessary to obtain or establish a certain mapping relationship in a specific calibration process, such as the mapping relationship between parallax and distance, the mapping relationship between imaging point difference and object distance, the mapping relationship between the focus positions of the main and auxiliary cameras, or the control relationship between ROI position, parallax, and lens position.

[0005] However, existing technologies still suffer from problems such as reliance on factory calibration, poor adaptability, dependence on explicit calibration processes, inability to meet usage needs, impact on normal user shooting experience, and lack of continuous optimization capabilities, thus affecting the overall focusing effect. Summary of the Invention

[0006] The embodiments of this disclosure propose a dual-camera self-calibration method and system, which can calibrate and correct the coordinate offset between the two cameras and the mapping relationship of the focusing motor position without the user's perception during normal focusing or shooting without factory calibration, thereby continuously optimizing focusing accuracy, reducing calibration costs and improving user experience.

[0007] In a first aspect of this disclosure, a dual-camera self-calibration method is provided. The method includes determining a first image captured by a first shooting module after focusing by a focusing module, and a second image captured by a second shooting module, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. The method further includes determining a target image corresponding to the first image, and determining a matching relationship between the target image and the second image based on the target image's sharpness satisfying a preset first condition. The method further includes determining matching coordinates between the first and second images based on the matching relationship satisfying a preset second condition, and based on the position coordinates of the target image in the first image and the matching relationship. Furthermore, the method includes determining a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and updating the current calibration parameters based on the calibration point.

[0008] In a second aspect of this disclosure, a dual-camera self-calibration system is provided. The system includes an image acquisition module configured to determine a first image acquired by a first shooting module after focusing by a focusing module, and a second image acquired by a second shooting module, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. The system also includes a matching determination module configured to determine a target image corresponding to the first image, and, based on the target image's sharpness satisfying a preset first condition, determine a matching relationship between the target image and the second image. The system further includes a coordinate determination module configured to, based on the matching relationship satisfying a preset second condition, determine matching coordinates between the first and second images based on the position coordinates of the target image in the first image and the matching relationship. Furthermore, the system includes a parameter update module configured to determine a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and update the current calibration parameters based on the calibration point.

[0009] In a third aspect of this disclosure, a computer program product is provided, comprising a computer program that is executed by a processor to implement the method according to the first aspect.

[0010] In a fourth aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0011] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram of a dual-camera self-calibration method according to some embodiments of the present disclosure is shown;

[0014] Figure 2 A schematic diagram showing the position of a target image in a first image according to some embodiments of this disclosure is illustrated:

[0015] Figure 3 This illustration shows a schematic diagram of the positions of a target image and a first image in a second image, according to some embodiments of the present disclosure.

[0016] Figure 4 This illustration shows a schematic diagram illustrating the effect of updating the current calibration parameters based on a calibration point according to some embodiments of this disclosure;

[0017] Figure 5 A block diagram of a dual-camera self-calibration system according to some embodiments of the present disclosure is shown; and

[0018] Figure 6 A block diagram of an electronic device that can implement several embodiments of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “when”, “in response to determination”, or “in response to detection”.

[0021] As mentioned above, the existing technology has the following main technical defects:

[0022] Reliance on factory calibration leads to poor adaptability: For dual-camera modules, due to assembly errors, optical axis deviations, baseline deviations, field-of-view differences, and individual lens differences between the two cameras, the corresponding position of the same target in the images of the two cameras is not fixed. Especially in dual-camera structures with partially overlapping fields of view, the target area in the first camera image corresponds to a local area in the second camera image that moves with the target distance. If only the pre-set mapping table (such as the pre-established parallax-distance mapping relationship or coordinate difference-object distance mapping relationship) is relied upon, the mapping relationship is prone to inaccuracy due to assembly differences, drops, vibrations, temperature changes, or component aging during actual use.

[0023] Relying on explicit calibration processes cannot meet usage requirements: Although some existing technologies (such as patent CN104730802B) create a mapping table between imaging point difference and object distance, this approach requires combining pre-stored dual-camera device parameters, object distance values, and optical axis angle calculation formulas to determine the optical axis angle and then build the mapping table based on this. In other words, this approach still focuses on solving geometric parameters, and the establishment of the mapping table depends on preset parameters and calculation models (although it involves the mapping relationship between imaging point difference and object distance, it still relies on pre-stored parameters, geometric formulas, or explicit calibration processes). It does not belong to a self-calibration method that gradually learns the relationship between parallax / coordinate offset and distance during natural user use.

[0024] Impact on normal shooting experience: Some existing technologies (such as patent application CN107071243A) will prompt the user to perform calibration when the clarity of the main and secondary cameras is inconsistent. The target object needs to be present within multiple distance ranges. If the conditions are not met, the user will be prompted to change the scene and retest. Although this method can reduce the requirements of factory assembly to a certain extent, it may still interrupt the user's normal shooting process and require the current scene to meet the target distribution within a specific distance range, that is, it needs to meet a specific scene or trigger an explicit calibration process.

[0025] Lack of continuous optimization capability: The actual mapping relationship of the dual-camera module may drift slightly with time, temperature, mechanical condition, usage posture or local assembly changes. If the mapping relationship is only established at the factory stage or a few explicit calibration stages, it will be difficult to reflect changes in the device status in a timely manner during subsequent use. For dual-camera focusing or depth estimation systems that rely on local area matching, once the mapping relationship deviates, it may lead to inaccurate setting of the second camera search area, matching failure, and increased distance estimation error, which in turn affects focusing speed, focusing accuracy or depth calculation stability.

[0026] To address this, embodiments of this disclosure propose a dual-camera self-calibration method. The method includes determining a first image captured by a first shooting module after focusing by a focusing module, and a second image captured by a second shooting module, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. The method further includes determining a target image corresponding to the first image, and determining a matching relationship between the target image and the second image based on the target image's sharpness satisfying a preset first condition. The method also includes determining matching coordinates between the first and second images based on the matching relationship satisfying a preset second condition, and based on the position coordinates of the target image in the first image and the matching relationship. Furthermore, the method includes determining a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and updating the current calibration parameters based on the calibration point.

[0027] This approach offers several beneficial effects:

[0028] (1) No need to perform traditional factory calibration process, reducing production costs and manufacturing complexity: In order to obtain the parallax-distance relationship between dual-camera images, existing dual-camera modules usually need to establish calibration curves in advance through calibration boards, fixed distance fixtures, special test scenarios or multi-distance sampling processes during the factory stage. This process requires additional calibration equipment, test space, tooling fixtures and production line time, increasing the manufacturing cost of dual-camera modules.

[0029] By automatically acquiring valid matching coordinates that meet the sharpness and matching relationship conditions during normal user focusing, and using the matching coordinates to generate calibration points to update the current calibration parameters, the dual-camera module can gradually complete the self-calibration between parallax or coordinate offset and the position of the focusing motor during use, thereby reducing or simplifying the fine calibration process for the parallax-distance relationship of the dual cameras at the factory stage.

[0030] (2) Effectively compensate for assembly errors and long-term use drift, and improve the accuracy of calibration parameters: Due to the problems of camera installation deviation, lens focal length difference, sensor assembly error and motor stroke difference in actual products, even if the parallax-distance mapping relationship has been established at the factory, the original calibration curve may still be inaccurate due to factors such as drop, vibration, temperature change, lens aging, and motor response drift.

[0031] By not relying entirely on the fixed calibration curve at the factory, but instead selecting reliable calibration points based on the sharpness evaluation results of the target image and the dual-camera matching score after each normal focus by the user, and using these calibration points to continuously correct the current calibration parameters, the calibration curve can be dynamically updated according to the actual state of the equipment, compensating for errors caused by assembly differences or drift during use.

[0032] (3) Avoid explicit user calibration process and greatly improve user experience: Some existing technical solutions require users to enter a special calibration mode, align with the calibration pattern, mobile device, take a picture of the specified target or wait for the system to complete the calibration. This kind of method will interrupt the user's normal use and has requirements on environment, target distance and operation method.

[0033] By performing the self-calibration process during the user's normal focusing or shooting process, the user does not need to actively enter the calibration interface, use a calibration board, or perform any additional operations. After focusing is completed, only the sharpness and matching score of the target image need to be judged. When both meet the preset conditions, the data is automatically used as a valid calibration point to correct the calibration curve, thereby completing the calibration curve optimization without the user's awareness or with low awareness.

[0034] (4) Enhance robustness in abnormal scenarios: In actual use scenarios, the target may have insufficient texture, motion blur, occlusion, reflection, low light or repeated texture, etc. These factors may cause dual-camera matching errors. If the incorrect matching results are directly used for calibration, it will contaminate the calibration curve.

[0035] Before generating calibration points, the system simultaneously determines whether the sharpness evaluation result and matching relationship of the target image meet the preset conditions. Calibration points are only generated and used to correct the calibration curve when the image is sufficiently sharp and the dual-camera matching result is sufficiently reliable. This allows the system to filter out low-quality focus samples and low-confidence matching samples, reducing the impact of mismatches on the calibration curve.

[0036] (5) Strong engineering adaptability: The calibration curve is not limited to a fixed form. For example, the calibration curve can be a polynomial curve, piecewise linear curve, spline curve, lookup table, compensation curve or machine learning model. The calibration curve can also be corrected by direct fitting, incremental update, moving average, recursive least squares, least mean square update, Kalman filter update, Bayesian update or lookup table local update. It can adapt to different hardware platforms and different computing power conditions. For embedded devices with low computing power, lookup table local update or exponential moving average can be used. For terminal devices with high computing power, weighted fitting, recursive estimation or model training can be used. Its flexibility is conducive to deployment in various product forms such as mobile phones, tablets, smart cameras, vehicle cameras, robots, and industrial testing equipment.

[0037] Please see Figure 1 , Figure 1 A flowchart of a dual-camera self-calibration method according to some embodiments of the present disclosure is shown. Method 100 may be executed by a processing terminal, such as the aforementioned mobile phone, tablet, smart camera, vehicle camera, robot, or industrial inspection equipment, and has at least a dual-camera module consisting of a first shooting module and a second shooting module, and a focusing module for focusing the first shooting module. Here, the first shooting module is, for example, an autofocus camera consisting of a lens group, an image sensor, and a focusing lens, used to acquire a focused image based on shooting commands; the focusing module consists, for example, a focusing motor and a transmission assembly between the focusing motor and the focusing lens, used to control the movement of the focusing lens along the optical axis based on focusing commands, so as to adjust the distance between the focusing lens and the image sensor (i.e., the focusing position, represented by the number of steps executed by the focusing motor), thereby enabling the first shooting module to focus on objects at different distances; the second shooting module is, for example, a fixed-focus camera (or a wide-angle camera with a fixed focal length, or a focusing camera, and not limited to this) consisting of a lens group and an image sensor, which does not have a focusing module, so as to acquire an image that is always in the design nominal focus state based on shooting commands (in this way, it is not necessary to perform additional focus state judgment on the image acquired by the second shooting module, thereby simplifying the self-calibration judgment logic). It is understandable that the structure and working principle of the first shooting module, the second shooting module, and the focusing module are all well-known technologies in the field, and will not be elaborated further here.

[0038] In the embodiments of this disclosure, the field of view of the first shooting module may not exceed the field of view of the second shooting module, that is, the field of view of the first shooting module does not exceed the field of view of the second shooting module, and there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. For example, the field of view of the second shooting module and the field of view of the first shooting module have at least a partial overlap area, or the field of view of the second shooting module completely covers the field of view of the first shooting module, so that the target image in the image captured by the first shooting module can have a corresponding area in the image captured by the second shooting module. Of course, the field of view of the first shooting module may also exceed the field of view of the second shooting module, that is, the field of view of the first shooting module exceeds the field of view of the second shooting module, and it is not limited to this.

[0039] like Figure 1 As shown in block 102, method 100 can determine a first image captured by the first shooting module after the focusing module completes focusing, and a second image captured by the second shooting module. Here, the processing terminal can focus the first shooting module through the focusing module when the focusing action is triggered. After the focusing module completes focusing, it controls the distance between the focusing lens of the first shooting module and the image sensor (i.e., the focusing position), which can be used as a reliable reference for generating calibration points (such as pre-stored in the non-volatile storage medium of the processing terminal). At the moment when focusing is completed, the first shooting module captures the focused image (i.e., the first image) after focusing is completed, and the second shooting module simultaneously (or approximately simultaneously) captures the focusing state image (i.e., the second image), and the timestamp difference between the first image and the second image does not exceed a preset threshold.

[0040] It is understood that the focusing action can be an explicit focusing operation performed by the user on the image display interface of the processing terminal (such as double-tap to focus, manual focus, or manual adjustment), or an automatic focusing operation performed by the processing terminal according to a preset focusing program. In addition, the focusing process of the processing terminal can be a technical means well known in the art, such as using a deep learning model to detect salient areas of the image to achieve automatic selection of the focus area, and is not limited to this.

[0041] In box 104, method 100 can determine a target image corresponding to the first image, and determine the matching relationship of the target image in the second image based on the target image's clarity meeting a preset first condition. Here, the processing terminal can determine the target image corresponding to the first image based on the application scenario targeted by the first image, through default selection, user interaction, or intelligent detection algorithms, to ensure flexible adaptability of target selection.

[0042] In some implementations, when the processing terminal determines the target image corresponding to the first image, if the application scenario targeted by the first image is a background or landscape scene without a significant subject, the overlapping area between the first image and the second image can be directly determined as the target image. It is necessary to ensure that the field of view of the second shooting module completely covers the field of view of the first shooting module (that is, the second image contains the complete first image). Alternatively, a portion of the first image that overlaps with the second image can be determined as the target image, ensuring that the second image also contains the target image. If the application scenario targeted by the first image is a dinner party interactive shooting scene, the area corresponding to the user's input operation can be cropped from the first image based on the user's operation as the target image. For example, the pixel position in the first image corresponding to the user's click (such as a single click or double click) can be used as the center, and the corresponding target image can be cropped according to a preset image size (such as 512*512 pixels). Alternatively, the area in the first image corresponding to the user's input selection box can be used as the target image, and it is necessary to ensure that the target image is within the overlapping area between the field of view of the second shooting module and the field of view of the first shooting module.

[0043] When the application scenario targeted by the first image is a face or an object, the target image can be determined from the first image based on an object detection algorithm. For example, taking a face as an example, a single-stage (or two-stage) deep learning object detection algorithm or a traditional object detection algorithm based on HOG and SVM can be used to automatically select the face region from the first image and use it as the target image. Similarly, taking an object as an example, traditional saliency detection methods such as GBVS, Itti-Koch, and HS, or deep learning saliency detection algorithms such as the CNN series and Transformer series, can be used to automatically select the main body region from the first image and use it as the target image. It is necessary to ensure that the target image is within the overlapping area between the field of view of the second shooting module and the field of view of the first shooting module. In the embodiments of this disclosure, the structures and training processes of all the detection algorithm types mentioned above are well-known techniques in the art and will not be elaborated upon here.

[0044] After identifying the target image corresponding to the first image, the sharpness of the target image can be compared with a preset sharpness threshold (e.g., 0.8) to determine whether the sharpness of the target image meets the preset first condition. Here, when the sharpness of the target image exceeds the preset sharpness threshold, it indicates that the sharpness of the target image is high (or the focus is successful, and it is a valid image), which means that the preset first condition is met; when the sharpness of the target image does not exceed the preset sharpness threshold, it indicates that the target image is relatively blurry (or the focus has not converged, and it is an invalid image), which means that the preset first condition is not met. In this case, the first shooting module needs to be refocused through the focusing module, and the above process is repeated.

[0045] In embodiments of this disclosure, the processing terminal can determine the sharpness of a target image based on hardware computing power and evaluation accuracy requirements, using either a pre-trained neural network prediction algorithm or a traditional sharpness calculation algorithm. For example, when hardware computing power is high and / or evaluation accuracy requirements are high, the sharpness of the target image can be determined using a pre-trained neural network prediction algorithm, such as a deep learning model from the MobileNet, Transformer, GhostNet, or ShuffleNet series. When hardware computing power is low and / or evaluation accuracy requirements are low, the sharpness of the target image can be determined using a traditional sharpness calculation algorithm, such as any one of Laplacian energy, Tenengrad gradient, image variance, FFT frequency domain method, DCT frequency domain method, and NIQE method.

[0046] For example, taking the lightweight MobileNetV3-Small model with a neural network prediction algorithm as an example (the original classification layer is removed at the end and replaced with a global average pooling layer, followed by a ReLU activation layer, and finally connected to a fully connected layer and activated by the Sigmoid function; the input is a target image of size 512*512, and the output is a floating-point number between 0 and 1, where 0 represents the most blurred and 1 represents the most sharp), during the training process, it can use multiple high-definition images (with corresponding sharpness labels) covering different scenes, and blurred images (with corresponding sharpness labels) obtained by blurring each high-definition image as training sets. The mean squared error loss function is used to calculate the loss between the predicted sharpness and the sharpness label, and the Adam optimizer is used to optimize the parameters until the model converges.

[0047] After determining that the sharpness of the target image meets the preset first condition, the matching relationship of the target image in the second image can be determined. Here, the matching relationship can be understood as the position coordinates of the region in the second image that has the highest similarity to the target image and the similarity between the region and the target image. The position coordinates can be, for example, the pixel position coordinates of the upper left corner vertex, center point, or centroid of the region in the second image, and are not limited to these.

[0048] In the embodiments of this disclosure, since the field of view and image resolution are different between the first shooting module and the second shooting module, in order to ensure that the matching process is at the same spatial scale, the target image and the second image can also be aligned in resolution to effectively improve the matching accuracy.

[0049] In some implementations, when determining the matching relationship between the target image and the second image, the processing terminal specifically determines the image to be processed from the target image and the second image based on the field-of-view ratio between the first and second shooting modules, and downsamples the image to be processed. Here, the field-of-view ratio can be a horizontal field-of-view ratio (i.e., the field-of-view ratio on the x-axis) and a vertical field-of-view ratio (i.e., the field-of-view ratio on the y-axis). For example, taking the horizontal field-of-view ratio as an example, its determination process can be expressed as follows:

[0050] scale_x = tan(HFOV 1 / 2) / tan(HFOV 2 / 2)

[0051] In the above formula, scale_x is the lateral field of view ratio, and HFOV is... 1 HFOV is the horizontal field of view of the first shooting module. 2 This refers to the horizontal field of view of the second shooting module.

[0052] After determining the field-of-view ratio, the image with the larger pixel resolution from the target image and the second image can be selected as the image to be processed based on this ratio, and then downsampled. Taking the second image as an example, well-known interpolation algorithms such as bicubic or Lanzos interpolation can be used to downsample the second image based on the field-of-view ratio. This ensures that the width of the downsampled second image is the product of the horizontal field-of-view ratio and the width of the original second image, and that the height of the downsampled second image is the product of the vertical field-of-view ratio and the height of the original second image. Alternatively, embodiments of this disclosure can also select the image with the smaller pixel resolution from the target image and the second image as the image to be processed, and upsample this image based on the field-of-view ratio. This also achieves resolution alignment between the target image and the second image, and will not be elaborated further here.

[0053] Subsequently, based on the downsampled image to be processed, the matching relationship of the target image in the second image is determined. It is understandable that, in order to maintain a trade-off between accuracy, computational complexity, and scene adaptability, various methods can be used to determine the matching relationship of the target image in the second image. In one example, when accuracy requirements are low and / or scene adaptability requirements are low, taking the image to be processed as the second image, the processing terminal can determine multiple candidate region images in the second image (i.e., the downsampled image to be processed) based on the target image. Here, the sliding window size (such as being the same as, but not limited to, the size of the target image) and the sliding step size (such as half the width of the target image in the width direction, and half the height of the target image in the height direction, but not limited to,) can be determined based on the size of the target image. Multiple candidate region images are then determined in the second image based on this sliding window size and sliding step size. The size of each candidate region image is the same as the sliding window size, and all candidate region images can form a complete downsampled second image.

[0054] Next, the similarity between the target image and each candidate region image is calculated. The position coordinates of the candidate region image corresponding to the maximum similarity in the second image, along with the maximum similarity, are used to determine the matching relationship of the target image in the second image. Here, the method for calculating the similarity between the target image and each candidate region image can be based on computational complexity requirements, such as cross-correlation or normalized cross-correlation, sum of absolute differences or sum of squared differences, mean absolute error or mean squared error, histogram distance index, mutual information, structural similarity index, or feature vector distance index, and is not limited to these.

[0055] In embodiments of this disclosure, to further improve accuracy and / or scene adaptability, the target image and the second image can be registered based on a feature registration method to obtain the position coordinates and similarity of the target image in the second image, which serves as the matching relationship between the target image and the second image. The feature registration method can be, for example, a SIFT, ORB, or SURF feature point matching method well-known in the art, and its process will not be elaborated upon here.

[0056] In one example, when high accuracy and / or scene adaptability are required, taking the image to be processed as the second image as an example, the processing terminal can also obtain the position coordinates and similarity of the target image in the second image based on the target image, the second image (i.e., the downsampled image to be processed), and a pre-trained deep learning model, and determine the position coordinates and similarity as the matching relationship of the target image in the second image. Here, the deep learning model can be understood as a deep learning registration network based on the Siamese algorithm, MatchNet, or SuperPoint and SuperGlue, which can directly predict the position coordinates and similarity of the target image in the second image based on the input target image and the second image. For example, taking a deep learning matching network based on the Siamese concept as an example, this deep learning model contains two identical feature extraction branches with shared weights. One branch extracts the corresponding depth features based on the input target image (the size of the input target image can be pre-scaled, such as 128*128), and the other branch extracts the corresponding depth features based on the input second image (the input second image can be pre-scaled, such as 256*256). The depth features extracted by the two branches are used to calculate the matching cost space through a cross-correlation layer, and the calculation result is input into a regression head composed of multiple convolutional layers and fully connected layers, and finally output... The model outputs a multi-dimensional vector (including the position coordinates of the top-left vertex of the target image and a matching score used to characterize the credibility of the matching result, and this matching score is used as the similarity). During training, the deep learning model can use multiple pairs of dual-camera image samples covering different scenes (each pair of dual-camera image samples has a target image sample, a second image sample, a position coordinate label of the top-left vertex of the target image sample, and a matching score label) as the training set. The loss function is used to calculate the weighted sum of the smoothing loss between the predicted position coordinates and the position coordinate labels, and the cross-entropy loss between the predicted similarity and the similarity labels. The Adam optimizer is used to optimize the parameters until the model converges.

[0057] In box 106, method 100 can determine the matching coordinates between the first image and the second image based on the position coordinates of the target image in the first image and the matching relationship, based on the matching relationship satisfying a preset second condition. It is understood that, although the clarity of the first image meets the condition mentioned above, the matching process may still result in mismatches due to factors such as repeated textures, partial occlusion, lighting differences, or imaging differences between shooting modules. The processing terminal can determine whether the matching relationship satisfies the preset second condition by comparing the similarity in the matching relationship with a preset similarity threshold (e.g., 0.7). Here, when the similarity in the matching relationship exceeds the preset similarity threshold, it indicates that the matching relationship is reliable, that is, the matching relationship satisfies the preset second condition; when the similarity in the matching relationship does not exceed the preset similarity threshold, it indicates that the matching relationship is unreliable, that is, the matching relationship does not satisfy the preset second condition. In this case, it is necessary to redetermine the matching relationship of the target image in the second image, or to refocus the first shooting module using the focusing module and repeat the above process.

[0058] After determining that the matching relationship satisfies the preset second condition, the matching coordinates between the first image and the second image can be determined based on the position coordinates of the target image in the first image and the matching relationship. Here, the position coordinates of the target image in the first image can be, for example, the pixel position coordinates of the top left corner vertex, center point, or centroid of the target image in the first image. These coordinates can be determined based on a Cartesian coordinate system established with the top left corner vertex of the first image as the origin and the two adjacent sides as coordinate axes. Furthermore, the position corresponding to the position coordinates in the target image is the same as the position corresponding to the position coordinates in the matching relationship in the target image, for example, both being the top left corner vertex of the target image, but not limited to this.

[0059] Please see Figure 2 The diagram shows the position of a target image in a first image according to an embodiment of the present disclosure. (See also...) Figure 2 As shown, a first image 203 is displayed, which establishes a Cartesian coordinate system with the top-left corner as the origin and the two adjacent sides as coordinate axes. This first image 203 contains a target image 202, and the position coordinates 201 (x, y, y) of the top-left corner of the target image 202 in the first image. 1 y 1 ).

[0060] Please see Figure 3 The diagram illustrates the positions of a target image and a first image in a second image according to some embodiments of the present disclosure. (See also...) Figure 3As shown, a second image 305 is illustrated, which is a Cartesian coordinate system established with the top-left vertex as the origin and the two adjacent sides as coordinate axes. This second image 305 has a first image 304 projected onto it after linear scaling. The position coordinates (x, y, y) of the top-left vertex of the first image 304 within the second image 305 are shown in the figure. 3 y 3 The target image 303, after linear scaling, is projected onto the second image 305, and the position coordinates 302 (x, y) of the upper left corner vertex of the target image 303 in the second image 305 are... 2 y 2 ).

[0061] In some implementations, when the processing terminal determines the matching coordinates between the first image and the second image based on the position coordinates and matching relationship of the target image in the first image, it specifically determines the resolution ratio between the first image and the second image. Here, the resolution ratio can be a width ratio (i.e., the resolution ratio on the x-axis) and a height ratio (i.e., the resolution ratio on the y-axis), and the expression for its determination process can be found as follows:

[0062] ratioX12=W 1 / W 2

[0063] ratioY12=H 1 / H 2

[0064] In the above formula, ratioX12 is the width ratio, W 1 W is the width of the first image. 2 The width of the second image (or the second image width after downsampling mentioned above), ratioY12 is the height ratio, and H... 1 H is the height of the first image. 2 This is the height of the second image (or the height of the second image after downsampling mentioned above).

[0065] Next, the position coordinates of the target image in the first image, the position coordinates of the target image in the second image, and the resolution ratio are substituted into the preset matching coordinate formula to obtain the matching coordinates between the first and second images. The expression of the preset matching coordinate formula can be seen as follows:

[0066]

[0067] In the above formula, The matching coordinates between the first and second images. and Here are the position coordinates of the target image in the second image. and Here, X12 represents the position coordinates of the target image within the first image, and Y12 represents the height ratio. This method ensures that subsequent calibration points reflect the overall relative position of the first image within the second image, rather than the local position of the target image, thus facilitating the consistent application of calibration results across different target scenarios.

[0068] In box 108, method 100 can determine a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and update the current calibration parameters based on the calibration point. Here, the processing terminal can control the distance (i.e., the focusing position) between the focusing lens of the first shooting module and the image sensor after focusing is completed by calling the focusing module, and use this focusing position and the matching coordinates together as the calibration point, as shown in (…). , ), Let i be the matching coordinates of the i-th calibration point. Let be the focus position at the i-th calibration point. In embodiments of this disclosure, considering that under ideal assembly, the main parallax between the two shooting modules is concentrated in the horizontal direction, and the deviation in the vertical direction is small and almost constant, the matching coordinates in the calibration point can be simplified to coordinates on the x-axis, but are not limited thereto.

[0069] It is understood that the current calibration parameters can be preset initial calibration parameters, or the latest calibration parameters obtained by updating the preset initial calibration parameters through multiple calibration points. The preset initial calibration parameters can be used to characterize the mapping relationship between the pixel coordinate position of the first image on the second image and the focus position, and are not calibrated for individual devices (that is, fixed preset parameters that do not need calibration). In one example, taking the mapping relationship between the pixel coordinate position of the first image on the second image and the focus position as a linear fitting form (f=k*p+b) as an example, the initial calibration parameters are k and b (general preset values ​​can be used so that the processing terminal can be put into use without calibration for individual devices at the factory. The self-calibration process of this disclosure can gradually correct the initial calibration parameters from the first time the focus is completed), and at this time the matching coordinates in the calibration points can be simplified to coordinates on the x-axis.

[0070] In the embodiments of this disclosure, the mapping relationship between the pixel coordinate position of the first image on the second image and the focus position can also be represented as a polynomial curve, a piecewise linear curve, or a spline curve. Of course, it can also be determined by a lookup table, a compensation curve, or a machine learning model, and is not limited thereto.

[0071] In this way, the current calibration parameters can be corrected each time a new reliable calibration point is generated, so that the calibration parameters can be dynamically updated with changes in the actual state, thereby effectively serving downstream applications such as dual-camera focusing, target positioning, or depth estimation.

[0072] In some implementations, the second shooting module can be a fixed-focus shooting module or it can have a focusing function; when there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module, a target image corresponding to the first image can be determined, and after the clarity of the target image meets the preset first condition, the target image is matched with the second image to determine the matching relationship of the target image in the second image.

[0073] Next, the similarity in the matching relationship can be compared with a preset similarity threshold. If the similarity exceeds the preset similarity threshold, it is determined that the matching relationship meets the preset second condition. Based on the position coordinates of the target image in the first image and the matching relationship, the matching coordinates between the first image and the second image are determined. If the similarity does not exceed the preset similarity threshold, it is determined that the matching relationship does not meet the preset second condition, thereby ending the calibration without generating calibration points.

[0074] It is understandable that the second image does not need to be in full focus. Even if the second image is out of focus, blurry, or has weakened texture to a certain extent, as long as a matching relationship with a similarity exceeding the preset similarity threshold can still be determined, the matching relationship can be used to determine the matching coordinates in the future.

[0075] In some implementations, when the processing terminal updates the current calibration parameters based on calibration points, it specifically updates the calibration points to a set of historical calibration points, and then fits and solves a preset mapping expression based on the updated set of historical calibration points to obtain the first fitting parameters. Here, the set of historical calibration points includes all calibration points determined from the first focus completion to the most recent focus completion (the matching coordinates in each calibration point can be simplified to coordinates on the x-axis), and the determination process of each calibration point can be referred to the above, so it will not be elaborated further here.

[0076] Understandably, the processing terminal can update the calibration points to the historical calibration point set, and then fit and solve the preset mapping expression based on the updated historical calibration point set. For example, if the preset mapping expression is f=k*p+b as mentioned above, the preset mapping expression can be fitted and solved by least squares fitting (i.e., line fitting), and the solved k and b can be used as the first fitting parameters.

[0077] In the embodiments of this disclosure, the preset mapping expression may also be in the form of other expressions, such as multi-segment linear form, polynomial form, exponential form, logarithmic form, power function form, or nonlinear form, etc. The corresponding fitting solution method may be one or more of multi-segment linear fitting, polynomial fitting, exponential fitting, logarithmic fitting, power function fitting, logistic regression fitting, Gaussian fitting, spline fitting, local regression, or nonlinear fitting methods based on machine learning, and is not limited to these.

[0078] Then, the current calibration parameters are updated based on the first fitted parameters. Here, after determining the first fitted parameters, the k value in the current calibration parameters can be updated based on the k value in the first fitted parameters, and the b value in the current calibration parameters can be updated based on the b value in the first fitted parameters. In this way, the calibration curve can be refitted as a whole with the update of the historical calibration point set, which is suitable for scenarios with high calibration accuracy requirements.

[0079] In some implementations, when the processing terminal updates the current calibration parameters based on the calibration point, it specifically substitutes the current calibration parameters into a preset mapping expression to obtain a first mapping expression, and determines the coordinate deviation based on the first mapping expression and the calibration point. Here, taking the preset mapping expression as f=k*p+b mentioned above as an example, the current calibration parameters (i.e., the values ​​of k and b) can be substituted into the preset mapping expression to obtain the first mapping expression; then, the matching coordinates in the calibration point (i.e., simplified to coordinates on the x-axis) are substituted into the first mapping expression to obtain the predicted focusing position, and the absolute value of the difference between the predicted focusing position and the focusing position in the calibration point is determined as the coordinate deviation.

[0080] Next, the current calibration parameters are updated based on the coordinate deviation. In one example, the coordinate deviation, calibration point, and current calibration parameters can be substituted into a preset parameter update expression to obtain the corrected fitting parameters. This preset parameter update expression can be seen as follows:

[0081] k_new = k + α * error * p_new

[0082] b_new=b+β*error

[0083] In the above formula, k_new is the k value in the corrected fitting parameters, k is the k value in the current calibration parameters, α and β are the preset learning rate parameters, error is the coordinate deviation, p_new is the matching coordinate in the calibration point, b_new is the b value in the corrected fitting parameters, and b is the b value in the current calibration parameters.

[0084] After determining the corrected fitting parameters, the k value in the current calibration parameters can be updated based on the k value in the corrected fitting parameters, and the b value in the current calibration parameters can be updated based on the b value in the corrected fitting parameters. In this way, parameter updates can be directly driven by the deviation, effectively shortening the calculation path and facilitating deployment on embedded platforms.

[0085] In some implementations, when the processing terminal updates the current calibration parameters based on calibration points, it specifically substitutes the initial calibration parameters into a preset mapping expression to obtain a second mapping expression, and determines a compensation calibration point set based on the updated historical calibration point set and the second mapping expression. Here, taking the preset mapping expression as f=k*p+b mentioned above as an example, the initial calibration parameters can be substituted into this preset mapping expression to obtain the second mapping expression (i.e., the initial calibration curve). Then, the matching coordinates of each calibration point in the updated historical calibration point set can be substituted into the second mapping expression to obtain the corresponding predicted focusing position. Next, the absolute value of the difference between the focusing position in each calibration point of the updated historical calibration point set and the corresponding predicted focusing position can be calculated, and the matching coordinates and the corresponding absolute values ​​of the differences in all calibration points can be integrated into a compensation calibration point set.

[0086] Next, the preset mapping expression is fitted and solved based on the set of compensation calibration points to obtain the second fitting parameters. These second fitting parameters are then substituted into the preset mapping expression to obtain the third mapping expression. Here, the preset mapping expression (f=k*p+b) can be fitted and solved using least-squares fitting based on the set of compensation calibration points. The solved k and b are then used as the second fitting parameters, which are substituted into the preset mapping expression to obtain the third mapping expression. It is understood that the third mapping expression can be used to characterize the deviation of the actual coordinate offset from the predicted coordinate offset of the initial calibration curve at different focusing positions. When this deviation is large (e.g., both k and b values ​​exceed preset thresholds), it indicates that there may be excessive calibration offset. In this case, the second mapping expression can also be directly used to update the current calibration parameters, and this is not a limitation.

[0087] Next, the third and second mapping expressions are superimposed to obtain the fourth mapping expression, and the current calibration parameters are updated based on the fourth mapping expression. Here, the process of superimposing the third and second mapping expressions can be understood as superimposing the k value in the third mapping expression with the k value in the second mapping expression to obtain the k value in the fourth mapping expression, and superimposing the b value in the third mapping expression with the b value in the second mapping expression to obtain the b value in the fourth mapping expression.

[0088] After determining the fourth mapping expression, the k value in the current calibration parameters can be updated based on the k value in the fourth mapping expression, and the b value in the current calibration parameters can be updated based on the b value in the fourth mapping expression. In this way, the general benchmark and individual deviation can be decoupled, so that the initial calibration curve can be retained as the reference baseline, and it is also convenient to revert to the initial state in case of excessive or abnormal calibration drift.

[0089] In some implementations, when the processing terminal updates the current calibration parameters based on calibration points, it specifically constructs a first parameter vector based on the calibration points and determines the gain vector based on the first parameter vector and the covariance matrix corresponding to the current calibration parameters. In embodiments of this disclosure, the current calibration parameters can also be incrementally corrected (i.e., updated) using an iterative update method. This iterative update method can be, for example, any one of the following: recursive least squares, weighted moving average update, exponential moving average update, stochastic gradient descent update, Kalman filter update, Bayesian update, robust iterative update, piecewise curve local update, neural network, or incremental update using a machine learning model, and is not limited thereto.

[0090] Here, taking the recursive least squares method as an example, a first parameter vector can be constructed based on the matching coordinates (i.e., simplified to coordinates on the x-axis) in the calibration points. The expression of this first parameter vector can be, for example, x=[p,1]^T. Then, the first parameter vector and the covariance matrix corresponding to the current calibration parameters are substituted into the preset gain vector calculation formula to obtain the gain vector. The expression of the preset gain vector calculation formula can be seen as follows:

[0091] K = P*x / (λ + x^T*P*x)

[0092] In the above formula, K is the gain vector, P is the covariance matrix corresponding to the current calibration parameters, x is the first parameter vector, and λ is the preset forgetting factor.

[0093] It is understood that the covariance matrix corresponding to the current calibration parameter can be a 2*2 diagonal matrix, which is determined based on the covariance matrix corresponding to the previous calibration parameter. In the embodiments of this disclosure, the initial covariance matrix can be expressed as, for example, P0 = [[1000, 0], [0, 1000]], and is not limited to this.

[0094] Next, a second parameter vector is constructed based on the current calibration parameters, and a third parameter vector is determined based on the second parameter vector, the gain vector, and the calibration point. Here, the expression for the second parameter vector can be, for example, θ = [k, b]^T. The third parameter vector is obtained by substituting this second parameter vector, the gain vector, and the focus position from the calibration point into a preset update parameter vector formula. An example of the expression for this preset update parameter vector formula is as follows:

[0095] θ_new=θ+K*(fx^T*θ)

[0096] In the above formula, θ_new is the third parameter vector, θ is the second parameter vector, K is the gain vector, f is the focusing position in the calibration point, and x is the first parameter vector.

[0097] Then, the current calibration parameters are updated based on the third parameter vector. It can be understood that the expression for the third parameter vector is the same as the expression for the second parameter vector mentioned above. The k value in the current calibration parameters is updated based on the k value corresponding to the third parameter vector, and the b value in the current calibration parameters is updated based on the b value pair corresponding to the third parameter vector. In this way, only the current calibration parameters are incrementally adjusted after each new calibration point is added, without needing to refit all historical calibration points, thus reducing computational overhead and making it suitable for continuously running online self-calibration scenarios.

[0098] Furthermore, the covariance matrix corresponding to the current calibration parameter can be updated based on the gain vector and the first parameter vector mentioned above to obtain the covariance matrix corresponding to the next calibration parameter. The expression for the update process can be found below:

[0099] P_new=(IK*x^T)*P / λ

[0100] In the above formula, P_new is the covariance matrix corresponding to the next calibration parameter, I is the identity matrix, K is the gain vector, x is the first parameter vector, P is the covariance matrix corresponding to the current calibration parameter, and λ is the preset forgetting factor.

[0101] Please see Figure 4 The illustration shows the effect of updating the current calibration parameters based on a calibration point according to some embodiments of the present disclosure. For example... Figure 4 As shown, the initial calibration curve 401 determined based on the initial calibration parameters, the newly added calibration point 403, and the corrected calibration curve 402 determined based on the calibration point 403 and the aforementioned recursive least squares method are illustrated.

[0102] In some implementations, the processing terminal is also used to determine whether the number of calibration points in the updated historical calibration point set exceeds a preset threshold. Here, this can be determined by comparing the number of calibration points in the updated historical calibration point set with a preset threshold (e.g., 90) to determine whether the updated historical calibration point set has reached its maximum number of calibration points. It is understood that when the number of calibration points exceeds the preset threshold, it indicates that the updated historical calibration point set has reached its maximum number of calibration points, and calibration points need to be removed; when the number of calibration points does not exceed the preset threshold, it indicates that the updated historical calibration point set has not reached its maximum number of calibration points, and the historical calibration point set can continue to be updated.

[0103] Subsequently, in response to the determination that the number of calibration points exceeds a preset threshold, it is determined whether outliers exist in the updated historical calibration point set. Furthermore, in response to the determination that outliers exist in the updated historical calibration point set, outlier removal is performed on the updated historical calibration point set. Here, after determining that the number of calibration points exceeds the preset threshold, the existence of outliers in the updated historical calibration point set can be determined based on an outlier removal method. This outlier removal method can be, for example, any one of Z-score, IQR, MAD, geometric consistency outlier removal, RANSAC, MLESAC, PROSAC, LMEdS, DBSCAN, LOF, and Isolation Forest. Once an outlier is determined to exist in the updated historical calibration point set, it is removed. In this way, as calibration points accumulate and the size of the historical calibration point set increases, outlier removal improves calibration efficiency and prevents the impact of abnormal calibration points on the calibration process.

[0104] In embodiments of this disclosure, the processing terminal may also persistently store the updated current calibration parameters in a non-volatile storage medium so that they can be loaded as the current calibration parameters after the processing terminal is powered on again. This enables the calibration results accumulated during long-term use to continue across power-on and power-off cycles, avoiding the need to start accumulating from the initial calibration parameters each time the power is turned on.

[0105] Figure 5 A block diagram of a dual-camera self-calibration system according to some embodiments of the present disclosure is shown. The various embodiments in this specification are described in a progressive manner, with reference to each other for similar or identical parts. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. Figure 5As shown, the dual-camera self-calibration system 500 may include at least an image acquisition module 502, configured to determine a first image acquired by the first shooting module after the focusing module has completed focusing, and a second image acquired by the second shooting module, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. The dual-camera self-calibration system 500 also includes a matching determination module 504, configured to determine a target image corresponding to the first image, and, based on the target image's sharpness satisfying a preset first condition, determine the matching relationship of the target image in the second image. The dual-camera self-calibration system 500 also includes a coordinate determination module 506, configured to, based on the matching relationship satisfying a preset second condition, determine the matching coordinates between the first image and the second image based on the position coordinates of the target image in the first image and the matching relationship. Furthermore, the dual-camera self-calibration system 500 also includes a parameter update module 508, configured to determine a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and update the current calibration parameters based on the calibration point.

[0106] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0107] Figure 6 Block diagrams of electronic devices that can implement various embodiments of the present disclosure are shown. For example... Figure 6As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 603 according to computer program instructions stored in read-only memory (ROM) 602. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0108] The various processes and procedures described above, such as method 100, can be executed by processor 601. For example, in some embodiments, method 100 may be implemented as a software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program may be loaded into and / or installed onto electronic device 600 via ROM 602. When the software program is loaded into RAM 603 and executed by processor 601, one or more actions of method 100 described above may be performed.

[0109] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0110] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] This disclosure can be a method, apparatus, system, and / or program product. The program product may include a machine-readable storage medium on which machine-readable program instructions for performing various aspects of this disclosure are loaded. The machine-readable program instructions described herein can be downloaded from the machine-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network and forwards them to the machine-readable storage medium in the respective computing / processing device.

[0112] Machine program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. Machine-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the machine-readable program instructions to implement various aspects of this disclosure.

[0113] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0114] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A dual-camera self-calibration method, characterized in that, include: The first image captured by the first shooting module after the focusing module has completed focusing, and the second image captured by the second shooting module are determined, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module; A target image corresponding to the first image is determined, and the matching relationship between the target image and the second image is determined based on the target image's clarity meeting a preset first condition. Based on the matching relationship satisfying a preset second condition, and based on the position coordinates of the target image in the first image and the matching relationship, the matching coordinates between the first image and the second image are determined; as well as Based on the matching coordinates and the focusing position corresponding to the focusing module, a calibration point is determined, and the current calibration parameters are updated based on the calibration point.

2. The method according to claim 1, characterized in that, Determining the target image corresponding to the first image includes: The target image is determined based on the overlapping area between the first image and the second image; or The target image is determined from the first image based on user actions; or The target image is determined in the first image based on the target detection algorithm.

3. The method according to claim 1, characterized in that, Determining the matching relationship between the target image and the second image includes: Based on the field-of-view ratio between the first and second shooting modules, an image to be processed is determined from the target image and the second image, and the image to be processed is downsampled; and Based on the downsampled image to be processed, the matching relationship between the target image and the second image is determined.

4. The method according to claim 1, characterized in that, Determining the matching relationship between the target image and the second image includes: Based on the target image, multiple candidate region images are determined in the second image; Calculate the similarity between the target image and each of the candidate region images, and determine the position coordinates of the candidate region image corresponding to the maximum similarity in the second image and the maximum similarity as the matching relationship of the target image in the second image; or Based on the target image, the second image, and the pre-trained deep learning model, the position coordinates and similarity of the target image in the second image are obtained, and the position coordinates and similarity are determined as the matching relationship of the target image in the second image.

5. The method according to claim 1, characterized in that, The matching relationship has the position coordinates of the target image in the second image; Determining the matching coordinates between the first image and the second image based on the position coordinates of the target image in the first image and the matching relationship includes: Determine the resolution ratio between the first image and the second image; and The position coordinates of the target image in the first image, the position coordinates of the target image in the second image, and the resolution ratio are substituted into a preset matching coordinate formula to obtain the matching coordinates between the first image and the second image.

6. The method according to any one of claims 1-5, characterized in that, The step of updating the current calibration parameters based on the calibration point includes: The calibration points are updated to the historical calibration point set, and a preset mapping expression is fitted and solved based on the updated historical calibration point set to obtain the first fitting parameters; and The current calibration parameters are updated based on the first fitted parameters.

7. The method according to claim 6, characterized in that, The step of updating the current calibration parameters based on the calibration point includes: Substituting the current calibration parameters into the preset mapping expression yields a first mapping expression, and based on the first mapping expression and the calibration point, the coordinate deviation is determined; and The current calibration parameters are updated based on the coordinate deviation.

8. The method according to claim 6, characterized in that, The step of updating the current calibration parameters based on the calibration point includes: Substitute the initial calibration parameters into the preset mapping expression to obtain the second mapping expression, and determine the compensation calibration point set based on the updated historical calibration point set and the second mapping expression; The preset mapping expression is fitted and solved based on the set of compensation calibration points to obtain a second fitting parameter. The second fitting parameter is then substituted into the preset mapping expression to obtain a third mapping expression. The third mapping expression and the second mapping expression are superimposed to obtain the fourth mapping expression, and the current calibration parameters are updated based on the fourth mapping expression.

9. The method according to any one of claims 1-5, characterized in that, The step of updating the current calibration parameters based on the calibration point includes: A first parameter vector is constructed based on the calibration points, and a gain vector is determined based on the first parameter vector and the covariance matrix corresponding to the current calibration parameters. A second parameter vector is constructed based on the current calibration parameters, and a third parameter vector is determined based on the second parameter vector, the gain vector, and the calibration point; and The current calibration parameters are updated based on the third parameter vector.

10. The method according to claim 6, characterized in that, The method further includes: Determine whether the number of calibration points in the updated set of historical calibration points exceeds a preset threshold. In response to determining that the number of calibration points exceeds the preset number threshold, it is determined whether there are outliers in the updated set of historical calibration points; and In response to the determination that outliers exist in the updated set of historical calibration points, outlier removal is performed on the updated set of historical calibration points.

11. A dual-camera self-calibration system, characterized in that, include: The image acquisition module is configured to determine a first image acquired by the first shooting module after the focusing module has completed focusing, and a second image acquired by the second shooting module, wherein there is an overlapping area between the field of view of the first shooting module and the field of view of the second shooting module. The matching determination module is configured to determine a target image corresponding to the first image, and to determine the matching relationship of the target image in the second image based on the target image's clarity meeting a preset first condition; The coordinate determination module is configured to determine the matching coordinates between the first image and the second image based on the position coordinates of the target image in the first image and the matching relationship, based on the matching relationship satisfying a preset second condition. as well as The parameter update module is configured to determine a calibration point based on the matching coordinates and the focusing position corresponding to the focusing module, and update the current calibration parameters based on the calibration point.

12. A computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the steps of the method as claimed in any one of claims 1-10.

13. An electronic device, characterized in that, include: One or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method as described in any one of claims 1-10.

Images