Binocular vision calibration method and device

Abstract

A binocular vision calibration method and device relates to the field of camera calibration. The method comprises: capturing multiple sets of left and right eye images in a first space using a binocular camera, inputting these images into a first processing model, and outputting two sets of light source position information, wherein the two sets of light source position information correspond to multiple left and right eye images, respectively; determining two light source motion paths based on the two sets of light source position information; matching the two light source motion paths to determine a first matching result; and calibrating the binocular camera based on the first matching result. Based on this solution, the left and right camera position offset can be detected during the use of the binocular camera after it leaves the factory, and the binocular camera can be calibrated based on the detection results, effectively suppressing error accumulation in the binocular camera to avoid failure of the depth of field modeling system. Furthermore, the method does not require additional auxiliary equipment and can be deployed on drones, resulting in low implementation costs.

Description

Technical Field

[0001] The present application relates to the field of camera calibration, and more specifically, to a method and apparatus for binocular vision calibration. Background Art

[0002] With the development of binocular depth-of-field technology, binocular cameras are also widely used in 3D visual modeling technology. Taking binocular cameras as an example, the camera's external parameters directly affect the camera's accuracy. Although binocular cameras undergo corresponding external parameter calibration operations in the factory laboratory environment before leaving the factory, this operation is also called static calibration. However, during the distribution, transportation, and application of binocular cameras, problems such as transportation vibration, high-frequency vibration during use, human-caused structural damage, and abnormal external parameter adjustments may occur. These problems can cause the binocular camera's focus to shift, ultimately causing the binocular camera to deviate from the original calibration position, ultimately affecting the visual modeling results. Therefore, dynamic calibration of binocular cameras during use is very necessary.

[0003] In view of this, a binocular vision calibration method applied to binocular cameras needs to be developed urgently. Summary of the Invention

[0004] This application provides a binocular vision calibration method and device that can accurately determine whether the external parameters of a binocular camera are appropriate and whether the camera head of the binocular camera has deviated from the original calibration position in a non-factory experimental environment. This allows for the corresponding calibration of the binocular camera, effectively suppressing error accumulation in the binocular camera and preventing failure of the depth of field modeling system. This method and device do not require additional auxiliary equipment, thus controlling implementation costs.

[0005] In a first aspect, a binocular vision calibration method is provided. The method includes: capturing a first image set in a first space using a binocular camera, wherein the first image set includes a first left-eye image and a first right-eye image, the first left-eye image and the first right-eye image include a first reference object, the first space includes a moving light source, and the binocular camera and the first reference object remain stationary; inputting the first left-eye image and the first right-eye image into a first processing model, respectively, and outputting first light source position information corresponding to the first reference object in the first left-eye image and second light source position information corresponding to the first reference object in the first right-eye image, wherein the first processing model is configured as follows:

[0006] Extracting a reference object included in an input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining a brightness value of each pixel in the pixel region; determining a light reflection type of the reference object based on the brightness value of each pixel; setting a plurality of sampling points on the surface of the reference object based on the light reflection type of the reference object; determining a plurality of light propagation paths based on imaging position information of the plurality of sampling points on an imaging device of a binocular camera; and determining light source position information corresponding to the reference object in the input image based on the position information of an intersection where the plurality of light propagation paths converge;

[0007] After the first time period, a plurality of second image sets are sequentially captured by the binocular camera, wherein the second image set includes a second left-eye image and a second right-eye image, and the plurality of second left-eye images and the plurality of second right-eye images all include the first reference object; the plurality of second left-eye images and the plurality of second right-eye images are input into a first processing model, and third light source position information corresponding to the first reference object in the plurality of second left-eye images and fourth light source position information corresponding to the first reference object in the plurality of second right-eye images are output; a first light source motion path is determined based on the first light source position information and the plurality of third light source position information, and a second light source motion path is determined based on the second light source position information and the plurality of fourth light source position information; the first light source motion path and the second light source motion path are matched to determine a first matching result; and a binocular camera is calibrated based on the first matching result.

[0008] For example, the first left-eye image and the first right-eye image may include multiple first reference objects, and the number may be related to the system computing power and computing resources. If the system computing power and computing resources are sufficient, the number of first reference objects may be appropriately increased.

[0009] For example, after determining the brightness value of each pixel in a pixel region, a weighted average operation can be performed on the brightness values ​​of the pixels in the region to obtain a first brightness result; alternatively, the maximum brightness value of the pixels in the region can be used as the first brightness result. When the first brightness result is less than a first brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is diffuse reflection; when the first brightness result is greater than or equal to the first brightness threshold and less than a second brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is glossy reflection; and when the first brightness result is greater than or equal to the second brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is specular reflection.

[0010] For example, the first processing model may also be a pre-trained neural network model, and in the process of iterative calculation based on different training samples, the model parameters are adjusted to increase the accuracy of data processing of the first processing model.

[0011] For example, the first time period may be a preset reasonable duration, such as 5 minutes, 10 minutes, etc.

[0012] For example, the binocular camera sequentially acquires multiple second image sets by acquiring one second image set every second time period. The second time period is also a preset reasonable duration, for example, 5 seconds, 15 seconds, etc. The specific second time period can be determined based on the movement speed of the light source. Furthermore, during the acquisition of multiple second image sets, the duration of the second time period can also be adaptively adjusted.

[0013] Based on this technical solution, the left and right camera offsets can be detected during post-production use. Calibration can then be performed based on the detection results, effectively suppressing error accumulation and preventing failure of the depth-of-field modeling system. Furthermore, without the need for additional auxiliary equipment, the system is low-cost, easy to deploy, and has broad applicability, particularly on smaller drones.

[0014] In combination with the first aspect, in certain implementations of the first aspect, before the first left-eye image and the first right-eye image are respectively input into the first processing model, the first left-eye image and the first right-eye image are cropped to retain part of the first reference object.

[0015] Based on the above technical solution, the features of the first reference object can be highlighted while reducing the subsequent overhead of performing corresponding calculations on the left and right eye images.

[0016] In combination with the first aspect, in certain implementations of the first aspect, when the light reflection type of the reference object is diffuse reflection, sampling points are evenly set on the surface of the reference object; or, when the light reflection type of the reference object is glossy reflection or mirror reflection, sampling points are set at the light reflection points on the surface of the reference object.

[0017] For example, it is known that glossy reflection is between diffuse reflection and specular reflection, so in terms of the number of sampling points set, the sampling points set in the diffuse reflection case are more than the sampling points set in the glossy reflection case, and the sampling points set in the glossy reflection case are more than the sampling points set in the specular reflection case.

[0018] Based on the above technical solution, the corresponding method of setting sampling points can be determined according to the type of light reflection on the surface of the reference object, which helps to avoid the waste of computing resources caused by setting too many sampling points, and also helps to avoid inaccurate calculated light propagation paths caused by setting too few sampling points.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned first matching result includes a first matching degree. When the first matching degree is lower than a first threshold, the first relative position information of the left and right cameras of the binocular camera set before leaving the factory is obtained; based on the first light source motion path and the second light source motion path, the second relative position information of the left and right cameras of the binocular camera is derived; based on the first relative position information and the second relative position information, the extrinsic parameter matrix of the binocular camera is calibrated, and the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

[0020] For example, the first threshold is a preset reasonable threshold. When the first matching degree is the output result of the intersection-over-union ratio function, the first threshold may be a value such as 0.7 or 0.8.

[0021] Based on the above technical solution, there is no need to construct an experimental environment in the factory. It is possible to directly determine the external parameter matrix used to compensate for the relative position offset of the left and right cameras of the binocular camera based on the deviation between the motion path of the first light source and the motion path of the second light source, so as to realize the dynamic visual calibration function of the binocular camera.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the binocular camera is fixedly installed on a drone.

[0023] In a second aspect, a binocular vision calibration device is provided, the device comprising:

[0024] a control unit, configured to control the binocular camera to capture a first image set in a first space, wherein the first image set includes a first left-eye image and a first right-eye image, the first left-eye image and the first right-eye image include a first reference object, the first space includes a moving light source, and the binocular camera and the first reference object remain stationary;

[0025] a processing unit, configured to input the first left image and the first right image into a first processing model, respectively, and output first light source position information corresponding to the first reference object in the first left image and second light source position information corresponding to the first reference object in the first right image, wherein the first processing model is configured as follows:

[0026] Extracting a reference object from an input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining brightness information of each pixel in the pixel region;

[0027] Determine the light reflection type of the reference object based on the brightness information of each pixel;

[0028] According to the light reflection type of the reference object, multiple sampling points are set on the surface of the reference object;

[0029] Determining multiple light propagation paths according to imaging position information of multiple sampling points on an imaging device of a binocular camera;

[0030] Determining the light source position information corresponding to the reference object in the input image based on the intersection position information of the multiple light propagation paths;

[0031] The control unit is further configured to, after the first period of time, control the binocular camera to sequentially capture a plurality of second image sets, wherein the second image set includes a second left-eye image and a second right-eye image, and the plurality of second left-eye images and the plurality of second right-eye images all include the first reference object;

[0032] The processing unit is further configured to input the plurality of second left-eye images and the plurality of second right-eye images into the first processing model, and output third light source position information corresponding to the first reference object in the plurality of second left-eye images and fourth light source position information corresponding to the first reference object in the plurality of second right-eye images;

[0033] The calibration unit is used to determine the motion path of the first light source based on the first light source position information and the multiple third light source position information, and to determine the motion path of the second light source based on the second light source position information and the multiple fourth light source position information; match the first light source motion path and the second light source motion path to determine a first matching result; and calibrate the binocular camera based on the first matching result.

[0034] In combination with the second aspect, in certain implementations of the second aspect, before the above-mentioned processing unit inputs the first left-eye image and the first right-eye image into the first processing model respectively, the above-mentioned processing unit is also used to: crop the first left-eye image and the first right-eye image to retain part of the first reference object.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing unit is specifically used to: when the light reflection type of the reference object is diffuse reflection, evenly set sampling points on the surface of the reference object; or, when the light reflection type of the reference object is glossy reflection or mirror reflection, set sampling points at the light reflection points on the surface of the reference object.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned first matching result includes a first matching degree, and the above-mentioned calibration unit is specifically used to: when the first matching degree is lower than a first threshold: obtain the first relative position information of the left and right cameras of the binocular camera set before leaving the factory; derive the second relative position information of the left and right cameras of the binocular camera based on the first light source motion path and the second light source motion path; calibrate the extrinsic parameter matrix of the binocular camera based on the first relative position information and the second relative position information, and the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the binocular camera is fixedly installed on a drone.

[0038] In a third aspect, a binocular vision calibration device is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation mode of the method design of the first aspect above.

[0039] In a fourth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to echo a signal from the memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes a method in any possible implementation of the method design of the first aspect above.

[0040] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program or instruction, which is executed by a processor to implement the method in any possible implementation of the method design of the first aspect.

[0041] In a sixth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic block diagram of a binocular vision calibration system 100 provided in an embodiment of the present application;

[0043] Figure 2 is a flow chart of a binocular vision calibration method 200 provided in an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of the principle of determining the path of light propagation;

[0045] Figure 4 is a schematic block diagram of a method 400 for setting sampling points proposed in an embodiment of the present application;

[0046] Figure 5 5 is a flowchart of a calibration method 500 proposed in an embodiment of the present application;

[0047] Figure 6 is a schematic diagram of a light source motion path matching result proposed in an embodiment of the present application;

[0048] Figure 7 It is a schematic block diagram of a binocular vision calibration device 700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0050] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0051] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0052] The binocular camera mentioned in the embodiments of this application can also be replaced by a binocular video camera or a laser-based binocular vision sensor. This embodiment of the application does not limit this. The method proposed in the embodiments of this application is applicable to devices based on binocular vision technology. For the sake of convenience, this application will be described in detail based on a binocular camera.

[0053] Binocular vision is an important form of machine vision. It uses the principle of parallax to obtain three-dimensional geometric information of an object through multiple images. A binocular vision system typically uses a binocular camera to simultaneously capture two images of the object from different angles. Based on the principle of parallax, the system recovers the object's three-dimensional geometric information and reconstructs the object's 3D outline and position.

[0054] The main factors affecting the accuracy of modeling based on binocular cameras include the following seven parameters:

[0055] 1. Baseline distance; 2. Camera focal length; 3. Angle between the optical axis and the baseline; 4. Camera distortion; 5. Number of calibration images; 6. Number of calibration checkerboard squares; 7. Position of the calibration plate.

[0056] Among them, parameters 1 to 3 are mainly the structural parameters of the binocular camera, which are completely determined after the camera is finalized. Therefore, it is necessary to determine whether these three parameters have defects before leaving the factory, and they cannot be changed during subsequent use. Parameter 4 is a property of the camera itself. Although it is a calibration parameter, it can be determined whether it has defects before leaving the factory, and it is difficult to change during subsequent use. Parameters 5 to 7 are external parameters that can be calibrated later. That is, parameters 5 to 7 are usually calibrated through calibration operations before leaving the factory. The calibration operation is also static calibration. However, during the sale, transportation and application of the binocular camera, there may be problems such as transportation vibration, high-frequency vibration during use, human-caused structural damage and abnormal external parameter adjustment, which may cause the focus of the binocular camera to shift, and ultimately cause the binocular camera to deviate from the original calibration position, ultimately affecting the visual modeling results.

[0057] It should be understood that the process of calibrating parameters 5 to 7 is essentially to calibrate the relative positions of the left and right cameras of the binocular camera. The calibrated relative position information can be used as the known attribute information of the binocular camera itself. However, during subsequent use, the relative positions of the left and right cameras may change. Since the subsequent use process is not in the factory experimental environment, it is difficult to calibrate the binocular camera using the static calibration method performed before leaving the factory. Therefore, it is necessary to calibrate it using a binocular camera calibration method in a non-experimental environment. Otherwise, long-term use will inevitably lead to error accumulation, resulting in a large error between the target model and the actual situation, making the binocular camera unusable.

[0058] An existing method for binocular vision calibration in a non-experimental environment is a bionic binocular vision extrinsic parameter continuous calibration method. This method uses a motor with position feedback to read the absolute position of the left and right cameras, and then calculates the relative matrix of the binocular cameras at the current moment. Since the parameter matrix of the initial relative position of the binocular cameras is known, the relative position relationship of the binocular cameras at any moment can be obtained using position feedback, thereby achieving continuous dynamic calibration. However, this method requires a motor device to be configured for the binocular cameras, increasing the cost of the overall equipment. In addition, if the space for binocular camera deployment is limited, it is difficult to configure a motor device for the binocular cameras, and it is even more difficult to use the motor device to drive the left and right cameras relative to each other. For example, a binocular camera with the above architecture is difficult to deploy on a small drone because the drone has a limited load capacity and the space for installing the binocular cameras is small. Therefore, the left and right cameras must be fixed and cannot be driven by the motor device.

[0059] In view of this, the embodiments of the present application propose a binocular vision calibration method, device and corresponding system, which estimates the light source displacement through multiple sets of images taken by left and right cameras and the brightness of the reference objects in the multiple sets of left and right images, and compares them with the static calibration results performed in the factory to achieve fast and continuous dynamic calibration.

[0060] Figure 1 1 is a schematic block diagram of a binocular vision calibration system 100 provided in an embodiment of the present application.

[0061] In some possible embodiments, the system 100 includes a binocular camera 110, an image processing platform 120, a computing platform 130, and a control platform 140. These devices may be connected via a wireless network to enable data exchange between the devices.

[0062] Among them, the binocular camera 110 is used to execute the image capture process mentioned in the binocular vision calibration method proposed in this application, obtain multiple sets of left and right eye images, and send these images to the image processing platform 120, or be actively obtained by the image processing platform 120.

[0063] The image processing platform 120 may include at least one neural network model dedicated to digital image processing, which performs image preprocessing, feature extraction, and other operations on the input image to make the image features more distinct. It also sends feature information related to visual calibration, such as the position change information of the light source corresponding to the reference object included in the image, to the computing platform 130.

[0064] The computing platform 130 is used to calculate the light source movement paths corresponding to multiple groups of left-eye images and the light source movement paths corresponding to multiple groups of right-eye images based on the received feature information related to the above-mentioned visual calibration, and to determine whether the left and right eye cameras are offset by determining the matching degree based on these two light source movement paths, and to compensate for the offset accordingly to achieve dynamic calibration of the binocular camera, suppress its error accumulation, and avoid failure of subsequent depth of field modeling.

[0065] The control platform 140 may be a remote server for controlling the binocular camera 110 , the image processing platform 120 , and the computing platform 130 to perform corresponding actions.

[0066] Based on the above system 100, an embodiment of the present application proposes a binocular vision calibration method.

[0067] Figure 2 This is a flowchart of a binocular vision calibration method 200 provided in an embodiment of the present application.

[0068] S210: In a first space, capture a first image set using a binocular camera.

[0069] The first image set includes a first left-eye image and a first right-eye image, the first left-eye image and the first right-eye image include a first reference object, and the first space includes a moving light source. The binocular camera and the first reference object remain stationary.

[0070] It should be understood that based on the operating mechanism of the binocular camera, the first left-eye image and the first right-eye image in the above-mentioned first image set should be taken at the same time.

[0071] In some possible embodiments, the binocular camera of the present application can be mounted on a drone, specifically an agricultural drone. Therefore, the first space is usually a farmland below the drone, and the moving light source is the sun.

[0072] In some possible embodiments, the first left-eye image and the first right-eye image may include multiple first reference objects, and the number may be related to the system computing power and computing resources. If the system computing power and computing resources are sufficient, the number of first reference objects may be appropriately increased.

[0073] In some possible embodiments, after acquiring the first left image and the first right image, image preprocessing operations may be performed on these two images. For example, a first reference object may be extracted, where the extraction criterion for the first reference object is that the object's surface has significant light reflection characteristics. Alternatively, after extracting the first reference object, the first left image and the first right image may be cropped based on the edge contour of the first reference object, retaining portions of the first reference object. This method can highlight the features of the first reference object while reducing the overhead of subsequent calculations performed on the left and right images.

[0074] S220: Input the first left eye image and the first right eye image into the first processing model respectively, and output the first light source position information corresponding to the first reference object in the first left eye image and the second light source position information corresponding to the first reference object in the first right eye image.

[0075] The first processing model is configured to perform the following steps:

[0076] S221: extracting a reference object included in the input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining a brightness value of each pixel in the pixel region.

[0077] S222: Determine the light reflection type of the reference object according to the brightness value of each pixel.

[0078] It should be understood that different types of light reflection have different brightness values ​​corresponding to the reflection points. For example, the brightness corresponding to the mirror reflection point is greater than the brightness corresponding to the glossy reflection point, and the brightness corresponding to the glossy reflection point is greater than the brightness corresponding to the diffuse reflection point.

[0079] In some possible embodiments, after determining the brightness value of each pixel in a pixel region, a weighted average operation may be performed on the brightness values ​​of the pixels in the region to obtain a first brightness result; alternatively, the maximum brightness value of the pixels in the region may be used as the first brightness result. When the first brightness result is less than a first brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is diffuse reflection; when the first brightness result is greater than or equal to the first brightness threshold and less than a second brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is glossy reflection; and when the first brightness result is greater than or equal to the second brightness threshold, it indicates that the light reflection type of the reference object corresponding to the pixel region is specular reflection.

[0080] S223: Setting a plurality of sampling points on the surface of the reference object according to the light reflection type of the reference object.

[0081] S224: Determine multiple light propagation paths according to imaging position information of multiple sampling points on the imaging device of the binocular camera.

[0082] S225: Determine the light source position information corresponding to the reference object in the input image according to the intersection position information where the multiple light propagation paths converge.

[0083] Figure 3 It is a schematic diagram of the principle of determining the path of light propagation.

[0084] refer to Figure 3 As shown, taking a sampling point as an example, due to the reflection of light at the sampling point, the light is finally refracted by the camera lens and falls on the imaging device deployed behind the camera, and the sampling point is mapped to the specified position of the imaging device (also called the imaging position). Then, after obtaining the imaging position of the sampling point, based on the principle of light propagation (reflection and refraction), the entire light path from the imaging device to the actual sampling point position, and then from the actual sampling point position to the light source can be inferred. The actual light propagation path is Figure 3 The focus of multiple optical fiber propagation paths is the actual position of the light source.

[0085] In some possible embodiments, the first processing model may also be a pre-trained neural network model, and the model parameters are adjusted during iterative calculations based on different training samples to increase the accuracy of data processing by the first processing model. The specific training process of the first processing model is as follows:

[0086] First, at least one training sample must be determined. This training sample includes a training image and a sample label for the training sample. The training image includes various reference objects. Specifically, the position information of sampling points corresponding to the reference objects can be extracted through the aforementioned one-dimensional color processing. The sample label of the training sample indicates the position information of each sampling point in the space depicted in the image and the light source position information corresponding to the position information mapped to the imaging device within the camera. This light source position information can be the distance relative to a preset reference point in the camera or the three-dimensional coordinates of a point in the three-dimensional space presented in the training image. A first processing model is then trained based on this at least one training sample, so that in subsequent practical applications, after acquiring an image, the first processing model can directly determine the light source position information in the scene depicted in the image.

[0087] S230: After the first period of time, a plurality of second image sets are captured in sequence by the binocular camera.

[0088] The second image set includes a second left-eye image and a second right-eye image, and the plurality of second left-eye images and the plurality of second right-eye images all include the first reference object.

[0089] In some possible embodiments, the first time period can be a preset reasonable duration, such as 5 minutes, 10 minutes, etc. Furthermore, S230 can be executed based on a timer, i.e., a timer trigger mechanism can be embedded in the controller corresponding to the binocular camera: when the controller decides to perform visual calibration on the binocular camera, it records the current moment as the first moment and simultaneously executes S210. When the controller determines that S220 has been completed and the first time period has passed since the current moment, it controls the binocular camera to execute S230.

[0090] In some possible embodiments, the controller may be a control chip installed in a binocular camera or a control chip installed in a remote server, and establishes a connection between the binocular camera, the data processing platform, and the computing platform, and performs remote control.

[0091] It should be understood that while both the binocular camera and the first reference object remain stationary during the aforementioned steps, in actual applications, the device on which the binocular camera is mounted may experience minor vibrations, such as vibrations. For example, a drone may remain stationary in mid-air, but the drone itself may experience vibrations due to the operation of its propellers. This may result in no displacement of the binocular camera as a whole, but the relative positions of the two cameras within the binocular camera may shift. Therefore, before executing S230, the binocular camera must remain stationary for a first period of time to allow the offset between the two cameras to accumulate, making it easier to detect the offset.

[0092] In some possible embodiments, the binocular camera sequentially acquires multiple second image sets, and the binocular camera may acquire a second image set every second period of time. The second period of time is also a preset reasonable duration, for example, 5 seconds, 15 seconds, etc. The specific second period of time may be determined based on the movement speed of the light source. Furthermore, during the process of acquiring multiple second image sets, the duration of the second period of time may also be adaptively adjusted.

[0093] S240: Input multiple second left-eye images and multiple second right-eye images into the first processing model, and output the third light source position information corresponding to the first reference object in the multiple second left-eye images and the fourth light source position information corresponding to the first reference object in the multiple second right-eye images.

[0094] It should be understood that S230 and S240 are essentially repeated steps S210 and S220, and by obtaining multiple position information of the light source, the motion trajectory of the light source can be determined. Specific embodiments are detailed in the above corresponding content and will not be repeated here.

[0095] S250: Determine a first light source movement path according to the first light source position information and the plurality of third light source position information, and determine a second light source movement path according to the second light source position information and the plurality of fourth light source position information.

[0096] It should be understood that the above-mentioned first light source position information and multiple third light source position information are all determined based on the left-eye image obtained by the left-eye camera, and therefore, the first light source motion path is associated with the left-eye camera; the above-mentioned second light source position information and multiple fourth light source position information are all determined based on the right-eye image obtained by the right-eye camera, and therefore, the second light source motion path is associated with the right-eye camera.

[0097] S260: Match the first light source motion path and the second light source motion path to determine a first matching result.

[0098] It should be understood that, when the relative positions of the left and right cameras of the binocular camera are not offset, the first light source motion path and the second light source motion path should be completely consistent. Conversely, when the relative positions of the left and right cameras of the binocular camera are offset, the first light source motion path and the second light source motion path should deviate, and the greater the relative position offset of the left and right cameras, the greater the deviation between the first light source motion path and the second light source motion path.

[0099] S270: Perform a calibration operation on the binocular camera according to the first matching result.

[0100] In some possible embodiments, after obtaining the first light source motion path and the second light source motion path, a first light source path schematic diagram and a second light source path schematic diagram can be generated respectively. The two schematic diagrams have a coordinate system of the same standard specifications, specifically a three-dimensional coordinate system for representing the specific motion path of the light source in space. Then, based on the intersection over union (IOU) function, a matching operation is performed on the two schematic diagrams to determine the above-mentioned first matching result. Among them, based on the basic principle of the intersection over union function, it can be known that if the first matching result is equal to 1, it means that the first light source motion path and the second light source motion path coincide with each other, that is, there is no relative position offset between the left and right eye cameras of the binocular camera; if the first matching result is not equal to 1, it means that there is a deviation between the first light source motion path and the second light source motion path, and there is a relative position offset between the left and right eye cameras of the binocular camera. It should be understood that the offset is the difference between the relative position between the left and right eye cameras at the current moment and the relative position between the left and right eye cameras determined during the static calibration before leaving the factory.

[0101] Based on this technical solution, the left and right camera offsets can be detected during post-production use. Calibration can then be performed based on the detection results, effectively suppressing error accumulation and preventing failure of the depth-of-field modeling system. Furthermore, without the need for additional auxiliary equipment, the system is low-cost, easy to deploy, and has broad applicability, particularly on smaller drones.

[0102] In some possible embodiments, as described in the above embodiments, light reflection can include the following three types: diffuse reflection, glossy reflection, and specular reflection. Based on the above S223, it can be seen that for different light reflection types, different sampling points are set on the reference object surface. The specific method for setting sampling points on the reference object surface is as follows.

[0103] Figure 4 4 is a schematic block diagram of a method 400 for setting sampling points proposed in an embodiment of the present application.

[0104] refer to Figure 4 It can be seen that, when the light reflection type of the reference object is diffuse reflection, sampling points are evenly set on the surface of the reference object; or,

[0105] When the light reflection type of the reference object is glossy reflection or specular reflection, a sampling point is set at the light reflection point on the surface of the reference object.

[0106] In some possible embodiments, when the light reflection type of the reference object is glossy reflection or mirror reflection, the image of the light source can be mapped onto the imaging device of the binocular camera. Then, in the left and right eye images, there will be a light spot corresponding to the light source on the surface of the reference object, and the pixel brightness corresponding to the light spot part in the image is at least greater than the first brightness threshold proposed in the above embodiment. It is only necessary to set the sampling point in the pixel corresponding to the light spot part.

[0107] It should be understood that it is known that glossy reflection is between diffuse reflection and specular reflection, so in terms of the number of sampling points set, the sampling points set in the diffuse reflection case are more than the sampling points set in the glossy reflection case, and the sampling points set in the glossy reflection case are more than the sampling points set in the specular reflection case.

[0108] Based on the above technical solution, the corresponding method of setting sampling points can be determined according to the type of light reflection on the surface of the reference object, which helps to avoid the waste of computing resources caused by setting too many sampling points, and also helps to avoid inaccurate calculated light propagation paths caused by setting too few sampling points.

[0109] In some possible embodiments, if there are multiple sampling points set on the surface of the reference object, then the final determined light propagation paths should also be multiple. Ideally, the determined light propagation paths should all be associated with a light source position. Since the light source is not a point in an abstract sense, there may be deviations between the light source positions associated with multiple light propagation paths. When the light source position is represented by three-dimensional coordinates, the light source coordinates associated with multiple light propagation paths can be added and averaged, and the resulting coordinates can be used as the final determined light source position information.

[0110] In some possible embodiments, after obtaining the light source coordinates associated with multiple light propagation paths, each light source coordinate can be subtracted from the other light source coordinates to determine a light source coordinate deviation value. When the light source coordinate deviation value exceeds a preset deviation threshold, it indicates that the light source coordinate value is abnormal and can be eliminated. This solution further improves the accuracy of the determined light source position information.

[0111] In some possible embodiments, the above S270 may be implemented by the following method.

[0112] Figure 5 5 is a flow chart of a calibration method 500 proposed in an embodiment of the present application. The first matching result includes a first matching degree, which may be the output result of the intersection-over-union function proposed in the above embodiment. When the first matching degree is lower than a first threshold, the following steps are performed:

[0113] S510: Acquire first relative position information of the left and right cameras of the binocular camera set before leaving the factory.

[0114] S520: Deriving second relative position information of the left and right cameras of the binocular camera according to the first light source motion path and the second light source motion path.

[0115] S530: Calibrate the extrinsic parameter matrix of the binocular camera according to the first relative position information and the second relative position information, where the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

[0116] In some possible embodiments, the first threshold is a preset reasonable threshold. When the first matching degree is the output result of the intersection-over-union function, the first threshold may be a value such as 0.7 or 0.8.

[0117] Figure 6 : is a schematic diagram of a light source motion path matching result proposed in an embodiment of the present application. The corresponding first threshold is 0.7.

[0118] The matching result 1 corresponds to a matching degree of 0.9, which is greater than the first threshold value. This indicates a good match, and the two light source motion paths (the first light source motion path and the second light source motion path) are substantially coincident.

[0119] The matching degree corresponding to matching result 2 is 0.7, which is exactly equal to the first threshold. It can be seen that the overlap of the two light source motion paths is worse than that of matching result 1.

[0120] The matching degree corresponding to the matching result 3 is 0.4, which is lower than the first threshold. It can be seen that the relative offset of the two light source motion paths is large, so the position offset between the cameras of the binocular camera is large, and the above method 500 needs to be used to calibrate the binocular camera.

[0121] Based on the above technical solution, there is no need to construct an experimental environment in the factory. It is possible to directly determine the external parameter matrix used to compensate for the relative position offset of the left and right cameras of the binocular camera based on the deviation between the motion path of the first light source and the motion path of the second light source, so as to realize the dynamic visual calibration function of the binocular camera.

[0122] In addition, embodiments of the present application also provide a device for implementing any of the above methods. Figure 7 : is a schematic block diagram of a binocular vision calibration apparatus 700 provided in an embodiment of the present application. The apparatus 700 includes:

[0123] A control unit 710 is configured to control a binocular camera to capture a first image set in a first space, wherein the first image set includes a first left image and a first right image, the first left image and the first right image include a first reference object, the first space includes a moving light source, and the binocular camera and the first reference object remain stationary;

[0124] The processing unit 720 is configured to input the first left image and the first right image into a first processing model, respectively, and output first light source position information corresponding to the first reference object in the first left image and second light source position information corresponding to the first reference object in the first right image, wherein the first processing model is configured as follows:

[0125] Extracting a reference object from an input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining brightness information of each pixel in the pixel region;

[0126] Determine the light reflection type of the reference object based on the brightness information of each pixel;

[0127] According to the light reflection type of the reference object, multiple sampling points are set on the surface of the reference object;

[0128] Determining multiple light propagation paths according to imaging position information of multiple sampling points on an imaging device of a binocular camera;

[0129] Determining the light source position information corresponding to the reference object in the input image based on the intersection position information of the multiple light propagation paths;

[0130] The control unit 710 is further configured to, after the first period of time, control the binocular camera to sequentially capture a plurality of second image sets, wherein the second image set includes a second left-eye image and a second right-eye image, and the plurality of second left-eye images and the plurality of second right-eye images all include the first reference object;

[0131] The processing unit 720 is further configured to input the plurality of second left-eye images and the plurality of second right-eye images into the first processing model, and output third light source position information corresponding to the first reference object in the plurality of second left-eye images and fourth light source position information corresponding to the first reference object in the plurality of second right-eye images.

[0132] The calibration unit 730 is used to determine the motion path of the first light source based on the first light source position information and the multiple third light source position information, and to determine the motion path of the second light source based on the second light source position information and the multiple fourth light source position information; match the first light source motion path and the second light source motion path to determine a first matching result; and calibrate the binocular camera based on the first matching result.

[0133] In some possible embodiments, before the processing unit 720 inputs the first left-eye image and the first right-eye image into the first processing model respectively, the processing unit 720 is further used to: crop the first left-eye image and the first right-eye image to retain part of the first reference object.

[0134] In some possible embodiments, the processing unit 720 is specifically used to: when the light reflection type of the reference object is diffuse reflection, set sampling points evenly on the surface of the reference object; or when the light reflection type of the reference object is glossy reflection or mirror reflection, set sampling points at the light reflection points on the surface of the reference object.

[0135] In some possible embodiments, the above-mentioned first matching result includes a first matching degree, and the above-mentioned calibration unit 630 is specifically used to: when the first matching degree is lower than a first threshold: obtain the first relative position information of the left and right cameras of the binocular camera set before leaving the factory; derive the second relative position information of the left and right cameras of the binocular camera based on the first light source motion path and the second light source motion path; calibrate the extrinsic parameter matrix of the binocular camera based on the first relative position information and the second relative position information, and the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

[0136] In some possible embodiments, the binocular camera used with the device 700 is fixedly installed on a drone.

[0137] The above-described device can detect the position offset of the left and right cameras during the use of the binocular camera after it leaves the factory. It can also calibrate the binocular camera based on the detection results, effectively suppressing the accumulation of binocular camera errors and preventing failure of the depth of field modeling system. Furthermore, without the need for additional auxiliary equipment, the implementation cost is low, deployment is easy, and applicability is broad, especially for deployment on smaller drones.

[0138] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory, a magnetic disk, or an optical disk.

[0143] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A binocular vision calibration method, characterized in that: Applied to a binocular camera, the method includes: In a first space, a first image set is captured by the binocular camera, where the first image set includes a first left-eye image and a first right-eye image, where the first left-eye image and the first right-eye image include a first reference object, the first space includes a moving light source, and the binocular camera and the first reference object remain stationary; The first left image and the first right image are respectively input into a first processing model, and first light source position information corresponding to the first reference object in the first left image and second light source position information corresponding to the first reference object in the first right image are output. The first processing model is configured as follows: Extracting a reference object included in an input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining a brightness value of each pixel in the pixel region; determining the light reflection type of the reference object according to the brightness value of each pixel; Setting a plurality of sampling points on the surface of the reference object according to the light reflection type of the reference object; determining a plurality of light propagation paths according to imaging position information of the plurality of sampling points on the imaging device of the binocular camera; Determining light source position information corresponding to the reference object in the input image based on the intersection position information where the multiple light propagation paths converge; After a first period of time, sequentially capturing a plurality of second image sets using the binocular camera, wherein the second image sets include a second left-eye image and a second right-eye image, and the plurality of second left-eye images and the plurality of second right-eye images all include the first reference object; Inputting a plurality of the second left-eye images and a plurality of the second right-eye images into the first processing model, and outputting third light source position information corresponding to the first reference object in the plurality of the second left-eye images and fourth light source position information corresponding to the first reference object in the plurality of the second right-eye images; Determine a first light source movement path based on the first light source position information and the plurality of third light source position information, and determine a second light source movement path based on the second light source position information and the plurality of fourth light source position information; Matching the first light source motion path and the second light source motion path to determine a first matching result, where the first matching result includes a first matching degree; When the first matching degree is lower than a first threshold: Obtaining first relative position information of the left and right cameras of the binocular camera set before leaving the factory; Derived second relative position information of the left and right cameras of the binocular camera according to the first light source motion path and the second light source motion path; The extrinsic parameter matrix of the binocular camera is calibrated according to the first relative position information and the second relative position information, where the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

2. The method according to claim 1, characterized in that Before inputting the first left-eye image and the first right-eye image into the first processing model respectively, the method further includes: The first left-eye image and the first right-eye image are cropped to retain a portion of the first reference object.

3. The method according to claim 1 or 2, characterized in that The step of setting a plurality of sampling points on the surface of the reference object according to the light reflection type of the reference object includes: In the case where the light reflection type of the reference object is diffuse reflection, sampling points are evenly arranged on the surface of the reference object; or, In a case where the light reflection type of the reference object is glossy reflection or specular reflection, a sampling point is set at the light reflection point on the surface of the reference object.

4. The method according to claim 1, wherein The binocular camera is fixedly installed on the drone.

5. A binocular vision calibration device, characterized in that: Applied to a binocular camera, the device comprises: a control unit, configured to control the binocular camera to capture a first image set in a first space, where the first image set includes a first left-eye image and a first right-eye image, where the first left-eye image and the first right-eye image include a first reference object, the first space includes a moving light source, and the binocular camera and the first reference object remain stationary; a processing unit, configured to input the first left image and the first right image into a first processing model, respectively, and output first light source position information corresponding to the first reference object in the first left image and second light source position information corresponding to the first reference object in the first right image, wherein the first processing model is configured as follows: Extracting a reference object included in an input image, performing one-dimensional processing on a pixel region corresponding to the reference object, and determining a brightness value of each pixel in the pixel region; determining the light reflection type of the reference object according to the brightness value of each pixel; Setting a plurality of sampling points on the surface of the reference object according to the light reflection type of the reference object; determining a plurality of light propagation paths according to imaging position information of the plurality of sampling points on the imaging device of the binocular camera; Determining light source position information corresponding to the reference object in the input image based on the intersection position information where the multiple light propagation paths converge; The control unit is further configured to, after a first period of time, control the binocular camera to sequentially capture a plurality of second image sets, wherein the second image sets include a second left-eye image and a second right-eye image, and both the plurality of second left-eye images and the plurality of second right-eye images include the first reference object; The processing unit is further configured to input the plurality of second left-eye images and the plurality of second right-eye images into the first processing model, and output third light source position information corresponding to each of the plurality of second left-eye images and fourth light source position information corresponding to each of the plurality of second right-eye images; A calibration unit is used to determine the first light source motion path based on the first light source position information and the multiple third light source position information, and determine the second light source motion path based on the second light source position information and the multiple fourth light source position information; match the first light source motion path and the second light source motion path to determine a first matching result, wherein the first matching result includes a first matching degree; when the first matching degree is lower than a first threshold: obtain the first relative position information of the left and right cameras of the binocular camera set before leaving the factory; derive the second relative position information of the left and right cameras of the binocular camera based on the first light source motion path and the second light source motion path; calibrate the extrinsic parameter matrix of the binocular camera based on the first relative position information and the second relative position information, wherein the extrinsic parameter matrix is ​​used to compensate for the relative position offset of the left and right cameras of the binocular camera.

6. The device according to claim 5, characterized in that Before the processing unit inputs the first left image and the first right image into the first processing model respectively, the processing unit is further configured to: The first left-eye image and the first right-eye image are cropped to retain a portion of the first reference object.

7. The device according to claim 5 or 6, characterized in that The processing unit is specifically configured to: In the case where the light reflection type of the reference object is diffuse reflection, sampling points are evenly arranged on the surface of the reference object; or, In a case where the light reflection type of the reference object is glossy reflection or specular reflection, a sampling point is set at the light reflection point on the surface of the reference object.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are executed by a processor to implement the method according to any one of claims 1 to 4.

Images