Binocular camera image splicing method and device, electronic equipment and storage medium

Abstract

The invention provides a binocular camera image stitching method and device, electronic equipment and a storage medium, and relates to the technical field of video surveillance, and the method comprises the steps: determining the corresponding working state angle deviation data of two sensors in different directions based on the working angle data corresponding to the two sensors in a binocular camera; under the condition that the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, based on the angle deviation calibration data and the working state angle deviation data, correction preprocessing is carried out on an output image corresponding to a target sensor in the target direction, and the target sensor belongs to two sensors; and splicing the corrected output images of the two sensors. According to the invention, the splicing effect between the output images corresponding to the two sensors after the structural member is deformed can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video surveillance, and in particular, to a binocular camera image stitching method, apparatus, electronic device, and storage medium. Background Art

[0002] In recent years, binocular cameras have been favored by users due to their advantages of wide monitoring range and small distortion. Binocular cameras capture images from two different perspectives simultaneously to obtain rich information.

[0003] However, when a binocular camera is used in a scene with a high ambient temperature, structural components such as screws, studs, and gaskets in the binocular camera will deform, resulting in a change in the relative position of the fields of view between the two sensors in the binocular camera compared to the initial relative position of the fields of view during calibration. As a result, the stitching effect between the output images corresponding to the two sensors is poor. Summary of the Invention

[0004] The present invention provides a binocular camera image stitching method, apparatus, electronic device, and storage medium to solve the defect in the prior art that the stitching effect is poor due to the deformation of structural components in a high-temperature environment, and to improve the stitching effect between the output images corresponding to the two sensors after the deformation of the structural components.

[0005] The present invention provides a binocular camera image stitching method, including:

[0006] Determining working state angle deviation data corresponding to the two sensors in different directions based on the working angle data corresponding to the two sensors in the binocular camera;

[0007] In the case where the working state angle deviation data in the target direction is inconsistent with the angle deviation calibration data, performing correction preprocessing on the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data, where the target sensor belongs to the two sensors;

[0008] Stitching the output images of the two sensors after correction.

[0009] According to the binocular camera image stitching method provided by the present invention, the performing correction preprocessing on the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data includes:

[0010] Obtaining the output images corresponding to the two sensors respectively;

[0011] Determining the target sensor based on the output images corresponding to the two sensors respectively;

[0012] Based on the angle deviation calibration data and the working state angle deviation data in the target direction, determine the target correction parameters corresponding to the output image of the target sensor;

[0013] Determine the target correction operation corresponding to the target direction, where the target correction operation is used to perform pre-correction processing on the output image of the target sensor;

[0014] Based on the target correction parameters, perform the target correction operation on the output image of the target sensor.

[0015] According to the binocular camera image stitching method provided by the present invention, the determining the target correction parameters corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0016] When the target direction is the first preset direction, based on the first difference between the angle deviation calibration data and the working state angle deviation data in the first preset direction, determine the target rotation angle corresponding to the output image of the target sensor;

[0017] Determine the target rotation angle as the target correction parameter corresponding to the output image of the target sensor.

[0018] According to the binocular camera image stitching method provided by the present invention, the determining the target correction parameters corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0019] When the target direction is the second preset direction, determine the second difference between the angle deviation calibration data and the working state angle deviation data in the second preset direction;

[0020] Based on the product of the first preset threshold and the tangent value corresponding to the second difference, determine the first offset pixel number of the output image of the target sensor in the second preset direction; the first preset threshold is used to characterize the mapping relationship between the change of the field of view angle and the number of pixel point offsets in the second preset direction;

[0021] Determine the first offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

[0022] According to the binocular camera image stitching method provided by the present invention, the determining the target correction parameters corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0023] When the target direction is the third preset direction, determine a third difference between the angle deviation calibration data and the working state angle deviation data in the third preset direction;

[0024] Based on the product of the second preset threshold and the tangent value corresponding to the third difference, determine a second offset pixel number of the output image of the target sensor in the third preset direction; the second preset threshold is used to represent the mapping relationship between the change of the field of view angle and the number of pixel point offsets in the third preset direction;

[0025] Determine the second offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

[0026] According to the binocular camera image stitching method provided by the present invention, the determining the target sensor based on the output images respectively corresponding to the two sensors includes:

[0027] Perform target detection on each of the output images to determine a motion area corresponding to each of the output images;

[0028] Based on the pixel proportion of each motion area in the corresponding output image, determine a motion proportion corresponding to each of the output images;

[0029] Determine the sensor corresponding to the minimum motion proportion as the target sensor.

[0030] According to the binocular camera image stitching method provided by the present invention, the stitching the output images of the two sensors after correction includes:

[0031] In the output image of the target sensor after correction, determine a valid pixel area at the same horizontal height as the output image corresponding to other sensors;

[0032] Based on the horizontal edge of the valid pixel area, respectively crop the output image of the target sensor after correction and the output image corresponding to other sensors to obtain a first image to be stitched corresponding to the output image of the target sensor after correction after cropping, and a second image to be stitched corresponding to the output image corresponding to other sensors;

[0033] Stitch the first image to be stitched and the second image to be stitched.

[0034] The present invention also provides a binocular camera image stitching device, including:

[0035] A determination module, configured to determine working state angle deviation data corresponding to the two sensors in different directions based on the working angle data respectively corresponding to the two sensors in the binocular camera;

[0036] A calibration preprocessing module is configured to, when there is an inconsistency between the working state angle deviation data and the angle deviation calibration data in the target direction, perform calibration preprocessing on the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data, where the target sensor belongs to two sensors;

[0037] A splicing module is configured to splice the output images of the two sensors after calibration.

[0038] The present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for stitching binocular camera images as described in any one of the above is implemented.

[0039] The present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for stitching binocular camera images as described in any one of the above is implemented.

[0040] The method, device, electronic device, and storage medium for stitching binocular camera images provided by the present invention obtain the working angle data of two sensors in a binocular camera in a working state, determine the working state angle deviation data corresponding to the two sensors in different directions, and perform calibration preprocessing on the output image corresponding to the target sensor along the target direction according to the comparison result of the angle deviation calibration data and the working state angle deviation data in different directions, reducing the influence of the deformation of the structural member in a high-temperature environment on the output image corresponding to the target sensor, thereby improving the image stitching effect corresponding to the two sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 is one of the flow diagrams of the method for stitching binocular camera images provided by the embodiments of the present invention;

[0043] Figure 2 is another flow diagram of the method for stitching binocular camera images provided by the embodiments of the present invention;

[0044] Figure 3 is one of the schematic diagrams of calibration preprocessing provided by the embodiments of the present invention;

[0045] Figure 4 It is the second schematic diagram of the calibration preprocessing provided by the embodiment of the present invention;

[0046] Figure 5 It is the third schematic diagram of the calibration preprocessing provided by the embodiment of the present invention;

[0047] Figure 6 It is the schematic diagram of cropping provided by the embodiment of the present invention;

[0048] Figure 7 It is the schematic structural diagram of the binocular camera image stitching device provided by the embodiment of the present invention;

[0049] Figure 8 It is the schematic structural diagram of the electronic device provided by the embodiment of the present invention. Specific Embodiments

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0051] Aiming at the problem of poor stitching effect caused by the deformation of structural components in a high-temperature environment in the prior art, the embodiment of the present invention provides a binocular camera image stitching method. Figure 1 It is the first schematic flowchart of the binocular camera image stitching method provided by the embodiment of the present invention. As Figure 1 shown, the method includes:

[0052] Step 110: Determine the working state angle deviation data corresponding to the two sensors in different directions based on the working angle data corresponding to the two sensors in the binocular camera.

[0053] Specifically, gyroscopes are respectively installed on the two sensors (sensors) in the binocular camera. Through the gyroscopes corresponding to the two sensors respectively, the angle calibration data corresponding to the two sensors at the time of factory production and the angle data corresponding to the working state can be obtained. According to the difference between the angle calibration data corresponding to the two sensors at the time of factory production, the angle deviation calibration data corresponding to the two sensors can be determined. Specifically: the two sensors are sensor A and sensor B respectively. Through the gyroscope A corresponding to sensor A, the first angle calibration data corresponding to sensor A after binocular stitching calibration during factory production of the binocular camera is obtained. The first angle calibration data is the first calibration angle data α of sensor A in the x-axis direction in the space coordinate system at the time of factory production.1 , the second calibration angle data β in the y-axis direction 1 and the third calibration angle data γ in the z-axis direction 1 . After the binocular camera is calibrated for binocular stitching at the factory, the second angle calibration data corresponding to sensor B is obtained through the gyroscope B corresponding to sensor B. The second angle calibration data is the fourth calibration angle data α in the x-axis direction, the fifth calibration angle data β in the y-axis direction 2 , and the sixth calibration angle data γ in the z-axis direction 2 corresponding to sensor B in the space coordinate system at the time of factory 2 . According to the difference between the first angle calibration data and the second angle calibration data, the angle deviation calibration data corresponding to sensor A and sensor B in different directions can be determined. Specifically: through the difference between the first calibration angle data α 1 and the fourth calibration angle data α 2 , the first angle deviation calibration data a in the x-axis direction between sensor A and sensor B can be determined 0 , that is, a 0 =α 1 -α 2 ; through the difference between the second calibration angle data β 1 and the fifth calibration angle data β 2 , the second angle deviation calibration data b in the y-axis direction between sensor A and sensor B can be determined 0 , that is, b 0 =β 1 -β 2 ; through the difference between the third calibration angle data γ1 and the sixth calibration angle data γ2, the third angle deviation calibration data c in the z-axis direction between sensor A and sensor B can be determined 0 , that is, c 0 =γ 1 -γ 2 . The angle deviation calibration data corresponding to sensor A and sensor B is composed of the first angle deviation calibration data a 0 , the second angle deviation calibration data b 0 and the third angle deviation calibration data c 0 .

[0054] . After that, according to the difference between the angle data of the two sensors in the working state, the working state angle deviation data corresponding to the two sensors can be determined. Specifically: the first angle data corresponding to sensor A in the working state of the binocular camera is obtained through the gyroscope A corresponding to sensor A. The first angle data is the first working angle data α' in the x-axis direction of sensor A in the space coordinate system in the working state 1, the second working angle data β' in the y-axis direction 1 and the third working angle data γ' in the z-axis direction 1 . The second angle data corresponding to sensor B of the binocular camera in the working state is obtained through the gyroscope B corresponding to sensor B. The second angle data is the fourth working angle data α' of sensor B in the spatial coordinate system in the x-axis direction respectively in the working state 2 , the fifth working angle data β' in the y-axis direction 2 and the sixth working angle data γ' in the z-axis direction 2 . According to the difference between the first angle data and the second angle data, the working state angle deviation data corresponding to sensor A and sensor B in different directions can be determined. Specifically: through the first working angle data α' 1 and the fourth working angle data α' 2 , the first working state angle deviation data a' of sensor A and sensor B in the x-axis direction can be determined 0 , that is, a' 0 =α' 1 -α' 2 ; through the second working angle data β' 1 and the fifth working angle data β' 2 , the second working state angle deviation data b' of sensor A and sensor B in the y-axis direction can be determined 0 , that is, b' 0 =β' 1 -β' 2 ; through the third working angle data γ' 1 and the sixth working angle data γ' 2 , the third working state angle deviation data c' of sensor A and sensor B in the z-axis direction can be determined 0 , that is, c' 0 =γ' 1 -γ' 2 . The working state angle deviation data corresponding to sensor A and sensor B is composed of the first working state angle deviation data a' 0 , the second working state angle deviation data b' 0 and the third working state angle deviation data c' 0 .

[0055] Step 120, in the case where the working state angle deviation data in the target direction is inconsistent with the angle deviation calibration data, based on the angle deviation calibration data and the working state angle deviation data, pre-correct the output image corresponding to the target sensor along the target direction, and the target sensor belongs to two sensors.

[0056] Specifically, Figure 2 is the second flowchart of the binocular camera image stitching method provided by the embodiment of the present invention. As Figure 2 shown, after determining the angle deviation calibration data and working state angle deviation data corresponding to the two sensors, compare the working state angle deviation data and angle deviation calibration data in the same direction to determine whether there is inconsistency between the working state angle deviation data and angle deviation calibration data in the target direction, that is, compare the first working state angle deviation data a' of sensor A and sensor B in the x-axis direction 0 with the first angle deviation calibration data a 0 and compare the second working state angle deviation data b' of sensor A and sensor B in the y-axis direction 0 with the second angle deviation calibration data b 0 and compare the third working state angle deviation data c' of sensor A and sensor B in the z-axis direction 0 with the third angle deviation calibration data c 0 On the one hand, if the working state angle deviation data and angle deviation calibration data of sensor A and sensor B in the three directions are all consistent, that is, a' 0 = a 0 , b' 0 = b 0 and c' 0 = c 0 hold simultaneously, it indicates that the relative position of the fields of view between sensor A and sensor B of the binocular camera in the working state is exactly the same as that during the factory stitching calibration, that is, the stitching effect between the output images corresponding to sensor A and sensor B in the working state is the same as that during the factory stitching calibration. At this time, no pre-correction processing is required for sensor A or sensor B. On the other hand, if there is inconsistency between the working state angle deviation data and angle deviation calibration data of sensor A and sensor B in the target direction, it can be understood that there is inconsistency between the working state angle deviation data and angle deviation calibration data in at least one of the x-axis direction, y-axis direction, and z-axis direction in the space coordinate system, that is, a' 0 = a 0 , b' 0 = b 0 and c' 0 = c 0When they do not hold simultaneously, all inconsistent directions are determined as the target directions. Then, the target sensor is determined from sensorA and sensorB. Subsequently, for each target direction, based on the angle deviation calibration data and the working state angle deviation data, the output images corresponding to the target sensor are corrected and preprocessed along the target direction to obtain the output images corresponding to the two sensors after correction. Then, the output images corresponding to the two sensors after correction are stitched together. By correcting the output images corresponding to the two sensors, the influence of the deformation of the structural members in the high-temperature environment on the stitching effect is corrected, and the image stitching effect corresponding to sensorA and sensorB is improved.

[0057] It should be noted that in the embodiments of the present invention, when correcting and preprocessing the output image corresponding to the target sensor, the output image corresponding to sensorA can be corrected and preprocessed with the output image corresponding to sensorB as the reference, that is, the target sensor is sensorA; or the output image corresponding to sensorB can be corrected and preprocessed with the output image corresponding to sensorA as the reference, that is, the target sensor is sensorB; or a reference parameter can be preset to correct and preprocess the output images corresponding to sensorA and sensorB respectively, that is, the target sensors are sensorA and sensorB. In the embodiments of the present invention, a solution description is given for correcting and preprocessing the output image corresponding to one sensor with the output image corresponding to sensorA or sensorB as the reference.

[0058] Further, the correcting and preprocessing the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data includes:

[0059] Obtain the output images corresponding to the two sensors respectively;

[0060] Based on the output images corresponding to the two sensors respectively, determine the target sensor;

[0061] Based on the angle deviation calibration data and the working state angle deviation data in the target direction, determine the target correction parameter corresponding to the output image of the target sensor;

[0062] Determine the target correction operation corresponding to the target direction, and the target correction operation is used to correct and preprocess the output image of the target sensor;

[0063] Based on the target correction parameter, perform the target correction operation on the output image of the target sensor.

[0064] Specifically, when it is determined that there is a discrepancy between the working state angle deviation data and the angle deviation calibration data in the target direction, the output image A corresponding to sensor A and the output image B corresponding to sensor B can be obtained first. Based on the object detection results of the output image A and the output image B, the target sensor can be determined from sensor A and sensor B. After determining the target sensor, the target correction parameter that needs to be adjusted for pre-correcting the output image of the target sensor in the target direction can be calculated according to the angle deviation calibration data and the working state angle deviation data in the target direction. After determining the target correction parameter, the target correction operation corresponding to the target direction can be performed on the output image of the target sensor to obtain the output image after the first pre-correction process.

[0065] It should be noted that if there is only one target direction, after performing the target correction operation on the output image of the target sensor, only the output image after the first correction corresponding to the target direction is obtained. If there are multiple target directions, after performing multiple pre-correction processes on the output image of the target sensor in sequence, the finally obtained image is the output image of the target sensor after correction, and the number of pre-correction processes is the same as the number of target directions. The embodiment of the present invention does not limit the execution order of the target correction operations corresponding to each target direction, as long as multiple pre-correction processes are performed on the output image corresponding to the target sensor in sequence. These multiple pre-correction processes are serial operations, that is, the target correction operation for the current target direction is to perform another pre-correction process on the image obtained in the previous target direction. For example, if both the x-axis direction and the y-axis direction are target directions, the output image corresponding to the target sensor can be pre-corrected along the x-axis direction first, and then another pre-correction process can be performed on the pre-corrected output image along the y-axis direction, and the finally obtained image is determined as the output image of the target sensor after correction.

[0066] Furthermore, as Figure 2 shown, determining the target sensor based on the output images respectively corresponding to the two sensors includes:

[0067] Performing object detection on each of the output images to determine the motion region corresponding to each of the output images;

[0068] Based on the pixel proportion of each of the motion regions in the corresponding output image, determining the motion proportion corresponding to each of the output images;

[0069] Determining the sensor corresponding to the minimum motion proportion as the target sensor.

[0070] Specifically, when determining the target sensor, target detection can be performed on the output images corresponding to sensor A and sensor B respectively. According to the target detection results, the moving objects corresponding to the output image A and the output image B can be determined, as well as the moving regions corresponding to each moving object. Through this moving region, the number of local pixels corresponding to the moving object can be determined. According to the pixel proportion of this moving region in the output image, the moving proportion corresponding to the output image A and the output image B can be determined respectively. This moving proportion can be understood as the proportion of the number of local pixels in the moving region to the total number of pixels in the entire output image. A small moving proportion means that the field of view angle of the lens corresponding to the sensor of this moving region is small, and vice versa, it means that the field of view angle of the lens corresponding to the sensor of this moving region is large. Then, compare the moving proportions corresponding to the output image A and the output image B respectively, and determine the sensor A or sensor B with a smaller moving proportion as the target sensor. For example, if the moving proportion corresponding to the output image A is less than the moving proportion corresponding to the output image B, then determine sensor A as the target sensor, and only perform calibration preprocessing on the output image A of this sensor A, while the output image B corresponding to sensor B remains unchanged. The above operations can reduce the loss of the field of view angle of the region corresponding to the moving object during calibration preprocessing.

[0071] Further, determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0072] When the target direction is the first preset direction, based on the first difference between the angle deviation calibration data and the working state angle deviation data in the first preset direction, determine the target rotation angle corresponding to the output image of the target sensor;

[0073] Determine the target rotation angle as the target correction parameter corresponding to the output image of the target sensor.

[0074] Specifically, when the target direction is the first preset reverse direction, the first preset direction is the x-axis direction, that is, the first working state angle deviation data a' in the x-axis direction 0 is inconsistent with the first angle deviation calibration data a 0 , according to the first angle deviation calibration data a in the x-axis direction 0 and the first difference between the first working state angle deviation data a' 0 , determine the target rotation angle θ corresponding to the output image of this target sensor, that is, θ = a 0 - a' 0, the target rotation angle is the target correction parameter, that is, when performing the target correction operation on the output image of the target sensor, the output image needs to be rotated by the target rotation angle. In addition, when the target direction is the x-axis direction, it can be determined that the target correction operation is to rotate the output image of the target sensor around the center of the output image by the target rotation angle, and the rotation direction can be determined according to the sign bit of the target rotation angle. For example, when the sign bit is positive, it rotates in the clockwise direction, and when the sign bit is negative, it rotates in the counterclockwise direction, or when the sign bit is negative, it rotates in the clockwise direction, and when the sign bit is positive, it rotates in the counterclockwise direction. The embodiments of the present invention do not limit this.

[0075] Exemplarily, taking sensorB as the target sensor, the target direction is the x-axis direction, the target rotation angle θ is negative, and when the sign bit is positive, it rotates in the clockwise direction, and when the sign bit is negative, it rotates in the counterclockwise direction as an example, the output image A corresponding to sensorA remains unchanged, and the right shadow area of the output image A is the overlapping area of the fields of view of the lens corresponding to sensorA and the lens corresponding to sensorB. Since the target rotation angle θ is negative, the output image B corresponding to sensorB is rotated counterclockwise around the center of the output image B, and the rotation degree is the absolute value of the target rotation angle θ. Figure 3 is one of the schematic diagrams of the correction preprocessing provided by the embodiments of the present invention. The corrected output image obtained after the output image B performs the above target correction operation is as Figure 3 shown by the dashed box on the right. The left shadow area in the output image B is the overlapping area of the fields of view of the lens corresponding to sensorB and the lens corresponding to sensorA after the correction preprocessing.

[0076] Further, determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0077] In the case where the target direction is the second preset direction, determining a second difference between the angle deviation calibration data and the working state angle deviation data in the second preset direction;

[0078] Based on the product of the first preset threshold and the tangent value corresponding to the second difference, determining a first offset pixel number of the output image of the target sensor in the second preset direction; the first preset threshold is used to characterize the mapping relationship between the change of the field of view angle and the pixel point offset number in the second preset direction;

[0079] Determining the first offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

[0080] Specifically, when the target direction is the second preset direction, the second preset direction is the y-axis direction, that is, the second working state angle deviation data b' in the y-axis direction 0 is inconsistent with the second angle deviation calibration data b 0 , the second angle deviation calibration data b in the y-axis direction can be determined 0 and the second working state angle deviation data b' 0 . Then, calculate the second difference between them and calculate the tangent value of the second difference. The tangent value of the second difference is: tan(b 0 - b' 0 ). After that, obtain the first preset threshold M that is pre-determined to represent the mapping relationship between the change in the field of view angle and the number of pixel offsets in the y-axis direction. Calculate the first offset pixel number p in the y-axis direction through the product of the tangent value of the second difference and the first preset threshold, that is, p = M × tan(b 0 - b' 0 ). The first offset pixel number p is the target correction parameter. That is, when performing the target correction operation on the output image of the target sensor, the output image of the target sensor needs to be offset by the first offset pixel number. In addition, when the target direction is the y-axis direction, it can be determined that the target correction operation is to offset the output image of the target sensor by the first offset pixel number along the y-axis direction. The offset direction can be determined according to the sign bit of the first offset pixel number. For example, when the sign bit is positive, it is offset along the positive half-axis of the y-axis, and when the sign bit is negative, it is offset along the negative half-axis of the y-axis. Or, when the sign bit is positive, it is offset along the negative half-axis of the y-axis, and when the sign bit is negative, it is offset along the positive half-axis of the y-axis. The embodiments of the present invention do not limit this

[0081] For example, taking sensorB as the target sensor, the target direction is the y-axis direction, the first offset pixel number is negative, and when the sign bit is positive, it is offset along the positive half-axis of the y-axis, and when the sign bit is negative, it is offset along the negative half-axis of the y-axis. The output image A corresponding to sensorA remains unchanged. Since the first offset pixel number is negative, the output image B corresponding to sensorB is offset along the negative half-axis of the y-axis, and the offset pixel number is the absolute value of the first offset pixel number Figure 4 is the second schematic diagram of the calibration preprocessing provided by the embodiments of the present invention. The corrected image obtained after the output image B performs the above target correction operation is as shown in Figure 4 the right dashed box

[0082] Further, the determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes:

[0083] When the target direction is the third preset direction, determine a third difference between the angle deviation calibration data and the working state angle deviation data in the third preset direction;

[0084] Based on the product of the second preset threshold and the tangent value corresponding to the third difference, determine a second offset pixel number of the output image of the target sensor in the third preset direction; the second preset threshold is used to represent the mapping relationship between the change in the field of view angle and the number of pixel point offsets in the third preset direction;

[0085] Determine the second offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

[0086] Specifically, when the target direction is the third preset direction, the third preset direction is the z-axis direction, that is, the third working state angle deviation data c' in the z-axis direction 0 is inconsistent with the third angle deviation calibration data c 0 , the third difference between the third angle deviation calibration data c in the z-axis direction 0 and the third working state angle deviation data c' 0 can be determined, and the tangent value of the third difference is calculated. The tangent value of the third difference is: tan(c 0 - c' 0 ). After that, obtain the second preset threshold N determined in advance for representing the mapping relationship between the change in the field of view angle and the number of pixel point offsets in the z-axis direction. Through the product of the tangent value of the third difference and the second preset threshold, calculate the second offset pixel number q in the z-axis direction, that is, q = N × tan(c 0 - c' 0 ). The second offset pixel number q is the target correction parameter. That is, when performing the target correction operation on the output image of the target sensor, the output image of the target sensor needs to be offset by the second offset pixel number. In addition, when the target direction is the z-axis direction, it can be determined that the target correction operation is to offset the output image of the target sensor along the z-axis direction by the second offset pixel number, and the offset direction can be determined according to the sign bit of the second offset pixel number. For example, when the sign bit is positive, it is offset along the positive half-axis of the z-axis, and when the sign bit is negative, it is offset along the negative half-axis of the z-axis, or when the sign bit is positive, it is offset along the negative half-axis of the z-axis, and when the sign bit is negative, it is offset along the positive half-axis of the z-axis. The embodiments of the present invention do not limit this.

[0087] Exemplarily, taking sensorB as the target sensor, the target direction being the z-axis direction, the number of second offset pixels being negative, and when the sign bit is positive, it is offset along the positive half-axis of the z-axis, and when the sign bit is negative, it is offset along the negative half-axis of the z-axis as an example, the output image A corresponding to sensorA remains unchanged. Since the number of second offset pixels is negative, the output image B corresponding to sensorB is offset along the negative half-axis of the z-axis, and the number of offset pixels is the absolute value of the number of second offset pixels. Figure 5 It is the third schematic diagram of the calibration preprocessing provided by the embodiment of the present invention. The calibrated image obtained after performing the above target calibration operation on the output image B is as Figure 5 shown by the dashed box on the right.

[0088] Step 130: Stitch the output images of the two sensors after calibration.

[0089] Specifically, after obtaining the output images corresponding to the two sensors after calibration, the output images of the two sensors after calibration can be stitched to obtain a stitched image corresponding to the complete field of view angle.

[0090] Furthermore, as Figure 2 shown, the stitching of the output images of the two sensors after calibration includes:

[0091] In the output image of the target sensor after calibration, determine the effective pixel region at the same horizontal height as the output image corresponding to other sensors;

[0092] Based on the horizontal edges of the effective pixel region, respectively crop the output image of the target sensor after calibration and the output image corresponding to the other sensors to obtain a first image to be stitched corresponding to the output image of the target sensor after calibration after cropping, and a second image to be stitched corresponding to the output image corresponding to the other sensors;

[0093] Stitch the first image to be stitched and the second image to be stitched.

[0094] Specifically, when stitching the output image of the target sensor after calibration and the output image corresponding to other sensors, the effective pixel region at the same horizontal height as the output image corresponding to the other sensors can be determined from the output image of the target sensor after calibration. For example, Figure 6 It is the schematic diagram of cropping provided by the embodiment of the present invention. Figure 6 In, the image in the dashed box on the right is the output image obtained after performing at least one calibration preprocessing on the output image B corresponding to sensorB. The output image B after calibration and Figure 6Compared with the output image A corresponding to sensorA on the left side, the upper and lower edges of the corrected output image B are both inclined, that is, the horizontal height on the left side is higher than that on the right side in both the upper and lower edges. The horizontal height of the upper edge of the corrected output image B is higher than that of the upper edge of the output image A, and the horizontal height of the lower edge of the corrected output image B is higher than that of the lower edge of the output image A. Therefore, to ensure the splicing effect of the corrected output image B and the output image A, the upper edge of the output image A constitutes the upper edge of the effective pixel region, and the horizontal line where the left horizontal height of the lower edge of the corrected output image B is located is the lower edge of the effective region. The effective pixel region is determined from the corrected output image B. After determining the effective pixel region, the corrected output image B and the output image A are cropped, that is, the corrected output image B is cropped according to the upper and lower edges of the effective pixel region to obtain the first image to be spliced, and the output image A is horizontally cropped according to the lower edge of the effective pixel region to obtain the second image to be spliced. The second image to be spliced is as shown in Figure 6 the left region A. After horizontal cropping, to ensure that the spliced image corresponding to the final output full field of view angle is rectangular, it is also necessary to crop along the vertical direction of the right endpoint of the lower edge of the first image to be spliced. Since the left shadow region of the first image to be spliced after cropping is the overlapping region of the field of view of the lens corresponding to sensorA and the lens corresponding to sensorB, only one crop of the first image to be spliced after cropping is required in the vertical direction to obtain the third image to be spliced. The third image to be spliced is as shown in Figure 6 the right region B. After cropping, the third image to be spliced and the second image to be spliced can be spliced to obtain the spliced image corresponding to the final output full field of view angle.

[0095] In addition, after splicing the third image to be spliced and the second image to be spliced, due to pixel loss caused by cropping, interpolation processing can be performed on the spliced image to ensure that the image resolution of the processed image is consistent with the initial output resolution of the two sensors when they leave the factory, improving the image effect.

[0096] Optionally, the interpolation algorithms for the above interpolation processing of the spliced image may include: bilinear interpolation algorithm, bicubic interpolation algorithm, nearest neighbor interpolation algorithm, cubic interpolation algorithm, and Lanczos interpolation, etc. The embodiments of the present invention do not limit this.

[0097] The binocular camera image stitching method provided by the embodiment of the present invention obtains the working angle data of two sensors in the binocular camera in the working state, determines the working state angle deviation data corresponding to the two sensors in different directions, and pre-corrects the output image corresponding to the target sensor along the target direction according to the comparison result of the angle deviation calibration data and the working state angle deviation data in different directions, so as to reduce the influence of the deformation of the structural member in the high-temperature environment on the output image corresponding to the target sensor, and further improve the image stitching effect corresponding to the two sensors.

[0098] The binocular camera image stitching device provided by the present invention will be described below. The binocular camera image stitching device described below can be correspondingly referred to the binocular camera image stitching method described above.

[0099] The embodiment of the present invention also provides a binocular camera image stitching device. Figure 7 is a schematic structural diagram of the binocular camera image stitching device provided by the embodiment of the present invention. As Figure 7 shown, the binocular camera image stitching device 700 includes: a determination module 710, a pre-correction processing module 720, and a stitching module 730, where:

[0100] The determination module 710 is configured to determine the working state angle deviation data corresponding to the two sensors in different directions based on the working angle data corresponding to the two sensors in the binocular camera;

[0101] The pre-correction processing module 720 is configured to, when the working state angle deviation data in the target direction is inconsistent with the angle deviation calibration data, pre-correct the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data, and the target sensor belongs to the two sensors;

[0102] The stitching module 730 is configured to stitch the output images of the two sensors after correction.

[0103] The binocular camera image stitching device provided by the embodiment of the present invention obtains the working angle data of two sensors in the binocular camera in the working state, determines the working state angle deviation data corresponding to the two sensors in different directions, and pre-corrects the output image corresponding to the target sensor along the target direction according to the comparison result of the angle deviation calibration data and the working state angle deviation data in different directions, so as to reduce the influence of the deformation of the structural member in the high-temperature environment on the output image corresponding to the target sensor, and further improve the image stitching effect corresponding to the two sensors.

[0104] Optionally, the pre-correction processing module 720 is specifically configured to:

[0105] Obtain the output images corresponding to the two sensors respectively;

[0106] Based on the output images corresponding to the two sensors respectively, determine the target sensor;

[0107] Based on the angle deviation calibration data and the working state angle deviation data in the target direction, determine the target correction parameter corresponding to the output image of the target sensor;

[0108] Determine the target correction operation corresponding to the target direction, where the target correction operation is used to perform pre-correction processing on the output image of the target sensor;

[0109] Based on the target correction parameter, perform the target correction operation on the output image of the target sensor.

[0110] Optionally, the correction preprocessing module 720 is specifically configured to:

[0111] In the case where the target direction is the first preset direction, based on the first difference between the angle deviation calibration data and the working state angle deviation data in the first preset direction, determine the target rotation angle corresponding to the output image of the target sensor;

[0112] Determine the target rotation angle as the target correction parameter corresponding to the output image of the target sensor.

[0113] Optionally, the correction preprocessing module 720 is specifically configured to:

[0114] In the case where the target direction is the second preset direction, determine the second difference between the angle deviation calibration data and the working state angle deviation data in the second preset direction;

[0115] Based on the product of the first preset threshold and the tangent value corresponding to the second difference, determine the first offset pixel number of the output image of the target sensor in the second preset direction; the first preset threshold is used to characterize the mapping relationship between the change of the field of view angle and the pixel point offset number in the second preset direction;

[0116] Determine the first offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

[0117] Optionally, the correction preprocessing module 720 is specifically configured to:

[0118] In the case where the target direction is the third preset direction, determine the third difference between the angle deviation calibration data and the working state angle deviation data in the third preset direction;

[0119] Determine the number of second offset pixels of the output image of the target sensor in the third preset direction based on the product of the second preset threshold and the tangent value corresponding to the third difference; the second preset threshold is used to characterize the mapping relationship between the change of the field of view angle in the third preset direction and the number of pixel point offsets;

[0120] Determine the number of second offset pixels as the target correction parameter corresponding to the output image of the target sensor.

[0121] Optionally, the calibration preprocessing module 720 is specifically configured to:

[0122] Perform target detection on each of the output images to determine the motion area corresponding to each of the output images;

[0123] Determine the motion ratio corresponding to each of the output images based on the pixel ratio of each of the motion areas in the corresponding output image;

[0124] Determine the sensor corresponding to the minimum motion ratio as the target sensor.

[0125] Optionally, the splicing module 730 is specifically configured to:

[0126] In the output image of the target sensor after calibration, determine the effective pixel area at the same horizontal height as the output images corresponding to other sensors;

[0127] Based on the horizontal edges of the effective pixel area, respectively crop the output image of the target sensor after calibration and the output images corresponding to other sensors to obtain a first image to be spliced corresponding to the output image of the target sensor after calibration after cropping, and a second image to be spliced corresponding to the output images corresponding to other sensors;

[0128] Splice the first image to be spliced and the second image to be spliced.

[0129] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 8 shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the binocular camera image splicing method, and the method includes:

[0130] Based on the working angle data corresponding to each of the two sensors in the binocular camera, determine the working state angle deviation data corresponding to the two sensors in different directions;

[0131] In the case where the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, based on the angle deviation calibration data and the working state angle deviation data, perform pre-correction processing on the output image corresponding to the target sensor along the target direction, where the target sensor belongs to the two sensors;

[0132] Stitch the output images of the two sensors after correction.

[0133] In addition, when the logic instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0134] On the other hand, the present invention also provides a computer program product, where the computer program product includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the binocular camera image stitching method provided by the above-mentioned various methods. The method includes:

[0135] Based on the working angle data corresponding to each of the two sensors in the binocular camera, determine the working state angle deviation data corresponding to the two sensors in different directions;

[0136] In the case where the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, based on the angle deviation calibration data and the working state angle deviation data, perform pre-correction processing on the output image corresponding to the target sensor along the target direction, where the target sensor belongs to the two sensors;

[0137] Stitch the output images of the two sensors after correction.

[0138] In another aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a binocular camera image stitching method provided by the above-mentioned various methods. The method includes:

[0139] Based on the working angle data corresponding to the two sensors in the binocular camera, determine the working state angle deviation data corresponding to the two sensors in different directions;

[0140] In the case where the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, based on the angle deviation calibration data and the working state angle deviation data, perform pre-correction processing on the output image corresponding to the target sensor along the target direction, and the target sensor belongs to the two sensors;

[0141] Stitch the output images of the two sensors after correction.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stitching binocular camera images, characterized in that, it includes: Based on the working angle data corresponding to the two sensors in the binocular camera, determine the working state angle deviation data corresponding to the two sensors in different directions; In the case where the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, based on the angle deviation calibration data and the working state angle deviation data, perform correction preprocessing on the output image corresponding to the target sensor along the target direction, and the target sensor belongs to the two sensors; Stitch the output images of the two sensors after correction.

2. The method for stitching binocular camera images according to claim 1, characterized in that, The performing correction preprocessing on the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data includes: Obtain the output images corresponding to the two sensors respectively; Based on the output images corresponding to the two sensors respectively, determine the target sensor; Based on the angle deviation calibration data and the working state angle deviation data in the target direction, determine the target correction parameter corresponding to the output image of the target sensor; Determine the target correction operation corresponding to the target direction, and the target correction operation is used to perform correction preprocessing on the output image of the target sensor; Based on the target correction parameter, perform the target correction operation on the output image of the target sensor.

3. The method for stitching binocular camera images according to claim 2, characterized in that, The determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes: In the case where the target direction is the first preset direction, based on the first difference between the angle deviation calibration data and the working state angle deviation data in the first preset direction, determine the target rotation angle corresponding to the output image of the target sensor; Determine the target rotation angle as the target correction parameter corresponding to the output image of the target sensor.

4. The method for stitching binocular camera images according to claim 2, characterized in that, The determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes: In the case where the target direction is the second preset direction, determine the second difference between the angle deviation calibration data and the working state angle deviation data in the second preset direction; Based on the product of the first preset threshold and the tangent value of the second difference, determine the first offset pixel number of the output image of the target sensor in the second preset direction; the first preset threshold is used to characterize the mapping relationship between the change of the field of view angle and the number of pixel point offsets in the second preset direction; Determine the first offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

5. The binocular camera image stitching method according to claim 2, characterized in that, determining the target correction parameter corresponding to the output image of the target sensor based on the angle deviation calibration data and the working state angle deviation data in the target direction includes: when the target direction is the third preset direction, determining a third difference between the angle deviation calibration data and the working state angle deviation data in the third preset direction; determining a second offset pixel number of the output image of the target sensor in the third preset direction based on the product of a second preset threshold and the tangent value corresponding to the third difference; the second preset threshold is used to represent the mapping relationship between the change of the field of view angle and the number of pixel point offsets in the third preset direction; determining the second offset pixel number as the target correction parameter corresponding to the output image of the target sensor.

6. The binocular camera image stitching method according to any one of claims 2-5, characterized in that, determining the target sensor based on the output images respectively corresponding to the two sensors includes: performing target detection on each of the output images to determine a motion area corresponding to each of the output images; determining a motion ratio corresponding to each of the output images based on the pixel ratio of each of the motion areas in the corresponding output image; determining the sensor corresponding to the minimum motion ratio as the target sensor.

7. The binocular camera image stitching method according to any one of claims 1-5, characterized in that, stitching the output images of the two sensors after correction includes: in the output image of the corrected target sensor, determining a valid pixel area at the same horizontal height as the output image corresponding to the other sensor; respectively cropping the output image of the corrected target sensor and the output image corresponding to the other sensor based on the horizontal edge of the valid pixel area to obtain a first image to be stitched corresponding to the output image of the corrected target sensor after cropping, and a second image to be stitched corresponding to the output image corresponding to the other sensor; stitching the first image to be stitched and the second image to be stitched.

8. A binocular camera image stitching device, characterized in that, comprising: a determining module, configured to determine working state angle deviation data corresponding to two sensors in different directions based on the working angle data respectively corresponding to the two sensors in the binocular camera; a correction preprocessing module, configured to perform correction preprocessing on the output image corresponding to the target sensor along the target direction based on the angle deviation calibration data and the working state angle deviation data when the working state angle deviation data and the angle deviation calibration data in the target direction are inconsistent, and the target sensor belongs to the two sensors; a stitching module, configured to stitch the output images of the two sensors after correction.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the binocular camera image stitching method according to any one of claims 1-7.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the binocular camera image stitching method according to any one of claims 1-7.

Images