Image splicing method, electronic equipment and storage medium
The camera external parameters are corrected through the measurement results of the inertial measurement unit, and the image splicing problem caused by camera movement or dust is solved, achieving accurate splicing effect in unclear or no overlap.
Patent Information
- Application Number
- CN202310487446.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-05
AI Technical Summary
Existing image stitching methods In the case of camera movement or severe dust, the images collected by multiple cameras may have no overlapping areas or are not clear enough, resulting in the inability to determine the correct stitching position, resulting in poor stitching effect.
Through the measurement results of each camera when collecting image frames, the historical camera external parameters of each camera are corrected to obtain the latest camera external parameters of each camera, avoiding the dependence on image frames for calibration, and ensuring that Determine the correct stitching position when stitching images.
Regardless of whether the image frames collected by the camera are clear or whether there are overlapping areas for adjacent camera image frames, the latest external parameters of the camera can be accurately determined and the stitching effect of image stitching can be improved.
Smart Images

Figure CN118898539A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image stitching method, an electronic device and a storage medium. Background Art
[0002] Image stitching technology refers to the technology of stitching two or more images with overlapping areas into a panoramic image containing the information of each image through image fusion technology. In the process of coal mining, multiple cameras can be deployed underground in the coal mine, and the images of the underground fully mechanized mining face of the coal mine collected by different cameras can be stitched into a panoramic image of the fully mechanized mining face through image stitching technology, so that workers can cut coal in the coal mine by watching the panoramic image of the fully mechanized mining face, avoiding the risk of cutting coal in the coal mine.
[0003] Among them, multiple cameras can be installed on multiple hydraulic supports of the fully mechanized mining face. During the coal cutting process of the fully mechanized mining face, the hydraulic support plays the role of supporting the coal seam, and the coal cutter of the fully mechanized mining face can work within the support range of the hydraulic support. As the coal cutting work proceeds, the hydraulic support and the coal cutter will be pushed forward. The process of the hydraulic support pushing forward is called pulling the support, and the process of the coal cutter pushing forward is called pushing. When the hydraulic support pulls the support and the coal cutter does not push forward, since the camera will move toward the direction of the coal cutter together with the hydraulic support, the images of the coal cutter collected synchronously by multiple cameras will have no overlapping areas. In addition, during the coal cutting work, a large amount of dust will be generated, which will also cause the images of the coal cutter collected synchronously by multiple cameras to be not clear enough.
[0004] It can be seen that with the current image stitching method, when the camera moves or there is severe dust, the multiple images of the coal cutter acquired synchronously by multiple cameras may not have overlapping areas or may not be clear enough, and the correct stitching position cannot be determined during image stitching, resulting in a poor stitching effect of stitching multiple images. Summary of the invention
[0005] The embodiments of the present application provide an image stitching method, an electronic device, and a storage medium, so as to improve the stitching effect of stitching multiple images.
[0006] In the first aspect, an embodiment of the present application provides an image stitching method, which can be performed by an electronic device connected to multiple cameras. Wherein, the multiple cameras are deployed in a target scene, and the multiple cameras include a first camera and a second camera, and the first camera and the second camera are deployed adjacent to each other in the target scene. The method includes: the electronic device can obtain multiple image frames captured by the multiple cameras; the multiple image frames are obtained by the multiple cameras synchronously capturing the target scene; the electronic device can correct the first external parameter of the first camera according to the first inertial measurement unit IMU measurement result of the first camera when capturing the first image frame, and obtain the second external parameter of the first camera; the camera posture information corresponding to the second external parameter is the camera posture information corresponding to the first IMU measurement result; the electronic device can perform image stitching on the first image frame and the second image frame captured by the second camera according to the second external parameter.
[0007] In the above scheme, the multiple image frames synchronously captured by multiple cameras in the related art may not be clear enough or have no overlapping areas, and the correct stitching position cannot be determined when the image is stitched, resulting in a poor stitching effect of the image stitching. The electronic device can correct the historical camera extrinsic parameters of each camera (such as the first extrinsic parameter of the first camera) through the IMU measurement results of each camera when the image frame is captured, and obtain the latest camera extrinsic parameters of each camera (such as the second extrinsic parameter of the first camera). Since it is not necessary to rely on the image frames captured by each camera to calibrate the camera extrinsic parameters of each camera, regardless of whether the image frames captured by each camera are clear or whether the image frames captured by two adjacent cameras have overlapping areas, the latest camera extrinsic parameters of each camera can be accurately determined, which helps to determine the correct stitching position when performing image stitching and improve the stitching effect of image stitching.
[0008] In a possible implementation, before the electronic device corrects the first extrinsic parameter of the first camera, the electronic device may also, when there is a third image frame, determine that the first extrinsic parameter is the extrinsic parameter of the first camera when the third image frame is acquired; the third image frame is an image frame acquired by the first camera before acquiring the first image frame; the electronic device may also, when there is no third image frame, determine that the first extrinsic parameter is a preset extrinsic parameter of the first camera, or obtain the first extrinsic parameter by calibrating the first camera according to the first image frame.
[0009] In the above implementation, the electronic device can determine the historical camera extrinsic parameters of the camera by determining whether the image frame currently captured by the camera is an initial image frame. If the image frame currently captured by the camera is an initial image frame, the preset camera extrinsic parameters of the camera are used as the historical camera extrinsic parameters of the camera, or, when the image frame currently captured by the camera can be used to calibrate the camera extrinsic parameters, the camera is calibrated by the image frame currently captured by the camera to obtain the historical camera extrinsic parameters of the camera; if the image frame currently captured by the camera is not an initial image frame, the latest camera extrinsic parameters of the image frame previously captured by the camera are used as the historical camera extrinsic parameters of the currently captured image frame, thereby ensuring the accuracy of the historical camera extrinsic parameters of the camera.
[0010] In a possible implementation, before the electronic device corrects the first external parameter of the first camera, the electronic device may also determine, based on the first IMU measurement result, that the moving distance of the first camera is greater than a first threshold.
[0011] In the above implementation, the electronic device can determine the movement distance of the camera through the IMU measurement result of the camera, and determine whether the historical camera extrinsic parameters of the camera have changed through the movement distance of the camera. When the historical camera extrinsic parameters of the camera change, the historical camera extrinsic parameters of the camera are corrected through the IMU measurement result of the camera to obtain the latest camera extrinsic parameters of the camera, thereby ensuring the accuracy of the latest camera extrinsic parameters of the camera.
[0012] In a possible implementation manner, the electronic device may further perform image stitching on the first image frame and the second image frame according to the first external parameter when determining that the moving distance of the first camera is less than or equal to the first threshold.
[0013] In the above implementation, when the historical camera extrinsic parameters of the camera have not changed, the electronic device can determine that the historical camera extrinsic parameters of the camera are the latest camera extrinsic parameters of the camera, and perform image stitching through the historical camera extrinsic parameters of the camera, thereby ensuring the accuracy of the latest camera extrinsic parameters of the camera.
[0014] In a possible implementation, when the electronic device determines the moving distance of the first camera based on the first IMU measurement result, the electronic device may determine the moving distance based on the acceleration in a first direction in the first IMU measurement result; the first direction is a preset moving direction of the first camera.
[0015] In the above implementation, the electronic device can determine the movement distance of the camera through the acceleration of the camera's movement direction preset in the camera's IMU measurement result, thereby ensuring the accuracy of the camera's movement distance determined by the electronic device.
[0016] In a possible implementation, the electronic device corrects the first external parameter according to the first IMU measurement result, and when obtaining the second external parameter, the electronic device may adjust the rotation matrix and the translation matrix in the first external parameter according to the angular velocity and acceleration in the first IMU measurement result to obtain the second external parameter; or, the electronic device may adjust the angular velocity, acceleration and IMU intrinsic parameters in the first IMU measurement result according to the two-dimensional coordinates and three-dimensional coordinates of each feature point in the first feature point set of the first image frame to obtain the second IMU measurement result, and adjust the rotation matrix and the translation matrix in the first external parameter according to the angular velocity and acceleration in the second IMU measurement result to obtain the second external parameter; the coordinate projection error corresponding to the angular velocity, acceleration and IMU intrinsic parameter in the second IMU measurement result is less than a second threshold, and the coordinate projection error is the error between the two-dimensional coordinate of each feature point and the two-dimensional coordinate obtained by projecting the three-dimensional coordinate of each feature point.
[0017] In the above implementation, the electronic device can directly correct the historical camera extrinsic parameters of the camera through the IMU measurement results of the camera to obtain the latest camera extrinsic parameters of the camera, or, when the image frame currently acquired by the camera can be used to calibrate the camera extrinsic parameters, the IMU measurement result of the camera is first corrected through the feature point set of the image frame currently acquired by the camera to eliminate the error of the IMU measurement result, and then the historical camera extrinsic parameters of the camera are corrected through the corrected IMU measurement results of the camera to obtain the latest camera extrinsic parameters of the camera, thereby ensuring the accuracy of the latest camera extrinsic parameters of the camera.
[0018] In a possible implementation, calibration objects and / or calibration cards may also be deployed in the target scene, and the first feature point set in the first image frame may include feature points used to describe the calibration objects and / or calibration cards; or, the matching rate between the first feature point set and the second feature point set may be greater than a third threshold, and the second feature point set may be the feature point set of the second image frame.
[0019] In the above implementation, when calibration objects and / or calibration cards are deployed in the target scene, the feature point set of the image frame currently captured by the camera may include feature points used to describe the calibration objects and / or calibration cards, or the matching rate of the feature point sets of the image frames currently captured by two adjacent cameras is high, thereby ensuring that the image frame currently captured by the camera can be used to calibrate the camera extrinsic parameters.
[0020] In a possible implementation, the second external parameter may satisfy the following formula:
[0021]
[0022] Wherein, R(t+Δt) is the rotation matrix in the second external parameter, T(t+Δt) is the translation matrix in the second external parameter, R(t) is the rotation matrix in the first external parameter, T(t) is the translation matrix in the first external parameter, Ω x is the angular velocity with the x-axis as the rotation axis in the first IMU measurement result, Ω y is the angular velocity with the y coordinate axis as the rotation axis in the first IMU measurement result, Ω z is the angular velocity with the z coordinate axis as the rotation axis in the first IMU measurement result, R(t) T is the transpose of R(t), a 0 is the acceleration in the first IMU measurement result, and g is the gravitational acceleration.
[0023] In the above implementation, the electronic device can correct the historical camera extrinsic parameters of the camera through the IMU measurement results of the camera to obtain the latest camera extrinsic parameters of the camera, thereby ensuring the accuracy of the latest camera extrinsic parameters of the camera.
[0024] In a possible implementation, the coordinate projection error may satisfy the following formula:
[0025]
[0026] Among them, c i is the error between the two-dimensional coordinates of the ith feature point in the first feature point set and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the ith feature point in the first feature point set, x i is the two-dimensional coordinate of the i-th feature point in the first feature point set, X i is the three-dimensional coordinate of the i-th feature point in the first feature point set, Ω 0 is the angular velocity in the first IMU measurement result, a 0 is the acceleration in the first IMU measurement result, Ω is the angular velocity in the second IMU measurement result, a is the acceleration in the second IMU measurement result, Proj() is used to project the three-dimensional coordinates into two-dimensional coordinates, and p is the IMU internal parameter.
[0027] In the above implementation, the electronic device can correct the IMU measurement result of the camera through the feature point set of the image frame currently captured by the camera, thereby eliminating the error of the IMU measurement result and ensuring the accuracy of the latest camera extrinsic parameters of the camera.
[0028] In a second aspect, an embodiment of the present application further provides an image stitching device, which has the function of implementing the method described in the first aspect or any possible design of the first aspect, and the function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions, such as an image acquisition unit, an external parameter determination unit, and an image stitching unit.
[0029] An image acquisition unit, configured to acquire a plurality of image frames acquired by a plurality of cameras; the plurality of cameras are deployed in a target scene, the plurality of image frames are acquired by the plurality of cameras synchronously acquiring the target scene, the plurality of cameras include a first camera and a second camera, the first camera and the second camera are adjacently deployed in the target scene;
[0030] an external parameter determination unit, configured to correct a first external parameter of the first camera according to a first inertial measurement unit (IMU) measurement result of the first camera when acquiring a first image frame, so as to obtain a second external parameter of the first camera; the camera attitude information corresponding to the second external parameter is the camera attitude information corresponding to the first IMU measurement result;
[0031] An image stitching unit is used to stitch the first image frame and the second image frame captured by the second camera according to the second external parameter.
[0032] In a possible implementation, the extrinsic parameter determination unit is further used to: when there is a third image frame, determine that the first extrinsic parameter is an extrinsic parameter of the first camera when the third image frame is acquired; the third image frame is an image frame acquired by the first camera before acquiring the first image frame; when there is no third image frame, determine that the first extrinsic parameter is a preset extrinsic parameter of the first camera, or calibrate the first camera according to the first image frame to obtain the first extrinsic parameter.
[0033] In a possible implementation manner, the external parameter determination unit is further used to: determine, according to the first IMU measurement result, that the movement distance of the first camera is greater than a first threshold.
[0034] In a possible implementation manner, the image stitching unit is further configured to: when it is determined that the moving distance is less than or equal to the first threshold, perform image stitching on the first image frame and the second image frame according to the first external parameter.
[0035] In a possible implementation, the external parameter determination unit is specifically used to determine the moving distance according to the acceleration in a first direction in the first IMU measurement result; the first direction is a preset moving direction of the first camera.
[0036] In one possible implementation, the external parameter determination unit is specifically used to: adjust the rotation matrix and translation matrix in the first external parameter according to the angular velocity and acceleration in the first IMU measurement result to obtain the second external parameter; or, adjust the angular velocity, acceleration and IMU intrinsic parameters in the first IMU measurement result according to the two-dimensional coordinates and three-dimensional coordinates of each feature point in the first feature point set of the first image frame to obtain the second IMU measurement result, and adjust the rotation matrix and translation matrix in the first external parameter according to the angular velocity and acceleration in the second IMU measurement result to obtain the second external parameter; the coordinate projection error corresponding to the angular velocity, acceleration and IMU intrinsic parameter in the second IMU measurement result is less than a second threshold, and the coordinate projection error is the error between the two-dimensional coordinate of each feature point and the two-dimensional coordinate obtained by projecting the three-dimensional coordinate of each feature point.
[0037] In a possible implementation manner, a calibration object and / or a calibration card is further deployed in the target scene, and the first feature point set in the first image frame includes feature points for describing the calibration object and / or the calibration card.
[0038] In a possible implementation, the second external parameter may satisfy the following formula:
[0039]
[0040] Wherein, R(t+Δt) is the rotation matrix in the second external parameter, T(t+Δt) is the translation matrix in the second external parameter, R(t) is the rotation matrix in the first external parameter, T(t) is the translation matrix in the first external parameter, Ω x is the angular velocity with the x-axis as the rotation axis in the first IMU measurement result, Ω y is the angular velocity with the y coordinate axis as the rotation axis in the first IMU measurement result, Ω z is the angular velocity with the z coordinate axis as the rotation axis in the first IMU measurement result, R(t) T is the transpose of R(t), a 0 is the acceleration in the first IMU measurement result, and g is the gravitational acceleration.
[0041] In a possible implementation, the coordinate projection error may satisfy the following formula:
[0042]
[0043] Among them, c i is the error between the two-dimensional coordinates of the ith feature point in the first feature point set and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the ith feature point in the first feature point set, x iis the two-dimensional coordinate of the i-th feature point in the first feature point set, X i is the three-dimensional coordinate of the i-th feature point in the first feature point set, Ω 0 is the angular velocity in the first IMU measurement result, a 0 is the acceleration in the first IMU measurement result, Ω is the angular velocity in the second IMU measurement result, a is the acceleration in the second IMU measurement result, Proj() is used to project the three-dimensional coordinates into two-dimensional coordinates, and p is the IMU internal parameter.
[0044] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements the above-mentioned first aspect or any possible design method of the above-mentioned first aspect.
[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method of the first aspect or any possible design of the first aspect.
[0046] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising computer executable instructions, wherein the computer executable instructions are used to enable a computer to execute the method of the first aspect or any possible design of the first aspect.
[0047] The beneficial effects of the above-mentioned second to fifth aspects and possible designs thereof can refer to the description of the beneficial effects of the method described in the above-mentioned first aspect and any possible design thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A schematic diagram of an underground coal mine scene provided in an embodiment of the present application;
[0049] Figure 2 A schematic diagram of connecting a camera and an electronic device provided in an embodiment of the present application;
[0050] Figure 3 A schematic diagram of a flow chart of an image stitching method provided in an embodiment of the present application;
[0051] Figure 4 A schematic diagram of determining a moving distance of a first camera provided in an embodiment of the present application;
[0052] Figure 5 A schematic diagram of a flow chart of another image stitching method provided in an embodiment of the present application;
[0053] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;
[0054] Figure 7 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;
[0055] Figure 8 A schematic diagram of a connection method between computing device clusters provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0057] Before introducing the specific solutions provided in the embodiments of the present application, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art, and the terms in the present application are not limited.
[0058] (1) Camera coordinate system: It is the reference coordinate system of the camera. For example, it is a three-dimensional rectangular coordinate system with the optical center of the camera lens as the origin, the horizontal axis and the vertical axis are parallel to the two sides of the camera lens surface, and the vertical axis is perpendicular to the camera lens surface. The units of the horizontal axis and the vertical axis are both centimeters.
[0059] Pixel coordinate system: It is a reference coordinate system for the pixels included in the image captured by the camera. For example, it is a two-dimensional rectangular coordinate system with the pixel point in the upper left corner of the image as the origin, and the horizontal and vertical axes are parallel to the two sides of the image. The units of the horizontal and vertical axes are both the number of pixels.
[0060] (2) Camera extrinsic parameters: a set of parameters used to convert the coordinates of the object photographed by the camera between the world coordinate system and the camera coordinate system. Camera extrinsic parameters can be understood as being only related to the external parameters of the camera, and the camera extrinsic parameters change with the change of the rigid body position. Exemplarily, the camera extrinsic parameters can be:
[0061] M=[RT]
[0062] Among them, R is a 3*3 rotation matrix, including the roll angle, pitch angle, and yaw angle of the camera, which can represent the direction of the coordinate axis of the world coordinate system relative to the coordinate axis of the camera coordinate system. T is a 3*1 translation matrix, which can represent the position of the origin of the world coordinate system in the camera coordinate system.
[0063] (3) Camera intrinsic parameters: a set of parameters used to convert the coordinates of the object photographed by the camera between the camera coordinate system and the pixel coordinate system. The camera intrinsic parameters can be understood as being only related to the internal parameters of the camera, and the camera intrinsic parameters do not change with the change of the rigid body position. For example, the camera intrinsic parameters can be:
[0064]
[0065] Among them, f x and f y Represents the focal length f of the camera on the x-axis and y-axis of the pixel coordinate system, x 0 and 0 It represents the offset of the intersection of the camera's lens optical axis and the lens surface (called the principal point) on the x-axis and y-axis of the pixel coordinate system, and s is the tilt factor of the camera's lens optical axis. For example, a camera with a focal length of f = 35mm, a maximum resolution of 4256×2832, and a sensor size of 36.0×23.9mm, then x 0 =4256 / 2=2128,y 0 =2832 / 2=1416, d x =36.0 / 4256, d y =23.9 / 2832, f x =f / d x =4137.8, f y =f / d y =4147.3.
[0066] In the embodiments of the present application, "multiple" refers to two or more than two. In view of this, in the embodiments of the present application, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, for example, one, two or more. For example, including at least one means including one, two or more, and there is no restriction on which ones are included. For example, including at least one of A, B and C, then A, B, C, A and B, A and C, B and C, or A and B and C may be included. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the previously associated objects are in an "or" relationship.
[0067] Unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of the multiple objects.
[0068] The image stitching method provided in the embodiment of the present application can be applied to a scenario where image stitching is performed in real time when the camera is moving or the image captured by the camera is unclear.
[0069] For example, the image stitching method provided in the embodiment of the present application can be used for stitching panoramic images of fully mechanized mining faces in underground coal mines. Figure 1 , is a schematic diagram of a coal mine underground scene provided by an embodiment of the present application. The fully mechanized mining face in the coal mine underground may include multiple hydraulic supports 10 and coal cutters 20. During the coal cutting process of the fully mechanized mining face, the hydraulic supports 10 play a role in supporting the coal seam, and the coal cutter 20 can work within the support range of the hydraulic supports 10. As the coal cutting work proceeds, the hydraulic supports 10 and the coal cutter 20 will be pushed forward. The process of pushing the hydraulic supports 10 is called pulling the support, and the process of pushing the coal cutter 20 is called pushing the coal cutter 20. Multiple cameras 30 can be installed on multiple hydraulic supports 20 to collect images of the coal cutter 20.
[0070] See also Figure 2 , is a schematic diagram of a camera and an electronic device connected in an embodiment of the present application, and multiple cameras 30 can also be connected to an electronic device 40. Among them, the electronic device 40 can be a terminal device or a server. For example, the electronic device 40 can be a mobile phone, a tablet computer, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a wearable device and other terminal devices. Alternatively, the electronic device 40 can also be a single server or a server cluster composed of multiple servers. The server can specifically be a cloud server, also known as a cloud, a cloud, a cloud server or a cloud controller. A cloud server is a general term for devices or devices with data processing capabilities, such as physical devices such as a host or a processor, virtual devices such as a virtual machine or a container, and chips or integrated circuits. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 40.
[0071] The electronic device 40 may be directly connected to the multiple cameras 30 via a wired or wireless manner, or may be indirectly connected to the multiple cameras 30 via one or more other devices, and the embodiment of the present application does not impose any specific limitation on this.
[0072] The electronic device 40 can stitch the images of the coal cutter 20 captured by the multiple cameras 30 into a panoramic image of the coal cutter 20 through the image stitching technology, so that workers can perform coal cutting operations by watching the panoramic image of the coal cutter 20 in the coal mine, avoiding the risk of performing coal cutting operations underground in the coal mine. For example, the electronic device 40 can send a synchronous acquisition signal to the multiple cameras 30, and the synchronous acquisition signal can be used to instruct the multiple cameras 30 to synchronously acquire images of the coal cutter 20. The multiple cameras 30 can acquire images of the coal cutter 20 according to the synchronous acquisition signal sent by the electronic device 40, compress and encode the acquired images of the coal cutter 20, add timestamps, generate video streams, and transmit the video streams to the electronic device 40. The electronic device 40 receives the video stream sent by any camera 30, decodes the video stream, and obtains the images of each coal cutter 20 acquired by the camera 30, and each image of the coal cutter 20 has a timestamp. The electronic device 40 can determine the multiple images of the coal cutter 20 synchronously captured by the multiple cameras 30 according to the timestamps of the images of each coal cutter 20 , and stitch the multiple synchronously captured images of the coal cutter 20 to obtain a panoramic image of the coal cutter 20 .
[0073] However, when the hydraulic support 10 is pulled and the coal cutter 20 is not pushed, the camera 30 will move toward the coal cutter 20 together with the hydraulic support 10, which will result in the multiple images of the coal cutter 20 synchronously captured by the multiple cameras 30 having no overlapping area. In addition, when the coal cutter 20 is working, a large amount of dust will be generated, which will also result in the multiple images of the coal cutter 20 synchronously captured by the multiple cameras 30 not being clear enough. The multiple images of the coal cutter 20 synchronously captured by the multiple cameras 30 having no overlapping area or not being clear enough will make it impossible to determine the correct splicing position when splicing the images, resulting in a poor splicing effect of the multiple images.
[0074] In order to improve the stitching effect of multiple image stitching, an embodiment of the present application provides an image stitching method for stitching multiple image frames obtained by synchronously capturing a target scene with multiple cameras, and multiple cameras are deployed in the target scene. In an embodiment of the present application, the historical camera extrinsic parameters of each camera can be corrected by the inertial measurement unit (IMU) measurement results of each camera when capturing image frames to obtain the latest camera extrinsic parameters of each camera. Since it is not necessary to rely on the image frames captured by each camera to calibrate the camera extrinsic parameters of each camera, regardless of whether the image frames captured by each camera are clear or whether the image frames captured by two adjacent cameras have overlapping areas, the latest camera extrinsic parameters of each camera can be accurately determined, which helps to determine the correct stitching position when performing image stitching and improve the stitching effect of image stitching.
[0075] The following is a detailed description of the image stitching method provided by the embodiment of the present application in combination with specific application scenarios. Figure 3 , is a flow chart of an image stitching method provided in an embodiment of the present application. The image stitching method can be applied to a scene where real-time image stitching is performed when the camera is moving or the image captured by the camera is unclear. The image stitching method can be Figure 2 As shown or with Figure 2 The present application is not specifically limited to electronic devices with similar functional structures.
[0076] The specific process of the image stitching method is described as follows. Figure 1 As an example, the coal mine underground scene shown in Figure 2 The electronic device shown in the figure is implemented as an example, for example, any one of the multiple cameras described below (for example, the first camera or the second camera) can be Figure 1 and Figure 2 The camera 30 shown, the electronic device described below may be Figure 2 The electronic device 30 shown. In the embodiment of the present application, the electronic device is used as an example to introduce the method, or the method can also be implemented by other devices or systems, such as a chip, a chip system or a processor applied to the electronic device. In the embodiment of the present application, the image stitching of two image frames synchronously captured by any two adjacent cameras (such as the first camera and the second camera) among multiple cameras is introduced as an example.
[0077] S301. Acquire multiple image frames captured by multiple cameras.
[0078] In an embodiment of the present application, an electronic device connected to multiple cameras can obtain multiple image frames captured by multiple cameras. Among them, multiple cameras can be deployed in a target scene, and multiple image frames can be obtained by multiple cameras synchronously capturing the target scene. For example, the target scene can be an underground coal mine scene, and multiple cameras are installed on multiple hydraulic supports. When the coal cutter is cutting coal, the multiple cameras will move toward the direction of the coal cutter together with the multiple hydraulic supports. For another example, the target scene can be a scene such as a sports track.
[0079] It is understandable that the target scene may be deployed with or without calibration objects. For example, in a coal mine underground scene, the calibration object may be Figure 1 Coal cutter 20 shown. Alternatively, a calibration plate may be deployed in the target scene, or may not be deployed. The calibration plate may be a calibration card, which is a two-dimensional code that can be detected and identified by a program and can be used to calibrate the camera external parameters of the camera. For example, in a coal mine underground scene, the calibration card can be pasted on Figure 1The middle part of the coal cutter 20 shown in the figure does not rotate when the coal cutter 20 is working, but the end of the coal cutter 20 and the rocker arm installed at the end will rotate. This embodiment of the present application is not limited to this. Therefore, the multiple image frames obtained by synchronously capturing the target scene by multiple cameras may contain calibration objects and / or calibration cards, or may not contain calibration objects and / or calibration cards. This embodiment of the present application is not limited to this.
[0080] In a specific implementation, the electronic device can obtain multiple image frames captured by multiple cameras in the following two ways:
[0081] In a first approach, the electronic device may obtain multiple video streams obtained by capturing a target scene with multiple cameras, and obtain multiple image frames with the same timestamp from the multiple video streams.
[0082] In a second method, the electronic device may send synchronous acquisition information to multiple cameras, and the synchronous acquisition signal may be used to instruct the multiple cameras to synchronously acquire the target scene. The multiple cameras may synchronously acquire the target scene according to the synchronous acquisition signal to obtain multiple image frames, and send the multiple image frames to the electronic device. The electronic device receives multiple image frames from the multiple cameras.
[0083] In some embodiments, after acquiring multiple image frames captured by multiple cameras, the electronic device may perform image stitching on two image frames obtained by synchronously capturing the target scene by any two adjacent cameras among the multiple cameras. For example, the multiple cameras may include a first camera and a second camera, and the first camera and the second camera may be deployed adjacent to each other in the target scene. The multiple image frames may include image frames captured by the first camera (e.g., referred to as first image frames) and image frames captured by the second camera (e.g., referred to as second image frames), and the electronic device may perform image stitching on the first image frame and the second image frame.
[0084] Before stitching the first image frame and the second image frame, the electronic device may perform feature extraction on the first image frame and the second image frame to obtain a feature point set of the first image frame (for example, referred to as a first feature point set) and a feature point set of the second image frame (for example, referred to as a second feature point set).
[0085] Among them, when the first image frame and the second image frame have an overlapping area or are clear enough, the first feature point set and the second feature point set can be used to calibrate the camera extrinsics of the first camera and the second camera. For example, when the first image frame and the second image frame have an overlapping area, the matching rate of the first feature point set and the second feature point set can be greater than the matching threshold. It can be understood that the matching threshold can be a pre-set threshold or a threshold set based on experience or experiments, and the embodiments of the present application are not limited to this. When the first image frame and the second image frame are clear enough, if the first image frame and the second image frame contain calibration objects and / or calibration plates, the first feature point set and the second feature point set can include feature points for describing the calibration objects and / or calibration plates.
[0086] When the first image frame and the second image frame have no overlapping area or are not clear enough, the first feature point set and the second feature point set cannot be used to calibrate the camera extrinsics of the first camera and the second camera. For example, when the first image frame and the second image frame have no overlapping area, the matching rate of the first feature point set and the second feature point set may be less than or equal to the matching threshold. When the first image frame and the second image frame are not clear enough, if the first image frame and the second image frame contain calibration objects and / or calibration plates, the first feature point set and the second feature point set may not contain feature points used to describe the calibration objects and / or calibration plates.
[0087] For example, in a coal mine underground scene, when the first camera and the second camera move toward the coal cutter together with the first hydraulic support and the second hydraulic support respectively, the first image frame and the second image frame may not have an overlapping area. When the coal cutter generates a large amount of dust due to the coal cutting operation, the first image frame and the second image frame may not be clear enough. If the first image frame and the second image frame have no overlapping area or are not clear enough, the electronic device will be unable to determine the camera extrinsics of the first camera and the second camera based on the first feature point set and the second feature point set, and further, the electronic device will be unable to determine the correct stitching position when stitching the first image frame and the second image frame, and the first image frame and the second image frame cannot be stitched or the stitching effect is poor.
[0088] When the first feature point set and the second feature point set can be used to calibrate the camera extrinsic parameters of the first camera and the second camera, the way in which the electronic device calibrates the camera extrinsic parameters of the first camera and the second camera according to the first feature point set and the second feature point set may include but is not limited to: feature point extraction based on calibration (tag) objects and / or calibration cards, feature point extraction based on binocular matching, feature point extraction based on training models, and other calibration methods.
[0089] S302: According to a first inertial measurement unit IMU measurement result when the first camera acquires a first image frame, correct a first extrinsic parameter of the first camera to obtain a second extrinsic parameter of the first camera.
[0090] In an embodiment of the present application, in order to avoid the situation where the electronic device cannot determine the camera extrinsics of the first camera and the second camera based on the first feature point set and the second feature point set because the first feature point set and the second feature point set cannot be used to calibrate the camera extrinsics of the first camera and the second camera, thereby causing the electronic device to be unable to determine the correct stitching position when stitching the first image frame and the second image frame. The electronic device can obtain the IMU measurement result (for example, referred to as the first IMU measurement result) of the first camera when capturing the first image frame.
[0091] Among them, the first IMU measurement result may include the angular velocity of the first camera when collecting the first image frame, the acceleration of the first camera when collecting the first image frame, and the IMU internal parameters, the angular velocity may include the angular velocity with the x-axis as the rotation axis, the angular velocity with the y-axis as the rotation axis, and the angular velocity with the z-axis as the rotation axis, the IMU internal parameters include random noise, bias, etc., and the IMU internal parameters can be obtained by calibrating the IMU sensor in advance. The first IMU measurement result may be the measurement result of the IMU sensor integrated in the first camera, or it may be the measurement result of the IMU sensor connected to the first camera rigid body, and the embodiment of the present application does not specifically limit this.
[0092] After obtaining the first IMU measurement result, the electronic device can correct the historical camera extrinsic parameters of the first camera (for example, called the first extrinsic parameters) according to the first IMU measurement result to obtain the latest camera extrinsic parameters of the first camera (for example, called the second extrinsic parameters). Among them, the camera attitude information corresponding to the second extrinsic parameter can be the camera attitude information corresponding to the first IMU measurement result, and the camera attitude information may include the coordinates of the camera in the world coordinate system, the orientation of the camera, etc. The above-mentioned electronic device obtains the latest camera extrinsic parameters of the camera by correcting the historical camera extrinsic parameters of the camera, and does not need to rely on the image frames captured by the camera to calibrate the camera extrinsic parameters of the camera, so as to avoid the situation where the image frames captured by the camera are not clear enough or the image frames captured by two adjacent cameras do not have overlapping areas, and the correct stitching position cannot be determined when performing image stitching, thereby improving the stitching effect of image stitching.
[0093] In some embodiments, the electronic device may determine whether there is a third image frame, the third image frame being an image frame captured by the first camera before the first image frame is captured, that is, the electronic device may determine whether the first image frame is an initial image frame captured by the first camera. When there is a third image frame (that is, the first image frame is not an initial image frame captured by the first camera), the electronic device may determine that the first extrinsic parameter is a camera extrinsic parameter of the first camera when the third image frame is captured. When there is no third image frame (that is, the first image frame is an initial image frame captured by the first camera), the electronic device may determine that the first extrinsic parameter is a pre-set camera extrinsic parameter of the first camera, or, when the first feature point set can be used to calibrate the camera extrinsic parameter of the first camera, the electronic device may determine that the first extrinsic parameter is a camera extrinsic parameter of the first camera obtained by calibrating the first camera according to the first feature point set.
[0094] For example, the image frames captured by the first camera may include image frame 1, image frame 2, and image frame 3. When the first camera is capturing image frame 1, the first extrinsic parameter may be a preset camera extrinsic parameter A, or, when the feature point set 1 of image frame 1 can be used to calibrate the camera extrinsic parameter of the first camera, the first extrinsic parameter may be a camera extrinsic parameter B obtained by calibrating the first camera according to the feature point set 1, and the second extrinsic parameter may be a camera extrinsic parameter C obtained by correcting the camera extrinsic parameter A or the camera extrinsic parameter B according to the IMU measurement result when the first camera is capturing image frame 1. When the first camera is capturing image frame 2, the first extrinsic parameter may be a camera extrinsic parameter C obtained by correcting the camera extrinsic parameter C according to the IMU measurement result when the first camera is capturing image frame 2. By analogy, when the first camera is capturing image frame 3, the first extrinsic parameter may be a camera extrinsic parameter D obtained by correcting the camera extrinsic parameter D according to the IMU measurement result when the first camera is capturing image frame 3.
[0095] In some embodiments, the electronic device can determine the movement distance of the first camera based on the first IMU measurement result. Figure 4 , which is a schematic diagram of determining the moving distance of the first camera provided in an embodiment of the present application, the electronic device can determine the moving distance of the first camera based on the acceleration of the moving direction of the first camera (for example, referred to as the first direction) preset in the first IMU measurement result.
[0096] Among them, the moving distance of the first camera satisfies the following formula:
[0097] T(t+Δt)=T(t)+R(t) T ∫(∫a 1 dt)dt
[0098] T(t) is the translation matrix in the first external parameter, R(t) T is the transpose of the rotation matrix in the first external parameter, a 1 is the acceleration in the first direction of the first IMU measurement result. The implementation principle of this formula is: first calculate a 1 The moving distance of the first camera in the camera coordinate system is obtained by quadratic integration, and then the moving distance of the first camera in the camera coordinate system is converted according to T(t) and R(t) to obtain the moving distance of the first camera in the world coordinate system.
[0099] After determining the moving distance of the first camera, the electronic device can determine whether the historical camera extrinsic parameter of the first camera has changed according to whether the moving distance of the first camera is greater than the distance threshold. It can be understood that the distance threshold can be a pre-set threshold or a threshold set according to experience or experiment, and the embodiment of the present application is not limited to this. When the moving distance of the first camera is less than or equal to the distance threshold, the electronic device can determine that the historical camera extrinsic parameter of the first camera has not changed, that is, the electronic device does not need to perform correction processing on the first extrinsic parameter according to the first IMU measurement result to obtain the second extrinsic parameter, and directly determine the second extrinsic parameter as the first extrinsic parameter, and then directly perform image stitching on the first image frame collected by the first camera and the second image frame collected by the second camera according to the first extrinsic parameter. When the moving distance of the first camera is greater than the distance threshold, the electronic device can determine that the historical camera extrinsic parameter of the first camera has changed, that is, the electronic device needs to perform correction processing on the first extrinsic parameter according to the first IMU measurement result to obtain the second extrinsic parameter.
[0100] For example, in a coal mine underground scene, the electronic device can determine whether the first camera moves toward the coal cutter together with the first hydraulic support, that is, whether the first hydraulic support is pulled, based on whether the moving distance of the first camera is greater than the distance threshold. When the moving distance of the first camera is less than or equal to the distance threshold, the electronic device can determine that the first camera does not move toward the coal cutter together with the first hydraulic support, that is, the first hydraulic support is pulled. When the moving distance of the first camera is greater than the distance threshold, the electronic device can determine that the first camera does not move toward the coal cutter together with the first hydraulic support, that is, the first hydraulic support is not pulled.
[0101] In a specific implementation, the electronic device can correct the first external parameter according to the measurement result of the first IMU in the following two ways to obtain the second external parameter:
[0102] In method 1, the electronic device can directly adjust the rotation matrix and the translation matrix in the first external parameter according to the angular velocity and acceleration in the first IMU measurement result to obtain the second external parameter.
[0103] Among them, the second external parameter satisfies the following formula:
[0104]
[0105] Among them, R(t+Δt) is the rotation matrix in the second external parameter, T(t+Δt) is the translation matrix in the second external parameter, R(t) is the rotation matrix in the first external parameter, T(t) is the translation matrix in the first external parameter, Ω x is the angular velocity with the x-axis as the rotation axis in the first IMU measurement result, Ω y is the angular velocity with the y axis as the rotation axis in the first IMU measurement result, Ω z is the angular velocity with the z axis as the rotation axis in the first IMU measurement result, R(t) T is the transpose of R(t), a 0 is the acceleration in the first IMU measurement result, and g is the gravitational acceleration.
[0106] Method 2: When the first feature point set can be used to calibrate the camera extrinsic parameters of the first camera, the electronic device can first adjust the angular velocity, acceleration and IMU intrinsic parameters in the first IMU measurement result according to the two-dimensional coordinates and three-dimensional coordinates of each feature point in the first feature point set to obtain the second IMU measurement result. Then, according to the angular velocity and acceleration in the second IMU measurement result, the rotation matrix and translation matrix in the first extrinsic parameter are adjusted to obtain the second extrinsic parameter.
[0107] Among them, the two-dimensional coordinates of each feature point are the coordinates of each feature point in the pixel coordinate system (the unit is the number of pixels), and the three-dimensional coordinates of each feature point are the coordinates of each feature point in the world coordinate system (the unit is centimeters). The two-dimensional coordinates of each feature point are obtained based on the image feature point detection algorithm, and the three-dimensional coordinates of each feature point are obtained by pre-detection. The coordinate projection error corresponding to the angular velocity, acceleration and IMU internal parameters in the second IMU measurement result is less than the error threshold. It can be understood that the error threshold can be a preset threshold or a threshold set according to experience or experiment, and the embodiments of the present application are not limited to this. The coordinate projection error is the error between the two-dimensional coordinates of each feature point and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of each feature point. The coordinate projection error satisfies the following formula:
[0108]
[0109] c i is the error between the two-dimensional coordinates of the ith feature point in the first feature point set and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the ith feature point in the first feature point set, x i is the two-dimensional coordinate of the i-th feature point in the first feature point set, X i is the three-dimensional coordinate of the i-th feature point in the first feature point set, Ω 0is the angular velocity in the first IMU measurement result, a 0 is the acceleration in the first IMU measurement result, Ω is the angular velocity in the second IMU measurement result, a is the acceleration in the second IMU measurement result, Proj() is used to project the three-dimensional coordinates into two-dimensional coordinates, and p is the IMU internal parameter.
[0110] In some embodiments, based on the above-mentioned electronic device correcting the first extrinsic parameter according to the first IMU measurement result to obtain the second extrinsic parameter, the electronic device can also obtain the IMU measurement result of the second camera when capturing the second image frame (for example, referred to as the third IMU measurement result), and correct the historical camera extrinsic parameter of the second camera (for example, referred to as the third extrinsic parameter) according to the third IMU measurement result to obtain the latest camera extrinsic parameter of the second camera (for example, referred to as the fourth extrinsic parameter). It will not be repeated here.
[0111] S303: Perform image stitching on a first image frame captured by the first camera and a second image frame captured by the second camera according to a second external parameter.
[0112] In an embodiment of the present application, the electronic device can perform image stitching on the first image frame captured by the first camera and the second image frame captured by the second camera according to the second extrinsic parameter and the fourth extrinsic parameter. For example, in a coal mine underground scene, the electronic device can project both the first image frame and the second image frame to the world coordinate system corresponding to the coal cutter, and then perform image stitching on the image frame after the first image frame is projected and the image frame after the second image frame is projected according to the second extrinsic parameter and the fourth extrinsic parameter using the scale-invariant feature conversion feature to obtain a panoramic image of the coal cutter. The above-mentioned electronic device uses the latest camera extrinsic parameters of the two adjacent cameras to perform image stitching on the two image frames obtained by synchronously capturing the target scene with the two adjacent cameras, thereby avoiding the situation where the historical camera extrinsic parameters of the camera are no longer applicable due to camera movement or the image captured by the camera is not clear enough, which in turn makes it impossible to determine the correct stitching position according to the historical camera extrinsic parameters of the camera when performing image stitching, thereby improving the stitching effect of image stitching.
[0113] For ease of understanding, the following is a specific example. Figure 3-Figure 4 The image stitching method shown is further introduced.
[0114] See also Figure 5 , Figure 5 A flowchart of another image stitching method provided in an embodiment of the present application, in which the method is applied to Figure 1 Taking the coal mine underground scenario shown in the figure as an example, the process includes:
[0115] 1. IMU sensor installation:
[0116] In this process, the IMU sensor is connected to the camera rigid body, or the IMU sensor is integrated into the camera. When the camera moves toward the coal cutter together with the hydraulic support, the IMU sensor can obtain the movement information of the camera, and then obtain the pulling information of the hydraulic support.
[0117] 2. Initial camera extrinsic calibration:
[0118] In this process, the coal cutter is flushed before the coal cutter is officially operated to ensure that the feature points of the coal cutter can be extracted from the initial image of the coal cutter captured by the camera. The camera extrinsic parameters are calibrated according to the feature points of the coal cutter in the initial image to obtain the initial camera extrinsic parameters of the camera. Since the coal cutter only needs to be flushed when it is run for the first time, compared with the related art that requires flushing the coal cutter every hour, the maintenance cost is reduced.
[0119] 3. Judgment of hydraulic support pulling frame:
[0120] In this process, after the coal cutter officially starts the coal cutting operation, the hydraulic support is pulled out based on the camera movement information obtained by the IMU sensor. For example, the hydraulic support is pulled out when the camera movement distance is greater than the distance threshold. After determining that the hydraulic support is not pulled out, the camera's historical camera extrinsic parameters (such as the initial camera extrinsic parameters mentioned above) are used for image stitching.
[0121] 4. Images can be used for calibration judgment:
[0122] In this process, after the hydraulic support is pulled, it is determined whether the feature point set of the image captured by the camera can be used to calibrate the camera extrinsic parameters. For example, based on the feature point extraction of the calibration object and / or calibration card: if the feature point set of the image captured by the camera contains feature points used to describe the calibration object and / or calibration card, it is determined that the feature point set of the image captured by the camera can be used to calibrate the camera extrinsic parameters of the camera; if the feature point set of the image captured by the camera does not contain feature points used to describe the calibration object and / or calibration card, it is determined that the feature point set of the image captured by the camera cannot be used to calibrate the camera extrinsic parameters of the camera. For another example, based on the feature point extraction of binocular matching: if the matching rate of the feature point set of two images synchronously captured by two adjacent cameras is greater than the matching threshold, it is determined that the feature point set of the image captured by the camera can be used to calibrate the camera extrinsic parameters of the camera; if the matching rate of the feature point set of two images synchronously captured by two adjacent cameras is less than or equal to the matching threshold, it is determined that the feature point set of the image captured by the camera cannot be used to calibrate the camera extrinsic parameters of the camera.
[0123] 5. Correction of historical camera external parameters:
[0124] In this process, when the hydraulic support is determined to be pulled and the feature point set of the image captured by the camera cannot be used to calibrate the camera extrinsic parameters, the camera movement information obtained by the IMU sensor is used as the correction value of the camera's historical camera extrinsic parameters, and the camera's historical camera extrinsic parameters are corrected to obtain the camera's latest camera extrinsic parameters. The camera movement information obtained by the IMU is corrected to eliminate the error of the IMU sensor; the camera's historical camera extrinsic parameters are corrected to obtain the camera's latest camera extrinsic parameters. The camera's latest camera extrinsic parameters are used for image stitching.
[0125] When the hydraulic support is determined to be pulled and the feature point set of the image captured by the camera can be used to calibrate the camera extrinsic parameters, the feature point set of the image captured by the camera is first used as the correction value of the camera movement information obtained by the IMU, and the camera movement information obtained by the IMU is corrected to eliminate the error of the IMU sensor. Then, the camera movement information obtained by the corrected IMU sensor is used as the correction value of the camera's historical camera extrinsic parameters, and the camera's historical camera extrinsic parameters are corrected to obtain the camera's latest camera extrinsic parameters. The latest camera extrinsic parameters of the camera are used for image stitching.
[0126] For example, the two-dimensional coordinates and three-dimensional coordinates of each feature point in the feature point set of the image captured by the camera are obtained. When the camera movement information obtained by the IMU sensor is accurate, the two-dimensional coordinates obtained by the three-dimensional coordinate projection of each feature point will coincide with the two-dimensional coordinates obtained by the detection of each feature point, that is, the coordinate projection error is 0. However, due to problems such as long-term operation or heating of the IMU sensor, the pre-calibrated IMU internal parameters are no longer applicable, which leads to errors in the camera movement information obtained. Therefore, the parameter values in the camera movement information obtained by the IMU sensor can be continuously iterated and updated based on optimization algorithms such as genetic algorithms and genetic annealing algorithms, with the minimum coordinate projection error as the goal, to obtain the optimal parameter value, thereby eliminating the error of the IMU sensor.
[0127] Based on the above embodiments, the present application also provides a computing device, such as Figure 6 6 is a schematic diagram of a computing device provided in an embodiment of the present application, wherein the computing device 600 includes: a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other via the bus 602. The computing device 600 may be an electronic device. It should be understood that the present application does not limit the number of processors and memories in the computing device 600.
[0128] The bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus 602 may include a path for transmitting information between various components of the computing device 600 (eg, the memory 606, the processor 604, and the communication interface 608).
[0129] The processor 604 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0130] The memory 606 may include a volatile memory, such as a random access memory (RAM). The memory 606 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0131] The memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the communication module and the processing module, thereby implementing the above method embodiment. Figure 3-Figure 5 That is, the memory 606 stores the image stitching method for executing the above method embodiment. Figure 3-Figure 5 Instructions for the image stitching methods in .
[0132] The communication interface 608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or communication networks.
[0133] Based on the above embodiments, the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be an electronic device.
[0134] like Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 606 in one or more computing devices 600 in the computing device cluster may store the same memory for executing the above method embodiment. Figure 3-Figure 5 Instructions for the image stitching methods in .
[0135] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store the memory 606 for executing the above method embodiment. Figure 3-Figure 5 In other words, the combination of one or more computing devices 600 can jointly execute the instructions for executing the above method embodiment. Figure 3-Figure 5 Instructions for the image stitching methods in .
[0136] It is understandable that the memory 606 in different computing devices 600 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the electronic device. That is, the instructions stored in the memory 606 in different computing devices 600 can implement the functions of one or more modules in the communication module and the processing module.
[0137] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 8 A possible implementation is shown. Figure 8 As shown, two computing devices 600A and 600B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 606 in the computing device 600A stores instructions for executing the functions of the communication module. At the same time, the memory 606 in the computing device 600B stores instructions for executing the functions of the processing module.
[0138] It should be understood that Figure 8 The functions of the computing device 600A shown in FIG. 6A may also be completed by multiple computing devices 600. Similarly, the functions of the computing device 600B may also be completed by multiple computing devices 600.
[0139] Based on the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, which may be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above method embodiments. Figure 3-Figure 5 Image stitching method in .
[0140] Based on the above embodiments, the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above method embodiment. Figure 3-Figure 5 Image stitching method in .
[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.
[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0145] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An image stitching method, characterized in that: include: Acquire multiple image frames captured by multiple cameras; The multiple cameras are deployed in a target scene, the multiple image frames are obtained by synchronously capturing the target scene by the multiple cameras, the multiple cameras include a first camera and a second camera, and the first camera and the second camera are adjacently deployed in the target scene; According to a first inertial measurement unit IMU measurement result when the first camera acquires the first image frame, a first extrinsic parameter of the first camera is corrected to obtain a second extrinsic parameter of the first camera; The camera attitude information corresponding to the second external parameter is the camera attitude information corresponding to the first IMU measurement result; Image stitching is performed on the first image frame and the second image frame captured by the second camera according to the second external parameter.
2. The method according to claim 1, characterized in that Before correcting the first external parameter of the first camera, the method further includes: When there is a third image frame, determining that the first extrinsic parameter is an extrinsic parameter of the first camera when acquiring the third image frame; the third image frame is an image frame acquired by the first camera before acquiring the first image frame; When the third image frame does not exist, the first extrinsic parameter is determined to be a preset extrinsic parameter of the first camera, or the first extrinsic parameter is obtained by calibrating the first camera according to the first image frame.
3. The method according to claim 1, characterized in that Before correcting the first external parameter of the first camera, the method further includes: According to the first IMU measurement result, it is determined that the movement distance of the first camera is greater than a first threshold.
4. The method according to claim 3, characterized in that The method further comprises: When it is determined that the moving distance is less than or equal to the first threshold, image stitching is performed on the first image frame and the second image frame according to the first external parameter.
5. The method according to claim 3 or 4, characterized in that Determining a moving distance of the first camera according to the first IMU measurement result includes: The moving distance is determined according to the acceleration in a first direction in the first IMU measurement result; the first direction is a preset moving direction of the first camera.
6. The method according to any one of claims 1 to 5, characterized in that: According to the first IMU measurement result, the first external parameter is corrected to obtain the second external parameter, including: According to the angular velocity and acceleration in the first IMU measurement result, the rotation matrix and the translation matrix in the first external parameter are adjusted to obtain the second external parameter; or, According to the two-dimensional coordinates and three-dimensional coordinates of each feature point in the first feature point set of the first image frame, the angular velocity, acceleration and IMU internal parameters in the first IMU measurement result are adjusted to obtain the second IMU measurement result, and according to the angular velocity and acceleration in the second IMU measurement result, the rotation matrix and translation matrix in the first external parameter are adjusted to obtain the second external parameter; the coordinate projection error corresponding to the angular velocity, acceleration and IMU internal parameter in the second IMU measurement result is less than a second threshold, and the coordinate projection error is the error between the two-dimensional coordinate of each feature point and the two-dimensional coordinate obtained by projecting the three-dimensional coordinate of each feature point.
7. The method according to claim 2 or 6, characterized in that A calibration object and / or a calibration card is also deployed in the target scene, and the first feature point set in the first image frame includes feature points used to describe the calibration object and / or the calibration card; or, A matching rate between the first feature point set and the second feature point set is greater than a third threshold, and the second feature point set is a feature point set of the second image frame.
8. The method according to claim 6, characterized in that The second external parameter satisfies the following formula: Wherein, R(t+Δt) is the rotation matrix in the second external parameter, T(t+Δt) is the translation matrix in the second external parameter, R(t) is the rotation matrix in the first external parameter, T(t) is the translation matrix in the first external parameter, Ω x is the angular velocity with the x-axis as the rotation axis in the first IMU measurement result, Ω y is the angular velocity with the y coordinate axis as the rotation axis in the first IMU measurement result, Ω z is the angular velocity with the z coordinate axis as the rotation axis in the first IMU measurement result, R(t) T is the transpose of R(t), a0 is the acceleration in the first IMU measurement result, and g is the gravitational acceleration.
9. The method according to claim 6, characterized in that The coordinate projection error satisfies the following formula: Among them, c i is the error between the two-dimensional coordinates of the ith feature point in the first feature point set and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the ith feature point in the first feature point set, x i is the two-dimensional coordinate of the i-th feature point in the first feature point set, X i is the three-dimensional coordinate of the i-th feature point in the first feature point set, Ω0 is the angular velocity in the first IMU measurement result, a0 is the acceleration in the first IMU measurement result, Ω is the angular velocity in the second IMU measurement result, a is the acceleration in the second IMU measurement result, Proj() is used to project the three-dimensional coordinates into two-dimensional coordinates, and p is the IMU internal parameter.
10. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The memory stores one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1 to 9.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.
12. A computer program product, characterized in that The invention comprises a computer program, which, when being executed on a computer, enables the computer to execute the method as claimed in any one of claims 1 to 9.
Citation Information
Cited By
Quick calibration method and system for spliced imaging
CN121170032A
