Binocular splicing method, binocular splicing system and machine readable storage medium

Abstract

The application provides a binocular splicing method, a binocular splicing system and a machine readable storage medium. The method comprises the following steps: in the case that it is determined that a splicing parameter updating condition is met, controlling a pattern projection device to project a pattern to an overlapping area of a field of view angle of a first camera and a second camera, acquiring a first image through the first camera, and acquiring a second image through the second camera; determining a latest splicing parameter according to the first image and the second image; and splicing images acquired through the first camera and the second camera according to the latest splicing parameter to obtain a spliced image. The method can expand the applicable scenarios of the binocular splicing scheme.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a binocular stitching method, a binocular stitching system, and a machine-readable storage medium. Background Technology

[0002] Binocular image stitching (or simply binocular stitching) refers to stitching two overlapping images of the same scene into a larger image, which is of great significance in fields such as medical imaging, computer vision, and satellite data.

[0003] Current binocular stitching solutions are generally applicable to binocular modules with fixed positions. The overlapping area of ​​the field of view of the two eyes in such devices is fixed, and the shooting range is naturally fixed after the product is installed, which is not flexible enough. Summary of the Invention

[0004] In view of this, this application provides a binocular stitching method, a binocular stitching system, and a machine-readable storage medium.

[0005] According to a first aspect of the embodiments of this application, a binocular stitching method is provided, comprising:

[0006] When it is determined that the stitching parameter update conditions are met, the pattern projection device is controlled to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, and a first image is acquired through the first camera, and a second image is acquired through the second camera.

[0007] Based on the first image and the second image, determine the latest stitching parameters;

[0008] The images acquired by the first camera and the second camera are stitched together according to the latest stitching parameters to obtain a stitched image.

[0009] According to a second aspect of the embodiments of this application, a binocular stitching system is provided, comprising: a pattern projection device, a first camera, a second camera, and a processor; wherein:

[0010] The pattern projection device is used for pattern projection;

[0011] The first camera is used for image acquisition;

[0012] The second camera is used for image acquisition;

[0013] The processor is configured to, when determining that the stitching parameter update conditions are met, control the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, acquire a first image through the first camera, and acquire a second image through the second camera; determine the latest stitching parameters based on the first image and the second image; and stitch the images acquired through the first camera and the second camera according to the latest stitching parameters to obtain a stitched image.

[0014] According to a third aspect of the present application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0015] According to a fourth aspect of the embodiments of this application, a storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0016] The binocular stitching method of this application embodiment controls the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera when it is determined that the stitching parameter update conditions are met. A first image is acquired through the first camera, and a second image is acquired through the second camera. Then, the latest stitching parameters can be determined based on the first image and the second image, and the images acquired through the first camera and the second camera are stitched together based on the latest stitching parameters to obtain a stitched image. This realizes the automatic determination of stitching parameters, can be applied to binocular systems with variable relative positions, and expands the applicable scenarios of binocular stitching schemes. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a binocular stitching method provided in an embodiment of this application;

[0018] Figure 2A and Figure 2B This is a schematic diagram of the field of view of the binocular stitching system provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the projection pattern provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram of a binocular stitching process provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of a binocular splicing system provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0024] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0025] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0026] Please see Figure 1 This is a flowchart illustrating a binocular stitching method provided in an embodiment of this application. The binocular stitching method can be applied to binocular stitching systems, such as binocular cameras, binocular access control systems, or binocular door locks. The binocular stitching system may include a pattern projection device and two cameras (which may be referred to as a first camera and a second camera). Figure 1 As shown, this binocular stitching method may include the following steps:

[0027] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0028] Step S100: When it is determined that the stitching parameter update conditions are met, the pattern projection device is controlled to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, and a first image is acquired through the first camera, and a second image is acquired through the second camera.

[0029] In this embodiment of the application, in order to realize the automatic determination of the splicing parameters of the binocular splicing system, a pattern projection device can be set in the binocular splicing system. The pattern projection device can be used to project a pattern onto the overlapping area of ​​the field of view of the binoculars (i.e., the first camera and the second camera) of the binocular splicing system to determine the splicing parameters.

[0030] For example, when the binocular stitching system determines that the stitching parameter update conditions are met, the pattern projection device can be controlled to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera. When a pattern is projected onto the overlapping area of ​​the field of view of the first camera and the second camera, images are acquired through the first camera and the second camera respectively (the image acquired through the first camera can be called the first image, and the image acquired through the second camera can be called the second image).

[0031] For example, when the first camera acquires the first image and the second camera acquires the second image, the projection state of the pattern projection device remains consistent.

[0032] For example, the first camera and the second camera can be controlled to simultaneously (including the acquisition time being the same or there being a tolerable error between the acquisition times) acquire the first image and the second image.

[0033] For example, the pattern projected by the above-mentioned pattern projection device can be a pattern with rich texture, which may include, but is not limited to, a checkerboard or a QR code.

[0034] It should be noted that the duration of pattern projection can be determined based on the camera's frame rate. For example, the duration of pattern projection can be the time during which both the first and second cameras can capture N (N≥1, the specific value can be set according to actual needs) frames of images when pattern projection is present.

[0035] Step S110: Determine the latest stitching parameters based on the first image and the second image.

[0036] In this embodiment of the application, if the first image and the second image are obtained in accordance with the method described in step S100, the latest stitching parameters can be determined based on the first image and the second image.

[0037] For example, if the latest splicing parameters are determined, it can be determined whether the splicing parameters have been saved. If the splicing parameters have been saved, the latest splicing parameters can be used to replace the old splicing parameters; if the splicing parameters have not been saved, the latest splicing parameters can be saved.

[0038] Step S120: Based on the latest stitching parameters, stitch the images obtained by the first camera and the second camera to obtain a stitched image.

[0039] In this embodiment of the application, the images obtained by the first camera and the second camera can be stitched together according to the latest stitching parameters determined in the manner described above to obtain a stitched image.

[0040] It can be seen that, in Figure 1 In the method flow shown, when the stitching parameter update conditions are met, the pattern projection device is controlled to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera. The first image is acquired through the first camera, and the second image is acquired through the second camera. Then, the latest stitching parameters can be determined based on the first image and the second image. The images acquired through the first camera and the second camera are stitched together based on the latest stitching parameters to obtain a stitched image. This realizes the automatic determination of stitching parameters and can be applied to binocular systems with variable relative positions, thus expanding the applicable scenarios of binocular stitching schemes.

[0041] In some embodiments, determining that the splicing parameter update conditions are met may include:

[0042] Confirm that the splicing function is enabled for the first time.

[0043] For example, when the stitching function is enabled for the first time, such as during the initial operation of the binocular stitching system, it can determine that the stitching parameter update conditions are met. In this case, the stitching parameters can be determined in the manner described in the above embodiments, and the determined stitching parameters (the latest stitching parameters) can be saved.

[0044] In some embodiments, determining that the splicing parameter update conditions are met may include:

[0045] Received a command to adjust the relative positions of the first and second cameras.

[0046] For example, considering that the relative positions of the first and second cameras change, the stitching parameters need to be updated to ensure the accuracy of binocular stitching.

[0047] Accordingly, the binocular stitching system can determine that the stitching parameter update conditions are met upon receiving a relative position adjustment command from the first camera and the second camera—for example, receiving a position adjustment command for the first camera and / or receiving a position adjustment command for the second camera. In this case, the stitching parameters can be determined in the manner described in the above embodiments. If stitching parameters (which can be referred to as old stitching parameters) are already stored, the latest determined stitching parameters can be used to replace the old stitching parameters.

[0048] In some embodiments, determining that the splicing parameter update conditions are met may include:

[0049] Received splicing parameter update instruction.

[0050] For example, users can update the stitching parameters as needed. For instance, a user can trigger an update of the stitching parameters if they determine that the stitched image does not meet the requirements.

[0051] Accordingly, when the binocular stitching system receives a stitching parameter update instruction, it can determine that the stitching parameter update conditions are met. In this case, the stitching parameters can be determined in the manner described in the above embodiments, and the determined latest stitching parameters can replace the old stitching parameters.

[0052] In some embodiments, controlling the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera may include:

[0053] The pattern projection device is controlled to project a pattern onto a target area within the overlapping field of view of the first camera and the second camera; wherein, when the relative positions of the first camera and the second camera change, the target area is located within the overlapping field of view of the first camera and the second camera.

[0054] For example, considering that the relative positions of the first camera and the second camera change, the overlapping area of ​​the field of view of the first camera and the second camera will usually change as well. In order to ensure that when the pattern projection device projects a pattern, the projected pattern can cover part or all of the overlapping area of ​​the field of view of the first camera and the second camera, so as to automatically determine the stitching parameters in accordance with the manner described in the above embodiments, it is possible to determine the area that is always in the overlapping area of ​​the field of view of the first camera and the second camera (referred to as the target area in this document) when the relative positions of the first camera and the second camera change. Therefore, when controlling the pattern projection device to project a pattern, the pattern projection device can be controlled to project a pattern onto the target area in the overlapping area of ​​the field of view of the first camera and the second camera, so as to ensure that the projected pattern can cover part or all of the overlapping area of ​​the field of view of the first camera and the second camera.

[0055] In some embodiments, determining the latest stitching parameters based on the first image and the second image may include:

[0056] Feature extraction is performed on the first image and the second image respectively to obtain the features of the first image and the features of the second image;

[0057] Based on the features of the first image and the features of the second image, the homography relationship between the first image and the second image is determined.

[0058] For example, the homography relationship between the first image and the second image can be determined by feature extraction.

[0059] Accordingly, features can be extracted from the first image and the second image respectively to obtain the features of the first image and the features of the second image. Based on the features of the first image and the features of the second image, the homography relationship between the first image and the second image can be determined, such as the homography matrix. The homography relationship is determined as the latest stitching parameters. Then, based on the homography matrix, the images acquired by the first camera and the images acquired by the second camera can be converted to the same viewpoint and the images can be stitched together.

[0060] In some embodiments, determining the latest stitching parameters based on the first image and the second image may include:

[0061] Based on the phase consistency of the first image and the second image, the affine transformation parameters of the first image and the second image are determined.

[0062] For example, affine transformation parameters can be determined based on the phase consistency between the first image and the second image, and the determined affine transformation parameters can be used as the latest stitching parameters. Then, based on the determined affine transformation parameters, affine transformation can be performed on the image acquired by the first camera and / or the image acquired by the second camera, and image stitching can be performed.

[0063] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below in conjunction with specific application scenarios.

[0064] In this embodiment, a schematic diagram of the binocular stitching system can be shown as follows: Figure 2A As shown, Figure 2A As shown, the binocular stitching system may include camera 1 (i.e., the first camera mentioned above), camera 2 (i.e., the second camera mentioned above), and a laser emitting device for emitting patterns (i.e., the pattern projection device mentioned above).

[0065] The field of view and resolution of camera 1 and camera 2 can be adjusted according to actual needs.

[0066] For example, when camera 1 needs to focus on the vertical area, its vertical field of view may be 120° and its image resolution may be 1800*2400; when camera 2 focuses on the horizontal area, its vertical field of view may be 80° but its horizontal field of view may be 120° and its image resolution may be 1920*1080.

[0067] For example, the relative positional changes of a binocular system can be seen in [reference needed]. Figure 2A and Figure 2B .

[0068] For example, the pattern projected by the laser projection device can be a pattern with rich texture, which may include, but is not limited to, a checkerboard pattern or a QR code, or a specially designed pattern. Its schematic diagram can be as follows: Figure 3 As shown.

[0069] In this embodiment, the implementation process of the binocular stitching scheme can be as follows: Figure 4 As shown, it may include the following steps:

[0070] The binocular stitching system acquires images through camera 1 and camera 2 respectively.

[0071] When it is determined that the conditions for updating the stitching parameters are met, the laser projection device is controlled to project a pattern onto the overlapping area of ​​the field of view of camera 1 and camera 2. When a pattern is projected onto the overlapping area of ​​the field of view of camera 1 and camera 2, images are acquired through camera 1 and camera 2 respectively (such as the first image and the second image mentioned above). The stitching parameters are determined based on the images acquired by camera 1 and camera 2 to obtain the latest stitching parameters.

[0072] If the stitching parameters are already saved, the latest stitching parameters are obtained to replace the saved stitching parameters, that is, to update the stitching parameters. Then, the images obtained by camera 1 and camera 2 are stitched together according to the latest stitching parameters to obtain the stitched image.

[0073] If no stitching parameters are currently saved, the latest stitching parameters are saved, and the images acquired by camera 1 and camera 2 are stitched together based on the latest stitching parameters to obtain a stitched image.

[0074] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0075] Please see Figure 5 This is a schematic diagram of the structure of a binocular stitching system provided in an embodiment of this application, as shown below. Figure 5 As shown, the binocular stitching system may include: a pattern projection device 510, a first camera 520, a second camera 530, and a processor 540; wherein:

[0076] The pattern projection device 510 is used for pattern projection;

[0077] The first camera 520 is used for image acquisition;

[0078] The second camera 530 is used for image acquisition;

[0079] The processor 540 is configured to, when determining that the stitching parameter update conditions are met, control the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, acquire a first image through the first camera, and acquire a second image through the second camera; determine the latest stitching parameters based on the first image and the second image; and stitch the images acquired through the first camera and the second camera according to the latest stitching parameters to obtain a stitched image.

[0080] In some embodiments, the processor 540 determines that the splicing parameter update conditions are met, including:

[0081] Confirm that the splicing function is enabled for the first time; or,

[0082] Received a command to adjust the relative positions of the first camera and the second camera;

[0083] Received splicing parameter update instruction.

[0084] In some embodiments, the processor 540 controls the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, including:

[0085] The pattern projection device is controlled to project a pattern onto a target area within the overlapping field of view of the first camera and the second camera; wherein, when the relative positions of the first camera and the second camera change, the target area is located within the overlapping field of view of the first camera and the second camera.

[0086] In some embodiments, the processor 540 determines the latest stitching parameters based on the first image and the second image, including:

[0087] Feature extraction is performed on the first image and the second image respectively to obtain the features of the first image and the features of the second image;

[0088] Based on the first image features and the second image features, the homography relationship between the first image and the second image is determined.

[0089] In some embodiments, the processor 540 determines the latest stitching parameters based on the first image and the second image, including:

[0090] Based on the phase consistency between the first image and the second image, the affine transformation parameters of the first image and the second image are determined.

[0091] Please see Figure 6This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 can communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions in the memory 602 corresponding to the binocular stitching control logic, the processor 601 can execute the binocular stitching method described above.

[0092] The memory 602 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0093] In some embodiments, a storage medium, such as Figure 6 The memory 602 in the storage medium stores machine-executable instructions, which, when executed by a processor, implement the binocular stitching method described above. For example, the machine-readable storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A binocular stitching method, characterized in that, A binocular stitching system including a pattern projection device, a first camera, and a second camera, wherein the pattern projection device is independent of the first and second cameras, the method comprising: When it is determined that the stitching parameter update conditions are met, the pattern projection device is controlled to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, and a first image is acquired through the first camera, and a second image is acquired through the second camera. Based on the first image and the second image, determine the latest stitching parameters; The images acquired by the first camera and the second camera are stitched together according to the latest stitching parameters to obtain a stitched image; The control pattern projection device projects a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, including: The pattern projection device is controlled to project a pattern onto a target area within the overlapping field of view of the first camera and the second camera; wherein the pattern projected by the pattern projection device is a pattern with rich texture, and the target area is a predetermined area that is always located within the overlapping field of view of the first camera and the second camera, regardless of changes in the relative positions of the first camera and the second camera.

2. The method according to claim 1, characterized in that, The determination that the splicing parameter update conditions are met includes: Confirm that the splicing function is enabled for the first time; or, Received a command to adjust the relative positions of the first camera and the second camera; Received splicing parameter update instruction.

3. The method according to claim 1, characterized in that, The step of determining the latest stitching parameters based on the first image and the second image includes: Feature extraction is performed on the first image and the second image respectively to obtain the features of the first image and the features of the second image; Based on the first image features and the second image features, the homography relationship between the first image and the second image is determined.

4. The method according to claim 1, characterized in that, The step of determining the latest stitching parameters based on the first image and the second image includes: Based on the phase consistency between the first image and the second image, the affine transformation parameters of the first image and the second image are determined.

5. A binocular stitching system, characterized in that, include: The pattern projection device, a first camera, a second camera, and a processor, wherein the pattern projection device is independent of the first camera and the second camera; wherein: The pattern projection device is used for pattern projection; The first camera is used for image acquisition; The second camera is used for image acquisition; The processor is configured to, when determining that the stitching parameter update conditions are met, control the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, acquire a first image through the first camera, and acquire a second image through the second camera; determine the latest stitching parameters based on the first image and the second image; and stitch the images acquired through the first camera and the second camera based on the latest stitching parameters to obtain a stitched image. The processor controls the pattern projection device to project a pattern onto the overlapping area of ​​the field of view of the first camera and the second camera, including: The pattern projection device is controlled to project a pattern onto a target area within the overlapping field of view of the first camera and the second camera; wherein the target area is a predetermined area that is always within the overlapping field of view of the first camera and the second camera, regardless of changes in the relative positions of the first camera and the second camera.

6. The binocular stitching system according to claim 5, characterized in that, The processor determines that the splicing parameter update conditions are met, including: Confirm that the splicing function is enabled for the first time; or, Received a command to adjust the relative positions of the first camera and the second camera; Received splicing parameter update instruction.

7. The binocular stitching system according to claim 5, characterized in that, The processor determines the latest stitching parameters based on the first image and the second image, including: Feature extraction is performed on the first image and the second image respectively to obtain the features of the first image and the features of the second image; Based on the first image features and the second image features, the homography relationship between the first image and the second image is determined.

8. The binocular stitching system according to claim 5, characterized in that, The processor determines the latest stitching parameters based on the first image and the second image, including: Based on the phase consistency between the first image and the second image, the affine transformation parameters of the first image and the second image are determined.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-4.

10. A storage medium, characterized in that, The storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-4.

Images