Image alignment anchoring method and device and electronic equipment

Abstract

The invention provides an image alignment anchoring method and device and electronic equipment, a first image collected by a first camera in a real view range is obtained, the real view range comprises a plurality of set feature objects, the first image comprises anchor points corresponding to the feature objects, and position information of the anchor points is obtained. And obtaining a second image collected by the second camera, determining a plurality of alignment points in the second image, and obtaining position information of each alignment point. And performing alignment processing on the first image and the second image based on the position information of each anchor point and the position information of each alignment point to obtain a superposed image formed by the first image and the second image after alignment. According to the scheme, alignment and fusion of the two images are achieved through the anchor points in the first image and the alignment points in the second image, alignment can be achieved by processing a very small number of points, the processing amount is greatly reduced, and the requirement for high real-time performance can be met.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an image alignment and anchoring method, device and electronic equipment. Background Art

[0002] In augmented reality (AR) display, virtual images and real images are mostly displayed as unrelated content. Real images and virtual images are displayed independently and together within the field of view. In a small number of scenarios, it is necessary to achieve a fusion display or superposition of virtual images and real images, so that the two overlap in whole or in part and are displayed synchronously in the field of view.

[0003] In the existing processing methods, artificial intelligence algorithms are usually used to align virtual images and real images through image recognition to achieve the fusion of the two. However, using artificial intelligence image recognition to align virtual images and real images as a whole requires very high computing power, the computing module consumes a lot of power, is costly, and takes a long time to calculate. Therefore, it is difficult to meet the real-time display requirements in most scenarios. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an image alignment and anchoring method, device and electronic device to reduce the processing volume and meet the requirements of high real-time performance.

[0005] In a first aspect, the present invention provides an image alignment anchoring method, the method comprising:

[0006] Acquire a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes anchor points corresponding to the feature objects;

[0007] Obtaining position information of each anchor point;

[0008] Acquire a second image captured by a second camera, determine a plurality of alignment points in the second image, and obtain position information of each of the alignment points, where the number of the alignment points is the same as the number of the anchor points;

[0009] The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image consisting of the second image and the first image.

[0010] In an optional implementation, the step of obtaining the position information of each anchor point includes:

[0011] Obtaining the relative position relationship between each of the feature objects and the first camera, and obtaining the position information of the first camera;

[0012] According to the posture information of the first camera and the relative position relationship between each of the feature objects and the first camera, the position information of the anchor point corresponding to each of the feature objects is obtained.

[0013] In an optional implementation, the step of aligning the first image and the second image based on the position information of each anchor point and the position information of each alignment point includes:

[0014] performing perspective correction processing on the second image;

[0015] The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point in the second image after the perspective correction processing.

[0016] In an optional implementation, the step of performing perspective correction processing on the second image includes:

[0017] Obtaining a relative position relationship between the first camera and the AR projection screen;

[0018] Converting the second image from the coordinate system of the second camera to the coordinate system of the first camera;

[0019] Based on the relative position relationship between the first camera and the AR projection screen, the converted second image is projected onto the AR projection screen to achieve perspective correction processing of the second image.

[0020] In an optional implementation, the step of obtaining the relative position relationship between the first camera and the AR projection screen includes:

[0021] Obtaining real-time position information of the first camera through an inertial navigation unit disposed in the first camera;

[0022] Obtaining the location information of the AR projection screen;

[0023] The relative position relationship between the two is obtained based on the real-time posture information of the first camera and the position information of the AR projection screen.

[0024] In an optional implementation, the step of converting the second image from the coordinate system of the second camera to the coordinate system of the first camera includes:

[0025] Based on the position information of the first camera and the position information of the second camera, obtaining a conversion matrix between a coordinate system of the first camera and a coordinate system of the second camera;

[0026] Obtaining position information of the second image in the coordinate system where the second camera is located;

[0027] Based on the conversion matrix and the position information of the second image in the coordinate system where the second camera is located, the position information of the second image in the coordinate system where the first camera is located is obtained.

[0028] In an optional implementation, the step of aligning the first image and the second image based on the position information of each anchor point and the position information of each alignment point in the second image after perspective correction processing includes:

[0029] Based on the position information of each anchor point and the position information of each alignment point in the second image after perspective correction, the second image is moved, rotated and scaled so that each alignment point in the second image is aligned with the corresponding anchor point in the first image.

[0030] In an optional embodiment, the plurality of anchor points include a first anchor point, a second anchor point, and a third anchor point, and the plurality of alignment points include a corresponding first alignment point, a second alignment point, and a third alignment point;

[0031] The step of moving, rotating and scaling the second image so that each alignment point in the second image is aligned with a corresponding anchor point in the first image comprises:

[0032] Performing movement processing on the second image after the perspective correction processing, so that the first alignment point is aligned with the corresponding first anchor point in the first image, and determining a first target point;

[0033] Rotate the second image so that the second alignment point, the second anchor point corresponding to the second alignment point, and the first target point are collinear;

[0034] Scaling the second image so that the second alignment point is aligned with the second anchor point to determine a second target point;

[0035] The second image is rotated around a straight line formed by the first alignment point and the second alignment point, so that the third alignment point and a third anchor point corresponding to the third alignment point are aligned, and a third target point is determined.

[0036] In a second aspect, the present invention provides an image alignment anchoring device, the device comprising:

[0037] A first acquisition module, used to acquire a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes an anchor point corresponding to each of the feature objects;

[0038] An acquisition module, used to obtain the position information of each anchor point;

[0039] A second acquisition module is used to acquire a second image captured by a second camera, determine a plurality of alignment points in the second image, and obtain position information of each of the alignment points, where the number of the alignment points is the same as the number of the anchor points;

[0040] The alignment processing module is used to align the first image and the second image based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image composed of the second image and the first image.

[0041] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of the method described in any one of the aforementioned implementation modes are implemented.

[0042] The beneficial effects of the embodiments of the present invention include, for example:

[0043] The present invention provides an image alignment and anchoring method, device and electronic device, which obtain a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, the first image includes anchor points corresponding to each feature object, and the position information of each anchor point is obtained. A second image captured by a second camera is obtained, a plurality of alignment points in the second image are determined, and the position information of each alignment point is obtained. The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image consisting of the first image and the second image. In this solution, the alignment and fusion of the two images are achieved through the anchor points in the first image and the alignment points in the second image, and the alignment can be achieved by processing a very small number of points, which greatly reduces the processing volume and can meet the requirements of high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of an application scenario of the image alignment anchor point method provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of alignment of a real image and a virtual image in an embodiment of the present invention;

[0047] Figure 3 A flowchart of an image alignment anchor point method provided by an embodiment of the present invention;

[0048] Figure 4 for Figure 3 A flowchart of the sub-steps included in S12;

[0049] Figure 5 for Figure 2 A flowchart of the sub-steps included in S14;

[0050] Figure 6 for Figure 5 A flowchart of the sub-steps included in S141;

[0051] Figure 7 for Figure 6 A flowchart of the sub-steps included in S1412;

[0052] Figure 8 for Figure 5 A flowchart of the sub-steps included in S142;

[0053] Fig. 9 A functional module block diagram of an image alignment anchor point device provided by an embodiment of the present invention;

[0054] Fig.10 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0056] See also Figure 1 , is a schematic diagram of an application scenario of the image alignment and anchoring method provided in an embodiment of the present invention, the application scenario includes a first camera, a second camera and a main controller, wherein the first camera and the second camera can be connected to the main controller respectively to interact with the main controller for data, instructions, etc.

[0057] The first camera can be used to capture a first image, which can be an image within the user's field of view, that is, it can be understood as a real image. The second camera can be used to capture a second image, which is an image in an area that cannot be directly observed by the user's eyes, that is, it can be understood as a virtual image.

[0058] In this embodiment, the fusion of real images and virtual images can be applied in a variety of scenarios. For example, taking the driving scenario as an example, the first camera can be a camera on a wearable device, such as a camera on a wearable helmet. When the user wears the wearable helmet, the shooting range of the first camera is consistent with the field of view of the user's human eyes, for example, it can be the range in front of the driver's seat (including the front and side front), side, etc. The second camera can be a camera set at other positions of the vehicle, used to capture images of areas that the user cannot directly observe from the driver's seat, such as images of the vehicle chassis position, images of the trunk position, etc.

[0059] In addition, the application scenario may also include an AR projection screen, which may move with the first camera or be separated from the first camera. For example, the AR projection screen may be set on a wearable helmet, such as the front of the wearable helmet. In this way, the AR projection screen moves with the first camera, or the AR projection screen may also be set on a vehicle, such as at the dashboard or windshield of the vehicle.

[0060] AR screen projection can be used to project and display a first image, a second image, and a superimposed image after alignment and fusion processing of the first image and the second image.

[0061] like Figure 2 As shown in , the first camera has a field of view consistent with that of the human eye, and the first camera captures a real image in which an image anchor point is calibrated. The image anchor point can be used as a reference for alignment and fusion of the real image and the virtual image, thereby achieving alignment of the real image and the virtual image and displaying it on the AR projection screen.

[0062] See also Figure 3 , is a flow chart of an image alignment anchoring method provided by an embodiment of the present invention, the image alignment anchoring method can be executed by an image alignment anchoring device, the image alignment anchoring device can be implemented by software and / or hardware, and can be configured in an electronic device, the electronic device can be a computer device, a server, a main controller, etc. The detailed steps of the image alignment anchoring method are introduced as follows.

[0063] S11, acquiring a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes anchor points corresponding to the feature objects.

[0064] S12: Obtaining position information of each anchor point.

[0065] S13, acquiring a second image captured by a second camera, determining a plurality of alignment points in the second image, and obtaining position information of each of the alignment points, wherein the number of the alignment points is the same as the number of the anchor points.

[0066] S14, performing alignment processing on the first image and the second image based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image consisting of the second image and the first image.

[0067] In this embodiment, after the first camera is started, the first camera can capture a first image within a shooting range. When the first camera is a camera set on a wearable helmet, the position of the first camera is consistent with the position of a human eye, so the shooting range of the first camera is consistent with the field of view of the human eye. In this way, the captured first image is an image within the actual field of view.

[0068] In order to achieve image alignment, in this embodiment, multiple feature objects are set within the real field of view, and the position of the feature objects only needs to be within the field of view of the human eye, for example, they can be located on the hood of the vehicle. The feature objects can be in various forms, such as raised markers, spheres, or other forms, as long as they can serve as markings.

[0069] A feature object actually existing in the real field of view has an image corresponding to the feature object in the first image after the first image is captured. In this embodiment, the feature object existing in the first image is called an anchor point.

[0070] After the anchor point in the first image is determined, position information of the anchor point may be obtained, where the position information is position information of the anchor point in the first image.

[0071] In addition, the second camera may also synchronously capture the second image. In this embodiment, the first camera and the second camera have the same capture frequency, that is, the first camera and the second camera capture images at the same time point.

[0072] Alignment points are determined in the second image, wherein the number of the alignment points is consistent with the number of the anchor points, and position information of the alignment points in the second image is preset and is set according to the position information of the anchor points in the first image. Specifically, each alignment point has a corresponding anchor point, and after determining the position information of each anchor point, based on the relationship between the first image and the second image in spatial position, the position information of the corresponding alignment point is set according to the position information of the anchor point.

[0073] In this way, when the position information of each anchor point in the first image and the position information of each alignment point in the second image are determined, the first image and the second image can be aligned based on the position information of each anchor point and the position information of each alignment point. After the alignment process, the first image and the second image can be fused to obtain a superimposed image, and finally the superimposed image can be displayed on the AR projection screen to achieve superimposed display of the real image and the virtual image.

[0074] The image alignment and anchoring method provided in this embodiment does not need to perform recognition processing on the image as in the traditional method to achieve image alignment and fusion based on the recognition processing results. Instead, it only needs to process a very small number of anchor points and alignment points in the image to achieve alignment and superposition of two images, which greatly reduces the processing workload and thus speeds up the processing speed, and can meet the needs of high real-time performance.

[0075] See also Figure 4 In this embodiment, the step of obtaining the position information of each anchor point can be implemented in the following manner:

[0076] S121, obtaining the relative position relationship between each of the feature objects and the first camera, and obtaining the position information of the first camera.

[0077] S122: Obtain position information of an anchor point corresponding to each of the feature objects according to the position information of the first camera and the relative position relationship between each of the feature objects and the first camera.

[0078] Since the position of each feature object is fixed, when the user wears a wearable helmet, the user's position is relatively fixed, such as being in the driving position, so the relative position relationship between the first camera on the wearable helmet and each feature object can be determined. In this embodiment, the relative position relationship between each feature object and the first camera can be saved in the main controller.

[0079] On this basis, the inertial navigation unit in the first camera can obtain the position information of the first camera, and the position information includes position information and attitude information. In this way, based on the position information of the first camera and the relative position relationship between the first camera and the feature object, the position information of the anchor point corresponding to the feature object can be obtained.

[0080] See also Figure 5 On the basis of the above, the step of aligning the first image and the second image based on the position information of each anchor point and the position information of each alignment point can be implemented in the following way:

[0081] S141: Perform perspective correction processing on the second image.

[0082] S142: aligning the first image with the second image based on the position information of each anchor point and the position information of each alignment point in the second image after the perspective correction process.

[0083] Since the second image is obtained by taking pictures with the second camera, and the first image is obtained by taking pictures with the first camera, and the coordinate systems of the first camera and the second camera are not the same, that is, the first image and the second image are in different coordinate systems. If the first image and the second image are to be superimposed and fused, it is necessary to ensure that the first image and the second image are in the same coordinate system to ensure the accuracy of the superposition. Therefore, in this embodiment, the second image is subjected to perspective correction processing so that the second image after perspective correction processing and the first image are based on the same coordinate system. On this basis, the first image and the second image are aligned.

[0084] See also Figure 6 In this embodiment, the step of performing perspective correction processing on the second image can be implemented in the following manner:

[0085] S1411, obtain the relative position relationship between the first camera and the AR projection screen.

[0086] S1412: Convert the second image from the coordinate system of the second camera to the coordinate system of the first camera.

[0087] S1413, based on the relative position relationship between the first camera and the AR projection screen, project the converted second image onto the AR projection screen to achieve perspective correction processing of the second image.

[0088] Because the first image and the second image need to be eventually projected onto the AR projection screen for display, when the perspective correction is performed on the second image, the correction is performed during the process of projecting the second image onto the AR projection screen.

[0089] Specifically, the relative position relationship between the first camera and the AR projection screen can be obtained. In a possible implementation, if the AR projection screen and the first camera move with each other, for example, the AR projection screen and the first camera are both set on a wearable helmet, then the relative position relationship between the AR projection screen and the first camera is fixed and unchanged after being determined.

[0090] In another possible implementation, the AR projection screen does not move with the first camera, for example, the AR projection screen is set near the dashboard or on the windshield. In this case, the position of the AR projection screen is fixed. In this case, the step of obtaining the relative position relationship between the first camera and the AR projection screen can be implemented in the following way:

[0091] The real-time posture information of the first camera is obtained by an inertial navigation unit arranged in the first camera; the position information of the AR projection screen is obtained; and the relative position relationship between the two is obtained according to the real-time posture information of the first camera and the position information of the AR projection screen.

[0092] When the AR projection screen is set near the dashboard or on the windshield, the position of the AR projection screen is fixed, and the position information of the AR projection screen can be saved in the main controller. Since the first camera is located on the wearable helmet, when the user wears the wearable helmet, the position of the first camera will change with the user's movements. The real-time position information of the first camera can be obtained through the inertial navigation unit in the first camera.

[0093] In this way, the relative position relationship between the two can be obtained based on the position information in the posture information of the first camera and the position information of the AR projection.

[0094] In this embodiment, when the positions of the first camera and the second camera are determined, the coordinate systems of the first camera and the second camera are fixed, that is, they can be coordinate systems constructed with the first camera and the second camera as coordinate origins respectively. The position information of the first image is the position information in the coordinate system of the first camera, and the position information of the second image is the position information in the coordinate system of the second camera.

[0095] Therefore, the second image can be converted from the coordinate system where the second camera is located to the coordinate system where the first camera is located.

[0096] See also Figure 7 In this embodiment, the step of converting the second image from the coordinate system of the second camera to the coordinate system of the first camera can be implemented in the following manner:

[0097] S14121. Based on the position information of the first camera and the position information of the second camera, obtain a transformation matrix between a coordinate system of the first camera and a coordinate system of the second camera.

[0098] S14122: Obtain position information of the second image in the coordinate system where the second camera is located.

[0099] S14123: Based on the transformation matrix and the position information of the second image in the coordinate system where the second camera is located, obtain the position information of the second image in the coordinate system where the first camera is located.

[0100] In this embodiment, the conversion matrix between the coordinate system of the first camera and the coordinate system of the second camera can be determined by pre-calibration. For example, the first camera and the second camera can be used to shoot a calibration plate (such as a chessboard) in advance, and the internal parameter matrix of the first camera and the second camera can be calculated based on the captured image using OpenC or a calibration algorithm, and the internal parameter matrix includes focal length, principal point coordinates, etc.

[0101] By matching the images taken by the first camera and the second camera (i.e., matching the corresponding points in the two images), a basic matrix (characterizing the relationship between the corresponding points) is obtained. The basic matrix and the intrinsic matrix are combined to obtain the essential matrix, and the essential matrix is ​​decomposed to obtain singular values. The rotation matrix and the translation vector are obtained based on the singular values. The transformation matrix between the first camera and the second camera can be obtained by combining the rotation matrix and the translation vector.

[0102] On this basis, when the position information of the second image in the coordinate system where the second camera is located is known, the position information of the second image can be converted to the position information in the coordinate system where the first camera is located according to the conversion matrix.

[0103] When the position information of the second image in the coordinate system of the first camera is obtained, the second image can be projected onto the AR projection screen based on the relative position relationship between the first camera and the AR projection screen. At this time, the second image projected on the AR projection screen has been processed with perspective correction and is in the same coordinate system as the first image projected on the AR projection screen.

[0104] On this basis, the first image and the second image can be aligned based on the position information of each anchor point in the first image and the position information of each alignment point in the second image after the perspective correction processing. Specifically, the second image after the perspective correction processing can be moved, rotated, and scaled so that each alignment point in the second image is aligned with the corresponding anchor point in the first image.

[0105] In this embodiment, the multiple anchor points in the first image include a first anchor point, a second anchor point, and a third anchor point, and the multiple alignment points in the second image include the corresponding first alignment point, second alignment point, and third alignment point.

[0106] See also Figure 8 When the alignment process of the first image and the second image is realized, it can be realized in the following ways:

[0107] S1421: Move the second image after the perspective correction process to align the first alignment point with the corresponding first anchor point in the first image, and determine a first target point.

[0108] S1422: Rotate the second image so that the second alignment point, the second anchor point corresponding to the second alignment point, and the first target point are collinear.

[0109] S1423: Scaling the second image to align the second alignment point with the second anchor point, and determining a second target point.

[0110] S1424: Rotate the second image around a straight line formed by the first alignment point and the second alignment point to align the third alignment point and a third anchor point corresponding to the third alignment point, and determine a third target point.

[0111] In this embodiment, after the first alignment point is aligned with the first anchor point, the alignment position can be marked as the first target point on the AR projection screen. Similarly, after the second alignment point is aligned with the second anchor point, the alignment position is marked as the second target point, and after the third alignment point is aligned with the third anchor point, the alignment position is marked as the third target point.

[0112] After the first image and the second image are aligned based on the above method, the superimposed image of the two is displayed on the AR projection screen.

[0113] In this embodiment, the first image and the second image at each time point on the time axis are superimposed in sequence in the above manner to obtain a superimposed image, so that a dynamically changing superimposed image is displayed on the AR projection screen.

[0114] In the image alignment and anchoring method provided in this embodiment, an image within the external real field of view is collected by a first camera, and the characteristic objects therein are identified to be marked as anchor points in the image as a reference for image positioning. Alignment is performed based on the preset alignment points in the virtual image and the anchor points of the reference to align and position the entire image. Moreover, fusion is achieved by aligning three anchor points with three alignment points, which satisfies full positioning of six degrees of freedom.

[0115] Based on the same inventive concept, please refer to Fig. 9 , showing a schematic diagram of functional modules of an image alignment anchoring device provided in an embodiment of the present invention, this embodiment can divide the functional modules of the image alignment anchoring device according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic, which is only a logical functional division, and there may be other division methods in actual implementation.

[0116] For example, when each functional module is divided into corresponding functional modules, Fig. 9 The image alignment anchor device shown is only a schematic diagram of the device. The image alignment anchor device may include a first acquisition module, an acquisition module, a second acquisition module and an alignment processing module. The functions of each functional module of the image alignment anchor device are described in detail below.

[0117] A first acquisition module, used to acquire a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes an anchor point corresponding to each of the feature objects;

[0118] An acquisition module, used to obtain the position information of each anchor point;

[0119] A second acquisition module is used to acquire a second image captured by a second camera, determine a plurality of alignment points in the second image, and obtain position information of each of the alignment points, where the number of the alignment points is the same as the number of the anchor points;

[0120] The alignment processing module is used to align the first image and the second image based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image composed of the second image and the first image.

[0121] It can be understood that the above-mentioned first acquisition module, acquisition module, second acquisition module and alignment processing module can be used to execute the above-mentioned S11 to S14. The detailed implementation method of the first acquisition module, acquisition module, second acquisition module and alignment processing module can refer to the relevant contents of the above-mentioned S11 to S14.

[0122] In a possible implementation, the above-mentioned obtaining module may be specifically used for:

[0123] Obtaining the relative position relationship between each of the feature objects and the first camera, and obtaining the position information of the first camera;

[0124] According to the posture information of the first camera and the relative position relationship between each of the feature objects and the first camera, the position information of the anchor point corresponding to each of the feature objects is obtained.

[0125] In a possible implementation, the alignment processing module may be used to:

[0126] performing perspective correction processing on the second image;

[0127] The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point in the second image after the perspective correction processing.

[0128] In a possible implementation, the alignment processing module may be used to perform perspective correction processing on the second image in the following manner:

[0129] Obtaining a relative position relationship between the first camera and the AR projection screen;

[0130] Converting the second image from the coordinate system of the second camera to the coordinate system of the first camera;

[0131] Based on the relative position relationship between the first camera and the AR projection screen, the converted second image is projected onto the AR projection screen to achieve perspective correction processing of the second image.

[0132] In a possible implementation, the alignment processing module may be used to obtain the relative position relationship between the first camera and the AR projection screen in the following manner:

[0133] Obtaining real-time position information of the first camera through an inertial navigation unit disposed in the first camera;

[0134] Obtaining the location information of the AR projection screen;

[0135] The relative position relationship between the two is obtained based on the real-time posture information of the first camera and the position information of the AR projection screen.

[0136] In a possible implementation, the alignment processing module is used to transform the second image from the coordinate system of the second camera to the coordinate system of the first camera in the following manner:

[0137] Based on the position information of the first camera and the position information of the second camera, obtaining a conversion matrix between a coordinate system of the first camera and a coordinate system of the second camera;

[0138] Obtaining position information of the second image in the coordinate system where the second camera is located;

[0139] Based on the conversion matrix and the position information of the second image in the coordinate system where the second camera is located, the position information of the second image in the coordinate system where the first camera is located is obtained.

[0140] In a possible implementation, the alignment processing module is used to perform alignment processing in the following manner:

[0141] Based on the position information of each anchor point and the position information of each alignment point in the second image after perspective correction, the second image is moved, rotated and scaled so that each alignment point in the second image is aligned with the corresponding anchor point in the first image.

[0142] In a possible implementation, the multiple anchor points include a first anchor point, a second anchor point, and a third anchor point, and the multiple alignment points include a corresponding first alignment point, a second alignment point, and a third alignment point;

[0143] The alignment processing module is used to perform alignment processing in the following manner:

[0144] Performing movement processing on the second image after the perspective correction processing, so that the first alignment point is aligned with the corresponding first anchor point in the first image, and determining a first target point;

[0145] Rotate the second image so that the second alignment point, the second anchor point corresponding to the second alignment point, and the first target point are collinear;

[0146] Scaling the second image so that the second alignment point is aligned with the second anchor point to determine a second target point;

[0147] The second image is rotated around a straight line formed by the first alignment point and the second alignment point, so that the third alignment point and a third anchor point corresponding to the third alignment point are aligned, and a third target point is determined.

[0148] See also Fig.10 , is a block diagram of an electronic device provided in an embodiment of the present invention, the electronic device may be the above-mentioned main controller, etc., and the electronic device includes a memory, a processor and a communication module. The memory, the processor and the communication module are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0149] The memory is used to store computer programs or data. The memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0150] The processor is used to read / write data or programs stored in the memory, and execute the image alignment and anchoring method provided by any embodiment of the present invention.

[0151] The communication module is used to establish a communication connection between the electronic device and other communication terminals through the network, and is used to send and receive data through the network.

[0152] It should be understood that Fig.10The structure shown is only a schematic diagram of the structure of the electronic device. The electronic device may also include Fig.10 More or fewer components as shown, or with Fig.10 Different configurations are shown.

[0153] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the image alignment and anchoring method provided in the above embodiment is implemented.

[0154] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned image alignment anchoring method can be executed. Regarding the process involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant description in the above-mentioned method embodiment, which will not be described in detail here.

[0155] In summary, the image alignment and anchoring method, device, and electronic device provided in the embodiments of the present invention obtain a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes anchor points corresponding to each feature object, and obtains the position information of each anchor point. A second image captured by a second camera is obtained, a plurality of alignment points in the second image are determined, and the position information of each alignment point is obtained. The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image consisting of the first image and the second image. In this solution, the alignment and fusion of the two images are achieved through the anchor points in the first image and the alignment points in the second image. Alignment can be achieved by processing a very small number of points, which greatly reduces the amount of processing and can meet the requirements of high real-time performance.

[0156] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0157] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] Furthermore, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0159] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0160] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0161] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image alignment anchoring method, characterized in that: The method comprises: Acquire a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes anchor points corresponding to the feature objects; Obtaining position information of each anchor point; Acquire a second image captured by a second camera, determine a plurality of alignment points in the second image, and obtain position information of each of the alignment points, where the number of the alignment points is the same as the number of the anchor points; The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image consisting of the second image and the first image.

2. The image alignment anchoring method according to claim 1, characterized in that: The step of obtaining the position information of each anchor point comprises: Obtaining the relative position relationship between each of the feature objects and the first camera, and obtaining the position information of the first camera; According to the posture information of the first camera and the relative position relationship between each of the feature objects and the first camera, the position information of the anchor point corresponding to each of the feature objects is obtained.

3. The image alignment and anchoring method according to claim 1, characterized in that: The step of aligning the first image and the second image based on the position information of each anchor point and the position information of each alignment point comprises: performing perspective correction processing on the second image; The first image and the second image are aligned based on the position information of each anchor point and the position information of each alignment point in the second image after the perspective correction processing.

4. The image alignment anchoring method according to claim 3, characterized in that: The step of performing perspective correction processing on the second image comprises: Obtaining a relative position relationship between the first camera and the AR projection screen; Converting the second image from the coordinate system of the second camera to the coordinate system of the first camera; Based on the relative position relationship between the first camera and the AR projection screen, the converted second image is projected onto the AR projection screen to achieve perspective correction processing of the second image.

5. The image alignment and anchoring method according to claim 4, characterized in that: The step of obtaining the relative position relationship between the first camera and the AR projection screen includes: Obtaining real-time position information of the first camera through an inertial navigation unit disposed in the first camera; Obtaining the location information of the AR projection screen; The relative position relationship between the two is obtained based on the real-time posture information of the first camera and the position information of the AR projection screen.

6. The image alignment anchoring method according to claim 4, characterized in that: The step of converting the second image from the coordinate system of the second camera to the coordinate system of the first camera includes: Based on the position information of the first camera and the position information of the second camera, obtaining a conversion matrix between a coordinate system of the first camera and a coordinate system of the second camera; Obtaining position information of the second image in the coordinate system where the second camera is located; Based on the conversion matrix and the position information of the second image in the coordinate system where the second camera is located, the position information of the second image in the coordinate system where the first camera is located is obtained.

7. The image alignment and anchoring method according to claim 3, characterized in that: The step of aligning the first image and the second image based on the position information of each anchor point and the position information of each alignment point in the second image after the perspective correction processing comprises: Based on the position information of each anchor point and the position information of each alignment point in the second image after perspective correction, the second image is moved, rotated and scaled so that each alignment point in the second image is aligned with the corresponding anchor point in the first image.

8. The image alignment anchoring method according to claim 7, characterized in that: The plurality of anchor points include a first anchor point, a second anchor point, and a third anchor point, and the plurality of alignment points include corresponding first alignment points, second alignment points, and third alignment points; The step of moving, rotating and scaling the second image so that each alignment point in the second image is aligned with a corresponding anchor point in the first image comprises: Performing movement processing on the second image after the perspective correction processing, so that the first alignment point is aligned with the corresponding first anchor point in the first image, and determining a first target point; Rotate the second image so that the second alignment point, the second anchor point corresponding to the second alignment point, and the first target point are collinear; Scaling the second image so that the second alignment point is aligned with the second anchor point to determine a second target point; The second image is rotated around a straight line formed by the first alignment point and the second alignment point, so that the third alignment point and a third anchor point corresponding to the third alignment point are aligned, and a third target point is determined.

9. An image alignment anchoring device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image within a real field of view captured by a first camera, wherein the real field of view includes a plurality of feature objects set, and the first image includes an anchor point corresponding to each of the feature objects; An acquisition module, used to obtain the position information of each anchor point; A second acquisition module is used to acquire a second image captured by a second camera, determine a plurality of alignment points in the second image, and obtain position information of each of the alignment points, where the number of the alignment points is the same as the number of the anchor points; The alignment processing module is used to align the first image and the second image based on the position information of each anchor point and the position information of each alignment point to obtain an aligned superimposed image composed of the second image and the first image.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images