Image registration method, device, terminal and storage medium

Abstract

The present disclosure relates to an image registration method, device, terminal and storage medium, the method comprising: obtaining a reference image of a photographed object and multiple target images; wherein the target image and the reference image are images generated by a terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information; and performing image registration on the multiple target images according to the depth information of the reference image. The algorithm of the method is simple, the amount of calculation is small, and it is applicable to a variety of extreme scenes (such as scenes with very dark light, scenes with many repeated features of the photographed object, scenes with unclear features of the photographed object, or scenes with large information differences of multiple images obtained by multiple camera modules, etc.), and can well complete the registration of multiple target images, facilitate the subsequent processing of multiple target images, and thus improve the shooting effect.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular to an image registration method, device, terminal and storage medium. Background Art

[0002] Currently, many terminals are equipped with dual camera modules or more camera modules. When taking images using a terminal equipped with multiple camera modules, multiple images taken by the multiple camera modules are often aligned to facilitate fusion or other processing of the multiple images.

[0003] In the related art, the registration is mainly completed by using the feature matching method, for example, the ISS feature point detection method and the FPFH feature descriptor extraction method are used to extract feature points and feature descriptors corresponding to the feature points, and then the registration of multiple images is completed based on the feature points and feature point descriptors.

[0004] However, the algorithm of the above method is completely dependent on the feature information of the image, the algorithm is relatively complex, the amount of calculation is large, and in some extreme scenarios (such as very dimly lit scenes, scenes with many repeated features of the subject, scenes with unclear features of the subject, or scenes with large differences in information between multiple images acquired by multiple camera modules, etc.), it is difficult to complete the registration of multiple images, which affects the subsequent processing of multiple images and further affects the shooting effect. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image registration method, device, terminal and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image registration method, which is applied to a terminal, and the method includes:

[0007] Acquire a reference image of a photographed object and a plurality of target images; wherein the target image and the reference image are images generated by a terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information;

[0008] Image registration is performed on the multiple target images according to the depth information of the reference image.

[0009] Optionally, performing image registration on the multiple target images according to the depth information of the reference image includes:

[0010] Determine a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module is a camera module that generates the reference image;

[0011] Image registration is performed on the multiple target images according to the point cloud image.

[0012] Optionally, performing image registration on the multiple target images according to the point cloud image includes:

[0013] Determine a first mapping relationship between coordinate information of a unit sampling point in the world coordinate system and position information thereof in the reference camera coordinate system according to the external parameter information of the reference camera module and the depth information of the reference image;

[0014] Determine a second mapping relationship between coordinate information of a unit sampling point in the reference camera coordinate system and its coordinate information in the target coordinate system according to relative position information between the target camera module and the reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module; wherein the target camera module is a camera module for generating the target image, and the target coordinate system is a camera coordinate system of the target camera module;

[0015] Image registration is performed on the multiple target images according to the point cloud image, the first mapping relationship, and the second mapping relationship.

[0016] Optionally, the target image is an RGB image or an IR image.

[0017] Optionally, the unit sampling point is a pixel point or a feature point.

[0018] According to a second aspect of an embodiment of the present disclosure, there is provided an image registration device, which is applied to a terminal, and the device includes:

[0019] An acquisition module, used to acquire a reference image of a photographed object and a plurality of target images; wherein the target image and the reference image are images generated by a terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information;

[0020] A registration module is used to perform image registration on the multiple target images according to the depth information of the reference image.

[0021] Optionally, the registration module is further used for:

[0022] Determine a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module is a camera module that generates the reference image;

[0023] Image registration is performed on the multiple target images according to the point cloud image.

[0024] Optionally, the registration module is further used for:

[0025] Determine a first mapping relationship between coordinate information of a unit sampling point in the world coordinate system and position information thereof in the reference camera coordinate system according to the external parameter information of the reference camera module and the depth information of the reference image;

[0026] Determine a second mapping relationship between coordinate information of a unit sampling point in the reference camera coordinate system and its coordinate information in the target coordinate system according to relative position information between the target camera module and the reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module; wherein the target camera module is a camera module for generating the target image, and the target coordinate system is a camera coordinate system of the target camera module;

[0027] Image registration is performed on the multiple target images according to the point cloud image, the first mapping relationship, and the second mapping relationship.

[0028] Optionally, the target image is an RGB image or an IR image.

[0029] Optionally, the unit sampling point is a pixel point or a feature point.

[0030] According to a third aspect of an embodiment of the present disclosure, a terminal is provided, the terminal comprising a plurality of camera modules, wherein the plurality of camera modules include at least one depth camera module, and the terminal further comprises:

[0031] processor;

[0032] a memory for storing processor-executable instructions;

[0033] The processor is configured to execute the method as described in the first aspect.

[0034] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the method described in the first aspect.

[0035] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: in the method, by introducing a reference image with depth information, the registration of multiple target images is completed according to the reference image. The algorithm of the method is simple, the amount of calculation is small, and it is applicable to a variety of extreme scenes (such as scenes with very dim light, scenes with many repeated features of the photographed objects, scenes with unclear features of the photographed objects, or scenes with large information differences between multiple images acquired by multiple camera modules, etc.). It can well complete the registration of multiple images, facilitate the subsequent processing of multiple images, and thus improve the shooting effect.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] Figure 1 The figure is a flowchart of an image registration method according to an exemplary embodiment.

[0039] Figure 2 The figure is a flowchart of an image registration method according to an exemplary embodiment.

[0040] Figure 3 The figure is a flowchart of an image registration method according to an exemplary embodiment.

[0041] Figure 4 The figure is a flowchart of an image registration method according to an exemplary embodiment.

[0042] Figure 5 is a block diagram of an image registration apparatus according to an exemplary embodiment.

[0043] Figure 6 is a block diagram of a terminal according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0045] The present disclosure provides an image registration method, which is applied to a terminal. In the method, by introducing a reference image with depth information, the registration of multiple target images is completed according to the reference image. The algorithm of the method is simple, the amount of calculation is small, and it is applicable to a variety of extreme scenes (such as scenes with very dark light, scenes with many repeated features of the photographed object, scenes with unclear features of the photographed object, or scenes with large information differences of multiple images obtained by multiple camera modules, etc.), and the registration of multiple images can be completed well, which is convenient for subsequent processing of multiple images, thereby improving the shooting effect.

[0046] In an exemplary embodiment, an image registration method is provided, which is applied to a terminal, and the terminal includes multiple camera modules. Figure 1 As shown, the method includes:

[0047] S110: Acquire a reference image of the photographed object and a plurality of target images.

[0048] The target image and the reference image are images generated when the terminal shoots the same object at the same shooting position, and the reference image includes depth information.

[0049] For example, the reference image is a depth image generated by a depth camera module, the target image is an RGB image (color image) generated by an RGB camera module, or the target image is an IR image (grayscale image) taken by an IR camera module (infrared camera module). In this method, multiple target images can be multiple RGB images or multiple IR images, or some can be RGB images and the other can be IR images. Of course, the target image can also be other images that do not include depth information, which will not be elaborated here.

[0050] In one embodiment, the terminal is provided with two RGB camera modules and one Depth camera module (depth camera module), wherein the Depth camera module includes a ToF sensor (laser ranging sensor). When a user uses the terminal to shoot a certain object, the two RGB camera modules generate two RGB images, and the Depth camera module generates a Depth image. The Depth image is used as a reference image, and the two RGB images are used as target images.

[0051] In one embodiment, the terminal is provided with an RGB camera module, an IR camera module and a Depth camera module. When a user uses the terminal to shoot a certain object, the RGB camera module generates an RGB image, the IR camera module generates an IR image, and the Depth camera module generates a Depth image. The Depth image is used as a reference image, and the RGB image and the IR image are used as target images.

[0052] S120: Perform image registration on multiple target images according to the depth information of the reference image.

[0053] Wherein, registration refers to the matching of geographical coordinates of different image graphics obtained by different imaging means in the same area. In this embodiment, the image registration between multiple target images is achieved by using the reference image as a reference benchmark.

[0054] Here, if an image generated by a camera module includes depth information, the camera module may be referred to as a depth camera module. For example, the camera module that generates the reference image may be a depth camera module.

[0055] In the depth camera module, since the depth image it generates includes depth information (the distance information between the photographed object and the depth camera module), the coordinate transformation of any point in the depth image in the camera coordinate system, image coordinate system, pixel coordinate system and world coordinate system corresponding to the depth camera module can be completed accurately according to the internal and external reference information of the depth camera module. Among them, the internal reference information may include an internal reference matrix and a distortion parameter matrix, which are used to describe the transformation method of transforming points between the pixel coordinate system, the image coordinate system and the camera coordinate system, and the external reference information includes a rotation matrix and a translation matrix, which are used to describe the transformation method of transforming points from the world coordinate system to the camera coordinate system.

[0056] In other camera modules (non-depth camera modules), since the images they generate do not include depth information, they can only complete the coordinate transformation of any point in the generated image in the camera coordinate system, image coordinate system and pixel coordinate system corresponding to the camera module, but it is difficult to accurately complete the coordinate transformation of any point in the above three coordinate systems and the world coordinate system.

[0057] In a terminal equipped with a depth camera module and multiple other camera modules, the relative distance information of each camera module can be obtained. According to the relative distance information between other camera modules and the depth camera module, as well as the internal and external reference information of other camera modules and the internal and external reference information of the depth camera module, the mapping relationship between the coordinates of any point in the camera coordinate system of other camera modules and its coordinates in the camera coordinate system of the depth camera module can be determined.

[0058] Based on the above principle, we can first determine the coordinate information of each unit sampling point in the reference image in the reference camera coordinate system of the reference camera module, the reference image coordinate system, the reference pixel coordinate system and any coordinate system of the world coordinate system, and then determine the coordinate information of each unit sampling point in the target image in the target camera coordinate system, the target image coordinate system and the target pixel coordinate system of the target camera. Then, based on the mapping relationship between the coordinate information of the unit sampling point in the target camera coordinate system and its coordinate information in the reference camera coordinate system, we can determine the unit sampling points corresponding to the same geographic coordinates in multiple target images and complete the registration of multiple target images.

[0059] The unit sampling point may be a pixel point or a feature point, wherein the feature point may be determined according to a method for determining the feature point when completing registration according to feature matching in the prior art, which will not be described in detail here. In this method, the principle of completing the registration of multiple target images according to pixel points is basically the same as that of completing the registration of multiple target images according to feature points, and the following mainly describes in detail pixel points as unit sampling points.

[0060] In one embodiment, the terminal is provided with two RGB camera modules and one Depth camera module; the two RGB camera modules are target camera modules, and the Depth camera module is a reference camera module. When a user uses the terminal to shoot a certain object, the two RGB camera modules generate two RGB images, and the Depth camera module generates a Depth image, and the Depth image is used as a reference image and the RGB image is used as a target image. Then, according to the depth information of the reference image, the above two RGB images are image registered, that is, the Depth image is used as a reference benchmark for image registration between the two RGB images.

[0061] In one embodiment, the terminal is provided with an RGB camera module, an IR camera module (infrared camera module) and a Depth camera module. When a user uses the terminal to shoot a certain object, the RGB camera module generates an RGB image, the IR camera module generates an IR image (grayscale image), and the Depth camera module generates a Depth image. The Depth image is used as a reference image, and then the RGB image and the IR image are image-aligned according to the depth information of the reference image. That is, the Depth image is used as a reference benchmark for image alignment between the RGB image and the IR image.

[0062] In this method, a reference image with depth information is introduced, and the registration of multiple target images is completed according to the reference image. The algorithm of this method is simple, the calculation amount is small, and it is applicable to a variety of extreme scenes (such as scenes with very dim light, scenes with many repeated features of the photographed objects, scenes with unclear features of the photographed objects, or scenes with large information differences between multiple images acquired by multiple camera modules, etc.). It can well complete the registration of multiple images, facilitate the subsequent processing of multiple images, and thus improve the shooting effect.

[0063] In an exemplary embodiment, an image registration method is provided, which is an improvement of step S120 in the above method. For example, referring to Figure 2 As shown, in this method, multiple target images are registered according to the depth information of the reference image, including:

[0064] S210, determining a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image;

[0065] S220: performing image registration on multiple target images according to the point cloud image.

[0066] In step S210 , the camera module that generates the reference image is recorded as a reference camera module.

[0067] Point cloud refers to a massive set of points that express the spatial distribution and surface characteristics of a target in the same spatial reference system. After obtaining the spatial coordinates of each unit sampling point on the surface of an object, a set of points is obtained, which is called a point cloud. The image formed by the point cloud is called a point cloud image.

[0068] In one embodiment, the first coordinate information of each pixel point in the reference image in the reference pixel coordinate system of the reference camera module can be determined first, and then the second coordinate information of each pixel point in the reference image coordinate system of the reference camera module can be determined based on the first coordinate information and the internal reference information of the reference camera module. Then, the third coordinate information of each pixel point in the reference camera coordinate system can be determined based on the second coordinate information and the internal reference information of the reference camera module. Finally, the fourth coordinate information of each pixel point in the world coordinate system can be determined based on the third coordinate information and the external reference information of the reference camera module, so as to determine the point cloud image of the reference image based on the world coordinate system. In the point cloud image, each point represents a pixel point, and the fourth coordinate information of each pixel point is the geographic coordinate. Therefore, the image registration of multiple target images can be performed based on the pixel points corresponding to the same point in the point cloud image in multiple target images.

[0069] It should be noted that in this method, the point cloud image of the reference image based on the reference camera coordinate system can also be determined according to the internal reference information of the reference camera module, and then the multiple target images can be registered according to the depth information of the point cloud image and the reference image. However, when "registering multiple target images according to the point cloud image and the depth information of the reference image", it is still necessary to determine the coordinate information of each unit sampling point in the point cloud image in the world coordinate system according to the point cloud image, the external reference information of the reference camera module and the depth information of the reference image, and then register the multiple target images according to the above coordinate information, the first mapping relationship and the second mapping relationship.

[0070] In this method, for an image generated by a certain depth camera module, a method for converting the coordinate information of each pixel in the image in the camera coordinate system, image coordinate system, pixel coordinate system and world coordinate system corresponding to the depth camera module can be achieved through existing technology, so it is not described in detail.

[0071] In an exemplary embodiment, an image registration method is provided, which is an improvement of step S220 in the above method. For example, referring to Figure 4 As shown, in the method, performing image registration on the multiple target images according to the point cloud image includes:

[0072] S310, determining a first mapping relationship between coordinate information of a unit sampling point in a world coordinate system and position information thereof in a reference camera coordinate system according to external parameter information of a reference camera module and depth information of a reference image;

[0073] S320, determining a second mapping relationship between coordinate information of a unit sampling point in a reference camera coordinate system and its coordinate information in a target coordinate system according to relative position information between the target camera module and the reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module;

[0074] S330: Perform image registration on multiple target images according to the point cloud image, the first mapping relationship, and the second mapping relationship.

[0075] In this method, the target camera module may be a camera module that generates a target image, and the target camera coordinate system may be a camera coordinate system of the target camera module.

[0076] In this method, steps S310 and S320 are not executed in any order, that is, the two can be performed simultaneously, sequentially, or in reverse order. In this method, after determining the first mapping relationship and the second mapping relationship, the target mapping relationship between the coordinate information of the sampling point in the world coordinate system and its coordinate information in the target coordinate system can be determined. Based on the above target mapping relationship, the coordinate information corresponding to any unit sampling point in the point cloud image in the target coordinate system can be determined, and the unit sampling point in the target coordinate system is in a one-to-one correspondence with the unit sampling point of the target image. Therefore, the unit sampling points corresponding to any unit sampling point in the point cloud image in multiple target images can be determined, and the image registration of multiple target images is completed.

[0077] In one implementation, the unit sampling point may be a pixel point.

[0078] The first step: first determine the coordinate information of each pixel point of the target image in the target coordinate system. This process can be implemented according to the existing technology.

[0079] For example: the fifth coordinate information in the target pixel coordinate system of the target camera module, and then the sixth coordinate information of each pixel point in the target image coordinate system of the target camera module is determined based on the fifth coordinate information and the internal parameter information of the target camera module, and finally the seventh coordinate information of each pixel point in the target camera coordinate system is determined based on the sixth coordinate information and the internal parameter information of the target camera module. The seventh coordinate information is the coordinate information of each pixel point of the target image in the target coordinate system.

[0080] For another example: first, based on the internal parameter information of the target camera module, the third mapping relationship between the fifth coordinate information of any pixel point in the target pixel coordinate system and its sixth coordinate information in the target image coordinate system can be determined, and then the fourth mapping relationship between the sixth coordinate information of any pixel point in the target image coordinate system and its seventh coordinate information in the target camera coordinate system can be determined. Then, based on the above third mapping relationship and fourth mapping relationship, the seventh coordinate information of each pixel point in the target image in the target camera coordinate system can be determined. The seventh coordinate information is the coordinate information of each pixel point of the target image in the target coordinate system.

[0081] Step 2: Determine the second mapping relationship between the coordinate information of the pixel point in the reference camera coordinate system and its coordinate information in the target coordinate system according to the relative position information between the target camera module and the reference camera module, the internal and external parameter information of the target camera module, and the internal and external parameter information of the reference camera module.

[0082] Step 3: Determine a first mapping relationship between the coordinate information of the pixel point in the world coordinate system and its position information in the reference camera coordinate system according to the external parameter information of the reference camera module and the depth information of the reference image.

[0083] Step 4: According to the first mapping relationship and the second mapping relationship, determine the target mapping relationship between the coordinate information of the pixel point in the world coordinate system and its position information in the target coordinate system. Based on this, the coordinate information of the pixel point in the target image in the world coordinate system can be determined, and the geographic coordinates of each pixel point in the target image can be determined. The pixel points corresponding to the same coordinate information in the world coordinate system in multiple target images are aligned to complete the image alignment of multiple target images.

[0084] The algorithm of this method is simple, the calculation amount is small, and it is applicable to a variety of extreme scenes (such as scenes with very dim light, scenes with many repeated features of the subject, scenes with unclear features of the subject, or scenes with large differences in information of multiple images acquired by multiple camera modules, etc.). It can well complete the registration of multiple target images, facilitate the subsequent processing of multiple target images, and thus improve the shooting effect.

[0085] In an exemplary embodiment, an image registration method is provided, referring to Figure 4 As shown, the method includes:

[0086] S410: Acquire a reference image of the photographed object and a plurality of target images.

[0087] S420, determining a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module may be a camera module for generating the reference image;

[0088] S430, determining coordinate information of each pixel point of the target image in a target coordinate system; wherein the target camera module may be a camera module for generating the target image, and the target coordinate system may be a camera coordinate system of the target camera module;

[0089] S440, determining a target second mapping relationship between coordinate information of any pixel point in the reference camera coordinate system and its coordinate information in the target coordinate system according to the relative distance information between the target camera module and the reference camera module, the internal reference information and the external reference information of the target camera module, and the internal reference information and the external reference information of the reference camera module;

[0090] S450, determining a first mapping relationship between coordinate information of any pixel point in the world coordinate system and position information thereof in the reference camera coordinate system according to the external reference information of the reference camera module and the depth information of the reference image;

[0091] S460, determining a target mapping relationship between coordinate information of any pixel point in the world coordinate system and position information thereof in the target coordinate system according to the first mapping relationship and the second mapping relationship;

[0092] S470: performing image registration on multiple target images according to the target mapping relationship and the coordinate information of each pixel point of the target image in the target coordinate system.

[0093] In this method, by introducing a reference image with depth information, the registration of multiple target images is achieved directly according to the pixel points and the reference image, and there is no need to use feature information for image matching. This makes the algorithm of this method simpler and less computationally intensive. It is also applicable to a variety of extreme scenarios (such as scenes with very dim light, scenes with many repeated features of the photographed objects, scenes with unclear features of the photographed objects, or scenes with large information differences between multiple images acquired by multiple camera modules, etc.). It can well complete the registration of multiple target images, facilitate subsequent processing of multiple target images, and thus improve the shooting effect.

[0094] For example, since the method completes the precise registration of multiple target images based on pixel points and reference images, after the multiple target images are fused, a clearer fused image can be obtained, thereby improving the image quality.

[0095] For another example, since this method does not need to achieve registration of multiple target images based on feature matching, but instead achieves registration of multiple target images based on pixel points and reference images, even in a very dark shooting environment, multiple target images can be denoised more effectively to obtain denoised images with higher image quality.

[0096] For another example, since the method realizes accurate registration of multiple target images based on pixel points and reference images, more accurate and clearer image information can be obtained when modeling and mapping are performed based on the multiple target images, thereby achieving better modeling and mapping effects.

[0097] The present disclosure also proposes an image registration device, which is applied to a terminal. The device is used to implement the above-mentioned image registration method. In the device, when performing registration of multiple target images, the algorithm is simple, the amount of calculation is small, and it is applicable to a variety of extreme scenes (such as scenes with very dark light, scenes with many repeated features of the photographed object, scenes with unclear features of the photographed object, or scenes with large information differences between multiple images obtained by multiple camera modules, etc.), which can well complete the registration of multiple target images, facilitate the subsequent processing of multiple target images, and thus improve the shooting effect.

[0098] In an exemplary embodiment, an image registration device is provided, referring to Figure 5 As shown, the device includes an acquisition module 101 and a registration module 102. In the process of implementing the above-mentioned image registration method, the device:

[0099] The acquisition module 101 is used to acquire a reference image of a photographed object and multiple target images; wherein the target image and the reference image are images generated by the terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information;

[0100] The registration module 102 is used to perform image registration on multiple target images according to the depth information of the reference image.

[0101] In an exemplary embodiment, an image registration device is provided, which is an improvement of the above device. For example, referring to Figure 5 As shown, in the device, the registration module 102 is also used for:

[0102] Determine a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module may be a camera module for generating the reference image;

[0103] Based on the point cloud image, multiple target images are registered.

[0104] In an exemplary embodiment, an image registration device is provided, which is an improvement of the above device. For example, referring to Figure 5 As shown, in the device, the registration module 102 is also used for:

[0105] Determine a first mapping relationship between coordinate information of a unit sampling point in a world coordinate system and position information thereof in a reference camera coordinate system according to external parameter information of the reference camera module and depth information of the reference image;

[0106] Determine a second mapping relationship between coordinate information of a unit sampling point in a reference camera coordinate system and its coordinate information in a target coordinate system according to relative position information between a target camera module and a reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module; wherein the target camera module may be a camera module for generating a target image, and the target coordinate system may be a camera coordinate system of the target camera module;

[0107] Image registration is performed on the multiple target images according to the point cloud image, the first mapping relationship and the second mapping relationship.

[0108] In an exemplary embodiment, an image registration device is provided, which is an improvement of the above-mentioned device. For example, in the device, the target image may be an RGB image or an IR image.

[0109] In an exemplary embodiment, an image registration device is provided, which is an improvement of the above-mentioned device. For example, in the device, the unit sampling point can be a pixel point or a feature point.

[0110] In an exemplary embodiment, a terminal is provided, for example, the terminal may be a mobile phone, a tablet computer, a laptop computer, a camcorder, a still camera, or the like having multiple (i.e., two or more than two) camera modules, wherein the multiple camera modules include a depth camera module.

[0111] refer to Figure 6 As shown, terminal 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0112] The processing component 402 generally controls the overall operation of the terminal 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0113] The memory 404 is configured to store various types of data to support operations at the terminal 400. Examples of such data include instructions for any application or method operating on the terminal 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0114] Power component 406 provides power to various components of terminal 400. Power component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to terminal 400.

[0115] The multimedia component 408 includes a screen that provides an output interface between the terminal 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera module and / or a rear camera module. When the terminal 400 is in an operating mode, such as a shooting mode or a video mode, the front camera module and / or the rear camera module may receive external multimedia data. Each front camera module and the rear camera module may be a fixed optical lens system or have a focal length and optical zoom capability.

[0116] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the terminal 400 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0117] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0118] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the terminal 400. For example, the sensor assembly 414 can detect the open / closed state of the terminal 400, the relative positioning of the components, such as the display and keypad of the terminal 400, and the sensor assembly 414 can also detect the position change of the terminal 400 or a component of the terminal 400, the presence or absence of contact between the user and the terminal 400, the orientation or acceleration / deceleration of the terminal 400, and the temperature change of the terminal 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0119] The communication component 416 is configured to facilitate wired or wireless communication between the terminal 400 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0120] In an exemplary embodiment, terminal 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0121] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of a terminal 400 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to perform the method shown in the above embodiment.

[0122] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

[0123] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image registration method, applied to a terminal, characterized in that: The method comprises: Acquire a reference image of a photographed object and a plurality of target images; wherein the target image and the reference image are images generated by a terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information; Performing image registration on the multiple target images according to the depth information of the reference image; The performing image registration on the multiple target images according to the depth information of the reference image comprises: Determine a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module is a camera module that generates the reference image; Image registration is performed on the multiple target images according to the point cloud image.

2. The image registration method according to claim 1, characterized in that: The performing image registration on the multiple target images according to the point cloud image comprises: Determine a first mapping relationship between coordinate information of a unit sampling point in the world coordinate system and position information thereof in the reference camera coordinate system according to the external parameter information of the reference camera module and the depth information of the reference image; Determine a second mapping relationship between coordinate information of a unit sampling point in the reference camera coordinate system and its coordinate information in the target coordinate system according to relative position information between the target camera module and the reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module; wherein the target camera module is a camera module for generating the target image, and the target coordinate system is a camera coordinate system of the target camera module; Image registration is performed on the multiple target images according to the point cloud image, the first mapping relationship, and the second mapping relationship.

3. The image registration method according to any one of claims 1 to 2, characterized in that: The target image is an RGB image or an IR image.

4. The image registration method according to claim 2, characterized in that: The unit sampling point is a pixel point or a feature point.

5. An image registration device, applied to a terminal, characterized in that: The device comprises: An acquisition module, used to acquire a reference image of a photographed object and a plurality of target images; wherein the target image and the reference image are images generated by a terminal photographing the same photographed object at the same photographing position, and the reference image includes depth information; A registration module, used for performing image registration on the multiple target images according to the depth information of the reference image; The registration module is further used for: Determine a point cloud image of the reference image based on a world coordinate system according to the internal reference information and external reference information of the reference camera module and the depth information of the reference image; wherein the reference camera module is a camera module that generates the reference image; Image registration is performed on the multiple target images according to the point cloud image.

6. The image registration device according to claim 5, characterized in that: The registration module is further used for: Determine a first mapping relationship between coordinate information of a unit sampling point in the world coordinate system and position information thereof in the reference camera coordinate system according to the external parameter information of the reference camera module and the depth information of the reference image; Determine a second mapping relationship between coordinate information of a unit sampling point in the reference camera coordinate system and its coordinate information in the target coordinate system according to relative position information between the target camera module and the reference camera module, internal reference information and external reference information of the target camera module, and internal reference information and external reference information of the reference camera module; wherein the target camera module is a camera module for generating the target image, and the target coordinate system is a camera coordinate system of the target camera module; Image registration is performed on the multiple target images according to the point cloud image, the first mapping relationship, and the second mapping relationship.

7. The image registration device according to any one of claims 5 to 6, characterized in that: The target image is an RGB image or an IR image.

8. The image registration device according to claim 6, characterized in that: The unit sampling point is a pixel point or a feature point.

9. A terminal, characterized in that: The terminal includes a plurality of camera modules, wherein the plurality of camera modules include at least one depth camera module, and the terminal further includes: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the image registration method as described in any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the image registration method as described in any one of claims 1 to 4.

Images