Method and apparatus for image registration

Abstract

A method and apparatus for image registration are disclosed. The image registration apparatus includes at least one processor and is configured to project a first image generated based on an image obtained from a first camera to a first model to generate a first intermediate image, map the first intermediate image to a first output model to generate a first output image, project a second image generated based on an image obtained from a second camera to a second model to generate a second intermediate image, map the second intermediate image to a second output model to generate a second output image, and determine a matching rate between the first output image and the second output image, and transform at least one of the first model and the second model based on the determined matching rate and a preset reference matching rate.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2021-0102897, filed on August 5, 2021, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This disclosure relates, in some embodiments, to methods and apparatus for image registration. Background Technology

[0004] The statements in this section provide background information in connection with this disclosure only and do not necessarily constitute prior art.

[0005] As is well known, vehicles have surround view monitoring (SVM) systems, which are functions of imaging and displaying the environment around the vehicle so that the driver can easily visually inspect the environment.

[0006] The vehicle SVM system uses cameras mounted at the front, rear, left, and right sides of the vehicle to capture images of the surrounding environment. These images are then registered in real-time as a top-down view and displayed on the in-vehicle output screen, as if the driver were looking at the vehicle from above. This helps the driver accurately assess the vehicle's surroundings by displaying images with ambient information, allowing for easier parking without needing to check side or rearview mirrors.

[0007] Newer vehicles employ a 3D vehicle SVM system, which can display its surroundings in a three-dimensional manner.

[0008] To provide a 3D image of the area around a vehicle, multiple images captured by multiple cameras are mapped onto a 3D model and output. A 3D image of the area around a vehicle can be generated by combining multiple 3D output images based on the position, orientation, and focal length of a predetermined camera viewpoint.

[0009] Three-dimensional images obtained by combining multiple images require combining these images and establishing good registration between adjacent images. However, because the images are captured by different cameras at different angles and positions, mismatches can occur in the boundary regions of adjacent images. This mismatch exacerbates perceptual inconsistencies in images of the vehicle's surroundings and reduces image discernibility, thus hindering the driver from accurately identifying the situation around the vehicle. Summary of the Invention

[0010] According to at least one embodiment, the present disclosure provides an image registration apparatus including at least one processor configured to: project a first image generated based on an image obtained from a first camera to a first model to generate a first intermediate image; map the first intermediate image to a first output model to generate a first output image; project a second image generated based on an image obtained from a second camera to a second model to generate a second intermediate image; map the second intermediate image to a second output model to generate a second output image; and determine a matching rate between the first output image and the second output image, and transform at least one of the first model and the second model based on the determined matching rate and a preset reference matching rate.

[0011] According to at least one embodiment, the present disclosure provides an image registration method including the following steps (not necessarily in the following order): (i) projecting a first image generated based on an image obtained from a first camera to a first model to generate a first intermediate image; (ii) mapping the first intermediate image to a first output model to generate a first output image; (iii) projecting a second image generated based on an image obtained from a second camera to a second model to generate a second intermediate image; (iv) mapping the second intermediate image to a second output model to generate a second output image; and (v) determining a matching rate between the first output image and the second output image, and transforming at least one of the first model and the second model based on the determined matching rate and a preset reference matching rate. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a diagram of a configuration of an image registration apparatus according to at least one embodiment of the present disclosure.

[0013] Figure 2 is a diagram of an arrangement of cameras according to at least one embodiment of the present disclosure.

[0014] Figure 3A and Figure 3B is a diagram of an output model implemented by an image registration apparatus according to at least one embodiment of the present disclosure.

[0015] Figure 4 is a diagram of a process performed by an image registration apparatus for mapping at least one image captured by at least one camera to an output model according to at least one embodiment.

[0016] Figure 5A and Figure 5B is a diagram of a process performed by an image registration apparatus for using new texture coordinates generated by adjusting first parameters as a basis for mapping an intermediate image to an output model according to at least one embodiment.

[0017] Figure 6A and Figure 6Bis a diagram of a process performed by an image registration apparatus for using new texture coordinates generated by adjusting a third parameter as a basis for mapping an intermediate image to an output model according to another embodiment of the present disclosure.

[0018] Figure 7A and Figure 7B is a diagram of a process performed by an image registration apparatus for using new texture coordinates generated by adjusting a third parameter as a basis for mapping an intermediate image to an output model according to another embodiment of the present disclosure.

[0019] Figure 8 is a diagram of a configuration of an image registration apparatus according to another embodiment of the present disclosure.

[0020] Figure 9 is a flowchart of an image registration method according to at least one embodiment of the present disclosure.

[0021] Figure 10 is a flowchart of an image registration method according to another embodiment of the present disclosure.

[0022] Reference numerals

[0023] 100: image registration apparatus 110: processor

[0024] 120: input / output interface module 130: memory DETAILED DESCRIPTION

[0025] According to some embodiments, the present disclosure is directed to an apparatus and method for video or image registration that eliminates mismatches occurring in 3D videos or images that are combined images.

[0026] According to some embodiments, the present disclosure is directed to an apparatus and method for video or image registration that reduces perceived inconsistencies in 3D videos or images between their constituent images and improves the legibility of images.

[0027] The problems to be solved by the present disclosure are not limited to those described above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

[0028] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, although elements are shown in different drawings, the same reference numerals preferably indicate the same elements. Also, in the following description of some embodiments, detailed descriptions of related known components and functions will be omitted for the sake of clarity and conciseness, when it is deemed that such an omission will not obscure the subject matter of the present disclosure.

[0029] Also, various terminologies (e.g., first, second, A, B, (a), (b), etc.) are used for the purpose of distinguishing one component from another component only and do not necessarily indicate or suggest or imply a material, positional, or sequential or chronological relationship or order between or among the components. In the entire specification, when a part "includes" or "comprises" a component, unless there is a specific description contrary thereto, the part is intended to further include other components, rather than excluding the component. The terminologies such as "unit", "module", etc. refer to a unit for processing at least one function or operation, which can be implemented by hardware, software, or a combination thereof.

[0030] The detailed description which follows explains the disclosure with reference to the accompanying drawings, in which Figure 1 The detailed description which follows explains the disclosure with reference to the accompanying drawings, in which

[0031] Figure 1 is a diagram of a configuration of an apparatus 100 for video or image registration according to at least one embodiment of the disclosure.

[0032] As Figure 2 shown, the image registration apparatus 100 according to at least one embodiment includes a processor 110, an input / output interface module 120, and a memory 130.

[0033] The processor 110, the input / output interface module 120, and the memory 130 can transmit data to each other in the image registration apparatus 100.

[0034] The input / output interface module 120 obtains a video or an image captured by a first camera (not shown) therefrom and provides the image to the processor 110. The processor 110 processes the captured image from the first camera using operations such as lens correction, rotation, horizontal translation, or vertical translation to generate a first image.

[0035] The processor 110 obtains a first intermediate image by projecting the first image onto a first model. Here, the initial shape of the first model can be the same as the initial shape of the first output model. The first model can be transformed based on preset parameters (e.g., a first parameter, a second parameter, a third parameter, etc.).

[0036] The first intermediate image includes first intermediate sub-images each obtained by projecting an image respectively corresponding to a frame constituting the first image onto the first model. The first intermediate image can be a composite image in which the respective frames constituting the first image are combined in a time sequence with their respective first intermediate sub-images.

[0037] The processor 110 obtains the first output image by mapping the first intermediate image to the first output model. Here, the mapping can be a texture mapping. The first output model corresponds to all or part of the preset 3D model. Here, the preset 3D model can have, but is not limited to, any one of a hemispherical shape and a bowl shape.

[0038] The input / output interface module 120 obtains an image photographed by the second camera and provides it to the processor 110. The processor 110 processes the captured image from the second camera using operations such as lens correction, rotation, horizontal translation, or vertical translation to generate a second image.

[0039] The processor 110 projects the second image onto the second model to obtain a second intermediate image. Here, the initial shape of the second model can be the same as the initial shape of the second output model. The second model can be transformed based on preset parameters.

[0040] The second intermediate image includes second intermediate sub-images each obtained by the processor 110 projecting an image corresponding to each frame constituting the second image onto the second model. The second intermediate image can be a composite image in which each frame constituting the second image is combined in a time sequence with its respective second intermediate sub-image.

[0041] The processor 110 obtains the second output image by mapping the second intermediate image to the second output model. The second output image is an image obtained by the processor 110 texture-mapping the second intermediate image to the second output model. The second output model corresponds to all or part of the preset 3D model. Here, the preset 3D model can have, but is not limited to, any one of a hemispherical shape and a bowl shape.

[0042] The first output image is arranged in the preset 3D model at a position corresponding to the first output model. The second output image is arranged in the preset 3D model at a position corresponding to the second output model. The first output model and the second output model can be arranged such that they are adjacent to each other with one side of the first output model forming a boundary line with one side of the second output model. Alternatively, the first output model and the second output model can be arranged such that they have some overlapping area.

[0043] The processor 110 determines a degree or rate of matching between the first output video or image and the second output video or image arranged in the three-dimensional model. The processor 110 can determine the matching rate between them by frames of the first output video and the second output video corresponding to a preset period. Here, the preset period can be, but is not limited to, 1 / 30 seconds. The matching rate between the first output image and the second output image can be at least one of a vertical matching degree, a curvature matching degree, and a scale matching degree.

[0044] The processor 110 obtains a first screen image, and obtains a two-dimensional first comparison image from the first screen image. The first screen image is a first output image appearing on a display included in the input / output interface module 120. The first comparison image can be a frame image corresponding to a preset time point of the first screen image. Here, the preset time point can be any one of a time point for determining a vertical matching rate between the first output image and the second output image, a time point for determining a curvature matching rate therebetween, and a time point for determining a scale matching rate therebetween.

[0045] The processor 110 obtains a second screen image, and obtains a two-dimensional second comparison image from the second screen image. The second screen image is a second output image appearing on a display included in the input / output interface module 120. The second comparison image can be a frame image corresponding to a preset time point of the second screen image. Here, the preset time point can be any one of a time point for determining a vertical matching rate between the first output image and the second output image, a time point for determining a curvature matching rate therebetween, and a time point for determining a scale matching rate therebetween.

[0046] According to at least one embodiment of the present disclosure, the processor 110 determines a matching rate between the first output image and the second output image based on a comparison of feature points between the first output image and the second output image. The processor 110 sets a first region of interest (ROI, hereinafter "region of interest") in the first comparison image, and sets a second region of interest in the second comparison image. Here, the first region of interest and the second region of interest can be set as an area in which the first comparison image and the second comparison image overlap.

[0047] The processor 110 extracts first feature points from the first region of interest and second feature points from the second region of interest by using a preset algorithm. Here, the preset algorithm can be a scale-invariant feature transform (SIFT) algorithm, a speeded up robust features (SURF) algorithm, a HARRIS corner algorithm, a SUSAN algorithm, etc.

[0048] The processor 110 compares the first feature points with the second feature points to determine a matching rate between the first output image and the second output image. Here, the first feature points and the second feature points can correspond to common points of the first region of interest and the second region of interest. The matching rate between the first output image and the second output image can be determined based on a positional difference between the first feature points and the second feature points.

[0049] According to another embodiment, the processor 110 determines a matching rate between the first output image and the second output image based on a difference image between the first output image and the second output image. The processor 110 sets a third region of interest in the first comparison image and a fourth region of interest in the second comparison image. Here, the third region of interest and the fourth region of interest can be set as an overlapping area between the first comparison image and the second comparison image.

[0050] The processor 110 obtains a difference image. The difference image is an image indicating a degree of mismatch between the third region of interest and the fourth region of interest. The processor 110 determines a matching rate between the first output image and the second output image based on a distribution of pixels included in the difference image. Here, the matching rate between the first output image and the second output image can be determined based on a number of pixels included in the difference image.

[0051] When the matching rate between the first output image and the second output image is equal to or greater than a preset reference matching rate, the processor 110 determines that the first output image and the second output image are in registration, and generates the first output image and the second output image based on the first model and the second model.

[0052] When the matching rate between the first output image and the second output image is less than the preset reference matching rate, the processor 110 determines that at least one of the first output image and the second output image needs to be corrected. Thereafter, the processor 110 changes a vertical component value of a model coordinate of any one of the first model and the second model using a preset parameter, thereby transforming at least one of the first model and the second model. Here, the model coordinate can be a model coordinate expressing any one of the first model and the second model.

[0053] The direction in which the processor 110 projects any one of the first image and the second image onto any one of the first model and the second model can be set as a vertical direction of the model. Here, the vertical direction of the model can be a Z-axis direction of a Cartesian coordinate system expressing any one of the first model and the second model.

[0054] When the vertical direction of the model is set as the Z-axis of the Cartesian coordinate system, the processor 110 can change a Z-axis component value of a model coordinate expressing any one of the first model and the second model using a preset parameter, thereby transforming at least one of the first model and the second model. The processor 110 performs model transformation by multiplying the Z-axis component value of the model coordinate expressing any one of the first model and the second model by a C value as a preset parameter, respectively.

[0055] As shown in Equation 1, the processor 110 can transform any one of the first model and the second model by using a preset parameter C value, and obtain the first intermediate image and the second intermediate image having new texture coordinates using any one of the transformed first model and the second model.

[0056]

[0057] Here, X, Y, and Z are coordinates of a Cartesian coordinate system expressing the first model and the second model. 'q' denotes a value related to a new texture coordinate obtained based on any one of the transformed first model and the second model. 'r' denotes a value related to image rotation in a process of generating the first image or the second image based on an image captured by a camera. 't' denotes a value related to horizontal or vertical image movement in the process of generating the first image or the second image based on the image captured by the camera. K is a value related to the camera.

[0058] When multiplied by the C value, a value of a Z-axis component of a model coordinate expressing any one of the first model and the second model is changed. Any one of the first model and the second model is transformed based on the changed value of the Z-axis component. The processor 110 can obtain the first intermediate image or the second intermediate image having the new texture coordinates by projecting any one of the first image and the second image onto any one of the transformed first model and the second model.

[0059] The processor 110 can transform any one of the first model and the second model in various ways by adjusting the C value so that a matching rate between the first output image and the second output image is equal to or greater than a preset reference matching rate. Here, the matching rate between the first output image and the second output image can be at least one of a vertical matching rate, a curvature matching rate, and a scale matching rate.

[0060] The processor 110 transforms the model by adjusting the first parameter to impart a new slope to a side wall constituting any one of the first model and the second model. Here, an initial shape of the first model can be the same as the first output model, and an initial shape of the second model can be the same as the second output model. In addition, the initial shape of any one of the first model and the second model can be a semi-spherical shape, a bowl shape, a portion of a semi-spherical shape, or a portion of a bowl shape.

[0061] The processor 110 transforms the first model or the second model by adjusting the first parameter so that a vertical matching rate between the first output image and the second output image is equal to or greater than a preset reference vertical matching rate. Here, a C value as the first parameter can be generated based on Equation 2.

[0062]

[0063] Here, Cvertical The value of θ can be a real number between 0 and 1. The value of θ can be between 0 and 90 degrees, depending on its position in the model coordinate system. However, the possible values ​​of each variable are not limited to these details.

[0064] The processor 110 transforms the model by adjusting a second parameter to impart a new curvature to the right or left portion of the sidewalls constituting either the first or second model. Here, the initial shape of the first model can be the same as the first output model, and the initial shape of the second model can be the same as the second output model. Alternatively, the initial shape of either the first or second model can be hemispherical, bowl-shaped, a portion of a hemispherical, or a portion of a bowl.

[0065] Processor 110 adjusts the second parameter to transform the first model or the second model such that the curvature matching rate between the first output image and the second output image is equal to or greater than a preset reference curvature matching rate. Here, the C value as the second parameter can be generated based on Equation 3.

[0066]

[0067] Here, θ can have values ​​between 0 and 90 degrees, depending on its position on the model's coordinates. curvature The value is generated based on Equation 4.

[0068]

[0069] here, It is a value related to the angle between the cross section containing the model coordinates to be transformed on the model and the left and right symmetry planes of the model. The values ​​can be used independently to transform either the left or right part of the model. Using the model's left and right symmetry planes as a reference (0 degrees), the left edge of the model is set to -90 degrees, and its right edge is set to 90 degrees. In this case, It can have values ​​between -90 degrees and 90 degrees. When transforming the left edge of the model, The value changes to -90 degrees, and when the right edge of the model is transformed, The value changes to 90 degrees. α is a value that determines the degree to which the target part is transformed in the model. The value of α can be a real value between, but is not limited to, 0 and 1.

[0070] The processor 110 transforms the model by adjusting the third parameter to increase or decrease the vertical length of the model at different rates according to the height of the side wall constituting any one of the first model and the second model. For example, any one of the first model and the second model can be transformed so that the vertical length of the side wall of one model becomes larger from the lower end to the upper end. Here, the initial shape of the first model can be the same as the first output model, and the initial shape of the second model can be the same as the second output model. In addition, the initial shape of any one of the first model and the second model can be a hemisphere, a bowl, a portion of a hemisphere, or a portion of a bowl.

[0071] The processor 110 adjusts the third parameter so that the ratio matching rate between the first output image and the second output image is equal to or greater than a preset reference ratio matching rate to transform the first model or the second model. Here, the C value as the third parameter can be generated based on Equation 5.

[0072]

[0073] Here, θ can have, but is not limited to, a value between 0 degrees and 90 degrees depending on the position on the model coordinates. C weight is a value generated based on Equation 6.

[0074] C weight = cos β θ Equation 6

[0075] Here, β is a value related to the shape and the ratio of the model transformation. θ is a value related to the degree to which the model coordinates are away from the Z axis. θ is an angle value formed with the Z axis, and can have, but is not limited to, a value between 0 degrees and 90 degrees.

[0076] When the first model is transformed, the processor 110 projects the first image onto the transformed first model to obtain a first intermediate image. The first intermediate image has new texture coordinates of the transformed first model. The processor 110 maps the first intermediate image to the first output model to obtain the first output image.

[0077] When the second model is transformed, the processor 110 projects the second image onto the transformed second model to obtain a second intermediate image. The second intermediate image has new texture coordinates based on the transformed second model. The processor 110 maps the second intermediate image to the second output model to obtain the second output image.

[0078] The processor 110 can repeatedly adjust the preset parameters until the matching rate between the first output image and the second output image is equal to or greater than a preset reference matching rate. Here, the matching rate between the first output image and the second output image can be any one of a vertical matching rate, a curvature matching rate, and a proportion matching rate. The preset parameters can be any one of the first parameter, the second parameter, and the third parameter.

[0079] Above, the processor 110 has been described as generating a final screen image based on the first image and the second image. However, according to another embodiment, the processor 110 generates a final screen image based on the first to third images. According to still another embodiment, the processor 110 generates a final screen image based on the first to fourth images.

[0080] The processor 110 has been described through a process of transforming the first model to change the matching rate thereof with the second model, but the disclosure is not limited to the above description, and the second model can be individually transformed or the first model and the second model can be simultaneously transformed into different shaped models, respectively.

[0081] The image registration apparatus 100 includes an input / output interface module 120. The input / output interface module 120 obtains image data from the first camera and the second camera and outputs an image generated by the processor 110. The input / output interface module 120 is connected to the first camera or the second camera through a wired / wireless communication network (not shown).

[0082] The input / output interface module 120 can be provided to be integrated with the image registration apparatus 100. Alternatively, the input / output interface module 120 can be provided to be separated from the image registration apparatus 100 or can be provided as a separate device connected to the image registration apparatus 100. The input / output interface module 120 can include a port (e.g., a USB port) for connection to an external device.

[0083] The input / output interface module 120 can include a monitor, a touch screen, a microphone, a keyboard, a camera, an image sensor, earphones, headphones, or a touchpad.

[0084] The image registration apparatus 100 includes a memory 130 capable of storing a program for processing or control of the processor 110 and various data for operation of the image registration apparatus 100. The memory 130 can store at least one or more images generated by the processor 110, including the first image, the second image, the first intermediate image, the second intermediate image, the first output image, the second output image, the first screen image, the second screen image, and the final screen image.

[0085] Figure 2 is a diagram of an arrangement of cameras according to at least one embodiment of the disclosure.

[0086] As Figure 3A indicated, at least one of the cameras 210, 220, 230, 240 is connected to the image registration device 100.

[0087] The cameras 210, 220, 230, 240 are respectively arranged at predetermined positions of the vehicle 200. The arrangement of the cameras can be changed according to factors such as an imaging purpose, the number of the cameras 210, 220, 230, 240, and the design of the vehicle 200, for example, its length or profile.

[0088] The cameras 210, 220, 230, 240 are arranged on the front, rear, left, and right sides of the vehicle 200, etc. Here, the front camera 210 can be arranged at the center of the radiator grille of the vehicle 200, and the right camera 220 and the left camera 240 can be respectively arranged at the edges or the bottoms of the side mirrors of the vehicle 200. In addition, the rear camera 230 can be arranged at the center above the rear bumper.

[0089] Any two of the cameras 210, 220, 230, and 240 are arranged such that their optical axes form a predetermined angle. The predetermined angle can be, but is not limited to, 90 degrees.

[0090] The lenses of the cameras 210, 220, 230, and 240 can have a large angle of view, for example, a wide-angle lens or a fisheye lens.

[0091] The cameras 210, 220, 230, 240 capture images of the surroundings of the vehicle 200. At least two of the cameras 210, 220, 230, 240 can simultaneously capture images of the surroundings of the vehicle 200.

[0092] The cameras 210, 220, 230, 240 transmit the captured images to the image registration device 100. For transmission, the cameras 210, 220, 230, and 240 can be equipped with a short-range wireless communication module, for example, a Wi-Fi module, a Bluetooth module, a Zigbee module, or a UWB module. The cameras 210, 220, 230, and 240 can be provided to be integrated with the image registration device 100, but they can be separately provided.

[0093] Figure 3B And Figure 3A is a diagram of an output model implemented by the image registration device 100 according to at least one embodiment of the disclosure.

[0094] Figure 3B A bottom surface of a bowl model, which is an output model implemented by the image registration device 100, is illustrated. Figure 3A A side wall constituting the output bowl model implemented by the image registration device 100 is illustrated.

[0095] The image registration apparatus 100 maps the first intermediate image to the first output model 310, maps the second intermediate image to the second output model 320, maps the third intermediate image to the third output model 330, and maps the fourth intermediate image to the fourth output model 340, respectively.

[0096] As shown in FIG. 10, the image registration apparatus 100 combines the plurality of output models 310, 320, 330, and 340 to generate a bowl model as a composite output model. Figure 3B

[0097] The first output model 310, the second output model 320, the third output model 330, and the fourth output model 340 are arranged to be laterally adjacent to each other.

[0098] The first output model 310, the second output model 320, the third output model 330, and the fourth output model 340 are arranged to partially overlap.

[0099] The image registration apparatus can determine, as a bottom surface 300 of the bowl model, surface regions of the output models 310, 320, 330, and 340 having zero slope in the surface of the bowl model and occupying some portions of the surface of the bowl model.

[0100] The image registration apparatus can generate a bottom image visualizing the surrounding ground by using images mapped to the output models 310, 320, 330, and 340 arranged on the bottom surface 300 of the bowl model, respectively.

[0101] As shown in FIG. 11, the image registration apparatus can determine, as a side wall 350 of the bowl model, surface regions (i.e., non-bottom surface) having a non-zero slope in the surface constituting the bowl model. Here, the side wall 350 of the bowl model can include the remaining portions of the first to fourth output models 310, 320, 330, 340 excluding the zero-slope surface region of the bottom surface 300, i.e., a first output model side face 360, a second output model side face 370, a third output model side face 380, and a fourth output model side face 390. In addition, the first output model side face 360 and the second output model side face 370 can have a partial overlap 352. In addition, the second output model side face 370 and the third output model side face 380 can have a partial overlap 353. The third output model side face 380 and the fourth output model side face 390 can have a partial overlap 354. The fourth output model side face 390 and the first output model side face 360 can have a partial overlap 351. Figure 4

[0102] ​​The image registration apparatus generates a side wall image visualizing the surrounding environment by using images respectively mapped to output model sides 360, 370, 380, 390 arranged on the side wall 350 of the bowl model. The image registration apparatus generates a final screen image by combining a bottom image generated based on the bottom surface 300 of the bowl model with the side wall image generated based on the side wall 350 of the bowl model.

[0103] Figure 4 is a diagram including steps 410, 420, and 430 for processing performed by an image registration apparatus for mapping at least one image captured by at least one camera to an output model according to at least one embodiment.

[0104] As shown in Figure 5A , the image registration apparatus generates an image 400 based on an image captured by a camera. The image registration apparatus processes the captured image from the camera by operations such as lens correction, rotation, horizontal translation, or vertical translation to generate the image 400.

[0105] In step 410, the image registration apparatus projects the image 400 onto a model 411 to generate an intermediate image 412. Here, the initial shape of the model 411 can be the same as the initial shape of the output model 421. The model 411 can be transformed based on preset parameters.

[0106] In step 420, the image registration apparatus maps the intermediate image 412 to the output model 421 to generate an output image. Here, the output model 421 can correspond to all or part of a preset 3D model. The preset 3D model can have any one of a hemispherical shape and a bowl shape. The output image can be generated by the image registration apparatus that texture maps the intermediate image 412 to the output model 421.

[0107] In step 430, the image registration apparatus places the output image 432, another output image 431, still another output image 433, and yet another output image (not shown) in positions corresponding to respective output models in a 3D model. Here, the shape of the 3D model can be any one of a hemispherical shape and a bowl shape.

[0108] The image registration apparatus determines a matching rate between the output image 432 and the another output image 431. Here, the matching rate can be at least one of a vertical matching rate between the output image 432 and the another output image 431, a curvature matching rate between the output image 432 and the another output image 431, and a proportional matching rate between the output image 432 and the another output image 431.

[0109] The image registration apparatus transforms the model 411 by adjusting the parameters based on a comparison result of a matching rate between the output image 432 and the other output image 431 and a preset reference matching rate. The image registration apparatus can project the image 400 onto the transformed model to obtain a new intermediate image having new texture coordinates. The image registration apparatus generates a new output image by texture mapping the new intermediate image to the output model 421. Between the new output image and the other output image 431, a more improved registration is achieved over the model 411 before transformation.

[0110] Figure 5B and Figure 5A is a diagram of a process performed by an image registration apparatus for using new texture coordinates generated by adjusting first parameters as a basis for mapping an intermediate image to an output model according to at least one embodiment.

[0111] Figure 5B A process performed by an image registration apparatus for projecting an image onto a model 500 before transformation to generate an intermediate image and mapping the generated intermediate image to an output model 510 is shown. Figure 5A A process performed by an image registration apparatus for projecting an image onto a model 520 transformed based on first parameters to generate an intermediate image and mapping the generated intermediate image to an output model 530 is shown.

[0112] As shown in Figure 5A , the image registration apparatus projects the image onto the model 500 before transformation to generate an intermediate image. The image registration apparatus obtains texture coordinates of the intermediate image respectively corresponding to model coordinates representing the model 500, and maps the intermediate image to the output model 510 based on the obtained texture coordinates to generate an output image.

[0113] As shown in Figure 5B and Figure 5B , the image registration apparatus can adjust a vertical component value of the model coordinates based on the first parameters to transform the model 500 into a new model 520. Here, the new model 520 can have a side wall slope value changed based on the vertical component value of the new model coordinates.

[0114] Referring back to Figure 6A , the image registration apparatus projects the image onto the model 520 after transformation to generate an intermediate image. The image registration apparatus obtains new texture coordinates respectively corresponding to model coordinates representing the model 520 after transformation. The image registration apparatus maps the intermediate image to the output model 530 based on the obtained texture coordinates to generate a new output image.

[0115] The new texture coordinates generated by the image registration device based on the transformed model 520 can be the same as the coordinates obtained by vertically shifting the texture coordinates generated based on the pre-transformed model 500.

[0116] The new output image generated by the image registration device that maps the intermediate image to the output model 530 based on the new texture coordinates is as good as the output image generated by mapping the intermediate image to the output model 510 based on the pre-transformed model 500 by vertically shifting the texture coordinates.

[0117] Figure 6B And Figure 6A is a diagram of a process performed by the image registration device for using the new texture coordinates generated by adjusting the second parameter as a basis for mapping the intermediate image to the output model according to at least one embodiment.

[0118] Figure 6B A process performed by the image registration device for projecting an image onto a model 600 to generate an intermediate image and mapping the generated intermediate image to an output model 610 is shown. Figure 6A A process performed by the image registration device for projecting an image onto a new model 620 transformed based on a second parameter and mapping the generated intermediate image to an output model 630 is shown.

[0119] As shown in Figure 6A The image registration device projects an image onto the model 600 before transformation to generate an intermediate image. The image registration device obtains texture coordinates of the intermediate image that respectively correspond to model coordinates representing the model 600, and maps the intermediate image to the output model 610 based on the obtained texture coordinates to generate an output image.

[0120] As shown in Figure 6B and Figure 6B The image registration device modifies the model 600 by adjusting the vertical component values of the model coordinates based on a second parameter to generate a new model 620. The shape of the left or right portion of the model can be independently transformed according to the value of the second parameter. The new model 620 transformed based on the second parameter is shaped based on the changed vertical component values of each model coordinate such that the left model portion remains unchanged while the model wall surface has a decreasing curvature towards the right edge of the model. The model 620 transformed based on the parameter becomes a shape in which the curvature of the model side wall decreases as the distance between the model coordinates forming the model side wall increases towards the right edge of the model.

[0121] The image registration apparatus generates a new intermediate image by projecting the image onto the new model 620. Here, the projection of the image onto the new model 620 is set to be in the vertical direction of the new model 620. The image registration apparatus obtains texture coordinates of the new intermediate image, which respectively correspond to the model coordinates representing the new model 620. Here, the new intermediate image projected onto the new model 620 includes a texture of a region that widens toward its right edge.

[0122] Referring back to Figure 7A , the image registration apparatus generates an intermediate image by projecting the image onto the new model 620. The image registration apparatus obtains texture coordinates, which respectively correspond to the model coordinates representing the new model 620. The image registration apparatus maps the intermediate image to the output model 630 based on the obtained texture coordinates to generate an output image.

[0123] As for the texture of the intermediate image mapped to the output model 630, the further vertical shift texture mapping occurs relative to the texture of the intermediate image before the model transformation, the closer to the right edge of the output model. This causes the curvature of the output image to increase toward the right edge of the output model, giving a visual effect that the line in the output image 632 after the model transformation is more curved than the line in the output image 612 before the model transformation.

[0124] Figure 7B And Figure 7A is a diagram of a process performed by the image registration apparatus for using new texture coordinates generated by adjusting the third parameter as a basis for mapping an intermediate image to an output model, according to at least one embodiment.

[0125] Figure 7B A process performed by the image registration apparatus for projecting an image onto a model 700 before transformation to generate an intermediate image and mapping the generated intermediate image to an output model 710 is shown. Figure 7A A process performed by the image registration apparatus for projecting an image onto a new model 720 transformed based on a third parameter to generate an intermediate image and mapping the generated intermediate image to an output model 730 is shown.

[0126] As shown in Figure 7A , the image registration apparatus projects an image onto a model 700 before transformation to generate an intermediate image. The image registration apparatus obtains texture coordinates of the intermediate image, which respectively correspond to the model coordinates representing the model 700, and maps the intermediate image to an output model 710 based on the obtained texture coordinates to generate an output image.

[0127] As shown in Figure 7B and Figure 7BAs shown in FIG. 7B, the image registration apparatus can change the vertical component values of the model coordinates based on the third parameter to transform the model 700 into a new model 720. The vertical length of the new model 720 increases toward the top of the model side wall. The vertical length of the new model 720 increases at a constant rate as it goes upward on the side wall. Here, the constant rate of the vertical length increase can vary according to the value of the third parameter.

[0128] As shown in FIG. 7B, the image registration apparatus can change the vertical component values of the model coordinates based on the third parameter to transform the model 700 into a new model 720. The vertical length of the new model 720 increases toward the top of the model side wall. The vertical length of the new model 720 increases at a constant rate as it goes upward on the side wall. Here, the constant rate of the vertical length increase can vary according to the value of the third parameter. Figure 8 As shown in FIG. 7B, the image registration apparatus can change the vertical component values of the model coordinates based on the third parameter to transform the model 700 into a new model 720. The vertical length of the new model 720 increases toward the top of the model side wall. The vertical length of the new model 720 increases at a constant rate as it goes upward on the side wall. Here, the constant rate of the vertical length increase can vary according to the value of the third parameter.

[0129] When compared with the output image mapped to the output model 710 based on the pre-transformed model 700, the output image on the new model 720 exhibits a vertical offset that increases toward the bottom thereof. In the case where the intermediate image is mapped to the output model 730 based on the texture coordinates generated using the new model 720, the output model 730 provides a well-textured vertically expanded lower portion with just the right vertical proportion on the lower portion of the new model 720, which has a gradually shortened vertical distance. Thus, the further vertical expansion texture mapping occurs closer to the bottom of the intermediate image.

[0130] The new output image portion 732 is obtained by vertically shifting the pre-transformed output image portion 712 by different lengths at different vertical positions. Here, the image vertically shifted by different lengths at different vertical positions is an image vertically shifted by lengths weighted at a constant rate according to the vertical position.

[0131] Figure 8 FIG. 8 is a diagram of an image registration apparatus 800 according to another embodiment of the disclosure.

[0132] As shown in FIG. 8, the image registration apparatus 800 includes a microcontroller unit (MCU) 830, a deserializer 840, a video processing unit (VPU) 850, and a serializer 860. Figure 9 The microcontroller unit 830 controls the operation of the image registration apparatus 800. Upon receiving an input external signal, the microcontroller unit 830 transmits a control signal to the video processing unit 850.

[0133] The external signal of the image registration apparatus 800 can be any one of a gear signal 813 and a switch signal 816. The gear signal 813 is information that the transmission of the vehicle has engaged a specific gear. The switch signal 816 is capable of controlling the operation of the image registration apparatus 800.

[0134] The external signal of the image registration apparatus 800 can be any one of a gear signal 813 and a switch signal 816. The gear signal 813 is information that the transmission of the vehicle has engaged a specific gear. The switch signal 816 is capable of controlling the operation of the image registration apparatus 800.

[0135] Upon receiving the gear signal 813 or the switch signal 816, the microcontroller unit 830 transmits a control signal for performing image registration to the video processing unit 850.

[0136] Upon receiving the gear signal 813 including information that the vehicle has shifted from reverse to non-reverse or receiving the switch signal 816 to terminate the operation of the image registration apparatus 800, the microcontroller unit 830 transmits a control signal for terminating image registration to the video processing unit 850.

[0137] The image registration apparatus 800 includes a deserializer 840 that receives images captured by the cameras 822, 824, 826, and 828 and transmits them to the video processing unit 850.

[0138] The first camera 822, the second camera 824, the third camera 826, and the fourth camera 828 take images of the surroundings of the vehicle in different directions to generate and transmit first, second, third, and fourth images to the deserializer 840. Here, the first image can be a front side image of the vehicle, the second image can be a rear side image of the vehicle, the third image can be a left side image of the vehicle, and the fourth image can be a right side image of the vehicle.

[0139] The deserializer 840 synchronizes the first, second, third, and fourth images input in parallel. The deserializer 840 serially transmits the synchronized first to fourth images to the video processing unit 850.

[0140] The video processing unit 850 performs image registration. When receiving the control signal for performing image registration from the microcontroller unit 830, the video processing unit 850 performs image registration based on the first to fourth images transmitted by the deserializer 840.

[0141] The video processing unit 850 projects the first, second, third, and fourth images to respective ones of first, second, third, and fourth models to generate first, second, third, and fourth intermediate images. Here, the first, second, third, and fourth intermediate images include texture coordinates of their projected images, respectively. The first, second, third, and fourth models can correspond to all or part of 3D models having any one of a semi-spherical shape and a bowl shape.

[0142] The video processing unit 850 texture-maps the first, second, third, and fourth intermediate images to a corresponding one of the first, second, third, and fourth output models to generate first, second, third, and fourth output images. Here, the first, second, third, and fourth output models can correspond to all or a portion of the 3D model.

[0143] The video processing unit 850 performs image registration by placing the first, second, third, and fourth output images on the 3-D model of the first, second, third, and fourth output models, respectively.

[0144] The video processing unit 850 determines a matching rate of the output images based on the first, second, third, and fourth output images. Here, the matching rate of the output images can be at least one of a vertical matching rate, a curvature matching rate, and a scale matching rate. The matching rate of the output images can be determined by comparing feature points between two adjacent images among the first, second, third, and fourth output images, or by using a difference image between the two adjacent images.

[0145] The video processing unit 850 determines whether the matching rate of two adjacent images among the first, second, third, and fourth output images is less than a preset reference matching rate, and if so, uses a preset parameter as a basis for transforming a model related to any one of the two adjacent images.

[0146] The video processing unit 850 projects the images onto the transformed model to generate intermediate images having new texture coordinates, and texture-maps the intermediate images to the output model based on the new texture coordinates to obtain new output images. Here, the new output images are output images whose matching rates with other adjacent output images are adjusted to be equal to or greater than a preset reference matching rate.

[0147] The video processing unit 850 transmits the registered output images to the serializer 860. The serializer 860 converts the registered output images into serial images and transmits them to the display 870 outside the image registration apparatus 800. The display 870 outputs the registered images of the surroundings of the vehicle.

[0148] Figure 9 is a flowchart of an image registration method according to at least one embodiment of the present disclosure.

[0149] As Figure 1As shown, the image registration device projects a first image generated based on the image captured by the first camera onto the first model to generate a first intermediate image (S900). Here, the initial shape of the first model can be the same as the initial shape of the first output model. The first model can be transformed based on a preset parameter.

[0150] The image registration device can project images respectively corresponding to frames constituting the first image onto the first model to obtain the first intermediate image. The first intermediate image includes first intermediate sub-images each obtained by the image registration device projecting images respectively corresponding to frames constituting the first image onto the first model. The first intermediate image can be a composite image in which the respective frames constituting the first image are combined in a time sequence with their respective first intermediate sub-images.

[0151] The image registration device maps the first intermediate image to the first output model to generate a first output image (S910). Here, the first output model can correspond to all or part of the preset 3D model. The preset 3D model can be any one of a hemispherical shape and a bowl shape. The first output image can be an image obtained by the image registration device texture-mapping the first intermediate image to the first output model.

[0152] The image registration device projects a second image generated based on the image captured by the second camera onto the second model to generate a second intermediate image (S920). Here, the initial shape of the second model can be the same as the initial shape of the second output model. The second model can be transformed based on a preset parameter.

[0153] The image registration device can project images respectively corresponding to frames constituting the second image onto the second model to obtain the second intermediate image. The second intermediate image includes second intermediate sub-images each obtained by the image registration device projecting images respectively corresponding to frames constituting the second image onto the second model. The second intermediate image can be a composite image in which the respective frames constituting the second image are combined in a time sequence with their respective second intermediate sub-images.

[0154] The image registration device maps the second intermediate image to the second output model to generate a second output image (S930). Here, the second output model can correspond to all or part of the preset 3D model. In addition, the preset 3D model can have any one of a hemispherical shape and a bowl shape. In addition, the second output image can be an image obtained by the image registration device texture-mapping the second intermediate image to the second output model.

[0155] The image registration apparatus places the first output image on the 3D model at a position corresponding to the first output model and places the second output image on the 3D model at a position corresponding to the second output model (S940). Here, the shape of the 3D model can be any one of a hemispherical shape and a bowl shape. The image registration apparatus can arrange the first output model and the second output model such that they have sides in contact with each other, forming a boundary line. In addition, the image registration apparatus can arrange the first output model and the second output model such that some areas of the first output model and some areas of the second output model overlap each other.

[0156] The image registration apparatus determines a matching rate between the first output video or image and the second output video or image (S950). The image registration apparatus determines a matching rate between two frames belonging to the first output video and the second output video and corresponding to a preset period based on the preset period. Here, the preset period can be, but is not limited to, 1 / 30 seconds. The matching rate between the first output image and the second output image can be at least one of a vertical matching rate, a curvature matching rate, and a proportional matching rate.

[0157] The image registration apparatus obtains a first screen image that is the first output image output on the display and obtains a first comparison image that is a two-dimensional image from the first screen image. The first comparison image can be a frame image corresponding to a preset time point. The image registration apparatus obtains a second screen image that is the second output image output on the display and obtains a second comparison image that is a two-dimensional image from the second screen image. The second comparison image can be a frame image corresponding to a preset time point. Here, the preset time point can be any one of a time point for determining a vertical matching rate between the first output image and the second output image, a time point for determining a curvature matching rate, and a time point for determining a proportional matching rate therebetween.

[0158] The image registration apparatus determines a matching rate between the first output image and the second output image by using a feature point comparison or a difference image. A method of determining a matching rate between the first output image and the second output image by a feature point comparison and a method of determining a matching rate therebetween based on a difference image can be equivalent to a method of determining a matching rate between the first output image and the second output image by referring to Figure 10 The given description, and need not be further elaborated.

[0159] The image registration apparatus transforms at least one of the first model and the second model based on a matching rate between the first output image and the second output image and a preset reference matching rate (S960).

[0160] When the matching rate between the first output image and the second output image is equal to or greater than the preset reference matching rate, the image registration apparatus generates a final screen image based on the first output image and the second output image.

[0161] When the image matching rate between the first output image and the second output image is less than a preset reference matching rate, the image registration apparatus adjusts a preset parameter to adjust a vertical component value of a model coordinate of any one of the first model and the second model to transform at least one of the first model and the second model.

[0162] The image registration apparatus can set a direction in which any one of the first image and the second image is projected onto any one of the first model and the second model as a vertical direction of the model. Here, the vertical direction of the model can be a Z-axis direction of a Cartesian coordinate system expressing any one of the first model and the second model.

[0163] The image registration apparatus changes a vertical component value of a model coordinate expressing any one of the first model and the second model by using a preset parameter. When the vertical direction of any one of the first model and the second model is set as a Z-axis of a Cartesian coordinate system expressing any one of the first model and the second model, the Z component value is changed in the model coordinate expressing any one of the first model and the second model.

[0164] The image registration apparatus performs model transformation by adjusting a first parameter so that a slope of a side wall constituting any one of the first model and the second model has a new slope. The image registration apparatus adjusts a second parameter to perform model transformation so that a curvature of a side wall constituting any one of the first model and the second model has a new curvature. The image registration apparatus adjusts a third parameter to perform model transformation so that a vertical length of the model is increased or decreased at a different ratio according to a height of a side wall constituting any one of the first model and the second model.

[0165] Here, the initial shape of the first model can be the same as the first output model, and the initial shape of the second model can be the same as the second output model. In addition, the initial shape of any one of the first model and the second model can be a hemispherical shape, a bowl shape, a portion of a hemispherical shape, or a portion of a bowl shape.

[0166] When the first model is transformed, the image registration apparatus projects the first image onto the transformed first model to generate a new first intermediate image. The first intermediate image has new texture coordinates corresponding to model coordinates of the transformed first model. The image registration apparatus texture-maps the first intermediate image to the first output model to obtain a new first output image.

[0167] When the second model is transformed, the image registration apparatus projects the second image onto the transformed second model to generate a new second intermediate image. The second intermediate image has new texture coordinates corresponding to model coordinates of the modified second model. The image registration apparatus texture-maps the second intermediate image to the second output model to obtain a new second output image.

[0168] The image registration apparatus can transform at least one of the first model and the second model so that a matching rate between the first output image and the second output image is equal to or greater than a preset reference matching rate. Here, the matching rate between the first output image and the second output image can be at least one of a vertical matching rate, a curvature matching rate, and a scale matching rate.

[0169] Figure 10 is a flowchart of an image registration method according to another embodiment of the disclosure.

[0170] As Figure 9 indicated, the image registration apparatus performs texture mapping and generates an output image based on a texture coordinate of an input image (S1000). The texture coordinate of the input image is a texture coordinate of an intermediate image obtained by projecting the input image onto a 3-D model. The image registration apparatus texture maps the intermediate image to an output model based on the texture coordinate to generate the output image.

[0171] The image registration apparatus determines a matching rate between two adjacent output images and compares the determined matching rate with a preset reference matching rate (S1010, S1020, S1030, S1030). Here, the determination of the matching rate can be at least one of a determination of a vertical matching rate, a determination of a left curvature matching rate, a determination of a right curvature matching rate, and a determination of a scale matching rate. The preset reference matching rate can be at least one of a reference vertical matching rate, a reference curvature matching rate, and a reference scale matching rate.

[0172] The image registration apparatus obtains 2-D images of two adjacent output images from a screen image. Here, the 2-D images can be frame images at a preset time point in the screen image. The preset time point can be at least one of a time point for determining a vertical matching rate between two adjacent output images, a time point for determining a curvature matching rate, and a time point for determining a scale matching rate therebetween.

[0173] The image registration apparatus determines a vertical matching rate between two adjacent output images based on 2-D images of the first output image and the second output image and compares the determined vertical matching rate with a reference vertical matching rate (S1010). When the vertical matching rate is less than the reference vertical matching rate, the image registration apparatus adjusts a first parameter (S1015) to transform the first model or the second model and generates new texture coordinates (S1050). Here, the first parameter is used to adjust the vertical matching rate of the output image.

[0174] The image registration apparatus generates a new output image mapped based on the new texture coordinates (S1000). The image registration apparatus obtains 2-D images of the new output image and two adjacent output images from a screen image.

[0175] The image registration apparatus determines a left curvature matching rate between the two adjacent output images based on the 2-D images of the new output image and the two adjacent output images, and compares the determined left curvature matching rate with a reference curvature matching rate (S1020). When the curvature matching rate between the left portion of one of the two adjacent output images and the right portion of the other adjacent output image is less than a predetermined reference curvature matching rate, the image registration apparatus adjusts a second parameter (S1025) to transform the left portion of one of the two adjacent output images and generate new texture coordinates (S1050). Here, the second parameter is used to adjust the curvature matching rate of the output image.

[0176] The image registration apparatus generates an output image based on the new texture coordinate mapping (S1000). The image registration apparatus obtains 2-D images of the output image and the adjacent output images from the frame images.

[0177] The image registration apparatus determines a right curvature matching rate between the two adjacent output images based on the 2-D images of the output image and the adjacent output images, and compares the determined right curvature matching rate with a reference curvature matching rate (S1030). When the curvature matching rate between the right portion of one of the two adjacent output images and the left portion of the other adjacent output image is less than a predetermined reference curvature matching rate, the image registration apparatus adjusts a right second parameter (S1035) to transform the right portion of any one of the output image models and generate new texture coordinates (S1050). Here, the right second parameter is used to adjust the right curvature matching rate of the output image.

[0178] The image registration apparatus generates a new output image based on the new texture coordinate mapping (S1000). The image registration apparatus obtains 2-D images of the new output image and the adjacent output images from the frame images.

[0179] The image registration apparatus determines a scale matching rate between the two adjacent output images based on the 2-D images of the new output image and the adjacent output images, and compares the determined scale matching rate with a reference scale matching rate (S1040). When the scale matching rate is less than a predetermined reference scale matching rate, the image registration apparatus adjusts a third parameter (S1045) to transform the first model or the second model and generate new texture coordinates (S1050). Here, the third parameter is used to adjust the scale matching rate of the model.

[0180] The image registration apparatus generates an output image based on the new texture coordinate mapping (S1000). The image registration apparatus obtains 2-D images of the output image and the adjacent output images from the frame images.

[0181] When the preset reference matching rate is equal to or exceeds any one of the vertical matching rate, the left curvature matching rate, the right curvature matching rate, and the proportional matching rate between them between two adjacent images of the output image, the image registration device can respond to determine the other one of the matching rates and compare the determined other matching rate with the reference matching rate. When the preset reference matching rate is equal to or exceeds the vertical matching rate, the left curvature matching rate, the right curvature matching rate, and the proportional matching rate, or when the time to end the operation of the image registration device, the image registration device generates the output image by texture mapping the intermediate image to the output model based on the current texture coordinates.

[0182] According to at least one embodiment, it has been described that steps S1010 to S1040 are each performed once, but other embodiments repeatedly perform steps S1010 to S1040, and each of steps S1010 to S1040 is independently, repeatedly, and in parallel.

[0183] Although Figure 10 and Figure 9 their respective steps are expressed as being performed in order, they merely exemplify the technical idea of some embodiments of the present disclosure. Therefore, a person of ordinary skill in the art can, in practicing the present disclosure, incorporate various modifications, additions, and substitutions of the order of steps shown, or by performing one or more of their steps in parallel, and thus, Figure 10 and Figure 9 the steps in Figure 10 and ​ are not limited to the time order shown.

[0184] Various embodiments of the systems and methods described herein can be implemented by digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation by one or more computer programs executable on a programmable system. A programmable system includes at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a storage system; at least one input device; and at least one output device, wherein the programmable processor can be a special purpose processor or a general purpose processor. A computer program (also known as a program, software, software application, or code) contains instructions for the programmable processor and is stored in a "computer-readable recording medium."

[0185] Computer-readable recording media include any type of recording device in which data that can be recorded and read by a computer system is stored. Examples of computer-readable recording media include non-volatile or non-transitory media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, optical disk / magnetic disk, storage device, etc. Computer-readable recording media also include transient media such as data transmission media. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network, where computer-readable code can be stored and executed in a distributed manner.

[0186] Various implementations of the systems and techniques described herein can be implemented using a programmable computer. Here, a computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or any other type of storage system or a combination thereof), and at least one communication interface. For example, a programmable computer can be one of a server, network device, set-top box, embedded device, computer expansion module, personal computer, laptop computer, personal data assistant (PDA), cloud computing system, and mobile device.

[0187] According to some embodiments of this disclosure, apparatus and methods for image registration can eliminate mismatches that occur in 3D images obtained by combining multiple images.

[0188] According to some embodiments of this disclosure, the apparatus and method for image registration can reduce perceived inconsistencies in a 3D image of combined images and improve image discernibility by improving vertical registration therein.

[0189] Although exemplary embodiments of this disclosure have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of the claimed invention. Therefore, exemplary embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of these embodiments is not limited by the illustrations. Therefore, those skilled in the art will understand that the scope of the claimed invention is not limited to the embodiments explicitly described above, but is limited by the claims and their equivalents.

Claims

1. An image registration apparatus, comprising at least one processor, said at least one processor being configured to: The first image is projected onto the first model to generate the first intermediate image. The first intermediate image is mapped to the first output model to generate the first output image. The second image is projected onto the second model to generate a second intermediate image. The second intermediate image is mapped to the second output model to generate the second output image, and The matching rate between the first output image and the second output image is determined, and the shape of at least one of the first model and the second model is transformed based on the determined matching rate and a preset reference matching rate. in, The first model corresponds to a first part of a preset 3D model, and the initial shape of the first model is the same as the shape of the first output model. The second model corresponds to a second part of the preset 3D model, and the initial shape of the second model is the same as the shape of the second output model. The first output image is positioned within the preset 3D model at a location corresponding to the first output model, and the second output image is positioned within the preset 3D model at a location corresponding to the second output model. The first and second output models are arranged adjacent to each other, wherein one side of the first output model and one side of the second output model form a boundary line, or the first and second output models are arranged to have an overlapping area. The transformation of the shape of at least one of the first model and the second model includes: adjusting at least one of the slope, curvature, or aspect ratio of the vertical dimension of the sidewalls of the preset 3D model formed by the first model and the preset 3D model formed by the second model, according to the matching criteria used to determine the matching rate.

2. The image registration apparatus according to claim 1, wherein the at least one processor is further configured to: Determine the vertical matching rate between the first output image and the second output image. Determine the curvature matching rate between the first output image and the second output image, and Determine the scaling ratio between the first output image and the second output image.

3. The image registration apparatus according to claim 2, wherein the at least one processor is further configured to: The first output image and the second output image are compared to determine the vertical matching rate between the first output image and the second output image in the vertical direction. Determine whether the determined vertical matching rate is less than a preset reference vertical matching rate, and if the determined vertical matching rate is less than the preset reference vertical matching rate, then modify at least one of the first model and the second model by adjusting the first parameter.

4. The image registration apparatus according to claim 2, wherein the at least one processor is further configured to: A comparison is made between the first output image and the second output image to determine the curvature matching rate between the first output image and the second output image with respect to curvature, and Determine whether the determined curvature matching rate is less than a preset reference curvature matching rate, and if the determined curvature matching rate is less than the preset reference curvature matching rate, then modify at least one of the first model and the second model by adjusting the second parameter.

5. The image registration apparatus according to claim 2, wherein the at least one processor is further configured to: A comparison is made between the first output image and the second output image to determine the proportional matching rate of the vertical dimensions between the first output image and the second output image regarding the horizontal-to-vertical ratio, and Determine whether the determined proportional matching rate is less than a preset reference proportional matching rate, and if the determined proportional matching rate is less than the preset reference proportional matching rate, then modify at least one of the first model and the second model by adjusting the third parameter.

6. The image registration apparatus according to claim 3, wherein, The first parameter is adjusted to form at least one of the slope of the sidewall of the preset 3D model formed by the first model and the slope of the sidewall of the preset 3D model formed by the second model.

7. The image registration apparatus according to claim 4, wherein, The second parameter adjusts at least one of the curvature of the sidewall of the preset 3D model formed by the first model and the curvature of the sidewall of the preset 3D model formed by the second model.

8. The image registration apparatus according to claim 5, wherein, The third parameter adjustment forms at least one of the vertical dimension in the aspect ratio of the sidewall of the preset 3D model formed by the first model and the vertical dimension in the aspect ratio of the sidewall of the preset 3D model formed by the second model.

9. An image registration method, comprising: The first image is projected onto the first model to generate a first intermediate image; The first intermediate image is mapped to the first output model to generate the first output image; The second image is projected onto the second model to generate a second intermediate image; The second intermediate image is mapped to the second output model to generate the second output image; as well as The matching rate between the first output image and the second output image is determined, and the shape of at least one of the first model and the second model is transformed based on the determined matching rate and a preset reference matching rate. The first model corresponds to a first part of a preset 3D model, and the initial shape of the first model is the same as the shape of the first output model. The second model corresponds to a second part of the preset 3D model, and the initial shape of the second model is the same as the shape of the second output model. The first output image is positioned within the preset 3D model at a location corresponding to the first output model, and the second output image is positioned within the preset 3D model at a location corresponding to the second output model. The first and second output models are arranged adjacent to each other, wherein one side of the first output model and one side of the second output model form a boundary line, or the first and second output models are arranged to have an overlapping area. The transformation of the shape of at least one of the first model and the second model includes: adjusting at least one of the slope, curvature, or aspect ratio of the vertical dimension of the sidewalls of the preset 3D model formed by the first model and the preset 3D model formed by the second model, according to the matching criteria used to determine the matching rate.

10. The image registration method according to claim 9, wherein, Determining the matching rate between the first output image and the second output image, and transforming at least one of the first model and the second model based on the determined matching rate and the preset reference matching rate, includes: Determine the vertical matching rate between the first output image and the second output image; Determine the curvature matching rate between the first output image and the second output image; and Determine the scaling ratio between the first output image and the second output image.

11. The image registration method according to claim 10, wherein, Determining the vertical matching rate between the first output image and the second output image includes: Compare the first output image and the second output image to determine the vertical matching rate between the first output image and the second output image in the vertical direction; and Determine whether the determined vertical matching rate is less than a preset reference vertical matching rate, and if the determined vertical matching rate is less than the preset reference vertical matching rate, then modify at least one of the first model and the second model by adjusting the first parameter.

12. The image registration method according to claim 10, wherein, Determining the curvature matching rate between the first output image and the second output image includes: Compare the first output image and the second output image to determine the curvature matching rate between the first output image and the second output image with respect to curvature; and Determine whether the determined curvature matching rate is less than a preset reference curvature matching rate, and if the determined curvature matching rate is less than the preset reference curvature matching rate, then modify at least one of the first model and the second model by adjusting the second parameter.

13. The image registration method according to claim 10, wherein, Determining the scaling ratio between the first output image and the second output image includes: Based on feature points or difference images, the first output image and the second output image are compared to determine the proportional matching rate of the vertical dimensions between the first output image and the second output image regarding the horizontal-to-vertical ratio; and Determine whether the determined proportional matching rate is less than a preset reference proportional matching rate, and if the determined proportional matching rate is less than the preset reference proportional matching rate, then modify at least one of the first model and the second model by adjusting the third parameter.

14. The image registration method according to claim 11, wherein, The first parameter is adjusted to form at least one of the slope of the sidewall of the preset 3D model formed by the first model and the slope of the sidewall of the preset 3D model formed by the second model.

15. The image registration method according to claim 12, wherein, The second parameter adjusts at least one of the curvature of the sidewall of the preset 3D model formed by the first model and the curvature of the sidewall of the preset 3D model formed by the second model.

16. The image registration method according to claim 13, wherein, The third parameter adjustment forms at least one of the vertical dimension in the aspect ratio of the sidewall of the preset 3D model formed by the first model and the vertical dimension in the aspect ratio of the sidewall of the preset 3D model formed by the second model.

Images