Image alignment method and device, electronic equipment, chip and medium

Abstract

The present disclosure relates to an image alignment method and device, electronic equipment, chip and medium, wherein the method comprises: obtaining a reference image corresponding to a source image and a plurality of intermediate alignment images; the plurality of intermediate alignment images are obtained by aligning the source image in combination with the reference image and a plurality of image alignment strategies; determining a predicted conversion matrix for converting the reference image according to the reference image and the plurality of intermediate alignment images; and converting the reference image according to the predicted conversion matrix to obtain a target alignment image corresponding to the source image; wherein different image alignment strategies are suitable for processing images in different fields, so that the above scheme can be applied to images in many fields and has strong generalization, thereby improving the image alignment efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image alignment method, apparatus, electronic device, chip, and medium. Background Technology

[0002] Currently, image alignment techniques can align a source image with a reference image through transformations such as spatial twisting, translation, and pixel mapping to obtain an aligned image. The transformation matrix is ​​determined based on both the source and reference images.

[0003] Image alignment technology involves image alignment algorithms, such as AI image alignment algorithms. AI image alignment algorithms, in particular, require training with domain-specific image data, resulting in large processing volumes and poor generalization, leading to low image alignment efficiency. Summary of the Invention

[0004] This disclosure provides an image alignment method, apparatus, electronic device, chip, and medium.

[0005] According to a first aspect of the present disclosure, an image alignment method is provided, the method comprising: acquiring a reference image corresponding to a source image and a plurality of intermediate alignment images; aligning the source image with the plurality of intermediate alignment images in combination with the reference image and a plurality of image alignment strategies; determining a prediction transformation matrix for transforming the reference image based on the reference image and the plurality of intermediate alignment images; and transforming the reference image based on the prediction transformation matrix to obtain a target alignment image corresponding to the source image.

[0006] In one embodiment of this disclosure, determining a prediction transformation matrix for transforming the reference image based on the reference image and the plurality of intermediate aligned images includes: inputting the reference image and the plurality of intermediate aligned images into a transformation matrix prediction model, and obtaining the prediction transformation matrix output by the transformation matrix prediction model.

[0007] In one embodiment of this disclosure, the transformation matrix prediction model is trained based on a sample reference image and multiple sample aligned images; the sample reference image and the multiple sample aligned images are obtained by image processing of the same original sample image.

[0008] In one embodiment of this disclosure, obtaining a reference image corresponding to a source image and a plurality of intermediate aligned images includes: obtaining the source image and a reference image corresponding to the source image; performing alignment processing on the source image by combining the reference image and the plurality of image alignment strategies to obtain a plurality of candidate aligned images; and selecting the plurality of intermediate aligned images from the plurality of candidate aligned images according to the reference image.

[0009] In one embodiment of this disclosure, selecting the plurality of intermediate alignment images from the plurality of candidate alignment images based on the reference image includes: determining the alignment score of the plurality of candidate alignment images based on the reference image; sorting the plurality of candidate alignment images in descending order based on the alignment score to obtain a sorting result; and determining the plurality of candidate alignment images that appear first in the sorting result as the intermediate alignment images.

[0010] In one embodiment of this disclosure, determining the alignment score of the plurality of candidate aligned images based on the reference image includes: determining a reference hash sequence corresponding to the reference image and a candidate hash sequence corresponding to the plurality of candidate aligned images based on a differential hashing algorithm; determining the distance between the candidate hash sequence of the candidate aligned image and the reference hash sequence for each candidate aligned image; and determining the alignment score of the plurality of candidate aligned images based on the distances corresponding to the plurality of candidate aligned images.

[0011] According to a second aspect of the present disclosure, a method for training a transformation matrix prediction model is also provided. The method includes: acquiring training data; the training data includes a sample reference image and a plurality of sample aligned images corresponding to the sample reference image; the sample reference image and the plurality of sample aligned images are obtained by processing the same original sample image; acquiring an initial transformation matrix prediction model; determining a prediction image based on the sample reference image, the plurality of sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model; and performing parameter adjustment processing on the transformation matrix prediction model based on the prediction image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model to achieve training.

[0012] In one embodiment of this disclosure, acquiring training data includes: acquiring the original sample image; performing image enhancement processing on the original sample image to obtain the sample reference image; and performing local region transformation processing on the original sample image multiple times to obtain multiple sample aligned images corresponding to the sample reference image.

[0013] In one embodiment of this disclosure, determining the predicted image based on the sample reference image, a plurality of sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model includes: inputting the sample reference image and the plurality of sample aligned images corresponding to the sample reference image into the transformation matrix prediction model to obtain a prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image; and transforming the sample reference image according to the prediction transformation matrix to obtain the predicted image.

[0014] According to a third aspect of the present disclosure, an image alignment apparatus is also provided. The apparatus includes: an acquisition module, configured to acquire a reference image corresponding to a source image and a plurality of intermediate alignment images; the plurality of intermediate alignment images are obtained by aligning the source image in combination with the reference image and a plurality of image alignment strategies; a determination module, configured to determine a prediction transformation matrix for performing transformation processing on the reference image based on the reference image and the plurality of intermediate alignment images; and a transformation processing module, configured to perform transformation processing on the reference image based on the prediction transformation matrix to obtain a target alignment image corresponding to the source image.

[0015] In one embodiment of this disclosure, the determining module is specifically used to input the reference image and the plurality of intermediate aligned images into the transformation matrix prediction model, and obtain the predicted transformation matrix output by the transformation matrix prediction model.

[0016] In one embodiment of this disclosure, the transformation matrix prediction model is trained based on a sample reference image and multiple sample aligned images; the sample reference image and the multiple sample aligned images are obtained by image processing of the same original sample image.

[0017] In one embodiment of this disclosure, the acquisition module includes: an acquisition unit, an alignment processing unit, and a selection unit; the acquisition unit is used to acquire the source image and a reference image corresponding to the source image; the alignment processing unit is used to perform alignment processing on the source image by combining the reference image and the plurality of image alignment strategies to obtain a plurality of candidate aligned images; the selection unit is used to select the plurality of intermediate aligned images from the plurality of candidate aligned images according to the reference image.

[0018] In one embodiment of this disclosure, the selection unit is specifically used to: determine the alignment score of the plurality of candidate alignment images based on the reference image; sort the plurality of candidate alignment images in descending order based on the alignment score to obtain a sorting result; and determine the plurality of candidate alignment images that are at the top of the sorting result as the intermediate alignment image.

[0019] In one embodiment of this disclosure, the selection unit is further configured to: determine a reference hash sequence corresponding to the reference image and a candidate hash sequence corresponding to the plurality of candidate aligned images according to a differential hashing algorithm; determine the distance between the candidate hash sequence of the candidate aligned image and the reference hash sequence for each candidate aligned image; and determine the alignment score of the plurality of candidate aligned images according to the distances corresponding to the plurality of candidate aligned images.

[0020] According to a fourth aspect of the present disclosure, a training apparatus for a transformation matrix prediction model is also provided. The apparatus includes: a first acquisition module for acquiring training data; the training data includes a sample reference image and a plurality of sample aligned images corresponding to the sample reference image; the sample reference image and the plurality of sample aligned images are obtained by processing the same original sample image; a second acquisition module for acquiring an initial transformation matrix prediction model; a determination module for determining a prediction image based on the sample reference image, the plurality of sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model; and a training module for performing parameter adjustment processing on the transformation matrix prediction model based on the prediction image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model to achieve training.

[0021] In one embodiment of this disclosure, the first acquisition module is specifically used to: acquire the original sample image; perform image enhancement processing on the original sample image to obtain the sample reference image; and perform local region transformation processing on the original sample image multiple times to obtain multiple sample aligned images corresponding to the sample reference image.

[0022] In one embodiment of this disclosure, the determining module is specifically used to input the sample reference image and a plurality of sample aligned images corresponding to the sample reference image into the transformation matrix prediction model, obtain the prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image; and perform transformation processing on the sample reference image according to the prediction transformation matrix to obtain the prediction image.

[0023] According to a fifth aspect of the present disclosure, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement the image alignment method as described above; or implement the training method of the transformation matrix prediction model as described above.

[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium is also provided, wherein when instructions in the storage medium are executed by a processor, the processor is able to perform the image alignment method as described above; or, perform the training method of the transformation matrix prediction model as described above.

[0025] According to a seventh aspect of the present disclosure, a chip is also provided, including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory of an electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, wherein when the processor executes the computer instructions, the electronic device performs the image alignment method as described above; or performs the training method of the transformation matrix prediction model as described above.

[0026] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0027] The process involves obtaining a reference image and multiple intermediate aligned images corresponding to the source image; aligning the source image using the multiple intermediate aligned images, combined with the reference image and multiple image alignment strategies; determining a prediction transformation matrix for transforming the reference image based on the reference image and the multiple intermediate aligned images; and transforming the reference image using the prediction transformation matrix to obtain the target aligned image corresponding to the source image. Different image alignment strategies are applicable to image processing in different domains, making the above scheme applicable to images in many domains, exhibiting strong generalization and thus improving image alignment efficiency.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a flowchart of an image alignment method according to an embodiment of the present disclosure;

[0031] Figure 2 This is a flowchart of an image alignment method according to another embodiment of the present disclosure;

[0032] Figure 3 This is a schematic diagram of image alignment.

[0033] Figure 4 A flowchart illustrating a training method for a transformation matrix prediction model according to an embodiment of this disclosure;

[0034] Figure 5 This is a schematic diagram of the structure of an image alignment device according to an embodiment of the present disclosure;

[0035] Figure 6 This is a schematic diagram of the structure of a training device for a transformation matrix prediction model according to an embodiment of the present disclosure;

[0036] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0037] Figure 8 This is a schematic diagram of the structure of a chip according to an embodiment of the present disclosure. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0039] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0040] Currently, image alignment techniques can align a source image with a reference image through transformations such as spatial twisting, translation, and pixel mapping to obtain an aligned image. The transformation matrix is ​​determined based on both the source and reference images.

[0041] Image alignment technology involves image alignment algorithms, such as AI image alignment algorithms. AI image alignment algorithms, in particular, require training with domain-specific image data, resulting in large processing volumes and poor generalization, leading to low image alignment efficiency.

[0042] Figure 1 This is a flowchart of an image alignment method according to an embodiment of the present disclosure. It should be noted that the image alignment method of this embodiment can be applied to an image alignment device, which can be configured in an electronic device or chip to enable the electronic device or chip to perform image alignment functions.

[0043] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0044] In addition, the image alignment device can also be software in an electronic device. Software, for example, includes image processing software. The following embodiments will be described using an electronic device as an example of the execution subject.

[0045] like Figure 1 As shown, the method includes the following steps:

[0046] Step 101: Obtain the reference image and multiple intermediate aligned images corresponding to the source image; the multiple intermediate aligned images are combined with the reference image and multiple image alignment strategies to align the source image.

[0047] In this embodiment of the disclosure, the applicable scenarios for image alignment processing may include at least one of the following: aligning images in an image sequence obtained by capturing multiple frames, aligning images in an image sequence obtained by high-dynamic-range (HDR) video recording, aligning images in an image sequence obtained by switching between multiple cameras, and aligning multiple images with high similarity selected from an image set.

[0048] In multi-frame photography or HDR video recording, the image sensor exposes multiple frames, resulting in an image sequence. Fusing these multiple images yields an image with low noise and rich detail. In multi-camera switching, as the camera zooms in or out, it uses cameras with different focal lengths. Since cameras with different focal lengths have misaligned optical axes and different fields of view, alignment is necessary to prevent image jitter.

[0049] In this sequence of images, one image can be selected as a reference image, and all other images in the sequence except the reference image can be used as source images. For each source image, the image alignment method described in this embodiment is executed.

[0050] Specifically, for multiple images with high similarity selected from the image set, one image can be chosen as a reference image, and the images other than the reference image can be used as source images. For each source image, the image alignment method in this embodiment is executed.

[0051] In addition, it should be noted that the applicable scenarios for image alignment processing can also include at least one of the following: performing image alignment processing in image denoising, performing image alignment processing in image super-resolution, etc.

[0052] In this embodiment, the image alignment strategy may include at least one of the following: an optical flow estimation method based on the Transformer architecture (FlowFormer optical flow alignment); an optical flow estimation method based on deep learning (open-source algorithm RAFT); an image alignment method based on global alignment (self-developed algorithm CVE), etc. It should be noted that the image alignment strategy is not limited to the above-listed methods; any existing image alignment strategies can be used, and the specific strategy can be set according to actual needs.

[0053] Step 102: Determine the prediction transformation matrix for transforming the reference image based on the reference image and multiple intermediate aligned images.

[0054] In this embodiment of the disclosure, the electronic device may perform step 102 by inputting a reference image and multiple intermediate aligned images into a transformation matrix prediction model, and obtaining the predicted transformation matrix output by the transformation matrix prediction model.

[0055] The transformation matrix prediction model, after acquiring the reference image and multiple intermediate aligned images, can use the reference image as the guide image to select the target local region that matches the local region of the guide image from the multiple intermediate aligned images; then perform mapping and raw image processing based on the multiple target local regions; and determine the prediction transformation matrix used to transform the reference image based on the generated image and the guide image.

[0056] The transformation matrix prediction model can be a feature extraction network. This feature extraction network can be, for example, a vectorized network, the feature extraction network in a Convolutional Neural Network (CNN), or the encoding network in a transformer model—any network that can output a vector matrix. No specific limitation is made here. This feature extraction network, capable of outputting a vector matrix, can be trained to output a vector matrix, i.e., a predicted transformation matrix, based on the input reference image and multiple intermediate aligned images.

[0057] In this embodiment of the disclosure, the transformation matrix prediction model can be trained based on a sample reference image and multiple sample aligned images; the sample reference image and multiple sample aligned images are obtained by image processing of the same original sample image.

[0058] The sample reference image and multiple sample aligned images can have the same size, for example, each consisting of M*N pixels, with each pixel corresponding to three channels: red (R), green (G), and blue (B). Correspondingly, the size of the output prediction transformation matrix can also be M*N, which can include M*N vectors.

[0059] Step 103: Transform the reference image according to the predicted transformation matrix to obtain the target aligned image corresponding to the source image.

[0060] In this embodiment of the disclosure, the electronic device can perform matrix multiplication on the prediction transformation matrix and the reference image to obtain the target aligned image corresponding to the source image.

[0061] In the image alignment method of this disclosure, a reference image and multiple intermediate aligned images corresponding to the source image are obtained; the source image is aligned by combining the multiple intermediate aligned images with the reference image and multiple image alignment strategies; a prediction transformation matrix for transforming the reference image is determined based on the reference image and the multiple intermediate aligned images; the reference image is transformed based on the prediction transformation matrix to obtain the target aligned image corresponding to the source image; wherein, different image alignment strategies are applicable to image processing in different fields, making the above scheme applicable to images in many fields, with strong generalization, thereby improving image alignment efficiency.

[0062] Figure 2 This is a flowchart illustrating another embodiment of the image alignment method disclosed herein. It should be noted that the image alignment method of this embodiment can be applied to an image alignment device, which can be configured in an electronic device or chip to enable the electronic device or chip to perform image alignment functions.

[0063] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0064] In addition, the image alignment device can also be software in an electronic device. Software, for example, includes image processing software. The following embodiments will be described using an electronic device as an example of the execution subject.

[0065] like Figure 2 As shown, the method includes the following steps:

[0066] Step 201: Obtain the source image and the corresponding reference image.

[0067] Step 202: Combine the reference image and multiple image alignment strategies to perform alignment processing on the source image to obtain multiple candidate aligned images.

[0068] Step 203: Select multiple intermediate alignment images from multiple candidate alignment images based on the reference image.

[0069] In one example of this disclosure, the electronic device performing step 203 may be as follows: determining the alignment scores of multiple candidate alignment images based on a reference image; sorting the multiple candidate alignment images in descending order based on the alignment scores to obtain a sorting result; and determining the multiple candidate alignment images that are at the top of the sorting result as the middle alignment images.

[0070] The process by which the electronic device determines the alignment score of multiple candidate aligned images based on a reference image can be, for example, as follows: determining the reference hash sequence corresponding to the reference image and the candidate hash sequences corresponding to the multiple candidate aligned images based on a differential hashing algorithm; determining the distance between the candidate hash sequence of the candidate aligned image and the reference hash sequence for each candidate aligned image; and determining the alignment score of the multiple candidate aligned images based on the distances between the multiple candidate aligned images.

[0071] The basic principle of the differential hashing algorithm is to reduce the size of the image and convert it to grayscale. For each row of pixels in the grayscale image, the left and right pixels are compared. If the left pixel is brighter than the right pixel (the left pixel value is greater than the right pixel value), it is recorded as 1; otherwise, it is recorded as 0. The hash values ​​obtained from each row of pixels are concatenated to obtain the hash sequence corresponding to the image.

[0072] The distance between the candidate hash sequence and the reference hash sequence of the candidate aligned image can be termed the Hamming distance. Hamming distance is a metric used to measure the distance (or difference) between two strings of equal length. Specifically, it represents the number of different characters at corresponding positions in two strings of equal length. From another perspective, Hamming distance also measures the minimum number of substitutions required to transform one string into another by replacing characters.

[0073] In this embodiment of the disclosure, after the electronic device determines the distances corresponding to multiple candidate alignment images, it can sort the multiple distances; and determine the alignment score of the multiple candidate alignment images based on the sorting result. Specifically, the larger the distance, the smaller the alignment score; and the smaller the distance, the larger the alignment score.

[0074] For example, an electronic device can divide multiple distance ranges into multiple sub-distance ranges. Each sub-distance range corresponds to an alignment score. The larger the distance within a sub-distance range, the smaller the alignment score; conversely, the smaller the distance within a sub-distance range, the larger the alignment score. For each candidate aligned image, the alignment score of the candidate image can be determined by querying each sub-distance range based on the distance corresponding to that candidate aligned image, and the alignment score of the sub-distance ranges including that distance can be used as the alignment score of that candidate aligned image.

[0075] In another example, the electronic device may perform step 203 as follows: for each candidate aligned image, determine the image similarity between the candidate aligned image and the reference image; and determine the alignment score of the candidate aligned image based on the image similarity.

[0076] Step 204: Determine the prediction transformation matrix for transforming the reference image based on the reference image and multiple intermediate aligned images.

[0077] Step 205: Transform the reference image according to the predicted transformation matrix to obtain the target aligned image corresponding to the source image.

[0078] Additionally, it should be noted that step 202 employs multiple image alignment strategies; step 203 employs an image selection strategy; and step 204 can be implemented using a transformation matrix prediction model. Different algorithms are used in different steps, and these algorithms are decoupled from each other. If one algorithm is updated, that algorithm can be updated directly without affecting other algorithms used in the image alignment method, thereby reducing algorithm maintenance costs and shortening maintenance time.

[0079] It should be noted that for detailed explanations of steps 204 to 205, please refer to [link / reference needed]. Figure 1 Steps 102 to 103 in the illustrated embodiment will not be described in detail here.

[0080] In the image alignment method of this disclosure, a source image and a corresponding reference image are obtained; the source image is aligned using the reference image and multiple image alignment strategies to obtain multiple candidate aligned images; multiple intermediate aligned images are selected from the multiple candidate aligned images based on the reference image; a prediction transformation matrix for transforming the reference image is determined based on the reference image and the multiple intermediate aligned images; the reference image is transformed using the prediction transformation matrix to obtain the target aligned image corresponding to the source image. Different image alignment strategies are applicable to images in different domains, making the above scheme applicable to images in many domains and exhibiting strong generalization. Selecting multiple intermediate aligned images from the multiple candidate aligned images based on the reference image can obtain multiple intermediate aligned images with better alignment effects, thereby further determining the accuracy of the obtained target aligned image and further improving image alignment efficiency.

[0081] The following example illustrates this. For example... Figure 3 The image shown is a schematic diagram of image alignment. Figure 3 In this process, the reference image and the source image are combined with alignment algorithms 1 to N (multiple image alignment strategies) to obtain the alignment. Figure 1 Up to N (multiple candidate aligned images); Alignment Figure 1 The algorithm for ranking by combining the results of N and the reference graph is as follows: Figure 1 To M (multiple center-aligned images); Figure 1 Input the M and reference image (as a guide image) into the AI-based alignment region mapping algorithm (transformation matrix prediction model and matrix multiplication formula) to obtain the alignment image (target alignment image).

[0082] Figure 4 This is a flowchart illustrating a training method for a transformation matrix prediction model according to an embodiment of the present disclosure. It should be noted that the training method for the transformation matrix prediction model in this embodiment can be applied to a training device for the transformation matrix prediction model. This device can be configured in an electronic device or chip, enabling the electronic device or chip to perform the training function of the transformation matrix prediction model.

[0083] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, vehicle controllers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touch screens, and / or displays.

[0084] Furthermore, the training device for the transformation matrix prediction model can also be software within an electronic device. Software, for example, includes model training software. The following embodiments will use an electronic device as an example for illustration.

[0085] like Figure 4 As shown, the method includes the following steps:

[0086] Step 401: Obtain training data; the training data includes a sample reference image and multiple sample aligned images corresponding to the sample reference image; the sample reference image and multiple sample aligned images are obtained by processing the same original sample image.

[0087] In this embodiment of the disclosure, the electronic device may perform step 401 as follows: acquire the original sample image; perform image enhancement processing on the original sample image to obtain a sample reference image; and perform local region transformation processing on the original sample image multiple times to obtain multiple sample aligned images corresponding to the sample reference image.

[0088] The original sample images can be images from an existing image set. An example of an existing image set is the DIV2K image set, which is a foundational dataset used for training algorithms such as image denoising and super-resolution reconstruction. To further improve the training accuracy of the transformation matrix prediction model, the original sample images can be real images from an existing image set. Real images can refer to images whose content represents facts; or images whose content represents the correct answer to a specific task or question.

[0089] In this embodiment of the disclosure, the image enhancement processing of the original sample image may include at least one of the following: adjusting the brightness of all or part of the image, adding noise to all or part of the image, blurring all or part of the image, sharpening all or part of the image, etc.

[0090] In the training data, multiple different sample reference images can be generated for each original sample image. The training data can include multiple batches of training sub-data; each batch of training sub-data is used to train the transformation matrix prediction model for one epoch. For each original sample image, the corresponding sample reference image can be different in different batches of training sub-data.

[0091] In this embodiment of the disclosure, the local region transformation processing performed on the original sample image may include at least one of the following: barrel pincushion distortion, affine transformation, optical flow mapping transformation, etc. The algorithm for performing the local region transformation processing may be, for example, an image transformation warp algorithm. Specifically, for each original sample image, at least one local region in the original sample image may be transformed; different local regions may employ the same or different transformation processing methods.

[0092] Step 402: Obtain the initial transformation matrix prediction model.

[0093] In this embodiment, the transformation matrix prediction model can be a feature extraction network. This feature extraction network can be, for example, a vectorized network, a feature extraction network in a Convolutional Neural Network (CNN), or an encoding network in a transformer model—a network that can output a vector matrix; no specific limitation is made here.

[0094] The initial transformation matrix prediction model can be a pre-trained transformation matrix prediction model.

[0095] Step 403: Determine the prediction image based on the sample reference image, multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model.

[0096] In this embodiment of the disclosure, the electronic device may perform step 403 as follows: input the sample reference image and multiple sample aligned images corresponding to the sample reference image into the transformation matrix prediction model, obtain the prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image, and perform transformation processing on the sample reference image according to the prediction transformation matrix to obtain the prediction image.

[0097] To facilitate processing by the transformation matrix prediction model, the sample reference image and multiple aligned sample images can be stitched together before being input into the model. This stitching process can be performed as follows: for each pixel in the sample reference image, obtain the pixel value; obtain the pixel values ​​at the corresponding positions in the multiple aligned sample images; stitch together the pixel values ​​in the sample reference image and the corresponding pixel values ​​in the multiple aligned sample images to obtain the stitched value of that pixel; and determine the stitched image based on the stitched values ​​of each pixel. The electronic device can then input the stitched image into the transformation matrix prediction model to obtain the predicted transformation matrix output by the model.

[0098] Step 404: Based on the original sample image corresponding to the predicted image and the sample reference image, and the loss function of the transformation matrix prediction model, the parameters of the transformation matrix prediction model are adjusted to achieve training.

[0099] In this embodiment of the disclosure, the loss function of the transformation matrix prediction model can be determined based on the image similarity function between the predicted image and the original sample image. The loss function can be, for example, the difference between the number 1 and the image similarity function. The electronic device performing step 404 can, for example, determine the value of the loss function based on the predicted image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model; and then, based on the value of the loss function, perform parameter adjustment processing on the transformation matrix prediction model to achieve training.

[0100] In the training method of the transformation matrix prediction model of this disclosure embodiment, training data is acquired. The training data includes a sample reference image and multiple sample aligned images corresponding to the sample reference image. The sample reference image and multiple sample aligned images are obtained by processing the same original sample image. An initial transformation matrix prediction model is acquired. A prediction image is determined based on the sample reference image, the multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model. The parameters of the transformation matrix prediction model are adjusted based on the prediction image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model to achieve training. The trained transformation matrix prediction model can be used to process the reference image and the intermediate aligned image determined by combining multiple image alignment strategies. Different image alignment strategies are suitable for processing images in different fields, making the transformation matrix prediction model applicable to image alignment in many fields, with strong generalization, thereby improving image alignment efficiency.

[0101] Figure 5 This is a schematic diagram of the structure of an image alignment device according to an embodiment of the present disclosure.

[0102] like Figure 5 As shown, the image alignment device may include: an acquisition module 501, a determination module 502, and a conversion processing module 503.

[0103] The acquisition module 501 is used to acquire a reference image and multiple intermediate aligned images corresponding to the source image; the multiple intermediate aligned images are obtained by aligning the source image in combination with the reference image and multiple image alignment strategies; the determination module 502 is used to determine a prediction transformation matrix for transforming the reference image based on the reference image and the multiple intermediate aligned images; the transformation processing module 503 is used to transform the reference image based on the prediction transformation matrix to obtain the target aligned image corresponding to the source image.

[0104] In one embodiment of this disclosure, the determining module 502 is specifically used to input the reference image and the plurality of intermediate aligned images into the transformation matrix prediction model, and obtain the predicted transformation matrix output by the transformation matrix prediction model.

[0105] In one embodiment of this disclosure, the transformation matrix prediction model is trained based on a sample reference image and multiple sample aligned images; the sample reference image and the multiple sample aligned images are obtained by image processing of the same original sample image.

[0106] In one embodiment of this disclosure, the acquisition module 501 includes: an acquisition unit, an alignment processing unit, and a selection unit; the acquisition unit is used to acquire the source image and a reference image corresponding to the source image; the alignment processing unit is used to perform alignment processing on the source image by combining the reference image and the plurality of image alignment strategies to obtain a plurality of candidate aligned images; the selection unit is used to select the plurality of intermediate aligned images from the plurality of candidate aligned images according to the reference image.

[0107] In one embodiment of this disclosure, the selection unit is specifically used to: determine the alignment score of the plurality of candidate alignment images based on the reference image; sort the plurality of candidate alignment images in descending order based on the alignment score to obtain a sorting result; and determine the plurality of candidate alignment images that are at the top of the sorting result as the intermediate alignment image.

[0108] In one embodiment of this disclosure, the selection unit is further configured to: determine a reference hash sequence corresponding to the reference image and a candidate hash sequence corresponding to the plurality of candidate aligned images according to a differential hashing algorithm; determine the distance between the candidate hash sequence of the candidate aligned image and the reference hash sequence for each candidate aligned image; and determine the alignment score of the plurality of candidate aligned images according to the distances corresponding to the plurality of candidate aligned images.

[0109] In the image alignment apparatus of this embodiment, a reference image corresponding to the source image and multiple intermediate aligned images are acquired; the multiple intermediate aligned images are combined with the reference image and multiple image alignment strategies to align the source image; a prediction transformation matrix for transforming the reference image is determined based on the reference image and the multiple intermediate aligned images; the reference image is transformed based on the prediction transformation matrix to obtain the target aligned image corresponding to the source image; wherein, different image alignment strategies are applicable to image processing in different fields, making the above scheme applicable to images in many fields, with strong generalization, thereby improving image alignment efficiency.

[0110] Figure 6 This is a schematic diagram of the structure of a training device for a transformation matrix prediction model according to an embodiment of this disclosure. Figure 6 As shown, the training device for the transformation matrix prediction model may include: a first acquisition module 601, a second acquisition module 602, a determination module 603, and a training module 604.

[0111] The system includes a first acquisition module 601 for acquiring training data, which includes a sample reference image and multiple sample aligned images corresponding to the sample reference image. The sample reference image and the multiple sample aligned images are obtained by processing the same original sample image. A second acquisition module 602 is used to acquire an initial transformation matrix prediction model. A determination module 603 is used to determine a prediction image based on the sample reference image, the multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model. A training module 604 is used to perform parameter adjustment processing on the transformation matrix prediction model based on the prediction image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model to achieve training.

[0112] In one embodiment of this disclosure, the first acquisition module 601 is specifically used to: acquire the original sample image; perform image enhancement processing on the original sample image to obtain the sample reference image; and perform local region transformation processing on the original sample image multiple times to obtain multiple sample aligned images corresponding to the sample reference image.

[0113] In one embodiment of this disclosure, the determining module 603 is specifically used to input the sample reference image and a plurality of sample aligned images corresponding to the sample reference image into the transformation matrix prediction model, obtain the prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image; and perform transformation processing on the sample reference image according to the prediction transformation matrix to obtain the prediction image.

[0114] In the training apparatus of the transformation matrix prediction model of this embodiment, training data is acquired. The training data includes a sample reference image and multiple sample aligned images corresponding to the sample reference image. The sample reference image and multiple sample aligned images are obtained by processing the same original sample image. An initial transformation matrix prediction model is acquired. A prediction image is determined based on the sample reference image, the multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model. The parameters of the transformation matrix prediction model are adjusted based on the prediction image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model to achieve training. The trained transformation matrix prediction model can be used to process the reference image and the intermediate aligned image determined by combining multiple image alignment strategies. Different image alignment strategies are suitable for processing images in different fields, making the transformation matrix prediction model applicable to image alignment in many fields, with strong generalization, thereby improving image alignment efficiency.

[0115] According to a fifth aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to: implement the image alignment method or the training method of the transformation matrix prediction model as described above.

[0116] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium.

[0117] When the instructions in the storage medium are executed by the processor, the processor is able to perform the image alignment method or the training method of the transformation matrix prediction model as described above.

[0118] To implement the above embodiments, this disclosure also provides a computer program product.

[0119] When the computer program product is executed by the processor of the electronic device, it enables the electronic device to perform the above-described method.

[0120] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0121] like Figure 7As shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 112 or a program loaded from memory 116 into random access memory (RAM) 113. The RAM 113 also stores various programs and data required for the operation of the electronic device 1000. The processor 111, ROM 112, and RAM 113 are interconnected via a bus 114. An input / output (I / O) interface 115 is also connected to the bus 114.

[0122] The following components are connected to I / O interface 115: memory 116 including hard disks, etc.; and communication section 117 including network interface cards such as local area network (LAN) cards, modems, etc., communication section 117 performs communication processing via a network such as the Internet; and driver 118 is also connected to I / O interface 115 as needed.

[0123] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 117. When the computer program is executed by processor 111, it performs the functions defined in the methods of this disclosure.

[0124] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 111 of the electronic device 1000 to perform the above-described method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0125] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0126] Figure 8 This is a schematic diagram of the structure of a chip according to an embodiment of this disclosure. Figure 8 As shown, the chip includes a processor 801 and an interface circuit 802. The number of processors 801 and the number of interface circuits 802 can be one or more.

[0127] Optionally, the chip also includes a memory 803 for storing necessary computer programs and data; an interface circuit 802 for receiving signals from the memory 803 and sending signals to the processor 801, the signals including computer instructions stored in the memory 803, which, when executed by the processor 801, cause the electronic device to execute the image alignment method or the training method of the transformation matrix prediction model described in the above embodiments of this disclosure.

[0128] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0130] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0131] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0132] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image alignment method, characterized in that, The method includes: Obtain a reference image corresponding to the source image and multiple intermediate aligned images; the multiple intermediate aligned images include images obtained by aligning the source image in combination with the reference image and multiple image alignment strategies. Based on the reference image and the plurality of intermediate aligned images, a prediction transformation matrix is ​​determined for transforming the reference image; The reference image is transformed according to the predicted transformation matrix to obtain the target aligned image corresponding to the source image; The step of determining the prediction transformation matrix for transforming the reference image based on the reference image and the plurality of intermediate aligned images includes: inputting the reference image and the plurality of intermediate aligned images into a transformation matrix prediction model, and obtaining the prediction transformation matrix output by the transformation matrix prediction model; The transformation matrix prediction model is trained based on a sample reference image and multiple sample aligned images; the sample reference image and the multiple sample aligned images are obtained by image processing of the same original sample image.

2. The method according to claim 1, characterized in that, The acquisition of the reference image corresponding to the source image and multiple intermediate aligned images includes: Obtain the source image and the reference image corresponding to the source image; By combining the reference image and the multiple image alignment strategies, the source image is aligned to obtain multiple candidate aligned images; Based on the reference image, the plurality of intermediate alignment images are selected from the plurality of candidate alignment images.

3. The method according to claim 2, characterized in that, The step of selecting the plurality of intermediate aligned images from the plurality of candidate aligned images based on the reference image includes: Based on the reference image, determine the alignment score of the plurality of candidate alignment images; Based on the alignment score, the multiple candidate alignment images are sorted in descending order to obtain the sorting result; The top-ranking candidate aligned images in the sorting results are determined as the intermediate aligned images.

4. The method according to claim 3, characterized in that, Determining the alignment score of the plurality of candidate alignment images based on the reference image includes: Based on the differential hashing algorithm, a reference hash sequence corresponding to the reference image and a candidate hash sequence corresponding to the plurality of candidate aligned images are determined. For each candidate aligned image, determine the distance between the candidate hash sequence of the candidate aligned image and the reference hash sequence; The alignment score of the multiple candidate alignment images is determined based on the distances corresponding to the multiple candidate alignment images.

5. A training method for a transformation matrix prediction model, characterized in that, The method includes: Acquire training data; the training data includes a sample reference image and multiple sample aligned images corresponding to the sample reference image; the sample reference image and the multiple sample aligned images are obtained by processing the same original sample image; Obtain the initial transformation matrix prediction model; The predicted image is determined based on the sample reference image, multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model; Based on the predicted image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model, the parameters of the transformation matrix prediction model are adjusted to achieve training. The step of determining the predicted image based on the sample reference image, multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model includes: inputting the sample reference image and multiple sample aligned images corresponding to the sample reference image into the transformation matrix prediction model to obtain the prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image; and transforming the sample reference image according to the prediction transformation matrix to obtain the predicted image.

6. The method according to claim 5, characterized in that, The acquisition of training data includes: Obtain the original image of the sample; The original image of the sample is subjected to image enhancement processing to obtain the reference image of the sample; The original sample image is subjected to local region transformation processing multiple times to obtain multiple sample aligned images corresponding to the sample reference image.

7. An image alignment device, characterized in that, The device includes: The acquisition module is used to acquire a reference image corresponding to the source image and multiple intermediate aligned images; the multiple intermediate aligned images include images obtained by aligning the source image in combination with the reference image and multiple image alignment strategies. The determining module is configured to determine a prediction transformation matrix for transforming the reference image based on the reference image and the plurality of intermediate aligned images; A transformation processing module is used to transform the reference image according to the predicted transformation matrix to obtain the target aligned image corresponding to the source image; The determining module is specifically used to input the reference image and the plurality of intermediate aligned images into the transformation matrix prediction model, and obtain the predicted transformation matrix output by the transformation matrix prediction model; the transformation matrix prediction model is trained based on the sample reference image and the plurality of sample aligned images; the sample reference image and the plurality of sample aligned images are obtained by image processing of the same original sample image.

8. A training device for a transformation matrix prediction model, characterized in that, The device includes: The first acquisition module is used to acquire training data; the training data includes a sample reference image and multiple sample aligned images corresponding to the sample reference image; the sample reference image and the multiple sample aligned images are obtained by processing the same original sample image; The second acquisition module is used to acquire the initial transformation matrix prediction model; The determination module is used to determine the prediction image based on the sample reference image, multiple sample aligned images corresponding to the sample reference image, and the transformation matrix prediction model; The training module is used to perform parameter adjustment processing on the transformation matrix prediction model based on the predicted image, the original sample image corresponding to the sample reference image, and the loss function of the transformation matrix prediction model, so as to achieve training. The determining module is specifically used to input the sample reference image and multiple sample aligned images corresponding to the sample reference image into the transformation matrix prediction model, obtain the prediction transformation matrix output by the transformation matrix prediction model for transforming the sample reference image, and perform transformation processing on the sample reference image according to the prediction transformation matrix to obtain the prediction image.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured as follows: The steps of implementing the image alignment method as described in any one of claims 1 to 4; or, the training method of the transformation matrix prediction model as described in any one of claims 5 to 6.

10. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor, the processor is able to perform an image alignment method as described in any one of claims 1 to 4; or, perform a training method for a transformation matrix prediction model as described in any one of claims 5 to 6.

11. A chip, characterized in that, It includes one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the image alignment method of any one of claims 1 to 4; or, to perform the training method of the transformation matrix prediction model of any one of claims 5 to 6.

Images