Image stitching method and device, electronic equipment and readable storage medium

Abstract

The application discloses an image splicing method and device, electronic equipment and a readable storage medium. In one aspect, the image splicing method provided by the application obtains target transformation parameters for registration by a prediction method, without obtaining feature points in the image, so that the calculation amount is lower than that of a traditional method, and real-time performance is better. In another aspect, the image splicing method provided by the application respectively enhances an initial image and a reference image, and then performs prediction according to the enhanced initial image and the enhanced reference image. In the prediction process, the enhanced features of target objects contained in the enhanced initial image and the enhanced reference image are used as prior information, so that the accuracy of the target transformation parameters can be improved, and the quality of the obtained panoramic image of the site is better.

Description

Technical Field

[0001] This application relates to the field of image stitching technology, specifically to an image stitching method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In recent years, with changes in people's shopping habits, the logistics industry has experienced rapid development, while also placing higher demands on all aspects of logistics transportation. For example, real-time monitoring and route planning of the entire site are required when goods are sorted at transit hubs. Due to the large area of ​​the site and the high cost of equipment for directly acquiring panoramic images, the most common approach is to install multiple cameras in the site to acquire multiple original images with overlapping areas, and then use image stitching technology to obtain a panoramic image for monitoring and cargo sorting route planning.

[0003] Currently, when performing image stitching, the method of extracting feature points is usually used to register the images to be stitched before stitching. However, this stitching method requires the extraction of a large number of feature points, resulting in a large amount of computation, low real-time performance, and low accuracy of the obtained registration parameters. Therefore, the quality of the panoramic image obtained after stitching is poor. Summary of the Invention

[0004] This application provides an image stitching method, apparatus, electronic device, and readable storage medium, aiming to solve the problem that the registration parameters obtained by current image stitching methods are not very accurate, resulting in poor quality of the panoramic image after stitching.

[0005] In a first aspect, this application provides an image stitching method, including:

[0006] Acquire the initial image and reference image of the target object;

[0007] Based on the enhanced initial image and the enhanced reference image, prediction is performed to obtain the target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object;

[0008] Based on the target transformation parameters, the initial image is registered to obtain the target site image;

[0009] The target site image and the reference image are stitched together to obtain a panoramic image of the site.

[0010] In one possible implementation of this application, the target object includes at least one of the following: site zoning markers, loading bays, parking lines, and site walls.

[0011] In one possible implementation of this application, before predicting the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image, the method further includes:

[0012] Edge detection is performed on a first preset region of the initial image and a second preset region of the reference image to obtain a first initial feature map and a second initial feature map of the target object.

[0013] The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object.

[0014] The first enhanced feature map is fused with the initial image, and the second enhanced feature map is fused with the reference image to obtain the enhanced initial image and the enhanced reference image, respectively.

[0015] In one possible implementation of this application, the step of predicting the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image includes:

[0016] The target transformation parameters are obtained by using a preset target parameter prediction model based on the enhanced initial image and the enhanced reference image.

[0017] The preset target parameter prediction model is trained through the following steps:

[0018] Obtain a training dataset, wherein the training dataset contains a first training image of the target object;

[0019] The enhanced first training image is transformed according to preset image transformation parameters to obtain an enhanced second training image, wherein the enhanced first training image contains the enhanced features of the target object;

[0020] Based on the enhanced first training image and the enhanced second training image, as well as the image transformation parameters, the initial parameter prediction model to be trained is trained to obtain the target parameter prediction model.

[0021] In one possible implementation of this application, acquiring the initial image and reference image of the target object includes:

[0022] Obtain multiple initial image pairs;

[0023] For each of the initial image pairs, detect the target object in the first preset region of the first candidate image and the second preset region of the second candidate image;

[0024] From each of the initial image pairs, extract the target image pairs that contain the target object in both the first and second preset regions;

[0025] The first candidate image and the second candidate image included in the target image pair are respectively set as the initial image and the reference image.

[0026] In one possible implementation of this application, there are multiple initial images;

[0027] The step of registering the initial image according to the target transformation parameters to obtain the target site image includes:

[0028] Based on the target transformation parameters corresponding to each initial image, the initial images are registered to obtain the first site image;

[0029] Calculate the similarity between the reference image and each of the first site images, and determine the target initial image corresponding to the first site image with the highest similarity.

[0030] According to the target transformation parameters corresponding to the target initial image, the initial images other than the target initial image are registered to obtain the second site image;

[0031] The first site image and each of the second site images of the target initial image are set as the target site images.

[0032] In one possible implementation of this application, after stitching the target site image and the reference image to obtain a panoramic site image, the method further includes:

[0033] Obtain the image quality assessment value of the panoramic image of the site;

[0034] If the image quality assessment value is lower than a preset assessment threshold, then the preset target parameter prediction model is trained using at least one of the initial image and the reference image to obtain an updated target parameter prediction model, wherein the preset target parameter prediction model is used to predict the target transformation parameters.

[0035] Secondly, this application provides an image stitching device, comprising:

[0036] The acquisition module is used to acquire the initial image and reference image of the target object;

[0037] The prediction module is used to predict the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object;

[0038] The registration module is used to register the initial image according to the target transformation parameters to obtain the target site image;

[0039] The stitching module is used to stitch the target site image and the reference image together to obtain a panoramic image of the site.

[0040] In one possible implementation of this application, the target object includes at least one of the following: site zoning markers, loading bays, parking lines, and site walls.

[0041] In one possible implementation of this application, the image stitching device further includes an enhancement module, which is used for:

[0042] Edge detection is performed on a first preset region of the initial image and a second preset region of the reference image to obtain a first initial feature map and a second initial feature map of the target object.

[0043] The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object.

[0044] The first enhanced feature map is fused with the initial image, and the second enhanced feature map is fused with the reference image to obtain the enhanced initial image and the enhanced reference image, respectively.

[0045] In one possible implementation of this application, the prediction module is further used for:

[0046] The target transformation parameters are obtained by using a preset target parameter prediction model based on the enhanced initial image and the enhanced reference image.

[0047] The preset target parameter prediction model is trained through the following steps:

[0048] Obtain a training dataset, wherein the training dataset contains a first training image of the target object;

[0049] The enhanced first training image is transformed according to preset image transformation parameters to obtain an enhanced second training image, wherein the enhanced first training image contains the enhanced features of the target object;

[0050] Based on the enhanced first training image and the enhanced second training image, as well as the image transformation parameters, the initial parameter prediction model to be trained is trained to obtain the target parameter prediction model.

[0051] In one possible implementation of this application, the acquisition module is further used for:

[0052] Obtain multiple initial image pairs;

[0053] For each of the initial image pairs, detect the target object in the first preset region of the first candidate image and the second preset region of the second candidate image;

[0054] From each of the initial image pairs, extract the target image pairs that contain the target object in both the first and second preset regions;

[0055] The first candidate image and the second candidate image included in the target image pair are respectively set as the initial image and the reference image.

[0056] In one possible implementation of this application, the initial image consists of multiple images, and the registration module is further configured to:

[0057] Based on the target transformation parameters corresponding to each initial image, the initial images are registered to obtain the first site image;

[0058] Calculate the similarity between the reference image and each of the first site images, and determine the target initial image corresponding to the first site image with the highest similarity.

[0059] According to the target transformation parameters corresponding to the target initial image, the initial images other than the target initial image are registered to obtain the second site image;

[0060] The first site image and each of the second site images of the target initial image are set as the target site images.

[0061] In one possible implementation of this application, the image stitching device further includes a retraining module, which is used for:

[0062] Obtain the image quality assessment value of the panoramic image of the site;

[0063] If the image quality assessment value is lower than a preset assessment threshold, then the preset target parameter prediction model is trained using at least one of the initial image and the reference image to obtain an updated target parameter prediction model, wherein the preset target parameter prediction model is used to predict the target transformation parameters.

[0064] Thirdly, this application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor calls the computer program in the memory, it executes the steps in any of the image stitching methods provided in this application.

[0065] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the image stitching methods provided in this application.

[0066] In summary, the image stitching method for logistics sites provided in this application includes: acquiring an initial image and a reference image of a target object; performing prediction based on the enhanced initial image and the enhanced reference image to obtain target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain enhanced features of the target object; registering the initial image according to the target transformation parameters to obtain a target site image; and stitching the target site image and the reference image together to obtain a panoramic image of the site. On one hand, the image stitching method provided in this application obtains the target transformation parameters for registration through prediction, without needing to acquire feature points in the image, thus resulting in lower computational load and better real-time performance compared to traditional methods. On the other hand, the image stitching method provided in this application enhances the initial image and the reference image separately, and then performs prediction based on the enhanced initial image and the enhanced reference image. During the prediction process, the enhanced features of the target object contained in the enhanced initial image and the enhanced reference image are used as prior information, thus improving the accuracy of the target transformation parameters and resulting in a higher quality panoramic image of the site. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the structure of the image stitching model provided in the embodiments of this application;

[0069] Figure 2 This is a schematic diagram illustrating an application scenario of the image stitching method provided in the embodiments of this application;

[0070] Figure 3 This is a schematic flowchart of an image stitching method provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of an image capture area provided in an embodiment of this application for acquiring an initial image and a reference image via a camera;

[0072] Figure 5 This is a schematic diagram of a process for obtaining an initial image and a reference image provided in an embodiment of this application;

[0073] Figure 6 This is a schematic diagram provided in this application embodiment of a method for registering initial images to obtain a target site image when there are multiple initial images;

[0074] Figure 7 This is a schematic diagram of a process for retraining a preset target parameter prediction model provided in an embodiment of this application;

[0075] Figure 8 This is a schematic diagram of an embodiment of the image stitching device provided in this application.

[0076] Figure 9 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0079] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0080] This application provides an image stitching method, apparatus, electronic device, and readable storage medium. The image stitching apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0081] The execution subject of the image stitching method in this application embodiment can be the image stitching device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the image stitching device. The image stitching device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0082] The electronic device can operate independently or in a cluster.

[0083] First, we introduce an image stitching model that can be used for image stitching in logistics sites, referencing... Figure 1 , Figure 1 This is a schematic diagram of a structure of an image stitching model 100. The image stitching model 100 includes:

[0084] Image preprocessing layer 101 is used to acquire video streams from pre-set monitoring equipment in the logistics site and select multiple images to be stitched from the video streams.

[0085] Image correction layer 102 is used to correct distortion in the images to be stitched together. Distortion correction refers to correcting image distortions such as wide-angle lens distortion and fisheye lens distortion caused by camera lenses.

[0086] The parameter prediction layer 103 is used to predict the registration parameters of the distortion-corrected images to be stitched together. The parameter prediction layer 103 includes a feature enhancement sublayer 1031 and a logistic regression (LR) sublayer 1032. The logistic regression sublayer 1032 predicts the registration parameters, while the feature enhancement sublayer 1031 performs feature enhancement processing on the distortion-corrected images to be stitched together. The enhanced images are then input into the feature enhancement sublayer 1031 to predict the registration parameters of the images to be registered within the stitched images.

[0087] Image registration layer 104 is used to register the image to be registered in the image to be stitched according to the registration parameters, so as to obtain the registered image.

[0088] Image stitching layer 105 is used to stitch the registered image and the reference image in the image to be stitched together to obtain a stitched panoramic image.

[0089] See Figure 2 , Figure 2This is a schematic diagram of a scenario for the image stitching system provided in an embodiment of this application. The image stitching system may include an electronic device 200, which integrates an image stitching device.

[0090] In addition, such as Figure 2 As shown, the image stitching system may also include a memory 201 for storing data, such as text data.

[0091] It should be noted that, Figure 2 The schematic diagram of the image stitching system shown is merely an example. The image stitching system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of image stitching systems and the emergence of new business scenarios, the technical solutions provided in this invention are also applicable to similar technical problems.

[0092] The image stitching method provided in this application embodiment is described below. In this application embodiment, an electronic device is used as the execution subject. For simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. The image stitching method includes: acquiring an initial image and a reference image of a target object; performing prediction based on the enhanced initial image and the enhanced reference image to obtain target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object; registering the initial image according to the target transformation parameters to obtain a target site image; and stitching the target site image and the reference image together to obtain a panoramic site image.

[0093] Reference Figure 3 , Figure 3 This is a flowchart illustrating an image stitching method provided in an embodiment of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. Specifically, the image stitching method may include the following steps 301-304, wherein:

[0094] 301. Obtain the initial image and reference image of the target object.

[0095] The initial image and the reference image are images to be stitched together. In this embodiment of the application, the image that is not registered during stitching is called the reference image, and the image that needs to be registered during stitching is called the initial image. Registration refers to mapping each pixel in the image to be registered from the space of the image to be registered to the space of the reference image.

[0096] In this embodiment, the initial image and the reference image can be images selected from the video stream. For example, the initial image and the reference image can be obtained by using the image preprocessing layer 101 in the image stitching model 100 to obtain images corresponding to the same shooting time point from two video streams captured by two adjacent cameras deployed in the logistics site.

[0097] Understandably, the initial image and the reference image should contain information about the same area within the logistics site. The following example illustrates this: initial and reference images are acquired using two adjacent cameras. Figure 4 ,exist Figure 4 In the diagram, region 401 within the logistics area 400 is the image capture area of ​​the first camera, and region 402 is the image capture area of ​​the adjacent second camera. Therefore, region 403 within the logistics area 400 is the logistics area captured by both the first and second cameras. Consequently, the initial image and the reference image will contain information about the same area within the logistics area. For ease of description, the image regions containing this information in the initial and reference images will be referred to as the overlapping region of the initial and reference images. To improve the accuracy of image registration, the initial and reference images can be images that contain the target object within the overlapping region.

[0098] In this embodiment of the application, the target object can refer to an object in the logistics site whose shape, location, and other attributes do not change over time. For example, the target object can be a site zoning sign, loading bay, parking line, site wall, etc. in the logistics site.

[0099] Site zoning signs are site markings used to distinguish different functional areas within a logistics site. For example, site zoning signs can be used to separate forklift unloading areas from non-unloading areas.

[0100] Loading bays refer to the bays used by transport vehicles in logistics sites during loading and unloading.

[0101] Parking lines are markings used to distinguish different parking areas.

[0102] Site walls refer to the walls in a logistics site, which can be load-bearing walls, side walls, etc.

[0103] 302. Based on the enhanced initial image and the enhanced reference image, a prediction is made to obtain the target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object.

[0104] The target transformation parameters refer to the parameters used when registering the initial image. For example, the target transformation parameters can refer to the parameters used when performing rigid transformations on the initial image. For instance, the target transformation parameters can include parameters used when performing various rigid transformations such as rotation and translation.

[0105] In this embodiment, feature enhancement can be performed on the initial image and the reference image first, and then the enhanced features can be used as prior information to improve the accuracy of the target transformation parameters. For example, the features of the target object in the initial image and the reference image can be enhanced, and then the enhanced features of the target object can be used as prior information to improve the accuracy of the target transformation parameters. This is because the shape, position, and other attributes of the target object do not change over time; therefore, using its features as prior information provides better stability. Conversely, using image features that change over time, such as those of workers or goods, as prior information would actually reduce the accuracy of the target transformation parameters. Taking the initial image and reference image obtained by two adjacent cameras as an example, if the image features that change over time, such as the characteristics of workers and goods, are used as prior information, the transmission and execution of the commands will have different delays for different cameras during the process of the electronic device issuing the shooting command. Therefore, even if the commands are issued at the same time, the actual time points corresponding to the initial image and the reference image will be different. As a result, the image features that change over time, such as the characteristics of workers and goods, contained in the overlapping area of ​​the initial image and the reference image will be different. Consequently, when these image features are used as prior information, the predicted target transformation parameters will be inaccurate. However, if the features that do not change over time, such as the characteristics of the site zoning marks, loading positions, parking lines, and site walls, are used as prior information, even if the actual time points corresponding to the initial image and the reference image are different, the prior information in the overlapping area of ​​the initial image and the reference image will not change. Therefore, the accuracy of the target transformation parameters can be improved.

[0106] In some embodiments, the edge features of the target object in the initial image and the reference image can be enhanced using the following methods to obtain the enhanced initial image and the enhanced reference image:

[0107] (i) Edge detection is performed on the first preset region of the initial image and the second preset region of the reference image respectively to obtain the first initial feature map and the second initial feature map of the target object.

[0108] The first preset region and the second preset region refer to the overlapping region in the initial image and the overlapping region in the reference image, respectively. Since the image capture area of ​​the camera is fixed in most logistics sites, the position of the overlapping region can be obtained in advance.

[0109] Edge detection refers to the detection of the edges of objects in an image. It involves identifying the edges of objects by analyzing the gradient of pixel values ​​across the image. Edge detection operators such as Sobel, Prewitt, Roberts, Canny, and Marr-Hildreth can be used to perform edge detection on a first preset region of the initial image and a second preset region of the reference image to obtain a first initial feature map and a second initial feature map of the target object.

[0110] In this embodiment of the application, the edges of the target object in the first preset region and the second preset region can be detected to obtain the first initial feature map and the second initial feature map of the target object. It can be understood that the first initial feature map contains the edge information of the target object in the first preset region, and the second initial feature map contains the edge information of the target object in the second preset region.

[0111] (ii) The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object.

[0112] In this embodiment of the application, the first initial feature map and the second initial feature map can be Gaussian blurred by a preset Gaussian filter kernel to obtain a first enhanced feature map and a second enhanced feature map with more obvious features.

[0113] Among them, the Gaussian filter kernel is a linear filter that can effectively suppress noise and smooth images.

[0114] (iii) The enhanced feature maps are fused with the corresponding images in the initial image and the reference image to obtain the enhanced initial image and the enhanced reference image.

[0115] In the embodiments of this application, each enhanced feature map can be fused with the corresponding images in the initial image and the reference image through mean fusion, median fusion, and other methods.

[0116] Understandably, the first enhanced feature map corresponds to the initial image, and the second enhanced feature map corresponds to the reference image. Fusing the first enhanced feature map and the initial image yields the enhanced initial image, and fusing the second enhanced feature map and the reference image yields the enhanced reference image.

[0117] For example, steps (i) to (iii) above can be achieved through the feature enhancement sublayer 1031 in the image stitching model 100.

[0118] As can be seen, through the above steps (i) to (iii), the edge features of the target object in the initial image and the reference image can be enhanced to highlight the prior information in the initial image and the reference image and improve the accuracy of the target transformation parameters.

[0119] In this embodiment, to reduce the computational load during image stitching and improve the real-time performance of the image stitching method, the traditional methods of extracting feature points (such as SIFT (Scale-invariant feature transform)) or extracting feature point pairs using deep learning algorithms and then calculating transformation parameters using algorithms such as RANSAC (Random Sample Consensus) are abandoned. Instead, the target transformation parameters are predicted based on a logistic regression model in deep learning. For example, the parameter prediction layer 103 in the image stitching model 100 can predict the target transformation parameters based on the enhanced initial image and the enhanced reference image. In this case, the step "predicting based on the enhanced initial image and the enhanced reference image to obtain the target transformation parameters of the initial image" can be implemented as follows: using a preset target parameter prediction model, the target transformation parameters are predicted based on the enhanced initial image and the enhanced reference image.

[0120] The target parameter prediction model can refer to a trained logistic regression model. Therefore, the layer parameters of the logistic regression sublayer 1032 in the image stitching model 100 can be used as the model parameters of the target parameter prediction model.

[0121] For example, a target parameter prediction model can be obtained by training a training dataset with the parameter prediction model to be trained. For instance, multiple training image pairs can be acquired through two adjacent cameras, and the label information of each training image pair can be manually labeled to obtain a training dataset. The parameter prediction model to be trained can then be trained using the training dataset, wherein there are overlapping regions between the training images in each training image pair.

[0122] In other embodiments, the training dataset may contain only multiple independent training images. By performing image transformations on each training image, image pairs and their label information can be obtained. This reduces the amount of data in the training dataset and lowers the equipment requirements for acquiring the training dataset, thereby reducing the cost of acquiring it. In this case, the preset target parameter prediction model can be trained using the following steps:

[0123] (1) Obtain a training dataset, wherein the training dataset contains a first training image of the target object.

[0124] This application does not limit the method for obtaining the training dataset. For example, the electronic device can download the training dataset from an open-source website, such as ImageNet. Alternatively, to improve the relevance of the first training image to the logistics site, a camera in the logistics site can be controlled to capture a first training image containing the target object.

[0125] ImageNet is a large visualization database used for research on visual object recognition software.

[0126] It should be noted that, in order to improve the training effect, images that do not contain features that change over time, such as those of staff and goods, can also be used as the first training images.

[0127] (2) Perform image transformation on the enhanced first training image according to the preset image transformation parameters to obtain the enhanced second training image, wherein the enhanced first training image contains the enhanced features of the target object.

[0128] In this embodiment, the image transformation can refer to a rigid transformation. Therefore, the preset image transformation parameters can refer to the parameters used in a rigid transformation. For example, the enhanced first training image can be subjected to various types of rigid transformations, such as rotation and translation, according to the preset image transformation parameters to obtain the enhanced second training image. It is understood that the second training image also contains the enhanced features of the target object.

[0129] The method for enhancing the first training image can refer to steps (i) to (iii). First, edge detection is performed on the first training image to obtain the third initial feature map of the target object in the first training image. Then, the third initial feature map is filtered by a preset Gaussian filter kernel to obtain the third enhanced feature map. Finally, the third enhanced feature map is fused with the first training image to obtain the enhanced first training image. The Gaussian filter kernel used during training can be the same as that in step (ii) or different.

[0130] (3) Based on the enhanced first training image and the enhanced second training image, as well as the image transformation parameters, train the initial parameter prediction model to be trained to obtain the target parameter prediction model.

[0131] Understandably, based on the first and second training images, as well as the image transformation parameters used as labels, the initial parameter prediction model to be trained can be trained to obtain a target parameter prediction model that can accurately acquire the target transformation parameters.

[0132] For example, steps (1)-(3) can be performed through the logistic regression sublayer 1032 in the image stitching model 100.

[0133] The following provides a process for implementing step 302 based on image stitching model 100, but this process should not be construed as a limitation on the embodiments of this application:

[0134] (a1) Obtain the image stitching model 100, and obtain multiple first training images containing target objects from the preset monitoring equipment in the logistics site through the image preprocessing layer 101 and image correction layer 102 in the image stitching model 100, so as to obtain the training dataset.

[0135] (a2) Edge detection is performed on the first training image through the feature enhancement sub-layer 1031 in the image stitching model 100 to obtain the third initial feature map of the target object in the first training image. Then, the third initial feature map is filtered by a preset Gaussian filter kernel to obtain the third enhanced feature map. Finally, the third enhanced feature map is fused with the first training image to obtain the enhanced first training image.

[0136] (a3) Perform rigid transformations such as rotation and translation on the enhanced first training image according to the preset image transformation parameters to obtain the second training image.

[0137] (a4) Input the first training image and the second training image into the logistic regression sub-layer 1032 in the image stitching model 100, and use the image transformation parameters as labels to train the logistic regression sub-layer 1032 to obtain the trained image stitching model 100.

[0138] (a5) Using the image preprocessing layer 101 and image correction layer 102 in the trained image stitching model 100, the initial image and the reference image containing the target object are obtained from two adjacent cameras in the logistics site.

[0139] (a6) By using the feature enhancement sub-layer 1031 in the trained image stitching model 100, edge detection is performed on the overlapping regions of the initial image and the overlapping regions of the reference image to obtain the first initial feature map and the second initial feature map of the target object. The first initial feature map and the second initial feature map are then filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object with more obvious features.

[0140] (a7) Through the feature enhancement sub-layer 1031 in the trained image stitching model 100, each enhanced feature map is fused with the corresponding image in the initial image and the reference image to obtain the enhanced initial image and the enhanced reference image.

[0141] (a8) Input the enhanced initial image and the enhanced reference image into the logistic regression sub-layer 1032 of the trained image stitching model 100 to predict the target transformation parameters of the initial image.

[0142] As can be seen, the trained image stitching model 100 can enhance the features of the target object in the initial image and the reference image. Then, based on the enhanced initial image and the enhanced reference image, the enhanced features of the target object are used as prior information to predict the target transformation parameters. Therefore, the obtained target transformation parameters are more accurate.

[0143] 303. Based on the target transformation parameters, register the initial image to obtain the target site image.

[0144] The target site image refers to the image obtained after registering the initial image. In this embodiment, the initial image can be transformed according to the target transformation parameters to achieve registration.

[0145] For example, the initial image can be registered using the image registration layer 104 in the trained image stitching model 100 to obtain the target site image. For instance, the target transformation parameters obtained in step (a8) and the initial image can be input into the image registration layer 104 in the trained image stitching model 100. Through the image registration layer 104, the initial image can be rotated, translated, or otherwise transformed according to the target transformation parameters to obtain the target site image.

[0146] 304. The target site image and the reference image are stitched together to obtain a panoramic image of the site.

[0147] In this embodiment, stitching refers to merging the overlapping areas between the target site image and the reference image to achieve stitching between the target site image and the reference image. It can be understood that the stitched panoramic image of the site simultaneously contains information from both the target site image and the reference image.

[0148] For example, the target site image and the reference image can be input into the image stitching layer 105 of the trained image stitching model 100, and the target site image and the reference image can be stitched together by the image stitching layer 105 to obtain a panoramic image of the site.

[0149] In summary, the image stitching method for logistics sites provided in this application includes: acquiring an initial image and a reference image of a target object; performing prediction based on the enhanced initial image and the enhanced reference image to obtain target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain enhanced features of the target object; registering the initial image according to the target transformation parameters to obtain a target site image; and stitching the target site image and the reference image together to obtain a panoramic image of the site. On one hand, the image stitching method provided in this application obtains the target transformation parameters for registration through prediction, without needing to acquire feature points in the image, thus resulting in lower computational load and better real-time performance compared to traditional methods. On the other hand, the image stitching method provided in this application enhances the initial image and the reference image separately, and then performs prediction based on the enhanced initial image and the enhanced reference image. During the prediction process, the enhanced features of the target object contained in the enhanced initial image and the enhanced reference image are used as prior information, thus improving the accuracy of the target transformation parameters and resulting in a higher quality panoramic image of the site.

[0150] In some embodiments, if the electronic device obtains an initial image and a reference image based on a video stream acquired by a camera, it can first pair the video frames in the video stream according to their capture time, and then filter the resulting video frame pairs, removing video frame pairs in which at least one corresponding video frame does not contain the target object, to obtain an initial image and a reference image containing the target object. (Reference) Figure 5 At this point, the step "Acquire the initial image and reference image of the target object" includes:

[0151] 501. Obtain multiple initial image pairs.

[0152] Each initial image pair contains a first candidate image and a second candidate image. The first candidate image and the second candidate image were captured at the same time, and there is an overlapping area between the first candidate image and the second candidate image.

[0153] For example, the image preprocessing layer 101 in the trained image stitching model 100 can be used to process two video streams captured by two adjacent cameras deployed in the logistics site according to the shooting time to obtain multiple initial image pairs composed of video frames.

[0154] 502. For each of the initial image pairs, detect the target object in the first preset region of the first candidate image and the second preset region of the second candidate image.

[0155] The explanation of the first preset region and the second preset region can be found above. The first preset region refers to the overlapping region in the first candidate image, and the second preset region refers to the overlapping region in the second candidate image.

[0156] For example, a trained classification model can be used to detect target objects.

[0157] The classification model can be a model composed of Convolutional Neural Networks (CNNs), which may include layers such as feature extraction layers and prediction layers.

[0158] The feature extraction layer is used to perform operations such as convolution and pooling on the input image to obtain the image features in the input image.

[0159] The prediction layer is used to classify objects in the input image based on the image features output by the feature extraction layer.

[0160] 503. Extract target image pairs from each of the initial image pairs, where both the first preset region and the second preset region contain the target object.

[0161] If the target object is contained in both the first preset region of the first candidate image and the second preset region of the second candidate image in an initial image pair, then the initial image pair can be used as the target image pair.

[0162] In some embodiments, there may be multiple candidate image pairs in the initial image pair that can serve as the target image pair. In this case, the target image pair can be selected based on the confidence scores obtained when detecting the first and second candidate images in each candidate image pair. For example, the average confidence scores of the first and second candidate images in each candidate image pair can be calculated during detection, and then the candidate image pair with the largest average confidence score can be selected as the target image pair.

[0163] 504. Set the first candidate image and the second candidate image included in the target image pair as the initial image and the reference image, respectively.

[0164] As can be seen, by using the methods in steps 501-504, initial image pairs that do not contain the target object can be filtered out, and the initial image and reference image containing the target object can be accurately obtained.

[0165] In some embodiments, there may be multiple initial images. In this case, the target transformation parameters used in the actual stitching can be determined based on the similarity between each registered initial image and the reference image. (Reference) Figure 6 At this point, the step "registering the initial image according to the target transformation parameters to obtain the target site image" includes:

[0166] 601. Based on the target transformation parameters corresponding to each initial image, register each initial image to obtain a first site image.

[0167] Understandably, since there are multiple initial images, there are also multiple predicted target transformation parameters.

[0168] The first site image includes images obtained by registering each initial image according to the corresponding target transformation parameters. Since each initial image is different, the predicted target transformation parameters are different, and therefore the registered images are also different.

[0169] 602. Calculate the similarity between the reference image and each of the first site images, and determine the target initial image corresponding to the first site image with the highest similarity.

[0170] 603. According to the target transformation parameters corresponding to the target initial image, register the initial images other than the target initial image to obtain the second site image.

[0171] 604. Set the first site image and each of the second site images of the target initial image as the target site image.

[0172] The similarity between each target site image and the reference image can be used to characterize the registration effect of each target transformation parameter. A higher similarity indicates a more similar overlapping area between the registered target site image and the reference image, thus resulting in a better registration effect for the corresponding target transformation parameters. Consequently, the stitching artifacts in the resulting panoramic site image are less noticeable after registration and stitching using these target transformation parameters. Therefore, the initial image corresponding to the first site image with the highest similarity can be obtained to obtain the target initial image. Then, according to the target transformation parameters corresponding to the target initial image, all images in the initial image except the target initial image are registered. The second site image obtained after registration and the first site image of the target initial image are then used as the target site image.

[0173] Since the target transformation parameters corresponding to the initial target image are the transformation parameters with the best registration effect, after stitching the target site image obtained from steps 601-604 with the reference image, a panoramic site image with inconspicuous stitching marks and high image quality can be obtained.

[0174] In some embodiments, the prediction performance of the target parameter prediction model in step 302 can also be evaluated based on the image quality of the panoramic site image. (See reference) Figure 7 At this point, after the step of "stitching the target site image and the reference image to obtain a panoramic site image", the method further includes:

[0175] 701. Obtain the image quality assessment value of the panoramic image of the site.

[0176] Image quality assessment values ​​are values ​​used to evaluate image quality. This application does not limit the types of image quality assessment values; for example, an image quality assessment value can be the peak signal-to-noise ratio (PSNR).

[0177] Peak signal-to-noise ratio (PSNR) is an engineering term that represents the ratio of the maximum possible power of a signal to the power of destructive noise that affects the accuracy of its representation.

[0178] 702. If the image quality assessment value is lower than a preset assessment threshold, then the preset target parameter prediction model is trained using at least one of the initial image and the reference image to obtain an updated target parameter prediction model, wherein the preset target parameter prediction model is used to predict the target transformation parameters.

[0179] If the image quality assessment value of the panoramic image of the site is lower than the preset assessment threshold, it indicates that the image quality of the panoramic image of the site is poor. There may be a large number of image defects such as stitching marks and uneven color. Therefore, at least one of the initial image and the reference image can be used as the training image to retrain the target parameter prediction model, so as to improve the target parameter prediction model's ability to predict the target transformation parameters under the logistics site and obtain the updated target parameter prediction model.

[0180] To better implement the image stitching method in the embodiments of this application, an image stitching device is also provided in the embodiments of this application, such as... Figure 8 The diagram shown is a structural schematic of one embodiment of the image stitching device in this application. The image stitching device 800 includes:

[0181] The acquisition module 801 is used to acquire the initial image and the reference image of the target object;

[0182] The prediction module 802 is used to predict the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object;

[0183] The registration module 803 is used to register the initial image according to the target transformation parameters to obtain the target site image;

[0184] The stitching module 804 is used to stitch the target site image and the reference image together to obtain a panoramic image of the site.

[0185] In one possible implementation of this application, the target object includes at least one of the following: site zoning markers, loading bays, parking lines, and site walls.

[0186] In one possible implementation of this application, the image stitching device 800 further includes an enhancement module 805, which is used for:

[0187] Edge detection is performed on a first preset region of the initial image and a second preset region of the reference image to obtain a first initial feature map and a second initial feature map of the target object.

[0188] The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object.

[0189] The first enhanced feature map is fused with the initial image, and the second enhanced feature map is fused with the reference image to obtain the enhanced initial image and the enhanced reference image, respectively.

[0190] In one possible implementation of this application, the prediction module 802 is further configured to:

[0191] The target transformation parameters are obtained by using a preset target parameter prediction model based on the enhanced initial image and the enhanced reference image.

[0192] The preset target parameter prediction model is trained through the following steps:

[0193] Obtain a training dataset, wherein the training dataset contains a first training image of the target object;

[0194] The enhanced first training image is transformed according to preset image transformation parameters to obtain an enhanced second training image, wherein the enhanced first training image contains the enhanced features of the target object;

[0195] Based on the enhanced first training image and the enhanced second training image, as well as the image transformation parameters, the initial parameter prediction model to be trained is trained to obtain the target parameter prediction model.

[0196] In one possible implementation of this application, the acquisition module 801 is further configured to:

[0197] Obtain multiple initial image pairs;

[0198] For each of the initial image pairs, detect the target object in the first preset region of the first candidate image and the second preset region of the second candidate image;

[0199] From each of the initial image pairs, extract the target image pairs that contain the target object in both the first and second preset regions;

[0200] The first candidate image and the second candidate image included in the target image pair are respectively set as the initial image and the reference image.

[0201] In one possible implementation of this application, the initial image consists of multiple images, and the registration module 803 is further configured to:

[0202] Based on the target transformation parameters corresponding to each initial image, the initial images are registered to obtain the first site image;

[0203] Calculate the similarity between the reference image and each of the first site images, and determine the target initial image corresponding to the first site image with the highest similarity.

[0204] According to the target transformation parameters corresponding to the target initial image, the initial images other than the target initial image are registered to obtain the second site image;

[0205] The first site image and each of the second site images of the target initial image are set as the target site images.

[0206] In one possible implementation of this application, the image stitching device 800 further includes a retraining module 806, which is used for:

[0207] Obtain the image quality assessment value of the panoramic image of the site;

[0208] If the image quality assessment value is lower than a preset assessment threshold, then the preset target parameter prediction model is trained using at least one of the initial image and the reference image to obtain an updated target parameter prediction model, wherein the preset target parameter prediction model is used to predict the target transformation parameters.

[0209] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0210] Since the image stitching device can perform the steps in the image stitching method in any embodiment, it can achieve the beneficial effects that the image stitching method in any embodiment of this application can achieve, as detailed in the preceding description, and will not be repeated here.

[0211] Furthermore, to better implement the image stitching method in the embodiments of this application, based on the image stitching method, the embodiments of this application also provide an electronic device, see below. Figure 9 , Figure 9 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 901. The processor 901 is used to execute the computer program stored in the memory 902 to implement the steps of the image stitching method in any embodiment; or, the processor 901 is used to execute the computer program stored in the memory 902 to implement, as shown in the following diagram. Figure 8 The functions of each module in the corresponding embodiment.

[0212] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 902 and executed by processor 901 to complete the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0213] The electronic device may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0214] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0215] The memory 902 can be used to store computer programs and / or modules. The processor 901 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902 and by calling data stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0216] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the image stitching device, electronic device and its corresponding modules described above can be referred to the description of the image stitching method in any embodiment, and will not be repeated here.

[0217] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a readable storage medium and loaded and executed by a processor.

[0218] Therefore, embodiments of this application provide a readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps in the image stitching method in any embodiment of this application. For specific operations, please refer to the description of the image stitching method in any embodiment, which will not be repeated here.

[0219] The readable storage medium may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0220] Since the instructions stored in the readable storage medium can execute the steps of the image stitching method in any embodiment of this application, the beneficial effects that the image stitching method in any embodiment of this application can achieve can be realized, as detailed in the preceding description, and will not be repeated here.

[0221] The foregoing has provided a detailed description of an image stitching method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for image stitching of a logistics site, characterized in that, include: Acquire an initial image and a reference image of the target object; the target object includes at least one of the following: site zoning markers, loading bays, parking lines, and site walls; Edge detection is performed on a first preset region of the initial image and a second preset region of the reference image to obtain a first initial feature map and a second initial feature map of the target object. The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object. The first enhanced feature map is fused with the initial image, and the second enhanced feature map is fused with the reference image to obtain the enhanced initial image and the enhanced reference image, respectively. Based on the enhanced initial image and the enhanced reference image, prediction is performed to obtain the target transformation parameters of the initial image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object; wherein the enhanced features of the target object are used as prior information. Based on the target transformation parameters, the initial image is registered to obtain the target site image; The target site image and the reference image are stitched together to obtain a panoramic image of the site.

2. The image stitching method according to claim 1, characterized in that, The step of predicting the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image includes: The target transformation parameters are obtained by using a preset target parameter prediction model based on the enhanced initial image and the enhanced reference image. The preset target parameter prediction model is trained through the following steps: Obtain a training dataset, wherein the training dataset contains a first training image of the target object; The enhanced first training image is transformed according to preset image transformation parameters to obtain an enhanced second training image, wherein the enhanced first training image contains the enhanced features of the target object; Based on the enhanced first training image and the enhanced second training image, as well as the image transformation parameters, the initial parameter prediction model to be trained is trained to obtain the target parameter prediction model.

3. The image stitching method according to claim 1, characterized in that, The acquisition of the initial image and reference image of the target object includes: Obtain multiple initial image pairs; For each of the initial image pairs, detect the target object in the first preset region of the first candidate image and the second preset region of the second candidate image; From each of the initial image pairs, extract the target image pairs that contain the target object in both the first and second preset regions; The first candidate image and the second candidate image included in the target image pair are respectively set as the initial image and the reference image.

4. The image stitching method according to claim 1, characterized in that, There are multiple initial images; The step of registering the initial image according to the target transformation parameters to obtain the target site image includes: Based on the target transformation parameters corresponding to each initial image, the initial images are registered to obtain the first site image; Calculate the similarity between the reference image and each of the first site images, and determine the target initial image corresponding to the first site image with the highest similarity. According to the target transformation parameters corresponding to the target initial image, the initial images other than the target initial image are registered to obtain the second site image; The first site image and each of the second site images of the target initial image are set as the target site images.

5. The image stitching method according to any one of claims 1-4, characterized in that, After stitching the target site image and the reference image to obtain a panoramic site image, the method further includes: Obtain the image quality assessment value of the panoramic image of the site; If the image quality assessment value is lower than a preset assessment threshold, then the preset target parameter prediction model is trained using at least one of the initial image and the reference image to obtain an updated target parameter prediction model, wherein the preset target parameter prediction model is used to predict the target transformation parameters.

6. An image stitching device, characterized in that, include: The acquisition module is used to acquire an initial image and a reference image of the target object; the target object includes at least one of the following: site zoning marks, loading positions, parking lines, and site walls; The enhancement module is used to perform edge detection on a first preset region of the initial image and a second preset region of the reference image, respectively, to obtain a first initial feature map and a second initial feature map of the target object; The first initial feature map and the second initial feature map are filtered by a preset Gaussian filter kernel to obtain the first enhanced feature map and the second enhanced feature map of the target object. The first enhanced feature map is fused with the initial image, and the second enhanced feature map is fused with the reference image to obtain the enhanced initial image and the enhanced reference image, respectively. The prediction module is used to predict the target transformation parameters of the initial image based on the enhanced initial image and the enhanced reference image, wherein the enhanced initial image and the enhanced reference image contain the enhanced features of the target object; wherein the enhanced features of the target object are used as prior information. The registration module is used to register the initial image according to the target transformation parameters to obtain the target site image; The stitching module is used to stitch the target site image and the reference image together to obtain a panoramic image of the site.

7. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the image stitching method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image stitching method according to any one of claims 1 to 5.

Images