Panoramic image splicing method and device, equipment and storage medium

Abstract

The invention provides a panoramic image splicing method and device, equipment and a storage medium, relates to the technical field of image processing, and can realize low memory occupation of image splicing and improve the image splicing speed. The method comprises the following steps: acquiring original image data of a preset image processing stage in a panoramic stitching process; if the original image data contains preset image data, quantizing a pixel storage format of the preset image data, the preset image data using a first pixel storage format, and the quantized preset image data using a second pixel storage format, the quantization processing is used for converting a pixel storage format of the image data from a first pixel storage format to a second pixel storage format; applying the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process; and performing panoramic stitching processing by using the intermediate image data to obtain a stitching result of the panoramic image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a panoramic image stitching method, device, equipment and storage medium. Background Art

[0002] In recent years, there are more and more devices that can capture digital images. With the development of digital image processing technology, more and more devices equipped with cameras support panoramic stitching functions to meet people's needs for large-scale scene shooting. The panoramic stitching function has become a standard feature of most smart terminals, allowing users to capture panoramic images of their environment anytime and anywhere. In contrast, larger scenes such as urban aerial photography will bring certain challenges to the panoramic stitching algorithm.

[0003] In the related technologies, the stitching of panoramic images is relatively complicated, mainly consisting of modules such as feature extraction, image pair matching, camera estimation, gain compensation, seam calculation, image fusion, and image cropping. This process has many modules and complex algorithms, and has certain requirements on the computing power of the drone main control, making it difficult for the drone main control to stitch panoramic images and unable to guarantee the stitching speed of panoramic images. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a panoramic image stitching method, device, equipment and storage medium that overcome the above problems or at least partially solve the above problems, which can solve the problem of high computing power of the drone main control in the related technology, so as to achieve low memory usage for the stitching of panoramic images and improve the stitching speed of panoramic images. The technical solution is as follows:

[0005] In a first aspect, a method for stitching a panoramic image is provided, the method comprising:

[0006] Obtaining original image data of a preset image processing stage in the panorama stitching process;

[0007] If the original image data includes preset image data, a pixel storage format of the preset image data is quantized, the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format, and the quantization process is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format;

[0008] Applying the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process;

[0009] The intermediate image data is used to perform panoramic stitching processing to obtain a stitching result of a panoramic image.

[0010] Furthermore, after obtaining the original image data of the preset image processing stage in the panoramic stitching process, the method further includes:

[0011] When there is image overlap in the original image data, the quantization range of the image weight in different pixel storage formats is determined according to the weight threshold in the image overlap area.

[0012] Furthermore, when there is image overlap in the original image data, determining the quantization range of the image weight in different pixel storage formats according to the weight threshold in the image overlap area includes:

[0013] When there is image overlap in the original image data, setting a threshold value for the number of image overlaps;

[0014] Calculating a weight threshold of image superposition in the image overlapping area according to the image overlapping quantity threshold;

[0015] The weight threshold of the image superposition is mapped to different pixel storage formats to determine the weight quantization range of the image weight in different pixel storage formats.

[0016] Furthermore, the quantizing process of the pixel storage format of the preset image data includes:

[0017] Obtaining the representation range of preset image data using different pixel storage formats;

[0018] Determining a mapping relationship between a first pixel storage format and a second pixel storage format according to a weight quantization range of the image weight in different pixel storage formats;

[0019] The mapping relationship is used to map the representation range of the preset image data using the first pixel storage format to the representation range of the second pixel storage format.

[0020] Furthermore, applying the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process includes:

[0021] Acquire the image format used by the quantized preset image data in the preset image processing stage;

[0022] If the image format is the first storage format, weighted processing is performed on the image weights and image pixels of the quantized preset image data in the preset image processing stage to obtain intermediate image data of the panoramic stitching process;

[0023] If the image format is the second storage format, in the process of weighted processing of the image weights and image pixels of the quantized preset image data in the preset image processing stage, the image pixel values ​​in the preset image data are zero-offset and zero-offset restored to obtain intermediate image data of the panoramic stitching process.

[0024] Furthermore, after obtaining the original image data of the preset image processing stage in the panoramic stitching process, the method further includes:

[0025] The original image data is stored in an external cache, so as to provide the image data to be processed to a preset image processing stage through the external cache, and the storage time corresponding to the original image data is greater than a preset time length;

[0026] Before the preset image processing stage, the original image data of the external cache is loaded into the memory, and the first stage data obtained by quantizing the preset image data in the original image data is stored in the memory, and the first stage data is suitable for image processing after the preset image processing stage;

[0027] In the preset image processing stage, the original image data of the external cache is loaded into the memory, and the second stage data obtained by quantizing the preset image data in the original image data is stored in the memory, and the second stage data is suitable for image processing in the preset image processing stage.

[0028] Furthermore, after storing the original image data in an external cache so as to provide the image data to be processed to a preset image processing stage through the external cache, the method further includes:

[0029] Splitting the total thread into two single threads using an asynchronous loading method, wherein the total thread is a single thread formed by the process of loading the original image data of the external cache into the memory;

[0030] The two single threads are executed under a parallel strategy to process the read and write operations and the synthesis operations respectively through the two single threads.

[0031] In a second aspect, a panoramic image stitching device is provided, the device comprising:

[0032] An acquisition unit, used for acquiring original image data of a preset image processing stage in the panoramic stitching process;

[0033] a quantization unit, configured to, if the original image data includes preset image data, perform quantization processing on the pixel storage format of the preset image data, wherein the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format, and the quantization processing is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format;

[0034] A processing unit, configured to apply the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process;

[0035] The stitching unit is used to perform panoramic stitching processing using the intermediate image data to obtain a stitching result of a panoramic image.

[0036] Furthermore, the device also includes:

[0037] A determination unit is used to determine the quantization range of image weights in different pixel storage formats according to a weight threshold in an image overlapping area after obtaining the original image data of a preset image processing stage in the panoramic stitching process, when the original image data has image overlap.

[0038] Furthermore, the determining unit is specifically configured to:

[0039] When there is image overlap in the original image data, setting a threshold value for the number of image overlaps;

[0040] Calculating a weight threshold of image superposition in the image overlapping area according to the image overlapping quantity threshold;

[0041] The weight threshold of the image superposition is mapped to different pixel storage formats to determine the weight quantization range of the image weight in different pixel storage formats.

[0042] Furthermore, the quantization unit is specifically used for:

[0043] Obtaining the representation range of preset image data using different pixel storage formats;

[0044] Determining a mapping relationship between a first pixel storage format and a second pixel storage format according to a weight quantization range of the image weight in different pixel storage formats;

[0045] The mapping relationship is used to map the representation range of the preset image data using the first pixel storage format to the representation range of the second pixel storage format.

[0046] Furthermore, the processing unit is specifically used for:

[0047] Acquire the image format used by the quantized preset image data in the preset image processing stage;

[0048] If the image format is the first storage format, weighted processing is performed on the image weights and image pixels of the quantized preset image data in the preset image processing stage to obtain intermediate image data of the panoramic stitching process;

[0049] If the image format is the second storage format, in the process of weighted processing of the image weights and image pixels of the quantized preset image data in the preset image processing stage, the image pixel values ​​in the preset image data are zero-offset and zero-offset restored to obtain intermediate image data of the panoramic stitching process.

[0050] Furthermore, the device also includes:

[0051] a storage unit, configured to store the original image data in an external cache after acquiring the original image data of the preset image processing stage in the panoramic stitching process, so as to provide the image data to be processed to the preset image processing stage through the external cache, wherein the storage time corresponding to the original image data is greater than a preset time length;

[0052] A first loading unit is used to load the original image data of the external cache into the memory before the preset image processing stage, and store the first stage data obtained by quantizing the preset image data in the original image data into the memory, wherein the first stage data is suitable for image processing after the preset image processing stage;

[0053] The second loading unit is used to load the original image data of the external cache into the memory in the preset image processing stage, and store the second stage data obtained by quantizing the preset image data in the original image data into the memory, wherein the second stage data is suitable for image processing in the preset image processing stage.

[0054] Furthermore, the device also includes:

[0055] a splitting unit, configured to, after storing the original image data in an external cache so as to provide the image data to be processed to a preset image processing stage through the external cache, split the total thread into two single threads using an asynchronous loading method, wherein the total thread is a single thread formed by the process of loading the original image data in the external cache into the memory;

[0056] The execution unit is used to execute the two single threads under a parallel strategy, so as to process the read and write operations and the synthesis operations respectively through the two single threads.

[0057] According to a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the panoramic image stitching method when executing the computer program.

[0058] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned panoramic image stitching method are implemented.

[0059] By means of the above technical scheme, the panoramic image stitching method, device, equipment and storage medium provided in the embodiment of the present application are compared with the panoramic image stitching method realized by high computing power stitching algorithm in the prior art. The present application obtains the original image data of the preset image processing stage in the panoramic stitching process; if the original image data contains the preset image data, the pixel storage format of the preset image data is quantized, where the preset image data uses the first pixel storage format, and the quantized preset image data uses the second pixel storage format, the quantization process is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format, the quantized preset image data is applied to the preset image processing stage of the panoramic stitching process, the intermediate image data of the panoramic stitching process is obtained, and the panoramic stitching process is performed using the intermediate image data to obtain the stitching result of the panoramic image. The whole process is quantized in the main control of the drone with limited memory, and the memory occupation of the image data is saved while meeting the task requirements of the panoramic stitching, thereby improving the image stitching speed to a certain extent.

[0060] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0062] Figure 1 It is a flowchart of a panoramic image stitching method provided in an embodiment of the present application;

[0063] Figure 2 is a flowchart of another panoramic image stitching method provided in an embodiment of the present application;

[0064] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step 105;

[0065] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step 103;

[0066] Figure 5 is a flowchart of a panoramic image fusion process provided by an embodiment of the present application;

[0067] Figure 6 It is a structural block diagram of a panoramic image stitching device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0069] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such use is interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0070] It should be noted that the panoramic image stitching method provided in this application can be applied to terminals, for example, the terminals can be various large tablets for commercial use, mobile phones or computers, ordinary consumer tablets, smart TVs, portable computer terminals, and can also be fixed terminals such as desktop computers. For the convenience of explanation, this application uses the terminal as the execution subject for example.

[0071] The embodiments of the present application can be applied to computer systems / servers, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0072] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0073] The embodiment of the present application provides a panoramic image stitching method, which can solve the problem of high computing power of the drone main control in the related art, so as to achieve low memory usage of the panoramic image stitching and improve the panoramic image stitching speed. Figure 1 As shown, the method may include:

[0074] 101. Obtain original image data of a preset image processing stage in a panoramic stitching process.

[0075] Considering that the panoramic stitching process is relatively complicated, it involves a large number of image processing stages, and each image processing stage requires the use of image data. For example, the feature extraction stage requires feature extraction of image data, the cropping stage requires cropping of image data, and the stitching calculation stage requires stitching of images. Due to the limitations of computing power, memory and battery life, it is difficult to perform panoramic stitching directly on the drone main control. Usually, the drone main control is used to collect panoramic image data, and then the panoramic image data is stitched on the terminal or the cloud. However, there is a certain delay in the transmission process of the panoramic image, which makes it difficult to meet the real-time requirements of panoramic stitching. Therefore, the embodiment of the present invention uses the drone main control to stitch the panoramic image data, but the stitching process will be affected by computing power and memory usage. For example, when the drone main control needs to stitch multiple panoramic images, there are at least 24 images in total when there are 8 horizontal positions and three vertical positions of upper, middle and lower. The size of each image is 4000*3000. In theory, 4000*3000*3*24≈824MB of space is required to store these images, and the panoramic stitching process also requires memory usage of other functional modules such as video and system modules, and the speed of panoramic stitching cannot be guaranteed.

[0076] In this embodiment, the original image data may be the image data to be fused in the image fusion stage during the image stitching process. The original image data is usually represented in the form of a two-dimensional array for storing pixel data. The specific pixel storage format is set and may be a double-precision type or a floating-point type.

[0077] The image fusion stage is taken as an example of the preset image processing stage. In the image fusion stage, after reading each image to be fused, multiple images to be fused need to be superimposed, and finally the average value is calculated according to the weight. In addition to superimposing the pixel values ​​of the images to be fused, the weights corresponding to the images to be fused also need to be superimposed. If the weights need to be stored in a 32-bit floating-point pixel storage format, the weight representation range of each image to be fused is 0-1, and the resolution of an image to be fused can be as high as 14400×7200. A single-precision array of the same resolution is required to store this part of the weights, which will make the memory usage of the drone main control very large. In the panoramic stitching process, most of the image data is stored in a floating-point pixel storage format. If the more original image data is stored, the more memory is occupied, which increases the difficulty of running the algorithm in the drone main control.

[0078] 102. If the original image data includes preset image data, quantize the pixel storage format of the preset image data.

[0079] Among them, the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format. The quantization processing is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format. Here, the image data obtained using the first pixel storage format occupies more space than the image data obtained using the second storage format.

[0080] Generally, considering that different image data in the original image data have different pixel storage formats, image data that occupies a space greater than a set value after using the first pixel storage format will be used as preset image data. By quantizing the pixel storage format of the preset image data, the pixel storage format of the preset image data can be converted, thereby reducing the memory occupancy of the original image data.

[0081] For example, the first pixel storage format is a 32-bit single-precision floating point number, and the second pixel storage format is an 8-bit fixed point number. The pixel storage format of the preset image data can be converted from a 32-bit single-precision floating point number to an 8-bit fixed point number through quantization. Specifically, for the mixing weight of each image, if a 32-bit single-precision floating point number is used to represent the mixing weight, the representation range is 0-1, which is mapped to the range of 0-85 and can be converted to an 8-bit fixed point number, thereby realizing the entire quantization process.

[0082] 103. Apply the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process.

[0083] It can be understood that the quantized preset image data has a lower memory usage and needs to be efficiently used by the algorithm of the preset image processing stage to obtain the intermediate image data of the panoramic stitching process. For different preset image processing stages, the processing method and processing flow of the preset image data are different, and the corresponding intermediate image data obtained are also different.

[0084] 104. Perform panoramic stitching processing using the intermediate image data to obtain a panoramic image stitching result.

[0085] In this embodiment, the intermediate image data is image data obtained through processing in a preset image processing stage, and the intermediate image data is quantized image data. Compared with the previous preset image data, the process of stitching using the intermediate image data can greatly reduce the space memory occupancy of the intermediate image data without affecting the stitching effect, thereby improving the stitching speed of the panoramic image.

[0086] The panoramic image stitching method provided in the embodiment of the present application is compared with the panoramic image stitching method implemented by high computing power stitching algorithm in the prior art. The present application obtains the original image data of the preset image processing stage in the panoramic stitching process; if the original image data contains the preset image data, the pixel storage format of the preset image data is quantized, where the preset image data uses the first pixel storage format, and the quantized preset image data uses the second pixel storage format, the quantization process is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format, the quantized preset image data is applied to the preset image processing stage of the panoramic stitching process, the intermediate image data of the panoramic stitching process is obtained, and the panoramic stitching process is performed using the intermediate image data to obtain the stitching result of the panoramic image. The whole process is quantized in the main control of the drone with limited memory, and the memory occupation of the image data is saved while meeting the task requirements of the panoramic stitching, thereby improving the image stitching speed to a certain extent.

[0087] Furthermore, considering the different image processing methods in the preset processing stage, the quantization ranges that can be provided for the preset image data are also different. The quantization range of the preset image data can be set according to the weight threshold of the processed preset image data. As an implementation method in this embodiment, after obtaining the original image data of the preset image processing stage in the panoramic stitching process, such as Figure 2 As shown, the method also includes the following steps:

[0088] 105. When there is image overlap in the original image data, determine the quantization range of the image weight in different pixel storage formats according to the weight threshold in the image overlap area.

[0089] Accordingly, step 102 specifically includes the following steps:

[0090] 1021. Obtain a representation range of preset image data using different pixel storage formats.

[0091] 1022. Determine a mapping relationship between a first pixel storage format and a second pixel storage format according to a weight quantization range of the image weight in different pixel storage formats.

[0092] 1023. Map the preset image data from a representation range of the first pixel storage format to a representation range of the second pixel storage format using the mapping relationship.

[0093] Specifically, Figure 3 As shown, step 105 includes the following steps:

[0094] 1051. When there is image overlap in the original image data, set a threshold value for the amount of image overlap.

[0095] 1052. Calculate a weight threshold of image superposition in the image overlapping area according to the image overlapping quantity threshold.

[0096] 1053. Map the weight threshold of the image superposition to different pixel storage formats, and determine the weight quantization range of the image weight in different pixel storage formats.

[0097] In this example, if the preset image processing stage requires image fusion, then there must be overlap between images in the original image data of the panoramic stitching. Theoretically, the more images overlap, the larger the image weight value. By setting the weight threshold in the overlapping area, the quantization range of the image weight in different pixel storage formats can be determined according to the weight threshold in the overlapping area.

[0098] For example, when the number of overlapping images does not exceed 3, the weight threshold is set to 3, and the 32-bit single-precision floating point is converted to an 8-bit fixed point number. The weight of each image can be quantized to [0,255]. That is, for 32-bit single-precision floating point and 8-bit fixed point numbers, the mapping relationship between these two pixel storage formats can be expressed as [0,85]. In other words, when the number of overlapping images is less than 3, the weight value of image superposition will not exceed 255, and the pixel storage format of the preset image data can be converted from 32-bit single-precision floating point to 8-bit fixed point. When the number of overlapping images does not exceed 257, the weight threshold is set to 257, and the 64-bit single-precision floating point is converted to 16-bit fixed point. The weight of each image can be quantized to [0,65535 / 3]. That is, for 64-bit single-precision floating point and 16-bit fixed point numbers, the mapping relationship between these two pixel storage formats can be expressed as [0,21845]. That is, when the number of image overlaps is less than 257, the weight value of the image superposition will not exceed 65535, and the pixel storage format of the preset image data can be converted from a 64-bit single-precision floating point to an 8-bit fixed point.

[0099] It is understandable that the quantization of the preset image data stored in 32-bit single-precision floating point to 8-bit fixed-point numbers will inevitably result in precision loss. Here, the single-precision floating point range [0.0, 3.0] is mapped to the fixed-point range [0, 85]. The floating point number represented by each fixed-point unit is: 1.0 / 85≈0.0118, which indicates that each unit has a precision loss of about 1.18%. Similarly, the quantization of the preset image data stored in 64-bit single-precision floating point to 16-bit fixed-point numbers also results in precision loss, but the precision loss here is not enough for the human eye to distinguish, which can meet the actual use needs of drone panoramic stitching.

[0100] The image fusion stage is taken as an example to illustrate the preset image processing stage. The image fusion stage includes image superposition and image merging. This process requires not only superposition of pixel values, but also superposition of corresponding weights. The final pixel value is obtained by weighting. The specific calculation formula is as follows:

[0101]

[0102] Among them, I is the pixel value of the fused image, I i For each image to be superimposed, w i is the weight value of the image to be superimposed, which is generated by the "Calculate Seams" stage. During the image fusion process, the numerator and denominator need to be saved separately until all images are processed.

[0103] Considering the application of the quantized preset image data in the preset image processing stage, specifically, Figure 4 As shown, step 103 includes the following steps:

[0104] 1031. Obtain an image format used by the quantized preset image data in a preset image processing stage.

[0105] 1032. If the image format is the first storage format, weighted processing is performed on the image weights and image pixels of the quantized preset image data in a preset image processing stage to obtain intermediate image data of a panoramic stitching process.

[0106] 1033. If the image format is the second storage format, in the process of weighted processing of the image weights and image pixels of the quantized preset image data in the preset image processing stage, zero point offset and zero point offset recovery are performed on the image pixel values ​​in the preset image data to obtain intermediate image data of the panoramic stitching process.

[0107] In this embodiment, the quantized preset image data needs to use different image formats in the preset image processing stage, and different image formats need to perform different preset processing operations on the quantized preset image data.

[0108] Continue to use the image fusion stage in the above text as an example to explain the preset image processing stage. Figure 5 It is a flowchart of the image fusion stage. Figure 5 In the image fusion stage, the image fusion stage mainly includes the following two image processing processes, the image superposition process and the image merging process. Specifically, each image to be fused is read, and the image superposition process is performed on the image to be fused. The image is first preprocessed, and then the preprocessed image is superimposed to determine whether the superposition of all images has been completed. If not, the above image preprocessing and image superposition processes are repeated for the unprocessed images. If yes, the image merging process is performed on the image to be fused, that is, the superimposed images are merged to obtain the image fusion result.

[0109] Specifically, during the image superposition process, the pixel values ​​and weight values ​​of each image are accumulated into the set image. After quantization, the storage weight of the preset image is converted into an 8-bit fixed-point number. The following formula is used to update the weight value during the superposition process. That is, the denominator of the calculation formula in the image fusion stage can be expressed as the following formula:

[0110] W=∑w*85

[0111] Among them, W is the setting image for storing weights, w is the 32-bit single-precision floating-point weight value of each image, and the above quantization process is to scale the 32-bit single-precision floating-point weight value by 85 times to obtain an image represented by an 8-bit fixed-point number, and sum the images represented by the 8-bit fixed-point numbers to obtain the setting image for storing weights.

[0112] Similarly, the numerator of the calculation formula in the image fusion stage can be expressed as follows:

[0113] P=∑w*85*c

[0114] Where c is the pixel value of the image. If the image format is the first storage format, for example, RGB format, the pixel value represents the pixel value corresponding to each channel. If the image format is the second storage format, for example, YUV NV12 format, for the UV plane, the pixel value needs to be offset from its zero point before scaling to avoid the overall green bias of the image. The calculation formula is as follows:

[0115] P uv =∑w*85*(c uv -128)

[0116] Correspondingly, in the image merging process, when all images are superimposed, the final pixel value is calculated. After quantization, the final pixel value can be expressed as the following formula:

[0117]

[0118] Similarly, if the image format is the second storage format, for example, the YUV NV12 format, then for the UV plane, the pixel value needs to be offset from its zero point before scaling. At this time, the final pixel value can be identified by the following formula:

[0119]

[0120] Furthermore, as an embedded device, the main control of the drone has limited memory capacity. The input image resolution of the panoramic stitching algorithm is high and the memory requirement is large, which can easily lead to insufficient memory and image stitching failure. In this embodiment, after obtaining the original image data of the preset image processing stage in the panoramic stitching process, the original image data can be stored in the external cache to provide the image data to be processed to the preset image processing stage through the external cache. Here, the storage time corresponding to the original image data is greater than the preset duration. Here, the preset duration can be at least one image processing stage. For example, the preset duration can be the sum of the processing durations corresponding to the image feature extraction stage and the image stitching stage. Before the preset image processing stage, the original image data in the external cache is loaded into the memory, and the first stage data obtained by quantizing the preset image data in the original image data is stored in the memory. Here, the first stage data is suitable for image processing after the preset image processing stage. In the preset image processing stage, the original image data in the external cache is loaded into the memory, and the second stage data obtained by quantizing the preset image data in the original image data is stored in the memory. Here, the second stage data is suitable for image processing in the preset image processing stage.

[0121] It can be understood that by storing a portion of recently unused data in an external cache to free up memory space, the original image data at different image processing stages and the preset image data obtained by quantization can be divided into the following parts: original image data, image data required for the fusion process, image data required for feature extraction, and image data required for solving seam exposure. The image data of these parts are processed through the external cache, which specifically involves the following processing flow:

[0122] The original image data is needed in the feature extraction stage and the image stitching stage. That is to say, this part of the image data will be loaded into the memory through external storage before the feature extraction stage and the image stitching stage, and put into the external storage after the feature extraction is completed. Before the image fusion, the original image data is read from the external storage. After the image fusion task is completed, the external cache is deleted.

[0123] Before the feature extraction stage, the original image data has been loaded into the memory. After quantizing the preset image data in the original image data, the quantized first-stage data is obtained, so that the first-stage data is stored in the memory for a long time for finding seams and exposure gain tasks, and then the external cache is deleted.

[0124] In the feature extraction stage, the original image data has been loaded into the memory. After quantizing the preset image data in the original image data, the quantized second-stage data is obtained. The second-stage data is only used in the feature extraction stage and is stored in the memory for a long time. The external cache is released after the task is completed.

[0125] Specifically, in the feature extraction stage, the original image data is quantized and stored in the memory. At the same time, the original image data is also stored in the external cache as an offline cache. Then, in the image fusion stage, the required image data is loaded into the memory asynchronously, thereby effectively reducing the memory pressure and releasing the memory space occupied by the drone.

[0126] It should be noted that, although the external storage method can reduce the memory pressure to a certain extent, so that the panoramic stitching algorithm with large memory requirements can also run on an embedded platform with small memory, the use of external cache needs to consider the speed of read and write operations. Since the image data to be loaded is usually a picture with a large amount of data, the image stitching and reading operations can be implemented synchronously by separating the read and write operations. Specifically, after the original image data is stored in the external cache, the image data to be processed can be provided to the preset image processing stage through the external cache. Then, the total thread can be split into two single threads using an asynchronous loading method. The total thread here is a single thread formed by the process of loading the original image data of the external cache into the memory, and then the two single threads are executed under the parallel strategy to process the read and write operations and the synthesis operations respectively through the two single threads. Of the two single threads, one single thread is used for the read and write operations of the external storage, and the other single thread is used for the synthesis operation of the existing image data.

[0127] In this embodiment, since the cache system is built on an external memory, the speed of the external memory is far slower than that of the built-in memory. If the cache is retrieved only when it is needed, the algorithm will spend most of the time waiting for the completion of the read and write operations, resulting in significant delays. This is due to the large difference in the reading speed between the external memory and the built-in memory. Generally, the built-in memory has a faster reading speed than the external memory.

[0128] Specifically, in the image fusion stage, the time to process an image is equal to the read and write time of the external memory plus the synthesis time of the fused image, which is a serial process. In embedded devices, read and write operations do not particularly occupy CPU resources, which means that during the process of reading the cache of the external memory, the CPU resources are idle, causing great waste. This is also why the algorithm spends most of its time waiting for the read and write operations to complete.

[0129] Assume that the time required for the synthesis operation is A, the time required for the read and write operation is B, and N images need to be fused. The time required for the entire fusion process is:

[0130] t st =(A+B)*N

[0131] In order to make full use of the read and write resources of the CPU and external storage, a parallel strategy is used here to optimize the read and write speed of the external cache system. During the thread execution of the parallel strategy, the read and write operations are separated from the merge operation of the image fusion process. Two single threads are used to process the read and write operations and the synthesis operation respectively. Assuming that the time required for the synthesis operation is A, the time required for the read and write operation is B, and N images need to be fused, the time required for the calculation process can be divided into the following cases:

[0132] Case 1: The time required for the synthesis operation is less than the time required for the read and write operations. The time required for the entire fusion process is: t mt1 =A+B*N

[0133] Case 2: The time required for the synthesis operation is greater than the time required for the read and write operations. The time required for the entire fusion process is: t mt2 =B+A*N

[0134] Case 3: The time required for the synthesis operation is equal to the time required for the read and write operations. The time required for both operations is C. The time required for the entire fusion process is: t mt3 =C*(N+1)

[0135] It is understandable that the time consumption of the parallel strategy is the same as that of the total thread when and only when the time required for the two operations is equal. Due to hardware reasons, the probability that the time required for the two operations is equal is very low. If the time required for the two operations is not equal, the time required for the parallel strategy is better than that of the total thread. In addition, the time required for the synthesis operation is related to the resolution of the transformed image. Although the read and write operations read images of the same size, due to system reasons, there are some differences in the time consumption of each read and write operation, so the time consumption in actual operation cannot be quantitatively calculated. The parallel strategy has a better processing effect than the serial method.

[0136] Further, based on the same inventive concept, the embodiment of the present application also provides a panoramic image stitching device, such as Figure 6 As shown, the device includes: an acquisition unit 21, a quantization unit 22, a processing unit 23, and a splicing unit 24.

[0137] An acquisition unit 21 is used to acquire original image data of a preset image processing stage in a panoramic stitching process;

[0138] a quantization unit 22, configured to, if the original image data includes preset image data, perform quantization processing on the pixel storage format of the preset image data, wherein the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format, and the quantization processing is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format;

[0139] A processing unit 23, configured to apply the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process;

[0140] The stitching unit 24 is used to perform panoramic stitching processing using the intermediate image data to obtain a stitching result of a panoramic image.

[0141] Compared with the stitching method of panoramic images realized by high computing power stitching algorithm in the prior art, the present invention provides a panoramic image stitching device. The present invention obtains the original image data of the preset image processing stage in the panoramic stitching process; if the original image data contains the preset image data, the pixel storage format of the preset image data is quantized, where the preset image data uses the first pixel storage format, and the quantized preset image data uses the second pixel storage format. The quantization process is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format, and the quantized preset image data is applied to the preset image processing stage of the panoramic stitching process to obtain the intermediate image data of the panoramic stitching process, and the intermediate image data is used to perform the panoramic stitching process to obtain the stitching result of the panoramic image. The entire process is performed by quantizing the pixel storage format of the image data in the main control of the drone with limited memory, while meeting the task requirements of the panoramic stitching, saving the memory occupation of the image data, and improving the image stitching speed to a certain extent.

[0142] In a specific application scenario, the device further includes:

[0143] A determination unit is used to determine the quantization range of image weights in different pixel storage formats according to a weight threshold in an image overlapping area after obtaining the original image data of a preset image processing stage in the panoramic stitching process, when the original image data has image overlap.

[0144] In a specific application scenario, the determining unit is specifically used to:

[0145] When there is image overlap in the original image data, setting a threshold value for the number of image overlaps;

[0146] Calculating a weight threshold of image superposition in the image overlapping area according to the image overlapping quantity threshold;

[0147] The weight threshold of the image superposition is mapped to different pixel storage formats to determine the weight quantization range of the image weight in different pixel storage formats.

[0148] In a specific application scenario, the quantization unit is specifically used for:

[0149] Obtaining the representation range of preset image data using different pixel storage formats;

[0150] Determining a mapping relationship between a first pixel storage format and a second pixel storage format according to a weight quantization range of the image weight in different pixel storage formats;

[0151] The mapping relationship is used to map the representation range of the preset image data using the first pixel storage format to the representation range of the second pixel storage format.

[0152] In a specific application scenario, the processing unit is specifically used to:

[0153] Acquire the image format used by the quantized preset image data in the preset image processing stage;

[0154] If the image format is the first storage format, weighted processing is performed on the image weights and image pixels of the quantized preset image data in the preset image processing stage to obtain intermediate image data of the panoramic stitching process;

[0155] If the image format is the second storage format, in the process of weighted processing of the image weights and image pixels of the quantized preset image data in the preset image processing stage, the image pixel values ​​in the preset image data are zero-offset and zero-offset restored to obtain intermediate image data of the panoramic stitching process.

[0156] In a specific application scenario, the device further includes:

[0157] a storage unit, configured to store the original image data in an external cache after acquiring the original image data of the preset image processing stage in the panoramic stitching process, so as to provide the image data to be processed to the preset image processing stage through the external cache, wherein the storage time corresponding to the original image data is greater than a preset time length;

[0158] A first loading unit is used to load the original image data of the external cache into the memory before the preset image processing stage, and store the first stage data obtained by quantizing the preset image data in the original image data into the memory, wherein the first stage data is suitable for image processing after the preset image processing stage;

[0159] The second loading unit is used to load the original image data of the external cache into the memory in the preset image processing stage, and store the second stage data obtained by quantizing the preset image data in the original image data into the memory, wherein the second stage data is suitable for image processing in the preset image processing stage.

[0160] In a specific application scenario, the device further includes:

[0161] a splitting unit, configured to, after storing the original image data in an external cache so as to provide the image data to be processed to a preset image processing stage through the external cache, split the total thread into two single threads using an asynchronous loading method, wherein the total thread is a single thread formed by the process of loading the original image data in the external cache into the memory;

[0162] The execution unit is used to execute the two single threads under a parallel strategy, so as to process the read and write operations and the synthesis operations respectively through the two single threads.

[0163] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the panoramic image stitching method of any one of the above embodiments when running.

[0164] Those skilled in the art can clearly understand that the specific working processes of the systems, devices, and modules described above can refer to the corresponding processes in the aforementioned method embodiments, and for the sake of brevity, they are not further described here.

[0165] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that within the spirit and principles of the present application, they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate from the protection scope of the present application.

Claims

1. A panoramic image stitching method, characterized in that: The method comprises: Obtaining original image data of a preset image processing stage in the panoramic stitching process; If the original image data includes preset image data, a pixel storage format of the preset image data is quantized, the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format, and the quantization process is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format; Applying the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process; The intermediate image data is used to perform panoramic stitching processing to obtain a stitching result of a panoramic image.

2. The method according to claim 1, characterized in that After obtaining the original image data of the preset image processing stage in the panoramic stitching process, the method further includes: When there is image overlap in the original image data, the quantization range of the image weight in different pixel storage formats is determined according to the weight threshold in the image overlap area.

3. The method according to claim 2, characterized in that When there is image overlap in the original image data, determining the quantization range of the image weight in different pixel storage formats according to the weight threshold in the image overlap area includes: When there is image overlap in the original image data, setting a threshold value for the number of image overlaps; Calculating a weight threshold of image superposition in the image overlapping area according to the image overlapping quantity threshold; The weight threshold of the image superposition is mapped to different pixel storage formats to determine the weight quantization range of the image weight in different pixel storage formats.

4. The method according to claim 2, characterized in that: The quantizing of the pixel storage format of the preset image data includes: Obtaining the representation range of preset image data using different pixel storage formats; Determining a mapping relationship between a first pixel storage format and a second pixel storage format according to a weight quantization range of the image weight in different pixel storage formats; The mapping relationship is used to map the representation range of the preset image data using the first pixel storage format to the representation range of the second pixel storage format.

5. The method according to any one of claims 1 to 4, characterized in that The step of applying the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process includes: Acquire the image format used by the quantized preset image data in the preset image processing stage; If the image format is the first storage format, weighted processing is performed on the image weights and image pixels of the quantized preset image data in the preset image processing stage to obtain intermediate image data of the panoramic stitching process; If the image format is the second storage format, in the process of weighted processing of the image weights and image pixels of the quantized preset image data in the preset image processing stage, the image pixel values ​​in the preset image data are zero-offset and zero-offset restored to obtain intermediate image data of the panoramic stitching process.

6. The method according to any one of claims 1 to 4, characterized in that After obtaining the original image data of the preset image processing stage in the panoramic stitching process, the method further includes: The original image data is stored in an external cache, so as to provide the image data to be processed to a preset image processing stage through the external cache, and the storage time corresponding to the original image data is greater than a preset time length; Before the preset image processing stage, the original image data of the external cache is loaded into the memory, and the first stage data obtained by quantizing the preset image data in the original image data is stored in the memory, and the first stage data is suitable for image processing after the preset image processing stage; In the preset image processing stage, the original image data of the external cache is loaded into the memory, and the second stage data obtained by quantizing the preset image data in the original image data is stored in the memory, and the second stage data is suitable for image processing in the preset image processing stage.

7. The method according to claim 6, characterized in that After storing the original image data in an external cache so as to provide the image data to be processed to a preset image processing stage through the external cache, the method further includes: Splitting the total thread into two single threads using an asynchronous loading method, wherein the total thread is a single thread formed by the process of loading the original image data of the external cache into the memory; The two single threads are executed under a parallel strategy to process the read and write operations and the synthesis operations respectively through the two single threads.

8. A panoramic image stitching device, characterized in that: The device comprises: An acquisition unit, used to acquire original image data of a preset image processing stage in the panoramic stitching process; a quantization unit, configured to, if the original image data includes preset image data, perform quantization processing on the pixel storage format of the preset image data, the preset image data uses a first pixel storage format, and the quantized preset image data uses a second pixel storage format, and the quantization processing is used to convert the pixel storage format of the image data from the first pixel storage format to the second pixel storage format; A processing unit, configured to apply the quantized preset image data to a preset image processing stage of a panoramic stitching process to obtain intermediate image data of the panoramic stitching process; The stitching unit is used to perform panoramic stitching processing using the intermediate image data to obtain a stitching result of a panoramic image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the panoramic image stitching method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the panoramic image stitching method according to any one of claims 1 to 7 are implemented.

Images