Image processing method, electronic device, and computer-readable storage medium

Abstract

This document provides an image processing method, an electronic device, and a computer-readable storage medium. The method includes: acquiring multiple first images associated with a first location, wherein the first images are obtained by capturing images of the first location from different viewpoints; adjusting the exposure features of the multiple first images based on reference features to obtain multiple adjusted first images; and generating a panoramic image associated with the first location based on the multiple adjusted first images, wherein pixels in the panoramic image are associated with pixels in the adjusted first images. Therefore, the panoramic image generated based on the adjusted first images can effectively avoid the problem of stitching and layering, and improve the image quality of the generated panoramic image.

Description

Technical Field

[0001] This article relates to the fields of image processing and artificial intelligence, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Panoramic street view image stitching is a key technology in fields such as autonomous driving, digital twins, and geographic information visualization. Its core objective is to generate a 360° seamless panoramic image by geometric transformation and pixel fusion from continuous, multi-view images, thus fully restoring the visual information of the scene.

[0003] However, current panoramic images generated from multi-view images often suffer from stitching and layering issues, affecting user experience and the reliability of information delivery. Summary of the Invention

[0004] This paper provides an image processing method, electronic device, and computer-readable storage medium to solve the technical problems of layering and blurring in panoramic images obtained by multi-view image stitching.

[0005] Firstly, this paper provides an image processing method, including:

[0006] Acquire multiple first images associated with the first location, wherein the first images are obtained by capturing images of the first location from different perspectives;

[0007] The exposure features of the multiple first images are adjusted based on reference features to obtain multiple adjusted first images;

[0008] A panoramic image associated with the first location is generated based on the multiple adjusted first images, wherein the pixels in the panoramic image are associated with the pixels in the adjusted first images.

[0009] Secondly, this paper provides an image processing apparatus, comprising:

[0010] The acquisition module is used to acquire multiple first images associated with the first location, wherein the first images are obtained by acquiring images of the first location from different perspectives;

[0011] An adjustment module is used to adjust the exposure features of the plurality of first images based on reference features to obtain a plurality of adjusted first images;

[0012] A generation module is configured to generate a panoramic image associated with the first location based on the plurality of adjusted first images, wherein the pixels in the panoramic image are associated with the pixels of the adjusted first images.

[0013] Thirdly, this article provides an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image processing method as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, this document provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the image processing method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, this document provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method described in the first aspect and various possible designs of the first aspect.

[0018] The image processing method, electronic device, and computer-readable storage medium presented in this paper adaptively adjust the exposure features of multiple first images acquired from multiple viewpoints based on reference features, thus ensuring that the multiple first images have the same exposure characteristics. Therefore, the panoramic image generated based on the adjusted first images can effectively avoid the stitching and layering problem, improving the image quality of the generated panoramic image. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this document or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this document. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a system architecture diagram based on some scenarios in this article;

[0021] Figure 2 A flowchart illustrating the image processing method presented in this paper;

[0022] Figure 3 This is another flowchart illustrating the image processing method presented in this paper;

[0023] Figure 4 This is another flowchart illustrating the image processing method presented in this paper;

[0024] Figure 5This is another flowchart illustrating the image processing method presented in this paper;

[0025] Figure 6 This is a schematic diagram of the image processing device provided in this article;

[0026] Figure 7 This is a schematic diagram of the electronic device provided in this article. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this document clearer, the technical solutions described below will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of this document.

[0028] It is understood that before using the technical solutions disclosed in the embodiments of this article, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this article in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as electronic devices, applications, servers, or storage media, that perform the operations described herein, based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation method described in this article. Other methods that comply with relevant laws and regulations may also be applied to the implementation method described in this article.

[0032] Panoramic street view stitching technology is a core supporting technology in fields such as autonomous driving, digital twins, and geographic information visualization. In autonomous driving scenarios, vehicles need to perceive their surroundings in real time through high-precision 360° street view images, such as identifying road signs, pedestrians, and obstacles, to achieve path planning and obstacle avoidance functions. The stitching quality of the street view image directly affects the accuracy and safety of the autonomous driving system's environmental perception. If there are stitching layering or blurring issues, it may lead to misjudgments or missed detections, thereby causing safety hazards. In the field of digital twins, panoramic street view images serve as the basic data for urban 3D models. They need to recreate real scenes through seamless, high-definition images to support applications such as urban planning and virtual simulation. For example, in digital twin systems, the stitching quality of the street view image determines the visual realism of the model and the accuracy of spatial data. Layering or blurring issues will cause model distortion and affect decision analysis. In the field of geographic information visualization, panoramic street view images are widely used in map services, tourist navigation, and other scenarios, with users relying on them to obtain intuitive spatial information. If the image has layering or blurring, it will seriously affect the user experience and the reliability of information transmission.

[0033] In related technologies, the pose and intrinsic parameters of multi-view images can be analyzed, and the image pixels can be superimposed according to cylindrical projection coordinates, with the last superimposed pixel as the final value. However, due to the exposure differences in multi-view images, panoramic images obtained by this method often have stitching layering problems.

[0034] In the process of solving the above-mentioned technical problems, the inventors discovered through research that in order to avoid the problem of stitching and layering in panoramic images, the exposure characteristics of multi-view images can be adaptively adjusted to remove the exposure differences in the adjusted multi-view images.

[0035] Furthermore, weight information can be determined based on the pixel positions in multi-view images and their associated exposure features. Pixel projection is then performed based on this weight information. This balances the pixel fusion effect and reduces blur. Ultimately, a panoramic image with both visual continuity and high definition is generated, meeting the high-precision requirements of fields such as autonomous driving and digital twins.

[0036] For example, Figure 1 Here are some system architecture diagrams used in this article, such as Figure 1 As shown, the system architecture includes at least a server 11 and a terminal device 12. The server 11 is communicatively connected to the terminal device 12, and an image processing device is installed in the server 11.

[0037] In some cases, based on the above system architecture, the terminal device 11 can acquire multiple first images at a first location and send them to the server 11. The server 11 can then adaptively adjust the exposure features of the multiple first images and determine weight information based on the position of each pixel in the first image and its associated exposure features. A projection operation is then performed based on the weight information associated with each pixel to obtain a panoramic image.

[0038] In other cases, terminal device 11 may perform at least part of the following operations: adaptively adjust the exposure features of multiple first images, and determine weight information based on the position of pixels in the first images and their associated exposure features. A projection operation is then performed based on the weight information associated with each pixel to obtain a panoramic image.

[0039] Figure 2 This is a flowchart illustrating the image processing method presented in this paper, such as... Figure 2 As shown, the method includes:

[0040] Step 201: Obtain multiple first images associated with the first location. The first images are obtained by capturing images of the first location from different perspectives.

[0041] In some cases, multiple first-view images captured at a first location can be obtained, allowing for the stitching of these images to create a panoramic image of that location. This panoramic image can then be applied in various fields such as autonomous driving and digital twins. The first image can be captured by rotating the image at the first location using an image acquisition device.

[0042] Step 202: Adjust the exposure features of multiple first images based on reference features to obtain multiple adjusted first images.

[0043] In some cases, during the rotational acquisition process, factors such as changes in lighting and automatic exposure adjustments by the equipment can cause inconsistencies in brightness and contrast among multiple first images. Consequently, directly stitching together multiple first images often results in layered stitching problems in the generated panoramic image. Therefore, adaptive adjustments can be made to multiple first images to ensure they have similar or identical exposure characteristics.

[0044] Optionally, in order to adjust the exposure features of the first image, a reference feature can be determined. This reference feature can be any exposure feature associated with the first image. Alternatively, the reference feature can be set by the user according to actual needs; this document does not impose any restrictions on this.

[0045] After determining the reference features, the exposure features of each first image can be adjusted based on the reference features to obtain the adjusted first image. These exposure features include, but are not limited to, the three-channel average brightness and the standard deviation of grayscale brightness of the first image.

[0046] Step 203: Generate a panoramic image with a first location association based on multiple adjusted first images, wherein the pixels in the panoramic image are associated with the pixels of the adjusted first images.

[0047] In some cases, after adjusting the exposure features of the first image, multiple adjusted first images may have the same exposure features. Therefore, a panoramic image associated with a first location can be generated based on multiple adjusted first images, thus eliminating the layering problem in this panoramic image.

[0048] Optionally, each pixel in the panoramic image can be associated with a pixel in the adjusted first image. For example, pixels in the adjusted first image can be projected onto a cylindrical panoramic coordinate system to obtain the pixels in the panoramic image.

[0049] The image processing method presented in this paper adaptively adjusts the exposure features of multiple first images acquired from multiple viewpoints based on reference features, thus ensuring that the multiple first images have the same exposure characteristics. Therefore, the panoramic image generated based on the adjusted first images can effectively avoid the stitching and layering problem, improving the image quality of the generated panoramic image.

[0050] Optionally, based on any of the above embodiments, step 203 includes:

[0051] Determine the weight information associated with pixels in the adjusted first image. The weight information is associated with the position of the pixel in the adjusted first image and the exposure features associated with the first image.

[0052] Based on the pixel-related weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic coordinate system to obtain a panoramic image.

[0053] In some cases, for pixels in the adjusted first image, pixel-associated weight information is determined. This weight information is determined based on the pixel's position in the adjusted first image and the exposure features associated with the first image.

[0054] In related technologies, pixel-weighted stitching is generally based on fixed cosine window weights. However, while weighted fusion can alleviate the layering problem in panoramic image stitching, the superposition of pixels from multiple perspectives leads to overall image blurring and loss of detail.

[0055] Therefore, in order to further solve the problem of stitching and layering and ensure the clarity of the panoramic image, we can combine the position of the pixel in the adjusted first image and the exposure features associated with the first image to generate comprehensive weight information, and then perform projection operation on the pixel based on this weight information.

[0056] Optionally, for pixels in the adjusted first image, the closer a pixel's display position is to the center region of the first image, the higher its weight. The smaller the brightness difference between a pixel's exposure features and the reference features, the higher its weight.

[0057] In other cases, in order to project a two-dimensional first image into a cylindrical panoramic coordinate system, it is also necessary to determine the pose parameters associated with that first image. This paper does not restrict the execution order of the pose parameter calculation steps. They can be calculated after acquiring multiple first images, or they can be calculated after adjusting the exposure parameters and determining the weight information for the first images.

[0058] Optionally, pixels can be projected onto a cylindrical panoramic coordinate system based on the pixel-associated weight information and the pose parameters associated with the first image to obtain a panoramic image associated with the first position.

[0059] In related technologies, inconsistent exposure of the first images from multiple perspectives directly results in brightness differences in the stitched areas. This leads to noticeable layering and seams in the panoramic image, resulting in poor visual continuity. Furthermore, the pixel overlay from multiple perspectives causes the generated panoramic image to be overall blurry and lose detail.

[0060] In some cases, the exposure features of multiple first images are adaptively adjusted based on reference features. The weight information of each pixel is determined by comprehensively considering its position in the first image and its associated exposure features, and pixel projection is performed based on this weight information. This ensures consistent global exposure, avoids layering issues in panoramic images, reduces local blurring caused by fixed weights, and improves the overall visual coherence of the image.

[0061] The image processing method presented in this paper determines the weight information of each pixel based on its position in the first image and its associated exposure features. Therefore, pixel projection based on this weight information can reduce local blurring caused by fixed weights and improve the visual coherence of the overall image. This results in a clearer and more coherent panoramic image.

[0062] Optionally, based on any of the above embodiments, the method further includes:

[0063] Obtain the pose file associated with the first image, and determine the extrinsic parameter matrix of the first image based on the pose file.

[0064] Obtain the intrinsic parameter file associated with the first image, and determine the image size and intrinsic parameter matrix of the first image based on the intrinsic parameter file.

[0065] The pose parameters associated with the first image are obtained based on the extrinsic matrix, intrinsic matrix, and image size.

[0066] In some cases, in order to project a two-dimensional first image into a cylindrical panoramic coordinate system, it is also necessary to determine the pose parameters associated with the first image.

[0067] Optionally, a pose file associated with the first image can be obtained; for example, the pose file can be a pose.txt file. Based on the pose file, the name, translation vector (X,Y,Z), and quaternions (qx, qy, qz, qw) of the first image are parsed and converted into an extrinsic parameter matrix.

[0068] Obtain the intrinsic parameter file associated with the first image, and parse the image size and intrinsic parameter matrix (3*3, fx, fy, fz, fw). Therefore, the pose parameters associated with the first image can be obtained based on the extrinsic parameter matrix, intrinsic parameter matrix, and image size.

[0069] The image processing method provided in this paper determines the pose parameters associated with the first image based on the pose file and intrinsic parameter file associated with the first image. Therefore, subsequent pixel-level projection operations can be accurately implemented based on these pose parameters.

[0070] Furthermore, based on any of the above embodiments, step 203 includes:

[0071] Based on the pixel's display position and pose parameters in the first image, the projection position of the pixel in the cylindrical panoramic image coordinate system is determined.

[0072] The projected pixel value associated with a pixel is determined based on the pixel value associated with the pixel and the weight information.

[0073] The pixels are projected to their projection positions in the cylindrical panoramic image coordinate system based on the projected pixel values.

[0074] In some cases, for each pixel in the adjusted first image, its projection position in the cylindrical panoramic coordinate system can be determined based on the pose parameters associated with that pixel and its display position in the first image. The projected pixel value is determined based on the pixel value associated with that pixel and weight information. Specifically, the projected pixel value can be obtained by calculating the product of the pixel value and the weight information.

[0075] Based on the projection position and projection pixel value, the pixel is projected onto the cylindrical panoramic coordinate system. After completing the projection operation on all pixels of multiple adjusted first images, the projected image is obtained.

[0076] The image processing method presented in this paper determines the projection position based on pose parameters, thus enabling accurate pixel projection. Furthermore, by determining pixel values ​​based on weight information combining position weights and exposure confidence weights, the pixel fusion effect can be balanced to reduce blur, further improving the image quality of panoramic images.

[0077] Figure 3 This is yet another flowchart illustrating the image processing method provided in this paper. Based on any of the above embodiments, such as... Figure 3 As shown, step 202 includes:

[0078] Step 301: Determine the exposure features associated with the first image.

[0079] Step 302: Based on the reference features, perform exposure feature adjustment operations on the first images other than the reference image among the multiple first images to obtain multiple adjusted first images.

[0080] In some cases, for each first image, the exposure features associated with that first image can be calculated. To unify the exposure features of multiple first images, one image can be selected as a reference image, and the exposure features corresponding to that reference image can be used as the reference image. Any one of the multiple first images can be selected as the reference image. Alternatively, the first image selected by the user from among the multiple first images can also be used as the reference image; this document does not impose any restrictions on this.

[0081] Furthermore, the exposure features of other first images can be adaptively adjusted based on the reference features of the reference image. For example, the exposure features of other first images can be adaptively adjusted pixel-by-pixel based on the reference features and a preset correction algorithm.

[0082] Optionally, the exposure features include the three-channel average brightness and the brightness standard deviation of the grayscale image. A first ratio can be calculated between the three-channel average brightness associated with the reference features and the three-channel average brightness associated with the first image, and a second ratio can be calculated between the brightness standard deviation associated with the reference image and the brightness standard deviation associated with the first image. For each pixel in the first image, the pixel is multiplied by the first ratio and the second ratio respectively to complete the correction of that pixel.

[0083] In other cases, multiple first images can be cropped based on a preset saturation range to output multiple adjusted first images with uniform color and exposure. This saturation range can be (0-255).

[0084] The image processing method presented in this paper selects any one of multiple first images as a reference image. Therefore, it can adaptively adjust the exposure features of the remaining first images based on the exposure features associated with the selected reference image, thereby ensuring consistent global exposure, avoiding the stitching and layering phenomenon in panoramic images, and improving the visual coherence of the overall image.

[0085] Furthermore, based on any of the above embodiments, step 301 includes:

[0086] The exposure features are obtained by calculating the three-channel average brightness associated with the first image and the brightness standard deviation of the grayscale image associated with the first image.

[0087] In some cases, for each first image, the average brightness of the RGB three channels of the first image can be calculated, and the standard deviation of the brightness of the grayscale image associated with the first image can be calculated. The average brightness and the standard deviation of the brightness are determined as the exposure characteristics of the first image.

[0088] The image processing method presented in this paper calculates the average brightness and grayscale standard deviation of the RGB channels of each first image, and uses the reference image as a benchmark to unify the exposure characteristics, thereby eliminating the exposure differences between first images from multiple perspectives and alleviating the problem of stitching and layering from the root.

[0089] Figure 4 This is yet another flowchart illustrating the image processing method provided in this paper. Based on any of the above embodiments, such as... Figure 4 As shown, step 203 includes:

[0090] Step 401: Calculate the first weight of pixel association based on the distance between the pixel's position in the adjusted first image and the center position of the adjusted first image, and the cosine window function.

[0091] Step 402: Calculate the second weight of pixel association based on the difference between the exposure features of the pixel and the reference features.

[0092] Step 403: Calculate the pixel association weight information based on the first weight and the second weight.

[0093] In some cases, pixel-weighted stitching can lead to image blurring due to excessive fusion. Therefore, in order to improve the clarity of panoramic images, the position weight and exposure weight of pixels can be combined to generate comprehensive weight information, and the pixels can be projected based on this weight information.

[0094] Optionally, for each pixel in the adjusted first image, its display position within the adjusted first image is determined. A first weight associated with the pixel is calculated based on the distance between the display position and the center of the adjusted first image, using a cosine window function. The closer the pixel's position is to the center region of the first image, the higher its weight.

[0095] Furthermore, a second weight for pixel association can be calculated based on the difference between the exposure features of a pixel and the reference features. The smaller the brightness difference, the higher the weight. Pixel association weight information is then calculated based on the first and second weights. This weight information can be obtained by multiplying the first and second weights. By dynamically adjusting the weight distribution, the excessive smoothing caused by traditional weighting strategies is avoided, thereby reducing blurring while alleviating layering.

[0096] The image processing method presented in this paper determines the weight information by combining the first weight associated with location and the second weight associated with exposure features. Therefore, it can balance the pixel fusion effect to reduce blur and further improve the image quality of panoramic images.

[0097] Figure 5 This is yet another flowchart illustrating the image processing method provided in this paper. Based on any of the above embodiments, such as... Figure 5 As shown, based on the pixel-associated weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic image coordinate system to obtain a panoramic image associated with the first position, including:

[0098] Step 501: Based on the pixel association weight information and the pose parameters associated with the first image, project the pixels onto the cylindrical panoramic image coordinate system to obtain the projected image.

[0099] Step 502: Input the projected image and prompt words into the diffusion model so that the diffusion model can perform multiple rounds of repair operations on the projected image based on the prompt words to obtain a panoramic image.

[0100] In some cases, for each pixel in each adjusted first image, its projection position in the cylindrical panoramic coordinate system can be determined based on the pose parameters associated with that pixel. The projection pixel value is then determined based on the pixel value associated with that pixel and weight information. The pixel is then projected onto the cylindrical panoramic coordinate system based on the projection position and projection pixel value. After completing the projection operation on all pixels of multiple adjusted first images, a projected image is obtained.

[0101] In other cases, after projecting multiple first images from multiple perspectives, the resulting projected image may contain blank areas or areas that are black and unfilled. Therefore, to ensure the integrity and accuracy of the panoramic image, a preset diffusion model can be used to repair and fill the projected image to obtain the panoramic image.

[0102] Optionally, the projected image and prompts can be input into the diffusion model, allowing the model to perform multiple rounds of restoration operations on the projected image based on the prompts to obtain a panoramic image. For example, the projected image and prompts can be input into the diffusion model, such as "eliminate image quality degradation," "suppress image loss," or "remove image degradation." The diffusion model iterates three times, with each round's input being the output of the previous round, except for the first round, until the inference operation is completed, resulting in the final panoramic image. This panoramic image has the characteristics of being unlayered and highly detailed.

[0103] The image processing method presented in this paper obtains a projected image by projecting pixels onto a cylindrical panoramic image coordinate system. Then, it performs repair and filling operations on the projected image based on a diffusion model, thus completing the missing content in the projected image and further improving the image quality of the panoramic image.

[0104] Furthermore, based on any of the above embodiments, the cylindrical panoramic image coordinate system includes at least one target projection position, and the target projection position simultaneously projects at least two pixels. The method further includes:

[0105] The first value is obtained by weighting the pixel values ​​and weight information associated with at least two pixels.

[0106] The weight information associated with at least two pixels is accumulated to obtain the second value.

[0107] At least two pixels at the target projection position are normalized based on the first and second values.

[0108] In some situations, when projecting multiple first images from multiple viewpoints, the target projection location may be simultaneously projected with multiple pixels. This pixel overlap can cause the displayed image of that target projection location to differ from other locations. Therefore, a pixel fusion operation can be performed on these multiple pixels.

[0109] Optionally, a first value can be obtained by weighting the pixel values ​​associated with at least two pixels at the target projection location and their weight information. A second value is obtained by summing the weight information associated with at least two pixels at the target projection location. Normalization is then performed on at least two pixels at the target projection location based on the first and second values. The ratio of the first value to the second value can be calculated to perform pixel fusion operation on the target projection location.

[0110] The image processing method presented in this paper reduces the blurriness of panoramic images by performing pixel fusion operations based on the pixel values ​​and weight information of multiple pixels when projecting multiple pixels at the same location.

[0111] Figure 6 This is a schematic diagram of the image processing device provided in this article, such as... Figure 6 As shown, the device includes: an acquisition module 61, an adjustment module 62, and a generation module 63. The acquisition module 61 is used to acquire multiple first images associated with a first location, wherein the first images are obtained by capturing images of the first location from different viewpoints. The adjustment module 62 is used to adjust the exposure features of the multiple first images based on reference features to obtain multiple adjusted first images. The generation module 63 is used to generate a panoramic image associated with the first location based on the multiple adjusted first images, wherein the pixels in the panoramic image are associated with the pixels of the adjusted first images.

[0112] Furthermore, based on any of the above embodiments, a generation module is used to determine the weight information associated with pixels in the adjusted first image. The weight information is associated with the position of the pixel in the adjusted first image and the exposure features associated with the first image. Based on the pixel-associated weight information and the pose parameters associated with the first image, the pixels are projected onto a cylindrical panoramic coordinate system to obtain a panoramic image.

[0113] Furthermore, based on any of the above embodiments, the generation module is configured to: calculate a first weight for pixel association based on the distance between the display position and the center position of the adjusted first image and a cosine window function; calculate a second weight for pixel association based on the difference between the exposure features of the pixel and the reference features; and calculate pixel association weight information based on the first weight and the second weight.

[0114] Furthermore, based on any of the above embodiments, a generation module is configured to: project pixels onto a cylindrical panoramic coordinate system based on pixel-associated weight information and pose parameters associated with the first image to obtain a projected image; and input the projected image and prompts into a diffusion model so that the diffusion model performs multiple rounds of repair operations on the projected image based on the prompts to obtain a panoramic image.

[0115] Furthermore, based on any of the above embodiments, the device further includes: a determining module, configured to determine any one of the plurality of first images as a reference image, and to determine the exposure features associated with the reference image as reference features. Alternatively, the determining module is further configured to acquire configuration exposure parameters and determine the configuration exposure parameters as reference features.

[0116] Furthermore, based on any of the above embodiments, the adjustment module is also used to determine the exposure features associated with the first image. Based on the reference features, an exposure feature adjustment operation is performed on the first images (excluding the reference image) among the multiple first images to obtain multiple adjusted first images.

[0117] Furthermore, based on any of the above embodiments, the adjustment module is used to: calculate the three-channel average brightness associated with the first image, and calculate the brightness standard deviation of the grayscale image associated with the first image to obtain exposure features.

[0118] Furthermore, based on any of the above embodiments, the cylindrical panoramic image coordinate system includes at least one target projection position, and the target projection position simultaneously projects at least two pixels. The image processing apparatus further includes: a calculation module, used to perform weighted calculation on the pixel values ​​associated with at least two pixels and weight information to obtain a first value; a processing module, used to perform cumulative calculation on the weight information associated with at least two pixels to obtain a second value; and a fusion module, used to perform normalization processing on at least two pixels at the target projection position based on the first value and the second value.

[0119] The device provided in this article can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0120] To implement the above embodiments, this document also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image processing method as described in any of the above embodiments.

[0121] To implement the above embodiments, this document also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method as described in any of the above embodiments.

[0122] To implement the above embodiments, this document also provides an electronic device, including: a processor and a memory;

[0123] The memory stores the instructions that the computer executes;

[0124] The processor executes computer execution instructions stored in memory, causing the processor to perform the image processing method as described in any of the above embodiments.

[0125] Figure 7 This is a schematic diagram of the structure of the electronic device 700 provided in this article. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of this article.

[0126] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0127] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments herein, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined above in the methods herein.

[0129] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this document, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0131] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0132] Computer program code for performing the operations described herein can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The units described herein can be implemented in software or hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0135] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0136] In the context of this document, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] In a first aspect, according to one or more embodiments herein, an image processing method is provided, comprising:

[0138] Acquire multiple first images associated with the first location, wherein the first images are obtained by capturing images of the first location from different viewpoints;

[0139] The exposure features of multiple first images are adjusted based on reference features to obtain multiple adjusted first images;

[0140] A first location-associated panoramic image is generated based on multiple adjusted first images, wherein pixels in the panoramic image are associated with pixels in the adjusted first images.

[0141] According to one or more embodiments of this document, generating a panoramic image associated with a first location based on multiple adjusted first images includes:

[0142] Determine the weight information associated with pixels in the adjusted first image. The weight information is associated with the position of the pixel in the adjusted first image and the exposure features associated with the first image.

[0143] Based on the pixel-related weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic coordinate system to obtain a panoramic image.

[0144] According to one or more embodiments of this document, determining the weight information associated with pixels in the adjusted first image includes:

[0145] The first weight of pixel association is calculated based on the distance between the pixel's position in the adjusted first image and the center position of the adjusted first image, and the cosine window function.

[0146] A second weight for pixel association is calculated based on the difference between the exposure features of a pixel and the reference features;

[0147] The pixel association weight information is calculated based on the first weight and the second weight.

[0148] According to one or more embodiments of this document, based on pixel-associated weight information and pose parameters associated with a first image, pixels are projected onto a cylindrical panoramic coordinate system to obtain a panoramic image associated with a first position, including:

[0149] Based on the pixel-related weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic image coordinate system to obtain the projected image.

[0150] The projected image and prompts are input into the diffusion model, which then performs multiple rounds of repair operations on the projected image based on the prompts to obtain a panoramic image.

[0151] According to one or more embodiments of this document, the method further includes:

[0152] Any one of the multiple first images is determined as the reference image, and the exposure features associated with the reference image are determined as the reference features;

[0153] or,

[0154] Obtain the configured exposure parameters and use them as the reference feature.

[0155] According to one or more embodiments of this document, adjusting the exposure features of multiple first images based on reference features to obtain multiple adjusted first images includes:

[0156] Determine the exposure features associated with the first image;

[0157] Based on the reference features, the exposure features of the first images other than the reference image are adjusted to obtain multiple adjusted first images.

[0158] According to one or more embodiments herein, determining exposure features associated with a first image includes:

[0159] The exposure features are obtained by calculating the three-channel average brightness associated with the first image and the brightness standard deviation of the grayscale image associated with the first image.

[0160] According to one or more embodiments of this article, the cylindrical panoramic image coordinate system includes at least one target projection position, which simultaneously projects at least two pixels;

[0161] The method also includes:

[0162] The first value is obtained by weighting the pixel values ​​and weight information associated with at least two pixels.

[0163] The weight information associated with at least two pixels is accumulated to obtain the second value;

[0164] At least two pixels at the target projection position are normalized based on the first and second values.

[0165] Secondly, according to one or more embodiments herein, an image processing apparatus is provided, comprising:

[0166] The acquisition module is used to acquire multiple first images associated with the first location. The first images are obtained by acquiring images of the first location from different perspectives.

[0167] The adjustment module is used to adjust the exposure features of multiple first images based on reference features to obtain multiple adjusted first images;

[0168] A generation module is used to generate a first location-associated panoramic image based on multiple adjusted first images, wherein pixels in the panoramic image are associated with pixels in the adjusted first images.

[0169] According to one or more embodiments herein, a generation module is configured to:

[0170] Determine the weight information associated with pixels in the adjusted first image. The weight information is associated with the position of the pixel in the adjusted first image and the exposure features associated with the first image.

[0171] Based on the pixel-related weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic coordinate system to obtain a panoramic image.

[0172] According to one or more embodiments herein, a generation module is configured to:

[0173] The first weight of pixel association is calculated based on the distance between the pixel's position in the adjusted first image and the center position of the adjusted first image, and the cosine window function.

[0174] A second weight for pixel association is calculated based on the difference between the exposure features of a pixel and the reference features;

[0175] The pixel association weight information is calculated based on the first weight and the second weight.

[0176] According to one or more embodiments herein, a generation module is configured to:

[0177] Based on the pixel-related weight information and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic image coordinate system to obtain the projected image.

[0178] The projected image and prompts are input into the diffusion model, which then performs multiple rounds of repair operations on the projected image based on the prompts to obtain a panoramic image.

[0179] According to one or more embodiments herein, the image processing apparatus further includes:

[0180] The determination module is used to determine any one of the multiple first images as a reference image and to determine the exposure features associated with the reference image as reference features;

[0181] or,

[0182] The determination module is also used to obtain the configuration exposure parameters and determine the configuration exposure parameters as reference features.

[0183] According to one or more embodiments herein, the module is modified for:

[0184] Determine the exposure features associated with the first image;

[0185] Based on the reference features, the exposure features of the first images other than the reference image are adjusted to obtain multiple adjusted first images.

[0186] According to one or more embodiments herein, the module is modified for:

[0187] The exposure features are obtained by calculating the three-channel average brightness associated with the first image and the brightness standard deviation of the grayscale image associated with the first image.

[0188] According to one or more embodiments of this article, the cylindrical panoramic image coordinate system includes at least one target projection position, which simultaneously projects at least two pixels;

[0189] The image processing apparatus also includes:

[0190] The calculation module is used to perform weighted calculations on the pixel values ​​and weight information associated with at least two pixels to obtain a first value;

[0191] The processing module is used to accumulate and calculate the weight information associated with at least two pixels to obtain a second value;

[0192] The fusion module is used to normalize at least two pixels at the target projection position based on a first value and a second value.

[0193] Thirdly, according to one or more embodiments herein, an electronic device is provided, comprising: at least one processor and a memory;

[0194] The memory stores the instructions that the computer executes;

[0195] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the image processing method as described in the first aspect above and various possible designs of the first aspect.

[0196] Fourthly, according to one or more embodiments herein, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the image processing method described in the first aspect and various possible designs of the first aspect.

[0197] Fifthly, according to one or more embodiments herein, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method as described in the first aspect above and various possible designs of the first aspect.

[0198] The above description is merely a preferred embodiment and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure herein is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed herein that have similar functions.

[0199] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this document. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0200] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, comprising: Acquire multiple first images associated with the first location, wherein the first images are obtained by capturing images of the first location from different perspectives; The exposure features of the multiple first images are adjusted based on reference features to obtain multiple adjusted first images; A panoramic image associated with the first location is generated based on the multiple adjusted first images, wherein the pixels in the panoramic image are associated with the pixels in the adjusted first images.

2. The method according to claim 1, wherein generating the panoramic image associated with the first location based on the plurality of adjusted first images comprises: Determine the weight information associated with pixels in the adjusted first image, the weight information being associated with the position of the pixel in the adjusted first image and the exposure features associated with the first image; The pixels are projected onto the cylindrical panoramic coordinate system based on the pixel-associated weight information and the pose parameters associated with the first image to obtain the panoramic image.

3. The method according to claim 2, wherein determining the weight information associated with pixels in the adjusted first image includes: The first weight associated with the pixel is calculated based on the distance between the position of the pixel in the adjusted first image and the center position of the adjusted first image, and a cosine window function. A second weight associated with the pixel is calculated based on the difference between the pixel's exposure features and the reference features; The weight information associated with the pixel is calculated based on the first weight and the second weight.

4. The method according to claim 2, wherein projecting the pixels onto a cylindrical panoramic coordinate system based on the pixel-associated weight information and the pose parameters associated with the first image to obtain the panoramic image associated with the first position includes: Based on the weight information associated with the pixels and the pose parameters associated with the first image, the pixels are projected onto the cylindrical panoramic coordinate system to obtain a projected image. The projected image and the prompt words are input into the diffusion model, so that the diffusion model performs multiple rounds of repair operations on the projected image based on the prompt words to obtain the panoramic image.

5. The method according to claim 1, further comprising: Any one of the plurality of first images is determined as a reference image, and the exposure feature associated with the reference image is determined as the reference feature; or, Obtain the configured exposure parameters and determine the configured exposure parameters as the reference feature.

6. The method according to claim 5, wherein adjusting the exposure features of the plurality of first images based on reference features to obtain a plurality of adjusted first images comprises: Determine the exposure features associated with the first image; Based on the reference features, an exposure feature adjustment operation is performed on the first images other than the reference image among the plurality of first images to obtain a plurality of adjusted first images.

7. The method according to claim 6, wherein determining the exposure features associated with the first image comprises: The exposure features are obtained by calculating the three-channel average brightness associated with the first image and the brightness standard deviation of the grayscale image associated with the first image.

8. The method according to any one of claims 4-7, wherein the cylindrical panoramic image coordinate system includes at least one target projection position, and the target projection position simultaneously projects at least two pixels; The method further includes: The pixel values ​​associated with the at least two pixels and the weight information are weighted and calculated to obtain a first value; The weight information associated with the at least two pixels is accumulated to obtain a second value; At least two pixels at the target projection position are normalized based on the first and second values.

9. An electronic device comprising: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the image processing method as described in any one of claims 1 to 8.

Images