A panoramic image determination method, device, apparatus and storage medium

Abstract

The application discloses a panoramic image determination method, device and equipment and a storage medium. The method comprises the following steps: acquiring initial images collected by multiple cameras, performing edge detection on the initial images to obtain image edges, performing image space alignment and image splicing based on the positional relationship of the initial images to obtain an initial panoramic image; determining an overlapping area in the initial panoramic image based on the image edges of the initial images, performing pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image; and performing pixel fusion on the overlapping area in the standard panoramic image based on pixel feature point matching to obtain a target panoramic image, which can solve the problem that the information described by the existing panoramic image determination method is not clear and accurate, and a panoramic image with clearer, more complete and more accurate information description is obtained, thereby improving the experience of users.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for determining panoramic images. Background Technology

[0002] With the development of technology, in-vehicle surround view systems are gradually being integrated into more and more vehicle models. In-vehicle surround view is a common driver assistance technology. Cameras are installed at the front, rear, and side mirrors of the vehicle, and image processing is used to obtain a panoramic image of the target area. This provides the driver with a rich perspective of the surrounding environment, thereby improving driving safety.

[0003] However, because there are overlapping areas between images captured by cameras installed in different locations, current methods for determining a target panoramic image by simply stitching together images captured by multiple cameras do not provide clear, complete, or accurate information, resulting in a poor user experience. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for determining panoramic images, in order to solve the problem that the information described by existing methods for determining panoramic images is not clear and accurate enough, and to provide a clearer, more complete, and more accurate information description of panoramic images, thereby improving the user experience.

[0005] According to one aspect of the present invention, a method for determining a panoramic image is provided, comprising:

[0006] Acquire initial images from multiple cameras, and perform edge detection on each initial image to obtain image edges;

[0007] An initial panoramic image is obtained by spatial alignment and image stitching based on the positional relationship of each initial image;

[0008] The overlapping region in the initial panoramic image is determined based on the image edges of each initial image, and the pixel adjustment of the overlapping region in the initial panoramic image is performed to obtain a standard panoramic image;

[0009] The target panoramic image is obtained by pixel fusion based on pixel feature point matching of the overlapping areas in the standard panoramic image.

[0010] According to another aspect of the present invention, a panoramic image determining device is provided, comprising:

[0011] The edge detection module is used to acquire initial images from multiple cameras and perform edge detection on each initial image to obtain the image edges.

[0012] The image alignment module is used to perform image spatial alignment and image stitching based on the positional relationship of each initial image to obtain an initial panoramic image;

[0013] The pixel adjustment module is used to determine the overlapping area in the initial panoramic image based on the image edges of each initial image, and to perform pixel adjustment on the overlapping area of ​​the initial panoramic image to obtain a standard panoramic image.

[0014] The pixel fusion module is used to perform pixel fusion based on pixel feature point matching on the overlapping areas in the standard panoramic image to obtain the target panoramic image.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the panoramic image determination method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the panoramic image determination method according to any embodiment of the present invention.

[0020] This invention discloses a method, apparatus, device, and storage medium for determining panoramic images. The method includes: acquiring initial images from multiple cameras; performing edge detection on each initial image to obtain image edges; performing image spatial alignment and image stitching based on the positional relationship of the initial images to obtain an initial panoramic image; determining overlapping regions in the initial panoramic image based on the image edges of each initial image; adjusting the pixels of the overlapping regions in the initial panoramic image to obtain a standard panoramic image; and performing pixel fusion based on pixel feature point matching on the overlapping regions in the standard panoramic image to obtain a target panoramic image. By adjusting and fusing the pixels of the overlapping regions, the information description of the overlapping regions is made clearer and more accurate, thereby solving the problem that the information described by existing panoramic image determination methods is not clear and accurate enough, resulting in a panoramic image with clearer, more complete, and more accurate information description, thus improving the user experience.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a method for determining a panoramic image provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a method for determining a panoramic image provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a panoramic image determination device provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the panoramic image determination method of this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1This is a flowchart illustrating a method for determining a panoramic image according to Embodiment 1 of the present invention. This embodiment is applicable to determining a target panoramic image from multiple images. The method can be executed by a panoramic image determining device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0031] S110. Acquire initial images from multiple cameras and perform edge detection on each initial image to obtain image edges.

[0032] The initial image can be understood as the raw images captured by multiple cameras. These cameras can be positioned in different locations based on the actual usage environment and needs to acquire images from different perspectives. For example, in a vehicle usage scenario, to allow the driver to see a full-view image of the vehicle's surroundings, multiple cameras can be positioned at the front, rear, and left and right rearview mirrors. The cameras can be planar cameras; to expand the camera's field of view, fisheye cameras can also be selected.

[0033] Specifically, initial images are acquired from multiple cameras installed at different locations, and edge detection is performed on each initial image to obtain image edges. Any existing edge detection model or algorithm can be used for image edge detection, and this embodiment of the invention does not impose any limitations on this.

[0034] S120. Based on the positional relationship of each initial image, perform image spatial alignment and image stitching to obtain the initial panoramic image.

[0035] The positional relationship of the initial images can be understood as the relationship between the initial images in terms of their location. This positional relationship can be determined based on the installation position of the camera that acquired each initial image, and may include information such as direction and distance. The initial panoramic image is a preliminary panoramic image obtained after aligning and stitching the initial images.

[0036] Specifically, the positional relationship of each initial image is determined based on its positional relationship. Spatially aligned, the initial images are then stitched together to ensure element alignment. This results in an initial panoramic image. While this initial panoramic image forms a preliminary target panoramic image, overlapping areas exist, leading to unclear and inaccurate information. Therefore, further pixel adjustments and fusion are needed to refine the overlapping areas and improve the clarity and accuracy of the target panoramic image.

[0037] S130. Based on the image edges of each initial image, determine the overlapping area in the initial panoramic image, and perform pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image.

[0038] A standard panoramic image can be understood as a panoramic image obtained by standardizing the pixels in the overlapping areas of an initial panoramic image.

[0039] Specifically, based on the image edges of each initial image, the overlapping area formed between the image edges can be determined. Since the color difference between the overlapping area and the image edges causes the image edges to protrude, the pixels in the overlapping area are adjusted to weaken the edge pixels and reduce the color difference, thus obtaining a standard panoramic image.

[0040] S140. Perform pixel fusion based on pixel feature point matching on the overlapping areas in the standard panoramic image to obtain the target panoramic image.

[0041] The target panoramic image can be understood as the final determined panoramic image.

[0042] Specifically, since the overlapping area in a standard panoramic image is composed of at least two initial images, pixel fusion of the initial images constituting the overlapping area is necessary to avoid seams. Pixel fusion can be achieved using pixel feature point matching. This involves extracting pixel feature points from the initial images constituting the overlapping area, performing feature point matching on these points, and then fusing pairs of pixels that meet the matching criteria.

[0043] The technical solution of this invention acquires initial images from multiple cameras, performs edge detection on each initial image to obtain image edges, performs image spatial alignment and image stitching based on the positional relationship of each initial image to obtain an initial panoramic image, determines the overlapping area in the initial panoramic image based on the image edges of each initial image, performs pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image, and performs pixel fusion based on pixel feature point matching on the overlapping area in the standard panoramic image to obtain a target panoramic image. This solution can solve the problem that the information described by existing panoramic image determination methods is not clear and accurate enough, resulting in a panoramic image with clearer, more complete and more accurate information description, thus improving the user experience.

[0044] Optionally, S110, the step of performing edge detection on each initial image to obtain image edges includes:

[0045] S111. For each initial image, determine the initial edge region of the initial image based on the pixel difference between different color channels of the initial image and the edge threshold.

[0046] The initial edge region is the edge region of the initial image determined based on the edge threshold. The edge threshold is a pre-set threshold used to distinguish edge regions, and the value of the edge threshold can be determined based on the actual situation. This embodiment of the invention does not impose any restrictions on this.

[0047] Specifically, the initial image captured by the camera is generally an image based on the RGB color channel. The initial image is split into RGB color channels, and the pixel values ​​of any two different color channels of each pixel in the initial image are subtracted to obtain the pixel difference. The pixel difference is compared with the edge threshold, and pixels that are greater than the edge threshold constitute the initial edge region.

[0048] For example, the pixel difference is obtained by subtracting the pixels in the R channel and the pixels in the G channel in the initial image, and then compared with the edge threshold to determine the initial edge region.

[0049] S112. The initial edge region is filtered and sharpened to obtain the target edge region.

[0050] The target edge region is the edge region obtained by further processing the initial edge region.

[0051] Specifically, the initial edge region determined by pixel difference is a roughly determined edge region. In order to further determine the edge region more accurately, the initial edge region is filtered to smooth the image and remove image noise. Then, the gradient intensity and direction of each pixel in the initial edge region are calculated, and the initial edge region is sharpened.

[0052] S113. Pixels in the target edge region whose pixel values ​​are higher than or equal to the first threshold are identified as edge pixels.

[0053] Among them, edge pixels are pixels located in the target edge region, determined based on a first threshold.

[0054] Specifically, a dual threshold is set to further determine the pixels in the target edge region to identify edge pixels. A first threshold is set as the upper limit threshold, and pixels in the target edge region with pixel values ​​higher than or equal to the first threshold are identified as edge pixels.

[0055] S114. Pixels in the target edge region whose pixel values ​​are lower than a first threshold and higher than a second threshold are identified as potential edge pixels; wherein the first threshold is greater than the second threshold.

[0056] Among them, potential edge pixels are pixels located in the target edge region determined based on the first threshold and the second threshold. Potential edge pixels can be further determined as edge pixels.

[0057] Specifically, a second threshold is set as the lower bound threshold. Pixels with pixel values ​​lower than the lower bound threshold are identified as non-edge pixels, and pixels with pixel values ​​higher than the second threshold (lower bound threshold) and lower than the first threshold (upper bound threshold) are identified as potential edge pixels.

[0058] S115. For the potential edge pixels, determine the edge pixels of the initial image based on the pixel values ​​in the neighborhood of the potential edge pixels.

[0059] Specifically, an 8-neighborhood is determined for potential edge pixels. If a pixel with a value higher than a second threshold exists within the 8-neighborhood, then the potential edge pixel is determined to be an edge pixel of the initial image. If no pixel with a value higher than the second threshold exists within the 8-neighborhood, then the potential edge pixel is determined not to be an edge pixel of the initial image.

[0060] S116. Perform edge fitting on the edge pixels to obtain the image edge.

[0061] For example, discontinuous pixels in the target edge region are obtained, and the positional relationship between each discontinuous pixel and the edge pixel is determined. If a discontinuous pixel is adjacent to an edge pixel, it is determined that the discontinuous pixel belongs to the edge pixel. If a discontinuous pixel is not adjacent to an edge pixel, it is determined that the discontinuous pixel does not belong to the edge pixel. All edge pixels are connected, the coordinates of each connected component are calculated, and the edges of the connected components are fitted to obtain the image edge.

[0062] Optionally, step S120, obtaining the initial panoramic image by performing image spatial alignment and image stitching based on the positional relationship of each initial image, includes:

[0063] S121. Obtain the positional relationship of each initial image in the world coordinate system.

[0064] The world coordinate system can be understood as the world coordinate system constructed within the space where each camera is located.

[0065] Specifically, since the initial images are acquired from cameras installed at different locations, it is necessary to determine the positional relationship of each initial image in the world coordinate system when stitching the initial images together.

[0066] S122. Perform fisheye distortion correction on each initial image.

[0067] Specifically, since most cameras use fisheye lenses, and the imaging characteristics of fisheye lenses cause image distortion, it is necessary to perform fisheye distortion correction on the initial images to reduce image distortion. This invention does not limit the algorithm for fisheye distortion correction; any existing distortion correction algorithm can be used.

[0068] S123. Project the corrected initial images from the world coordinate system to their respective image coordinate systems.

[0069] Specifically, for each initial image, after distortion correction, the world coordinates of each pixel in the initial image are projected to the image coordinates in their respective image coordinates through a coordinate system parameter matrix. This embodiment of the invention does not elaborate on the coordinate transformation between the world coordinate system and the image coordinate system; this can be determined based on the camera intrinsic and extrinsic parameter matrices.

[0070] S124. Based on the positional relationship and the image coordinates of each initial image in its respective image coordinate system, perform image alignment and image stitching to obtain an initial panoramic image.

[0071] Specifically, based on the positional relationship of each camera in the world coordinate system, the image coordinate system corresponding to each initial image is transformed to the same coordinate system. The image coordinates of each initial image in its own image coordinate system are transformed to the same coordinate system. Thus, the initial images can be aligned and stitched together to determine the initial panoramic image based on the image coordinates of the initial images in the same coordinate system.

[0072] Example 2

[0073] Figure 2 This is a flowchart of a method for determining a panoramic image according to Embodiment 2 of the present invention. This embodiment further refines steps S220 and S230 of the above embodiment. For example... Figure 2 As shown, the method includes:

[0074] S201. Acquire initial images from multiple cameras and perform edge detection on each initial image to obtain image edges.

[0075] S202. Based on the positional relationship of each initial image, perform image spatial alignment and image stitching to obtain the initial panoramic image.

[0076] S203. Determine the overlapping area in the initial panoramic image based on the image edges of each initial image.

[0077] S204. Perform pixel color equalization processing on the overlapping areas in the initial panoramic image.

[0078] Specifically, the overlapping areas in the initial panoramic image are formed by the superposition of pixels from two initial images. The unavoidable color difference between the two initial images leads to pixel color imbalance, affecting the final panoramic image's presentation. Therefore, pixel color equalization processing is required for the overlapping areas in the initial panoramic image.

[0079] Optionally, S204, performing pixel color equalization processing on the overlapping regions in the initial panoramic image, includes:

[0080] S2041. Determine the pixel color balance value based on the color difference of each pixel in the overlapping area of ​​the initial panoramic image.

[0081] Specifically, for the overlapping region formed by the two initial images, the pixel color equalization value is determined as follows: ;

[0082] in, It is the pixel color balance value corresponding to the p-th pixel. To determine the brightness difference between pixels in two initial images corresponding to the same pixel location, The distance metric function is represented by Subset, which is a subset of pixels in the overlapping region of the initial panoramic image. r(*) is the saturation function, which must be an odd function. This step can adapt to local image contrast. r(*) can amplify small differences and enrich large differences, expanding or compressing the dynamic range according to local content.

[0083] S2043. Compare the pixel color balance values ​​of pixels in the overlapping region with the pixel values ​​of pixels in the neighborhood, and discard the pixel color balance values ​​that are less than the pixel values ​​of all pixels in the neighborhood.

[0084] Specifically, the color balance value of the pixels in the overlapping area is compared with the pixel values ​​of all image edges. The color balance values ​​of pixels with values ​​smaller than those of the image edges are filtered out, thus filtering out pixels in the overlapping area whose pixel values ​​are smaller than those of the surrounding pixels and reducing the computational load of the equalization process.

[0085] S2042. Perform color equalization processing on the corresponding pixels in the overlapping area based on the color equalization values ​​of the unremoved pixels.

[0086] Specifically, color balance processing of overlapping areas is achieved by adjusting the colors of corresponding pixels in the overlapping areas based on the average pixel color of each unremoved pixel.

[0087] S205. Obtain the tilt angle between the image edges of the two initial images that constitute the overlapping area, and rotate the two initial images in the initial panoramic image according to the tilt angle to obtain the first image.

[0088] Specifically, in the overlapping region, the edges of the two initial images have a certain tilt angle relative to each other. The enclosing matrix of the contour is found, and the image edges are considered as baselines in their directions to obtain the tilt angle between the edges of the two initial images. Based on this tilt angle, the two images are rotated and corrected in the overlapping region. Since the overlapping region consists of two images with different colors, performing rotational correction before overlapping, and adding the pixel values, can appropriately reduce the color difference in the overlapping region.

[0089] S206. Adjust the pixel brightness of the overlapping area in the first image to obtain a standard panoramic image.

[0090] Specifically, the brightness distribution map of the overlapping area in the first image is obtained. Since the initial image actually acquired has light and dark deviations, the brightness of the overlapping area is unified by the standard image by changing the mean and standard deviation of the brightness distribution map to obtain a standard panoramic image.

[0091] The brightness distribution of a standard panoramic image is as follows:

[0092] ;

[0093] in, This represents the brightness distribution of a standard panoramic image. Let m0 and s0 represent the brightness distribution of the first image, m0 and s0 represent the mean and standard deviation of the pixel distribution of the standard image, and m and s are the mean and standard deviation of the first image.

[0094] S207. Extract features from the two images that constitute the overlapping area in the standard panoramic image to obtain the set of pixel feature points to be matched.

[0095] The set of pixel feature points to be matched is a collection of pixel features extracted from the image pixels that constitute the overlapping region.

[0096] Specifically, after determining the standard panoramic image, feature extraction is performed on the two images that constitute the overlapping area in the standard panoramic image. The pixel features extracted from each image are used as a set of pixel feature points to be matched, thus obtaining two sets of pixel feature points to be matched.

[0097] For example, the RGB color channels of the two images constituting the overlapping region are converted into grayscale images, and the gradients of the two images are calculated respectively. For each image, the OpenCV SURF operator is used to filter and precisely locate feature points based on the image gradient to obtain the set of pixel feature points to be matched. The specific implementation process of the SURF operator is not described in detail in this embodiment of the invention.

[0098] S208. Within the overlapping region, determine the matching window region based on the sliding matching window, and calculate the image matching degree for the set of pixel feature points to be matched contained in the matching window region.

[0099] Specifically, a sliding matching window is used to slide within the overlapping region at certain step sizes. After each slide of the sliding matching window, the image matching degree is calculated based on the set of feature points of the pixels to be matched contained in the matching window region formed by the sliding matching window. The image matching degree can be represented by the correlation between pixels.

[0100] For example, the image matching degree can be calculated as follows:

[0101] ;

[0102] Where NCC(p,d) represents the image matching degree, with a value ranging from [-1,1]. p I1(x,y) represents the matching window region formed by the sliding matching window. I1(x,y) represents the pixel value of the first path image within the overlapping region. x ,p y I2(x+d,y) represents the average pixel value of the first path image within the overlapping region. I2(x+d,y) represents the pixel value of the target image determined by sliding the sliding matching window corresponding to the first path image within the overlapping region by sliding it by d in the x-direction; I2(p x +d,p y ) represents the pixel mean of the target image. If NCC=-1, it means that the two matching windows are completely unrelated; conversely, if NCC=1, it means that the two matching windows are highly correlated.

[0103] S209. Perform median filtering on the pixels contained in the matching window region where the image matching degree is greater than the matching threshold.

[0104] Specifically, median filtering is used to address abrupt changes in pixel values ​​within overlapping regions. For pixels within a matching window region where the image matching degree is greater than the matching threshold, median filtering is used to remove pixels above the upper limit of the filtering threshold to eliminate abrupt changes in pixel values ​​and maintain the continuity of light intensity; pixels below the lower limit of the filtering threshold are removed to eliminate surrounding dissimilar pixels.

[0105] S210. Perform pixel fusion on the overlapping areas in the filtered standard panoramic image to obtain the target panoramic image.

[0106] Specifically, to avoid seams at the edges of overlapping areas between multiple images, pixel fusion is performed on the overlapping areas in the filtered standard panoramic image and then onto the target panoramic image.

[0107] Optionally, S210, the step of performing pixel fusion on the overlapping regions in the filtered standard panoramic image to obtain the target panoramic image, includes:

[0108] The first weighted pixel value of the first path image constituting the overlapping region in the filtered standard panoramic image is determined based on the first weighting factor.

[0109] The second weighted pixel value of the second path image constituting the overlapping region in the filtered standard panoramic image is determined based on the second weighting factor; wherein, the first weighting factor and the second weighting factor decrease linearly from the center pixel point to the boundary pixel point of the overlapping region;

[0110] The target panoramic image is obtained by summing the first and second weighted pixel values ​​of the overlapping regions in the filtered standard panoramic image.

[0111] Specifically, for the overlapping region in the filtered standard panoramic image, the pixel values ​​of the first path image constituting the overlapping region are weighted to obtain a first weighted pixel value, and the pixel values ​​of the second path image constituting the overlapping region are weighted to obtain a second weighted pixel value. The first weighted pixel value and the second weighted pixel value are summed for each pixel position in the overlapping region of the filtered standard panoramic image to obtain the target pixel value at that pixel position, thereby determining the target panoramic image.

[0112] Understandably, the first and second weighting factors can be represented using a weighted map. This weighted map is the same size as the overlapping region, and the weight values ​​of this map decrease linearly from the center pixel to the boundary pixel of the overlapping region. For example, the value at the center pixel is 1, and becomes 0 after linearly decreasing with the boundary pixels.

[0113] For example, the calculation formula for pixel fusion based on weighted summation is as follows:

[0114]

[0115] in, For the target panoramic image in The pixel at the location; For the first image in The pixel at the location; This is the first weighted pixel value; For the second image in The pixel at the location; This is the second weighted pixel value.

[0116] The technical solution of this invention acquires initial images from multiple cameras, performs edge detection on each initial image to obtain image edges, performs image spatial alignment and image stitching based on the positional relationship of each initial image to obtain an initial panoramic image, determines the overlapping area in the initial panoramic image based on the image edges of each initial image, performs pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image, and performs pixel fusion based on pixel feature point matching on the overlapping area in the standard panoramic image to obtain a target panoramic image. This solution can solve the problem that the information described by existing panoramic image determination methods is not clear and accurate enough, resulting in a panoramic image with clearer, more complete and more accurate information description, thus improving the user experience.

[0117] Example 3

[0118] Figure 3 This is a schematic diagram of a panoramic image determination device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an edge detection module 310, an image alignment module 320, a pixel adjustment module 330, and a pixel fusion module 340; wherein,

[0119] The edge detection module 310 is used to acquire initial images from multiple cameras and perform edge detection on each initial image to obtain image edges.

[0120] The image alignment module 320 is used to perform image spatial alignment and stitching based on the positional relationship of each initial image to obtain an initial panoramic image;

[0121] The pixel adjustment module 330 is used to determine the overlapping area in the initial panoramic image based on the image edges of each initial image, and to perform pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image.

[0122] The pixel fusion module 340 is used to perform pixel fusion based on pixel feature point matching on the overlapping areas in the standard panoramic image to obtain the target panoramic image.

[0123] Optionally, the edge detection module 310 is specifically used for:

[0124] For each initial image, the initial edge region of the initial image is determined based on the pixel difference between different color channels of the initial image and the edge threshold;

[0125] The initial edge region is filtered and sharpened to obtain the target edge region;

[0126] Pixels in the target edge region whose pixel values ​​are higher than or equal to a first threshold are identified as edge pixels.

[0127] Pixels in the target edge region whose pixel values ​​are lower than a first threshold and higher than a second threshold are identified as potential edge pixels; wherein, the first threshold is greater than the second threshold;

[0128] For the potential edge pixels, the edge pixels of the initial image are determined based on the pixel values ​​in the neighborhood of the potential edge pixels.

[0129] The image edges are obtained by edge fitting of the edge pixels.

[0130] Optionally, the image alignment module 320 is specifically used for:

[0131] Obtain the positional relationship of each initial image in the world coordinate system;

[0132] Fisheye distortion correction is performed on each initial image;

[0133] The corrected initial images are projected from the world coordinate system to the image coordinate system of their respective initial images.

[0134] Based on the positional relationship and the image coordinates of each initial image in its respective image coordinate system, image alignment and image stitching are performed to obtain the initial panoramic image.

[0135] Optionally, the pixel adjustment module 330 includes:

[0136] An equalization processing unit is used to perform pixel color equalization processing on the overlapping areas in the initial panoramic image.

[0137] An image rotation module is used to obtain the tilt angle between the image edges of two initial images that constitute an overlapping area, and to rotate the two initial images in the initial panoramic image according to the tilt angle to obtain a first image.

[0138] The brightness adjustment module is used to adjust the pixel brightness of the overlapping areas in the first image to obtain a standard panoramic image.

[0139] Optionally, the equalization processing unit is specifically used for:

[0140] The pixel color balance value is determined based on the color difference of each pixel in the overlapping area of ​​the initial panoramic image;

[0141] Compare the pixel color balance values ​​of pixels in the overlapping region with the pixel values ​​of the image edges, and discard the pixel color balance values ​​that are less than those of all image edges;

[0142] Color equalization processing is performed on the corresponding pixels in the overlapping region based on the color equalization values ​​of the unremoved pixels.

[0143] Optionally, the pixel fusion module 340 includes:

[0144] The feature extraction unit is used to extract features from the two images that constitute the overlapping area in the standard panoramic image to obtain a set of pixel feature points to be matched;

[0145] An image matching calculation unit is used to determine a matching window region based on a sliding matching window within the overlapping region, and to calculate the image matching degree of the set of pixel feature points to be matched contained in the matching window region.

[0146] The filtering unit is used to perform median filtering on pixels contained in the matching window region where the image matching degree is greater than the matching threshold.

[0147] The pixel fusion unit is used to perform pixel fusion on the overlapping areas in the filtered standard panoramic image to obtain the target panoramic image.

[0148] Optionally, the pixel fusion unit is specifically used for:

[0149] The first weighted pixel value of the first path image constituting the overlapping region in the filtered standard panoramic image is determined based on the first weighting factor.

[0150] The second weighted pixel value of the second path image constituting the overlapping region in the filtered standard panoramic image is determined based on the second weighting factor; wherein, the first weighting factor and the second weighting factor decrease linearly from the center pixel point to the boundary pixel point of the overlapping region;

[0151] The target panoramic image is obtained by summing the first and second weighted pixel values ​​of the overlapping regions in the filtered standard panoramic image.

[0152] The panoramic image determination device provided in this embodiment of the invention can execute the panoramic image determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0153] Example 4

[0154] Figure 4A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0155] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0156] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method for determining panoramic images.

[0158] In some embodiments, the method for determining the panoramic image may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the panoramic image determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the panoramic image determination method by any other suitable means (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0163] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0164] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0165] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0166] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining a panoramic image, characterized in that, include: Acquire initial images from multiple cameras, and perform edge detection on each initial image to obtain image edges; An initial panoramic image is obtained by spatial alignment and image stitching based on the positional relationship of each initial image; The overlapping region in the initial panoramic image is determined based on the image edges of each initial image, and the pixel adjustment of the overlapping region in the initial panoramic image is performed to obtain a standard panoramic image; The target panoramic image is obtained by pixel fusion based on pixel feature point matching of the overlapping areas in the standard panoramic image; The step of performing edge detection on each initial image to obtain image edges includes: For each initial image, the initial edge region of the initial image is determined based on the pixel difference between different color channels of the initial image and the edge threshold; The initial edge region is filtered and sharpened to obtain the target edge region; Pixels in the target edge region whose pixel values ​​are higher than or equal to a first threshold are identified as edge pixels. Pixels in the target edge region whose pixel values ​​are lower than a first threshold and higher than a second threshold are identified as potential edge pixels; wherein, the first threshold is greater than the second threshold; For the potential edge pixels, the edge pixels of the initial image are determined based on the pixel values ​​in the neighborhood of the potential edge pixels. Obtain discontinuous pixels in the target edge region and determine the positional relationship between each discontinuous pixel and the edge pixels. If a discontinuous pixel is adjacent to an edge pixel, it is determined that the discontinuous pixel belongs to the edge pixel. If a discontinuous pixel is not adjacent to an edge pixel, it is determined that the discontinuous pixel does not belong to the edge pixel. Connect all edge pixels and fit the edges of the connected regions according to the coordinates of each connected region to obtain the image edge. The target panoramic image is obtained by pixel fusion based on pixel feature point matching of overlapping regions in the standard panoramic image, including: Feature extraction is performed on the two images that constitute the overlapping area in the standard panoramic image to obtain a set of pixel feature points to be matched; Within the overlapping area, a matching window region is determined based on a sliding matching window, and the image matching degree is calculated for the set of pixel feature points to be matched contained in the matching window region. Median filtering is applied to pixels within the matching window region where the image matching degree is greater than the matching threshold. The target panoramic image is obtained by pixel fusion of the overlapping areas in the filtered standard panoramic image.

2. The method according to claim 1, characterized in that, The process of obtaining an initial panoramic image by performing image spatial alignment and image stitching based on the positional relationships of each initial image includes: Obtain the positional relationship of each initial image in the world coordinate system; Fisheye distortion correction is performed on each initial image; The corrected initial images are projected and transformed from the world coordinate system to the image coordinate system of their respective image coordinate systems. Based on the positional relationship and the image coordinates of each initial image in its respective image coordinate system, image alignment and image stitching are performed to obtain the initial panoramic image.

3. The method according to claim 1, characterized in that, To obtain a standard panoramic image, pixel adjustments are made to the overlapping regions in the initial panoramic image, including: Pixel color equalization processing is performed on the overlapping areas in the initial panoramic image; Obtain the tilt angle between the image edges of the two initial images that constitute the overlapping region, and rotate the two initial images in the initial panoramic image according to the tilt angle to obtain the first image; A standard panoramic image is obtained by adjusting the pixel brightness of the overlapping areas in the first image.

4. The method according to claim 3, characterized in that, The pixel color equalization processing of the overlapping regions in the initial panoramic image includes: The pixel color balance value is determined based on the color difference of each pixel in the overlapping area of ​​the initial panoramic image; Compare the pixel color balance values ​​of pixels in the overlapping region with the pixel values ​​of the image edges, and discard the pixel color balance values ​​that are less than those of all image edges; Color equalization processing is performed on the corresponding pixels in the overlapping region based on the color equalization values ​​of the unremoved pixels.

5. The method according to claim 1, characterized in that, The step of fusing pixels in the overlapping regions of the filtered standard panoramic image to obtain the target panoramic image includes: The first weighted pixel value of the first path image constituting the overlapping region in the filtered standard panoramic image is determined based on the first weighting factor. The second weighted pixel value of the second path image constituting the overlapping region in the filtered standard panoramic image is determined based on the second weighting factor; wherein, the first weighting factor and the second weighting factor decrease linearly from the center pixel point to the boundary pixel point of the overlapping region; The target panoramic image is obtained by summing the first and second weighted pixel values ​​of the overlapping regions in the filtered standard panoramic image.

6. A device for determining a panoramic image, characterized in that, include: The edge detection module is used to acquire initial images from multiple cameras and perform edge detection on each initial image to obtain the image edges. The image alignment module is used to perform image spatial alignment and image stitching based on the positional relationship of each initial image to obtain an initial panoramic image; The pixel adjustment module is used to determine the overlapping area in the initial panoramic image based on the image edges of each initial image, and to perform pixel adjustment on the overlapping area in the initial panoramic image to obtain a standard panoramic image. The pixel fusion module is used to perform pixel fusion based on pixel feature point matching on the overlapping areas in the standard panoramic image to obtain the target panoramic image; The edge detection module is specifically used for: For each initial image, the initial edge region of the initial image is determined based on the pixel difference between different color channels of the initial image and the edge threshold; The initial edge region is filtered and sharpened to obtain the target edge region; Pixels in the target edge region whose pixel values ​​are higher than or equal to a first threshold are identified as edge pixels. Pixels in the target edge region whose pixel values ​​are lower than a first threshold and higher than a second threshold are identified as potential edge pixels; wherein, the first threshold is greater than the second threshold; For the potential edge pixels, the edge pixels of the initial image are determined based on the pixel values ​​in the neighborhood of the potential edge pixels. Obtain discontinuous pixels in the target edge region and determine the positional relationship between each discontinuous pixel and the edge pixels. If a discontinuous pixel is adjacent to an edge pixel, it is determined that the discontinuous pixel belongs to the edge pixel. If a discontinuous pixel is not adjacent to an edge pixel, it is determined that the discontinuous pixel does not belong to the edge pixel. Connect all edge pixels and fit the edges of the connected regions according to the coordinates of each connected region to obtain the image edge. The pixel fusion module includes: The feature extraction unit is used to extract features from the two images that constitute the overlapping area in the standard panoramic image to obtain a set of pixel feature points to be matched; An image matching calculation unit is used to determine a matching window region based on a sliding matching window within the overlapping region, and to calculate the image matching degree of the set of pixel feature points to be matched contained in the matching window region. The filtering unit is used to perform median filtering on pixels contained in the matching window region where the image matching degree is greater than the matching threshold. The pixel fusion unit is used to perform pixel fusion on the overlapping areas in the filtered standard panoramic image to obtain the target panoramic image.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the method for determining the panoramic image according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for determining a panoramic image as described in any one of claims 1-5.

Images