Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and the method comprises the steps: projecting each image in an image sequence to a panorama camera space, determining a common-view relationship between the images, constructing an undirected graph, taking a color value of a target node as a reference color value, carrying out the color value propagation through the undirected graph, and obtaining a target color value; and determining color propagation coefficients corresponding to other nodes, performing image color processing on the images corresponding to the nodes by using the color propagation coefficients, so that the color values of the nodes in the undirected graph tend to be consistent, and when the colors and the brightness of the images collected at the same moment are inconsistent due to different shooting angles and shooting contents of the camera, performing image color processing on the images collected at the same moment. According to the technical scheme, the color value processing can be performed on the image, so that the color values of the images with the common-view relationship are similar, the problems of uneven coloring and poor preview effect of the three-dimensional point cloud are solved, the problem of large difference of the color values of the points in the same point cloud region is avoided, and the visual coherence and authenticity of the color values of the point cloud are improved.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology and image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) scanning devices typically incorporate a laser radar (LiDAR) and a camera. The LiDAR scans the surrounding environment to acquire three-dimensional spatial information and generate a corresponding point cloud, while the camera captures images of the surrounding environment and uses the color values ​​of each point in the image to colorize the point cloud generated by the LiDAR.

[0003] During 3D reconstruction, due to the limited field of view of a single camera, scanning equipment typically uses multiple cameras for image acquisition to obtain a wider field of view. The images captured by these cameras are then stitched together to create a complete view and perform point cloud coloring. However, at the same moment, different cameras capture different angles and content, resulting in variations in white balance and exposure. This leads to inconsistent color and brightness across different areas of the stitched image, resulting in poor point cloud coloring. Summary of the Invention

[0004] In response to the above technical problems, embodiments of the present disclosure provide an image processing method, apparatus, device, and storage medium.

[0005] One aspect of an embodiment of the present disclosure provides an image processing method, including:

[0006] projecting corresponding images in an image sequence into a panoramic camera space based on camera intrinsic parameters of at least one camera among the multiple cameras and determining a co-viewing relationship between the images, wherein the image sequence includes images captured by the multiple cameras at the same image capture moment, the panoramic camera space is a three-dimensional space corresponding to a panoramic camera model that performs panoramic image capture of a target scene, and the images having the co-viewing relationship have an intersection in their corresponding image capture ranges in the panoramic camera space;

[0007] constructing an undirected graph corresponding to the image sequence with the images as nodes and the co-viewing relationships between the images as edges;

[0008] Taking the color value of the target node as the reference color value, the color propagation coefficient corresponding to at least one node is determined through the undirected graph, and the image corresponding to the node is subjected to image color processing using the color propagation coefficient. The color propagation coefficient is used to characterize the degree of adjustment required to adjust the color value of the image to the reference color value, and the color value includes at least one of a chromaticity value and a brightness value.

[0009] Optionally, the method of determining a color propagation coefficient corresponding to at least one node through the undirected graph using the color value of the target node as a reference color value, and performing image color processing on the image corresponding to the node using the color propagation coefficient includes:

[0010] Taking the target node as the color propagation starting point, determining the color propagation coefficients of the nodes connected to the target node, and performing image color processing on the images corresponding to the nodes connected to the target node;

[0011] Marking the target node and the node where the image color processing is completed as propagated nodes;

[0012] Based on the structure of the undirected graph, the color propagation coefficients of the nodes connected to the propagated nodes are determined, and image color processing is performed on the images corresponding to the nodes connected to the propagated nodes until the nodes in the undirected graph are marked as the propagated nodes.

[0013] Optionally, determining the color propagation coefficient of the node connected to the propagated node based on the structure of the undirected graph, and performing image color processing on the image corresponding to the node connected to the target node includes:

[0014] Determining, based on the structure of the undirected graph, non-propagated nodes connected to the propagated nodes;

[0015] For at least one propagated node, determining the color propagation coefficient of the non-propagated node connected to the propagated node based on the first color value of the propagated node and the second color value of the non-propagated node connected to the propagated node;

[0016] Image color processing is performed on an image of an unpropagated node connected to the propagated node based on a color propagation coefficient, wherein a color value of the unpropagated node after image color processing is a product of the second color value and the color propagation coefficient.

[0017] Optionally, determining the color propagation coefficient of the non-propagated node connected to the propagated node based on the first color value of the propagated node and the second color value of the non-propagated node connected to the propagated node includes:

[0018] In response to a ratio of the first color value to the second color value being less than or equal to a coefficient threshold, determining the ratio of the first color value to the second color value as the color propagation coefficient;

[0019] In response to the ratio of the first color value to the second color value being greater than the coefficient threshold, the color propagation coefficient is determined based on a coefficient compression curve, wherein the coefficient compression curve is a logarithmic curve with the ratio of the first color value to the second color value as an independent variable.

[0020] Optionally, before determining a color propagation coefficient corresponding to at least one node using the undirected graph based on the color value of the target node as a reference color value and performing image color processing on the image corresponding to the node using the color propagation coefficient, the method includes:

[0021] Determine a target node from the undirected graph based on a preset screening condition, wherein the preset screening condition includes at least one of the following:

[0022] The image color variance is the smallest, where the image color variance is the variance of the color values ​​of the pixels in the image corresponding to the node;

[0023] The color time series variation value of the corresponding camera is the smallest. The color time series variation value is used to represent the change in the average color value of the images captured by the same camera at different image capture times. The average color value is the average color value of the pixels in the image.

[0024] The node distance value is the smallest, and the node distance value is the sum of the differences between the average color values ​​of the node and other nodes in the undirected graph.

[0025] Optionally, before determining a color propagation coefficient corresponding to at least one node using the undirected graph based on the color value of the target node as a reference color value and performing image color processing on the image corresponding to the node using the color propagation coefficient, the method includes:

[0026] For each two nodes connected by an edge, determining a common view area of ​​at least one node by image feature matching, wherein the common view area is an area with the same image content in the images having a common view relationship;

[0027] The average of the color values ​​of the pixels in the common viewing area is determined as the color value of the node.

[0028] Optionally, before determining the mean of the color values ​​of the pixels within the common view area as the color value of the node, the method includes:

[0029] Determine a matching block of at least one pixel based on a preset matching block size, where the matching block is an image region centered on the pixel and having a size equal to the preset matching block size;

[0030] The average of the color values ​​of the pixels in the matching block is determined as the color value of the central pixel of the matching block.

[0031] Another aspect of the present disclosure provides an image processing apparatus, including:

[0032] a first determining module, configured to project corresponding images in an image sequence into a panoramic camera space based on camera intrinsic parameters of at least one camera among the multiple cameras, and determine a co-viewing relationship between the images, wherein the image sequence includes images captured by the multiple cameras at the same image capture moment, the panoramic camera space is a three-dimensional space corresponding to a panoramic camera model that performs panoramic image capture of a target scene, and the images having the co-viewing relationship have an intersection in their corresponding image capture ranges in the panoramic camera space;

[0033] A construction module, configured to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-viewing relationships between the images as edges;

[0034] A processing module is used to use the color value of the target node as a reference color value, determine the color propagation coefficient corresponding to at least one node through the undirected graph, and use the color propagation coefficient to perform image color processing on the image corresponding to the node, wherein the color propagation coefficient is used to represent the degree of adjustment required to adjust the color value of the image to the reference color value, and the color value includes at least one of a chromaticity value and a brightness value.

[0035] Another aspect of the present disclosure provides an electronic device, including:

[0036] memory for storing computer programs;

[0037] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method described in the above aspects is implemented.

[0038] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the above aspects is implemented.

[0039] Another aspect of the embodiments of the present disclosure provides a computer program, comprising computer program instructions, which implement the method described in the above aspects when executed by a processor.

[0040] Based on the embodiment of the present disclosure, each image in the image sequence is projected into the panoramic camera space, the common view relationship between the images is determined and an undirected graph is constructed, the color value of the target node is used as the reference color value, the color value is propagated using the undirected graph, the color propagation coefficients corresponding to other nodes are determined, and the image corresponding to the node is processed with the color propagation coefficients, so that the color values ​​of each node in the undirected graph tend to be consistent. When the color and brightness of images collected at the same time are inconsistent due to different camera shooting angles and shooting contents, the image can be processed with color values ​​so that the color values ​​of images with a common view relationship are similar, which solves the problems of uneven coloring of three-dimensional point clouds and poor preview effects, avoids the problem of large differences in point color values ​​within the same point cloud area, and improves the visual consistency and authenticity of the point cloud color values.

[0041] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0044] Figure 1 This is a flowchart of an embodiment of the image processing method disclosed herein;

[0045] Figure 2 An effect diagram of projecting an image into a panoramic camera space provided by an embodiment of the present disclosure;

[0046] Figure 3 A schematic diagram of an undirected graph provided in an embodiment of the present disclosure;

[0047] Figure 4 This is a flowchart of another embodiment of the image processing method disclosed herein;

[0048] Figure 5 This is a flowchart of another embodiment of the image processing method disclosed herein;

[0049] Figure 6 This is a schematic structural diagram of an embodiment of the image processing device disclosed herein;

[0050] Figure 7 This is a schematic structural diagram of another embodiment of the image processing device disclosed herein;

[0051] Figure 8 The figure is a schematic structural diagram of an application embodiment of the electronic device disclosed herein. DETAILED DESCRIPTION

[0052] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0053] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0054] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0055] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0056] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0057] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0058] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0059] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0060] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0061] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0062] Figure 1This is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure. The image processing method of the present embodiment can be implemented using an electronic device with image processing capabilities. For example, the electronic device may include, but is not limited to, a Simultaneous Localization and Mapping (SLAM) scanner, a smartphone, a laptop computer, a desktop computer, and other devices. This disclosure uses a SLAM scanner as an example.

[0063] like Figure 1 As shown, the method includes the following steps:

[0064] Step 101 : projecting corresponding images in an image sequence into a panoramic camera space based on the camera intrinsic parameters of at least one camera among a plurality of cameras and determining a common viewing relationship between the images.

[0065] Among them, the image sequence includes images captured by multiple cameras at the same image capture time. The panoramic camera space is the three-dimensional space corresponding to the panoramic camera model that performs panoramic image capture of the target scene. The images with a common view relationship have an intersection in the corresponding image capture ranges in the panoramic camera space. Figure 2 The figure shows the effect of projecting images taken by 6 cameras into the panoramic camera space.

[0066] In one possible implementation, the image sequence is obtained by capturing images of the target scanning scene at the same time by multiple cameras in a SLAM scanning device. To avoid missing information, the shooting angles of two adjacent cameras are typically close, resulting in overlapping content between two or more adjacent images. For example, the left half of the first image and the right half of the second image may capture the same area of ​​the target scanning scene. By projecting each image in the image sequence into the panoramic camera space, it is possible to determine whether there is content overlap between the images, that is, whether there is a common view relationship.

[0067] Optionally, the co-viewing relationship between images can be determined by calculating the point cloud coordinate range corresponding to each image. There is a co-viewing relationship between images with intersections in their point cloud coordinate ranges. The point cloud coordinate range of an image can be calculated using parameters such as the coordinates of each point in the point cloud in the world coordinate system, the camera pose, the lidar-camera extrinsic parameters, and the camera intrinsic parameters. The camera pose is used to characterize the position and attitude of the camera in the world coordinate system, and the lidar-camera extrinsic parameters are used to characterize the relative position and attitude of the lidar relative to the camera in the world coordinate system. The co-viewing relationship between images is determined based on whether there is an intersection between the point cloud coordinate ranges corresponding to the images. For example, two images whose point cloud size of the intersection is larger than the preset point cloud size can be determined as a pair of images with a co-viewing relationship.

[0068] Step 102 : construct an undirected graph corresponding to the image sequence, using images as nodes and co-viewing relationships between images as edges.

[0069] In one possible implementation, an undirected graph is constructed with images as nodes and co-visibility relationships between images as edges. Each edge in the undirected graph connects nodes corresponding to two images with a co-visibility relationship. Optionally, if an independent node exists, that is, it is not connected to any other node by an edge, then the node can be removed from the undirected graph. The image corresponding to the node is not co-visible with any other image, so there is no need to adjust the color value. The node in the undirected graph is connected to at least one of the other nodes.

[0070] Indicative, Figure 3 shows an undirected graph. Figure 3 As shown, in this undirected graph, the image corresponding to node 1 has a co-viewing relationship with the images corresponding to nodes 0, 2, 3, and 4, and the image of node 6 only has a co-viewing relationship with the image of node 3.

[0071] Step 103 : Taking the color value of the target node as the reference color value, determine the color propagation coefficient corresponding to at least one node through the undirected graph, and perform image color processing on the image corresponding to the node using the color propagation coefficient.

[0072] Among them, the color propagation coefficient is used to characterize the degree of adjustment required to adjust the color value of the image to the reference color value, and the color value includes at least one of the chromaticity value and the brightness value (for example, it can be the chromaticity value, the brightness value, or a vector composed of the chromaticity value and the brightness value).

[0073] Optionally, the color value may be a YCrCb value, ie, a YUV value, where Y is a brightness value, and U and V are chrominance values ​​(used to describe hue and saturation).

[0074] Optionally, the target node can be any one or more nodes in the undirected graph, or it can be one or more nodes filtered out from the nodes of the undirected graph based on preset filtering conditions (for example, the node corresponding to the image taken by the camera with the smallest difference in color values ​​of the images in the time series, that is, the node with the highest reliability of color value).

[0075] The color propagation coefficient represents the degree of adjustment required for the color value. For example, if the color propagation coefficient obtained by solving node A is 0.8, the target color value of the image corresponding to node A can be calculated by multiplying the original color value of the image corresponding to node A by the color propagation coefficient of 0.8, and the image data of the image corresponding to node A can be modified based on the target color value to obtain the image after image color processing. Since the color propagation coefficient represents the degree of adjustment required to adjust the color value of the image to the reference color value, after image color processing is performed based on the color propagation coefficient, the color value of the image corresponding to each node is converted to the color value corresponding to the target node, that is, the color value of the image corresponding to each node in the undirected graph is consistent.

[0076] Based on the embodiment of the present disclosure, each image in the image sequence is projected into the panoramic camera space, the common view relationship between the images is determined and an undirected graph is constructed, the color value of the target node is used as the reference color value, the color value is propagated using the undirected graph, the color propagation coefficients corresponding to other nodes are determined, and the image corresponding to the node is processed with the color propagation coefficients, so that the color values ​​of each node in the undirected graph tend to be consistent. When the color and brightness of images collected at the same time are inconsistent due to different camera shooting angles and shooting contents, the image can be processed with color values ​​so that the color values ​​of images with a common view relationship are similar, which solves the problems of uneven coloring of three-dimensional point clouds and poor preview effects, avoids the problem of large differences in point color values ​​within the same point cloud area, and improves the visual consistency and authenticity of the point cloud color values.

[0077] In a possible implementation, when determining the color propagation coefficient of at least one node, the structure of an undirected graph can be used to continuously propagate the color value of the target node starting from the target node. Figure 4 As shown, the above step 103 may specifically include the following steps:

[0078] Step 103a: Taking the target node as the starting point of color propagation, determining the color propagation coefficients of the nodes connected to the target node, and performing image color processing on the images corresponding to the nodes connected to the target node.

[0079] Optionally, after the target node is selected, the first color value propagation is performed. First, the nodes connected to the target node are determined. Then, based on the baseline color value and the color value of the node connected to the target node, the color propagation coefficient corresponding to the node connected to the target node is determined, and image color processing is performed on the image corresponding to the node connected to the target node.

[0080] Indicative, such as Figure 3 As shown, the target node is node 1, and the first color value propagation is performed on nodes 0, 2, 3, and 4 to calculate the color propagation coefficient and process the image color.

[0081] Step 103b: Mark the target node and the node that has completed image color processing as propagated nodes.

[0082] After the first and subsequent color value propagation is complete, the target node and the node that has completed image color processing are marked as propagated nodes, and the remaining nodes in the undirected graph are marked as non-propagated nodes. For example, in step 103a, nodes 1, 0, 2, 3, and 4 are propagated nodes, while nodes 5 and 6 are non-propagated nodes.

[0083] Step 103c, based on the structure of the undirected graph, determine the color propagation coefficients of the nodes connected to the propagated nodes, and perform image color processing on the images corresponding to the nodes connected to the propagated nodes until the nodes in the undirected graph are marked as propagated nodes.

[0084] After the first color value propagation is performed based on the target node, the target node and other propagated nodes can be used to determine the color propagation coefficients of the nodes connected to the propagated node and perform image color processing, update them to the propagated nodes, and continue to iterate until the nodes in the undirected graph are marked as propagated nodes. For example, the iteration is stopped when a preset number of nodes are marked as propagated nodes, or when all nodes with a common viewing relationship are marked as propagated nodes.

[0085] In a possible implementation, the color propagation coefficient may be calculated based on the color value of the propagated node and the color value of the non-propagated node connected to the propagated node. Step 103c may include the following steps:

[0086] Step 1: Based on the structure of the undirected graph, determine the non-propagated nodes connected to the propagated nodes.

[0087] Illustratively, as shown in the example of step 103 a , the undirected graph further includes a non-propagated node 5 connected to the propagated node 4 , and a non-propagated node 6 connected to the propagated node 3 .

[0088] Step 2: for at least one propagated node, determine a color propagation coefficient of the non-propagated node connected to the propagated node based on the first color value of the propagated node and the second color value of the non-propagated node connected to the propagated node.

[0089] In one possible implementation, the color propagation coefficient of the non-propagated node connected to the propagated node can be determined based on the ratio of the first color value to the second color value. For example, taking brightness Y as an example, when node A is a propagated node, the color propagation coefficient ω corresponding to node B can be obtained based on the brightness Ya of node A and the brightness Yb of node B connected to node A. y =Ya / Yb, where ω yA floating point number greater than or equal to 0. The brightness of node A is propagated to node B, that is, the color propagation coefficient ω based on node B. y Perform image color processing. The calculation formula of the image color processing process is as follows:

[0090] Yb'=ω y ·Yb (1)

[0091] Wherein, Yb' is the color value of node B after image color processing.

[0092] In another possible implementation, in order to prevent the color value of the image corresponding to the node after image color processing from exceeding a preset color value range (for example, brightness exceeding 255) when the color propagation coefficient is large, the embodiment of the present disclosure provides a color value compression curve, which can calculate the color propagation coefficient corresponding to the non-propagated node based on the color value compression curve. Specifically, in response to the ratio of the first color value to the second color value being less than or equal to a coefficient threshold, the ratio of the first color value to the second color value is determined as the color propagation coefficient; in response to the ratio of the first color value to the second color value being greater than the coefficient threshold, the color propagation coefficient is determined based on the coefficient compression curve, wherein the coefficient compression curve is a logarithmic curve with the ratio of the first color value to the second color value as an independent variable.

[0093] Schematically, taking brightness as an example, the coefficient compression curve is expressed as follows:

[0094]

[0095] Using the color propagation coefficient calculated using the above coefficient compression curve to perform image color processing can make the image brightness tend to be consistent while preventing the image from being overexposed.

[0096] Step three, performing image color processing on the image of the non-propagated node connected to the propagated node based on the color propagation coefficient, wherein the color value of the non-propagated node after the image color processing is the product of the second color value and the color propagation coefficient.

[0097] In one possible implementation, the target node can be one or more nodes arbitrarily selected from an undirected graph. To further optimize the point cloud coloring effect, the image with the best color effect can be selected as the color propagation benchmark. Before step 103 above, the image processing method can further include the following steps:

[0098] Determine the target node from the undirected graph based on a preset filtering condition, where the preset filtering condition includes at least one of the following:

[0099] A. The image color variance is the smallest. The image color variance is the variance of the color values ​​of the pixels in the image corresponding to the node.

[0100] B. The color timing change value of the corresponding camera is the smallest. The color timing change value is used to represent the change in the average color value of the image captured by the same camera at different image acquisition times. The average color value is the average color value of the pixels in the image.

[0101] C. The node distance value is the smallest. The node distance value is the sum of the differences between the node and the average color values ​​of other nodes in the undirected graph.

[0102] Color values ​​can be selected from the red, green, and blue (RGB) color channels. Using preset filter condition A, you can filter out images with relatively uniform color values. These images, with their uniform color distribution and minimal color variation, can avoid camera overexposure due to sudden color changes, and their color values ​​are relatively reliable. Using preset filter condition B, you can filter out images with minimal acquisition environment differences and stable camera parameters. Using preset filter condition C, you can filter out images with the smallest total color difference from other nodes, minimizing the degree of color change in the image.

[0103] In a possible implementation, since the embodiment of the present disclosure optimizes the color differences of different images in the common viewing area and determines the degree of adjustment of the color value, for a pair of nodes with a common viewing relationship, the average color value of the pixel points in the common viewing area of ​​the corresponding images can be determined as the color value of the image (node). Figure 5 As shown, before the above step 103, the image processing method may further include the following steps:

[0104] Step 501: For each two nodes connected by an edge, determine a common viewing area of ​​at least one node by image feature matching.

[0105] The common viewing area is an area with the same image content in the images having a common viewing relationship.

[0106] Optionally, for each pair of images with a common view relationship (i.e., the two nodes connected by each edge), the common view area of ​​the pair of images can be determined by feature matching, that is, the area in the two images where the overlapping image content corresponds. Feature matching can be achieved through recognition models or boundary feature matching methods. Among them, for the same image, the common view area between it and different images may be located at different positions in the image. For example, the common view area between image a and image b is located in the left half of image a, and the common view area between image a and image c is located in the right half of image a.

[0107] Step 502: Determine the average color value of the pixels in the common view area as the color value of the node.

[0108] Alternatively, for two nodes in a common view relationship, the color values ​​of the pixels in their respective common view regions can be directly determined as their corresponding color values. The color value mean is the ratio of the sum of the color values ​​of the pixels in the common view region to the number of pixels.

[0109] Optionally, considering the color value error and robustness of a single pixel, a local window, i.e., a matching block, can be used to determine the color value of the pixel, thereby improving the accuracy of the image color value and the gain coefficient. Before step 502, the embodiment of the present disclosure may also include the following steps:

[0110] A matching block of at least one pixel is determined based on a preset matching block size; and an average value of color values ​​of the pixels in the matching block is determined as the color value of the central pixel of the matching block.

[0111] The matching block is an image area centered on a pixel point and having a size equal to a preset matching block size.

[0112] For example, the preset matching block size is 3*3, where 3 represents 3 pixels. For any pixel in the common viewing area, the pixel and the 8 pixels around it are determined as the matching block corresponding to the pixel, and the average of the actual color values ​​of the 9 pixels in the matching block is determined as the color value of the pixel.

[0113] Therefore, in step 502, the color value of the pixel participating in the image color value calculation is the average of the actual color values ​​of the pixels in the matching block corresponding to the pixel.

[0114] In a possible implementation, in order to further improve the accuracy of the color value of the node and avoid interference from special pixels, step 502 may further include the following steps:

[0115] Step 502a: using the interquartile range method to determine the first pixel point in the common viewing area.

[0116] The first pixel point includes a pixel point whose pixel color difference is greater than the upper quartile, and the pixel color difference is the difference in color values ​​of a common viewing pixel pair, and a common viewing pixel pair refers to a pair of pixel points in corresponding images at the same point in the intersection of the point clouds.

[0117] In the disclosed embodiment, the quartiles are obtained by arranging the pixel color differences of the common view pixel pairs within the common view area in ascending order, dividing the area into four equal parts, and the pixel color differences at the three dividing points, including the first quartile (i.e., the lower quartile, located at the 25% position), the second quartile (i.e., the median), and the third quartile (i.e., the upper quartile, located at the 75% position). The pixel point with a pixel color difference greater than the upper quartile is determined as the first pixel point.

[0118] Step 502b: perform object recognition on the image, and in response to the presence of a target object belonging to a preset object type, determine a second pixel point corresponding to the target object.

[0119] The preset object types may include, but are not limited to, glass, mirrors, screens, transient objects, and other object types with less reliable color values. Optionally, the image corresponding to each node may be input into an object recognition model. The object recognition model outputs the various object types contained in the image and the image regions corresponding to each object type. The pixel within the image region corresponding to the preset object type is then determined as the second pixel.

[0120] It is worth mentioning that there is no strict order in which steps one and two should be executed.

[0121] Step 502c: Eliminate the first pixel point and the second pixel point in the common viewing area to obtain the target pixel point in the common viewing area.

[0122] Step 502d: Determine the average of the color values ​​of the target pixel points as the color value of the node.

[0123] Filtering and removing the first pixel can reduce the interference of abnormal pixels in image acquisition on the color value calculation, while filtering and removing the second pixel can prevent the interference of pixels with less reliable color values ​​on the color value calculation. The color value corresponding to the node is obtained based on the average of the color values ​​of the retained target pixels. This can improve the reliability of the color value corresponding to the node, thereby improving the accuracy of the subsequent gain coefficient solution.

[0124] Based on the embodiment of the present disclosure, the color value of the pixel point is determined by using a matching block, and the pixels with large color value differences and low color value reliability in the common view area are filtered out, and then the image color value corresponding to the node is determined based on the mean of the color values ​​of the pixels in the common view area. This can reduce the impact of errors on the color value of the node, improve robustness, and thereby improve the reliability of the color propagation coefficient.

[0125] Figure 6 The following is a block diagram of an image processing apparatus provided by an exemplary embodiment of the present disclosure. The image processing apparatus includes:

[0126] A first determining module 601 is configured to project corresponding images in an image sequence into a panoramic camera space based on camera intrinsic parameters of at least one camera among the multiple cameras and determine a co-viewing relationship between the images, wherein the image sequence includes images captured by the multiple cameras at the same image capture time, the panoramic camera space is a three-dimensional space corresponding to a panoramic camera model that captures panoramic images of a target scene, and images having a co-viewing relationship have image capture ranges corresponding to the images in the panoramic camera space that intersect;

[0127] A construction module 602 is configured to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-visibility relationships between the images determined by the first determination module 601 as edges;

[0128] The processing module 603 is used to determine the color propagation coefficient corresponding to at least one node using the color value of the target node as the reference color value through the undirected graph constructed by the construction module 602, and use the color propagation coefficient to perform image color processing on the image corresponding to the node. The color propagation coefficient is used to represent the degree of adjustment required to adjust the color value of the image to the reference color value, wherein the color value includes at least one of a chromaticity value and a brightness value.

[0129] Optionally, in a possible implementation, the processing module 603 may also be used to:

[0130] Taking the target node as the starting point of color propagation, the color propagation coefficients of the nodes connected to the target node are determined, and image color processing is performed on the images corresponding to the nodes connected to the target node;

[0131] Mark the target node and the node that completes the image color processing as propagated nodes;

[0132] Based on the structure of the undirected graph, the color propagation coefficients of the nodes connected to the propagated nodes are determined, and image color processing is performed on the images corresponding to the nodes connected to the propagated nodes until the nodes in the undirected graph are marked as propagated nodes.

[0133] Optionally, in a possible implementation, the processing module 603 may also be used to:

[0134] Based on the structure of the undirected graph, determine the non-propagated nodes connected to the propagated nodes;

[0135] For at least one propagated node, determining a color propagation coefficient of a non-propagated node connected to the propagated node based on a first color value of the propagated node and a second color value of a non-propagated node connected to the propagated node;

[0136] Image color processing is performed on an image of an unpropagated node connected to a propagated node based on a color propagation coefficient, wherein a color value of the unpropagated node after the image color processing is a product of a second color value and the color propagation coefficient.

[0137] Optionally, in a possible implementation, the processing module 603 may also be used to:

[0138] In response to the ratio of the first color value to the second color value being less than or equal to a coefficient threshold, determining the ratio of the first color value to the second color value as a color propagation coefficient;

[0139] In response to the ratio of the first color value to the second color value being greater than a coefficient threshold, a color propagation coefficient is determined based on a coefficient compression curve, wherein the coefficient compression curve is a logarithmic curve with the ratio of the first color value to the second color value as an independent variable.

[0140] Optionally, in a possible implementation manner, the image processing device further includes:

[0141] A screening module is configured to determine a target node from an undirected graph based on a preset screening condition, wherein the preset screening condition includes at least one of the following:

[0142] The image color variance is the smallest, which is the variance of the color values ​​of the pixels in the image corresponding to the node;

[0143] The color time series variation value of the corresponding camera is the smallest. The color time series variation value is used to represent the change in the average color value of the images captured by the same camera at different image acquisition times. The average color value is the average color value of the pixels in the image.

[0144] The node distance value is the smallest, which is the sum of the differences between the node and the average color values ​​of other nodes in the undirected graph.

[0145] Optionally, in a possible implementation, as Figure 7 As shown, the image processing device also includes:

[0146] A second determining module 701 is configured to determine, for each edge connecting two nodes, a common viewing area of ​​at least one node by image feature matching, where the common viewing area is an area with the same image content in the images having a common viewing relationship;

[0147] The third determining module 702 is configured to determine the average color value of the pixels in the common view area as the color value of the node.

[0148] Optionally, in a possible implementation manner, the image processing device further includes:

[0149] A fourth determining module is configured to determine a matching block of at least one pixel point based on a preset matching block size, where the matching block is an image region centered on the pixel point and having a size equal to the preset matching block size;

[0150] The fifth determining module is configured to determine the average color value of the pixels in the matching block as the color value of the center pixel of the matching block.

[0151] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The parts of the embodiments that are the same, similar, or corresponding can be referred to each other. Since the method, device, system, and equipment embodiments are basically corresponding, the relevant parts can be referred to the description of the corresponding parts. The methods, devices, systems, and equipment of the embodiments of the present disclosure also correspond to each other in terms of specific implementation methods and beneficial technical effects. The relevant contents can be referenced to each other and will not be repeated here.

[0152] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0153] Memory for storing computer programs;

[0154] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the image processing method described in any one of the above embodiments of the present disclosure is implemented.

[0155] Figure 8 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed in the present invention. Figure 8 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0156] like Figure 8 As shown, the electronic device includes one or more processors and memory.

[0157] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0158] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the image processing methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0159] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0160] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0161] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0162] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0163] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the image processing method according to various embodiments of the present disclosure described in the above part of this specification.

[0164] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0165] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the image processing method according to various embodiments of the present disclosure described in the above part of this specification.

[0166] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0167] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0168] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0169] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0170] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0171] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0172] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0173] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0174] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An image processing method, characterized in that: include: projecting corresponding images in an image sequence into a panoramic camera space based on camera intrinsic parameters of at least one camera among the multiple cameras and determining a co-viewing relationship between the images, wherein the image sequence includes images captured by the multiple cameras at the same image capture moment, the panoramic camera space is a three-dimensional space corresponding to a panoramic camera model that performs panoramic image capture of a target scene, and the images having the co-viewing relationship have an intersection in their corresponding image capture ranges in the panoramic camera space; constructing an undirected graph corresponding to the image sequence with the images as nodes and the co-viewing relationships between the images as edges; Taking the color value of the target node as the reference color value, the color propagation coefficient corresponding to at least one node is determined through the undirected graph, and the image corresponding to the node is subjected to image color processing using the color propagation coefficient. The color propagation coefficient is used to characterize the degree of adjustment required to adjust the color value of the image to the reference color value, and the color value includes at least one of a chromaticity value and a brightness value.

2. The method according to claim 1, characterized in that The method of determining a color propagation coefficient corresponding to at least one node using the undirected graph based on the color value of the target node as a reference color value, and performing image color processing on the image corresponding to the node using the color propagation coefficient includes: Taking the target node as the color propagation starting point, determining the color propagation coefficients of the nodes connected to the target node, and performing image color processing on the images corresponding to the nodes connected to the target node; Marking the target node and the node where the image color processing is completed as propagated nodes; Based on the structure of the undirected graph, the color propagation coefficients of the nodes connected to the propagated nodes are determined, and image color processing is performed on the images corresponding to the nodes connected to the propagated nodes until the nodes in the undirected graph are marked as the propagated nodes.

3. The method according to claim 2, characterized in that The step of determining the color propagation coefficients of the nodes connected to the propagated nodes based on the structure of the undirected graph, and performing image color processing on the images corresponding to the nodes connected to the target node, includes: Determining, based on the structure of the undirected graph, non-propagated nodes connected to the propagated nodes; For at least one propagated node, determining the color propagation coefficient of the non-propagated node connected to the propagated node based on the first color value of the propagated node and the second color value of the non-propagated node connected to the propagated node; Image color processing is performed on an image of an unpropagated node connected to the propagated node based on a color propagation coefficient, wherein a color value of the unpropagated node after image color processing is a product of the second color value and the color propagation coefficient.

4. The method according to claim 3, characterized in that The determining, based on the first color value of the propagated node and the second color value of the non-propagated node connected to the propagated node, the color propagation coefficient of the non-propagated node connected to the propagated node comprises: In response to a ratio of the first color value to the second color value being less than or equal to a coefficient threshold, determining the ratio of the first color value to the second color value as the color propagation coefficient; In response to the ratio of the first color value to the second color value being greater than the coefficient threshold, the color propagation coefficient is determined based on a coefficient compression curve, wherein the coefficient compression curve is a logarithmic curve with the ratio of the first color value to the second color value as an independent variable.

5. The method according to any one of claims 1 to 4, characterized in that: Before determining the color propagation coefficient corresponding to at least one node using the undirected graph based on the color value of the target node as the reference color value and performing image color processing on the image corresponding to the node using the color propagation coefficient, the method includes: Determine a target node from the undirected graph based on a preset screening condition, wherein the preset screening condition includes at least one of the following: The image color variance is the smallest, where the image color variance is the variance of the color values ​​of the pixels in the image corresponding to the node; The color time series variation value of the corresponding camera is the smallest. The color time series variation value is used to represent the change in the average color value of the images captured by the same camera at different image capture times. The average color value is the average color value of the pixels in the image. The node distance value is the smallest, and the node distance value is the sum of the differences between the average color values ​​of the node and other nodes in the undirected graph.

6. The method according to any one of claims 1 to 4, characterized in that: Before determining the color propagation coefficient corresponding to at least one node using the undirected graph based on the color value of the target node as the reference color value and performing image color processing on the image corresponding to the node using the color propagation coefficient, the method includes: For each two nodes connected by an edge, determining a common view area of ​​at least one node by image feature matching, wherein the common view area is an area with the same image content in the images having a common view relationship; The average of the color values ​​of the pixels in the common viewing area is determined as the color value of the node.

7. The method according to claim 6, characterized in that Before determining the mean of the color values ​​of the pixels within the common view area as the color value of the node, the method includes: Determine a matching block of at least one pixel based on a preset matching block size, where the matching block is an image region centered on the pixel and having a size equal to the preset matching block size; The average of the color values ​​of the pixels in the matching block is determined as the color value of the central pixel of the matching block.

8. An image processing device, characterized in that: include: a first determining module, configured to project corresponding images in an image sequence into a panoramic camera space based on camera intrinsic parameters of at least one camera among the multiple cameras, and determine a co-viewing relationship between the images, wherein the image sequence includes images captured by the multiple cameras at the same image capture moment, the panoramic camera space is a three-dimensional space corresponding to a panoramic camera model that performs panoramic image capture of a target scene, and the images having the co-viewing relationship have an intersection in their corresponding image capture ranges in the panoramic camera space; A construction module, configured to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-viewing relationships between the images as edges; A processing module is used to use the color value of the target node as a reference color value, determine the color propagation coefficient corresponding to at least one node through the undirected graph, and use the color propagation coefficient to perform image color processing on the image corresponding to the node, wherein the color propagation coefficient is used to represent the degree of adjustment required to adjust the color value of the image to the reference color value, and the color value includes at least one of a chromaticity value and a brightness value.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is configured to execute a computer program stored in the memory, and when the computer program is executed, implements the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images