Image processing method, device, equipment and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, device and storage medium, which determine the common view relationship between images based on the corresponding point cloud coordinate ranges between the images and construct an undirected graph, generate a corresponding loss function based on the connectivity structure of the undirected graph, and the function value corresponding to the loss function represents the total color difference of the undirected graph after the color value is adjusted by the gain coefficient, and constrain the color value of each node to be consistent through preset constraints. The loss function is iteratively calculated to solve the gain coefficient corresponding to each node when the function value meets the preset constraints, and the gain coefficient is used to optimize the color value of the image, so that the color values ​​of images with a common view relationship are similar, thereby solving the problems of uneven coloring and poor preview effect of three-dimensional point clouds, avoiding the problem of large differences in point color values ​​within the same point cloud area, and improving the visual consistency and authenticity of the point cloud color values.

Description

Technical Field

[0001] This disclosure relates to three-dimensional reconstruction technology, image processing technology, and in particular an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) scanning equipment typically includes a lidar unit and a camera. The lidar unit performs laser scanning of the surrounding environment and generates a corresponding point cloud, while the camera acquires images of the surrounding environment to colorize the point cloud generated by the lidar unit using the color values ​​of each point in the image.

[0003] During 3D reconstruction, the camera moves with the SLAM device. As the camera moves, changes in the environment alter the white balance and exposure during image acquisition, leading to inconsistencies in color and brightness in the acquired image sequence. When using image sequences for point cloud coloring, multiple images may overlap. If these images have inconsistent colors and brightness, it will result in uneven 3D point cloud coloring and poor preview quality. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this disclosure provide an image processing method, apparatus, device, and storage medium.

[0005] One aspect of this disclosure provides an image processing method, including:

[0006] Based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence, the co-view relationship between the images is determined, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection;

[0007] An undirected graph corresponding to the image sequence is constructed using the images as nodes and the co-visibility relationships between the images as edges.

[0008] A loss function is generated for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge.

[0009] The loss function is iteratively solved to determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is minimized and the function value is less than the target value.

[0010] The image corresponding to the node is processed using the gain coefficient.

[0011] Optionally, the loss function for generating the undirected graph includes:

[0012] For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds in the image corresponding to the node.

[0013] The average color value of the pixels within the shared viewing area is determined as the color value of the node;

[0014] A loss function for the undirected graph is generated based on the color value of each node and the gain coefficient, wherein the gain coefficient is a variable to be solved in the loss function.

[0015] Optionally, before determining the average color value of the pixels within the shared viewing area as the color value of the node, the method includes:

[0016] The matching block for each pixel is determined based on a preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size.

[0017] The average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block.

[0018] Optionally, determining the average color value of the pixels within the shared viewing area as the color value of the node includes:

[0019] The first pixel in the common viewing area is determined by the interquartile range method. The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds.

[0020] Object recognition is performed on the image, and in response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined;

[0021] Remove the first pixel and the second pixel within the shared viewing area to obtain the target pixel within the shared viewing area;

[0022] The average color value of the target pixel is determined as the color value of the node.

[0023] Optionally, the loss function for generating the undirected graph based on the color value of each node and the gain coefficient includes:

[0024] The loss weight of the edge is determined based on the number of pixels in the shared viewing area corresponding to the edge, and the loss weight is proportional to the number of pixels.

[0025] The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0026] Optionally, the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference in color values ​​between the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

[0027] Optionally, the iterative solution of the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints includes:

[0028] The gradient descent method is used to iteratively solve the loss function based on the preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

[0029] Another aspect of this disclosure provides an image processing apparatus, comprising:

[0030] The first determining module is used to determine the co-view relationship between the images based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection;

[0031] The generation module is used to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-view relationships between the images as edges.

[0032] A function construction module is used to generate a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge.

[0033] The second determining module is used to iteratively solve the loss function and determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the function value being the minimum and the function value being less than the target value.

[0034] The processing module is used to perform image color processing on the image corresponding to the node using the gain coefficient.

[0035] In another aspect of this disclosure, an electronic device is provided, comprising:

[0036] memory for storing computer programs;

[0037] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the methods described above.

[0038] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described above.

[0039] In another aspect of this disclosure, a computer program is provided, including computer program instructions that, when executed by a processor, implement the method described above.

[0040] Based on the embodiments of this disclosure, the co-view relationship between images is determined based on the corresponding point cloud coordinate range between images, and an undirected graph is constructed. A corresponding loss function is generated based on the connected structure of the undirected graph. The function value corresponding to the loss function represents the total color difference of the undirected graph after color value adjustment using gain coefficients. The color values ​​of each node are constrained to be consistent through preset constraints. The gain coefficient corresponding to each node is solved by iteratively calculating the loss function to satisfy the preset constraints. The color value of the image is optimized using the gain coefficient. When environmental changes cause inconsistent color and brightness of overlapping content in multiple images, the color value of the image can be processed to make the color values ​​of images with co-view relationships similar. This solves the problems of uneven coloring and poor preview effect of 3D point clouds, avoids the problem of large differences in point color values ​​within the same point cloud area, and improves the visual coherence and realism of point cloud color values.

[0041] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0043] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0044] Figure 1 A flowchart of one embodiment of the image processing method of this disclosure;

[0045] Figure 2 A schematic diagram of an undirected graph provided for an embodiment of this disclosure;

[0046] Figure 3 A flowchart of another embodiment of the image processing method of this disclosure;

[0047] Figure 4 A flowchart of another embodiment of the image processing method of this disclosure;

[0048] Figure 5 This is a schematic diagram of the structure of one embodiment of the image processing apparatus of this disclosure;

[0049] Figure 6 This is a schematic diagram of another embodiment of the image processing apparatus of this disclosure;

[0050] Figure 7 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0051] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0052] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0053] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0055] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0056] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0057] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0058] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0059] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0061] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure. The image processing method of this embodiment can be implemented using an electronic device with image processing capabilities. For example, the electronic device may include, but is not limited to, a Simultaneous Localization and Mapping (SLAM) scanning device, a smartphone, a laptop computer, a desktop computer, etc. This disclosure uses a SLAM scanning device as an example for illustration.

[0062] like Figure 1 As shown, the method includes the following steps:

[0063] Step 101: Determine the co-view relationship between images based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence.

[0064] Among them, the point cloud coordinate ranges corresponding to images with co-view relationships overlap.

[0065] In one possible implementation, the image sequence is obtained by a camera in a SLAM scanning device acquiring images of the target scanning scene at a preset frequency. To avoid missing information, images are typically acquired at a higher frequency, resulting in overlapping content between adjacent images. For example, the left half of the first image and the right half of the second image may capture the same area of ​​the target scanning scene. By determining the point cloud coordinate range corresponding to each image, it can be determined whether there is content overlap between the images, i.e., whether there is a shared view relationship. Here, the point cloud coordinate range corresponding to an image is the coordinate range of the point cloud projected onto that image in the world coordinate system.

[0066] The point cloud coordinate range of an image can be calculated using parameters such as the coordinates of each point in the point cloud in the world coordinate system, camera pose, LiDAR-camera extrinsic parameters, and camera intrinsic parameters. The camera pose characterizes the camera's position and orientation in the world coordinate system, while the LiDAR-camera extrinsic parameters characterize the LiDAR's relative position and orientation relative to the camera in the world coordinate system. The co-viewing relationship between images is determined by whether there is an intersection between the corresponding point cloud coordinate ranges. For example, two images whose intersection point cloud size is larger than a preset point cloud size can be identified as a pair of images with a co-viewing relationship.

[0067] Step 102: Construct an undirected graph corresponding to the image sequence, using images as nodes and the common-view relationships between images as edges.

[0068] In one possible implementation, an undirected graph is constructed using images as nodes and the co-view relationships between images as edges. Each edge in this undirected graph connects the nodes corresponding to two images with a co-view relationship. Optionally, if there exists an independent node, that is, a node that is not connected to any other node by any edge, then that node can be removed from the undirected graph. The image corresponding to that node is not co-viewed with any other image, so there is no need to adjust the color value. Nodes in the undirected graph are connected to at least one of the other nodes.

[0069] Indicative Figure 2 An undirected graph is shown. For example... Figure 2 As shown, in this undirected graph, the image corresponding to node 1 has a shared view relationship with the images corresponding to nodes 0, 2, 3, and 4, while the image of node 6 only has a shared view relationship with the image of node 3.

[0070] Step 103: Generate the loss function for the undirected graph.

[0071] The loss function value is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color value includes at least one of the chromaticity value and the luminance value (for example, it can be the chromaticity value, the luminance value, or a vector composed of the chromaticity value and the luminance value). The total color difference includes the sum of the color difference corresponding to each edge, and the color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge.

[0072] In one possible implementation, the gain coefficient of each node can be set as a variable to be solved or optimized, and a loss function for the undirected graph can be constructed. The value of this loss function is used to characterize the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Thus, by constraining this function value, the value of the gain coefficient can be continuously adjusted to solve for the ideal gain coefficient corresponding to each node.

[0073] Optionally, color values ​​can be YCrCb values, i.e. YUV values, where Y is the luminance value and U and V are the chromaticity values ​​(used to describe hue and saturation).

[0074] Step 104: Iteratively solve the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints.

[0075] The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is the minimum and the function value is less than the target value.

[0076] Optionally, the constraints for solving the loss function include minimizing the function value and the function value being less than at least one of the target value. During the iterative solution of the loss function, the values ​​of the gain coefficients can be continuously changed and the corresponding function values ​​can be calculated. When the function values ​​corresponding to a certain set of gain coefficients satisfy the constraints, that set of gain coefficients is determined as the final solution result.

[0077] Step 105: Perform image color processing on the image corresponding to the node using the gain coefficient.

[0078] The gain coefficient represents the degree to which the color value needs adjustment. For example, if the gain coefficient obtained from solving node A is 0.8, the target color value of the image corresponding to node A can be calculated by multiplying the original color value of the image corresponding to node A by the gain coefficient 0.8. Based on the target color value, the image data of the image corresponding to node A can be modified to obtain the image after color processing.

[0079] Based on the embodiments of this disclosure, the co-view relationship between images is determined based on the corresponding point cloud coordinate range between images, and an undirected graph is constructed. A corresponding loss function is generated based on the connected structure of the undirected graph. The function value corresponding to the loss function represents the total color difference of the undirected graph after color value adjustment using gain coefficients. The color values ​​of each node are constrained to be consistent through preset constraints. The gain coefficient corresponding to each node is solved by iteratively calculating the loss function to satisfy the preset constraints. The color value of the image is optimized using the gain coefficient. When environmental changes cause inconsistent color and brightness of overlapping content in multiple images, the color value of the image can be processed to make the color values ​​of images with co-view relationships similar. This solves the problems of uneven coloring and poor preview effect of 3D point clouds, avoids the problem of large differences in point color values ​​within the same point cloud area, and improves the visual coherence and realism of point cloud color values.

[0080] In one possible implementation, since this embodiment optimizes the color differences of different images in a shared viewing area and determines the degree of color value adjustment, for a pair of nodes with a shared viewing relationship, the average color value of the pixels in the shared viewing area of ​​the corresponding images can be determined as the color value of that image (node). Figure 3 As shown, step 103 above may specifically include the following steps:

[0081] Step 103a: For the two nodes connected by each edge, determine the common viewing region of each node based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds onto the image corresponding to the node.

[0082] Optionally, after determining the point cloud coordinate range corresponding to each image, for each pair of images with a co-view relationship (i.e., two nodes connected by each edge), the co-view region of each image in the pair can be determined based on the projection of the point cloud intersection onto the pair of images. For the same image, the co-view region with different images may be located at different positions within the image. For example, the co-view region of image a with image b is located in the left half of image a, and the co-view region of image a with image c is located in the right half of image a.

[0083] Step 103b: Determine the average color value of the pixels within the shared viewing area as the color value of the node.

[0084] Optionally, for two nodes in a shared viewing relationship, their respective average color values ​​can be determined directly by the average color value of the pixels within the shared viewing area. The average color value is the ratio of the sum of the color values ​​of all pixels within the shared viewing area to the number of pixels.

[0085] Optionally, considering the color value error and robustness of individual pixels, a local window, i.e., a matching block, can be used to determine the color value of a pixel, thereby improving the accuracy of the image color value and gain coefficient. Prior to step 103b, this embodiment may also include the following steps:

[0086] The matching block for each pixel is determined based on the preset matching block size; the average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block.

[0087] Among them, the matching block is an image region centered on a pixel and with a size that is a preset matching block size.

[0088] For example, if the preset matching block size is 3*3, where 3 represents 3 pixels, then for any pixel in the common viewing area, the pixel and the 8 pixels around it are determined as the matching block corresponding to the pixel, and the average of the actual color values ​​of the 9 pixels in the matching block is determined as the color value of the pixel.

[0089] Therefore, for step 103b, the color value of the pixel participating in the image color value calculation is the average of the actual color values ​​of the pixels within the corresponding matching block.

[0090] In one possible implementation, to further improve the accuracy of the node's color values ​​and avoid interference from special pixels, such as Figure 4 As shown, step 103b may also include the following steps:

[0091] Step b1: Use the interquartile range method to determine the first pixel point within the common viewing area.

[0092] The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of a co-viewing pixel pair. A co-viewing pixel pair refers to a pair of pixels in the corresponding image that are the same point in the intersection of point clouds.

[0093] In this embodiment of the disclosure, the quartiles are determined by arranging the pixel color differences of all common-view pixel pairs within the common-view area in ascending order and dividing them into four equal parts. The pixel color differences at the three dividing points include the first quartile (i.e., the lower quartile, located at the 25th percentile), the second quartile (i.e., the median), and the third quartile (i.e., the upper quartile, located at the 75th percentile). Pixels with a pixel color difference greater than the upper quartile are identified as the first pixel.

[0094] Step b2: Perform object recognition on the image. In response to the existence of a target object belonging to a preset object type, determine the second pixel point corresponding to the target object.

[0095] The preset object types can include, but are not limited to, object types with low reliability of color values ​​such as glass, mirrors, screens, and transient objects. Optionally, the images corresponding to each node can be input into the object recognition model, and the object recognition model can output the various object types contained in the image and the image regions corresponding to each object type. The pixels in the image regions corresponding to the preset object types are determined as the second pixel.

[0096] It is worth mentioning that there is no strict order of execution between steps one and two.

[0097] Step b3: Remove the first and second pixels within the shared viewing area to obtain the target pixels within the shared viewing area.

[0098] Step b4: Determine the average color value of the target pixel as the color value of the node.

[0099] Filtering and removing the first pixel reduces interference from abnormal pixels in image acquisition on color value calculation. Filtering and removing the second pixel avoids interference from pixels with low color value reliability. Obtaining the color value corresponding to the node based on the average color values ​​of the retained target pixels improves the reliability of the node's color value, thereby increasing the accuracy of subsequent gain coefficient calculation.

[0100] Step 103c: Generate the loss function of the undirected graph based on the color value and gain coefficient of each node.

[0101] Here, the gain coefficient is the variable to be solved in the loss function.

[0102] By setting the gain coefficient of each node as the variable to be solved or optimized, a loss function for the undirected graph is constructed. The value of this loss function is used to characterize the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Thus, by constraining this function value, the value of the gain coefficient can be continuously adjusted to solve for the ideal gain coefficient corresponding to each node.

[0103] Optionally, the strength of different co-visibility relationships varies. For node pairs with larger co-visibility regions, the co-visibility relationship is stronger; for node pairs with smaller co-visibility regions, the co-visibility relationship is weaker, and the edge stability is lower. By reducing the influence of the color values ​​of node pairs with weaker co-visibility relationships on the function values, the obtained gain coefficients can better match the image processing requirements of the image sequence. Step 103c may include the following steps:

[0104] The loss weight of an edge is determined based on the number of pixels in the shared viewing area corresponding to that edge. A loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient. The loss weight is proportional to the number of pixels.

[0105] Optionally, a range of values ​​for the loss weight can be preset, and the value of the loss weight can be determined from this range based on the number of pixels in the shared viewing area. An illustrative expression for the loss weight is as follows:

[0106] W <i,j> =min(1.0, M) <i , j> / M mean (1)

[0107] Among them, nodes i and j have a common view relationship, denoted as an edge.<i,j> W <i,j> For the edge<i,j> The loss weight, M <i,j> For the edge<i,j> The number of pixels in the corresponding shared viewing area, M mean This represents the average number of pixels in the shared region corresponding to all edges in an undirected graph. As shown in formula (1), when the edge...<i,j> When the number of pixels in the corresponding shared region exceeds the average, the edge<i,j> The value is 1, otherwise it is the ratio of the number of pixels in the shared viewing area to the average value.

[0108] Optionally, the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference between the color values ​​of the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and the preset parameter, where the preset parameter is a positive integer.

[0109] The expression for the loss function is illustrated below:

[0110]

[0111] i, j ∈ [0, N] and i ≠ j

[0112] Where L is the function value, W <i,j> For the edge<i,j> loss weights, For the edge<i,j> The color value of node i, w i Let be the gain coefficient of node i. For the edge<i,j> The image color value of node j, w j Let be the gain coefficient of node j. Let λ be the regularization term, where λ is the regularization coefficient. This regularization term can constrain the gain coefficient w. k Not all values ​​are 0, to prevent the obtained gain coefficient from being meaningless.

[0113] Optionally, in the process of solving the above loss function to determine the gain coefficient, the gradient descent method can be used to iteratively solve the loss function based on the constraints to determine the gain coefficient of each node. The preset constraints include at least one of the following: minimizing the gain coefficient and ensuring that not all gain coefficients are zero. By constraining the gain coefficient to be minimized, the degree of color value adjustment in the image can be reduced to the greatest extent possible, preserving the original display effect of the image as much as possible while keeping the color values ​​of shared images similar.

[0114] Based on the embodiments of this disclosure, the color value of a pixel is determined by using a matching block, and pixels with large differences in color values ​​and low reliability of color values ​​in the shared viewing area are filtered out. Then, the image color value corresponding to the node is determined based on the mean of the color values ​​of the pixels in the shared viewing area. This can reduce the impact of errors on the color value of the node, improve robustness, and thus improve the accuracy of the gain coefficient. The loss weight of the edge is determined based on the size of the shared viewing area corresponding to the edge, which can reduce the influence of edges with weak shared viewing relationships on the function value and improve the reliability of the gain coefficient.

[0115] Figure 5 A structural block diagram of an image processing apparatus provided in an exemplary embodiment of the present disclosure is shown. The image processing apparatus includes:

[0116] The first determining module 501 is used to determine the co-view relationship between images based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence, wherein the point cloud coordinate ranges corresponding to images with co-view relationship have an intersection.

[0117] Construction module 502 is used to construct an undirected graph corresponding to the image sequence, with images as nodes and the co-view relationship between images determined by the first determining module 501 as edges;

[0118] The generation module 503 is used to generate a loss function for the undirected graph constructed by the construction module 502. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color value includes at least one of chromaticity value and luminance value. The total color difference includes the sum of the color difference corresponding to each edge. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge.

[0119] The second determining module 504 is used to iteratively solve the loss function generated by the generating module 503 and determine the gain coefficient of each node when the function value meets the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is the minimum and the function value is less than the target value.

[0120] The processing module 505 is used to perform image color processing on the image corresponding to the node using the gain coefficient determined by the second determining module 504.

[0121] Optionally, in one possible implementation, the generation module 503 described above can also be used for:

[0122] For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds onto the image corresponding to the node.

[0123] The average color value of the pixels within the shared viewing area is determined as the color value of the node;

[0124] A loss function for an undirected graph is generated based on the color value and gain coefficient of each node, where the gain coefficient is the variable to be solved in the loss function.

[0125] Optionally, in one possible implementation, such as Figure 6 As shown, the image processing apparatus also includes:

[0126] The third determining module 601 is used to determine the matching block of each pixel based on the preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size.

[0127] The fourth determining module 602 is used to determine the average color value of the pixels within the matching block as the color value of the center pixel of the matching block.

[0128] Optionally, in one possible implementation, the generation module 503 described above can also be used for:

[0129] The first pixel within the common viewing area is determined using the interquartile range method. This first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds.

[0130] Object recognition is performed on the image. In response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined.

[0131] Remove the first and second pixels within the shared viewing area to obtain the target pixels within the shared viewing area;

[0132] The average color value of the target pixel is used as the color value of the node.

[0133] Optionally, in one possible implementation, the generation module 503 described above can also be used for:

[0134] The loss weight of an edge is determined based on the number of pixels in the shared viewing area corresponding to the edge, and this loss weight is proportional to the number of pixels.

[0135] The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0136] Optionally, in one possible implementation, the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference between the color values ​​of the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

[0137] Optionally, in one possible implementation, the second determining module 504 may also be used for:

[0138] The gradient descent method is used to iteratively solve the loss function based on preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar, identical, or corresponding parts between embodiments can be referred to mutually. Since the method, apparatus, system, and device embodiments are basically corresponding, relevant parts can be referred to the corresponding descriptions. The methods, apparatus, systems, and devices in the embodiments of this disclosure also correspond to each other in specific implementation and beneficial technical effects; related content can be referred to mutually and will not be repeated here.

[0140] In addition, this disclosure also provides an electronic device, including:

[0141] memory for storing computer programs;

[0142] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the image processing method described in any of the above embodiments of the present disclosure.

[0143] Figure 7 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 7 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0144] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0145] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0146] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the image processing methods of the various embodiments of this disclosure described above and / or other desired functions.

[0147] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0148] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0149] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0150] Of course, to simplify, Figure 7 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0151] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0152] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0154] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0156] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0158] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0159] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0160] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0161] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0162] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, characterized in that, include: Based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence, the co-view relationship between the images is determined, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection; An undirected graph corresponding to the image sequence is constructed using the images as nodes and the co-visibility relationships between the images as edges. A loss function is generated for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge. The loss function is iteratively solved to determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the following: the function value is minimized and the function value is less than the target value. The image corresponding to the node is processed using the gain coefficient.

2. The method according to claim 1, characterized in that, The loss function for generating the undirected graph includes: For each edge connecting two nodes, the common viewing region of each node is determined based on the intersection of the point clouds. The common viewing region is the projection region of the intersection of the point clouds in the image corresponding to the node. The average color value of the pixels within the shared viewing area is determined as the color value of the node; A loss function for the undirected graph is generated based on the color value of each node and the gain coefficient, wherein the gain coefficient is a variable to be solved in the loss function.

3. The method according to claim 2, characterized in that, Before determining the average color value of the pixels within the shared viewing area as the color value of the node, the method includes: The matching block for each pixel is determined based on a preset matching block size. The matching block is an image region centered on the pixel and with a size equal to the preset matching block size. The average color value of the pixels within the matching block is determined as the color value of the center pixel of the matching block.

4. The method according to claim 3, characterized in that, Determining the average color value of the pixels within the shared viewing area as the color value of the node includes: The first pixel in the common viewing area is determined by the interquartile range method. The first pixel includes pixels whose pixel color difference is greater than the upper quartile. The pixel color difference is the difference in color values ​​of the common viewing pixel pair. The common viewing pixel pair is a pair of pixels in the corresponding image that share the same point in the intersection of the point clouds. Object recognition is performed on the image, and in response to the existence of a target object belonging to a preset object type, the second pixel point corresponding to the target object is determined; Remove the first pixel and the second pixel within the shared viewing area to obtain the target pixel within the shared viewing area; The average color value of the target pixel is determined as the color value of the node.

5. The method according to any one of claims 2 to 4, characterized in that, The loss function for generating the undirected graph based on the color value of each node and the gain coefficient includes: The loss weight of the edge is determined based on the number of pixels in the shared viewing area corresponding to the edge, and the loss weight is proportional to the number of pixels. The loss function is constructed based on the loss weight of each edge, the color value of each node, and the gain coefficient.

6. The method according to claim 5, characterized in that, The loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference in color values ​​between the two nodes corresponding to the edge after color value adjustment based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regular distances. The sum of the regular distances is the sum of the regular distances corresponding to all nodes in the undirected graph. The regular distance is the distance between the gain coefficient and a preset parameter, where the preset parameter is a positive integer.

7. The method according to any one of claims 1 to 4, characterized in that, The iterative solution of the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraints includes: The gradient descent method is used to iteratively solve the loss function based on the preset constraints to determine the gain coefficient of each node. The preset constraints also include at least one of the following: the gain coefficient is minimized and the gain coefficient is not all zero.

8. An image processing apparatus, characterized in that, include: The first determining module is used to determine the co-view relationship between the images based on the point cloud coordinate range corresponding to each image in the world coordinate system of the image sequence, wherein the point cloud coordinate ranges corresponding to the images with co-view relationship have an intersection; The construction module is used to construct an undirected graph corresponding to the image sequence, using the images as nodes and the co-view relationships between the images as edges. A generation module is used to generate a loss function for the undirected graph. The function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values ​​of the nodes. The color values ​​include at least one of chromaticity value and luminance value. The total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to each edge is the difference between the color values ​​of the two nodes connected by the edge. The second determining module is used to iteratively solve the loss function and determine the gain coefficient of each node when the function value satisfies the preset constraints. The gain coefficient is used to characterize the degree of adjustment of the color value. The preset constraints include at least one of the function value being the minimum and the function value being less than the target value. The processing module is used to perform image color processing on the image corresponding to the node using the gain coefficient.

9. An electronic device, characterized in that, include: memory for storing computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images