Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and the method comprises the steps: determining a common-view relation between images based on a corresponding point cloud coordinate range between the images, constructing an undirected graph, and generating a corresponding loss function based on a communication structure of the undirected graph. The function value corresponding to the loss function represents a total color difference value of the undirected graph after color value adjustment is carried out by utilizing the gain coefficient, and the color values of the nodes are restrained to tend to be consistent through a preset constraint condition, and the gain coefficient corresponding to each node when the function value meets the preset constraint condition is solved by carrying out iterative calculation on the loss function; according to the invention, the gain coefficient is used to optimize the color value of the image, so that the color values of the images with the common-view relationship are similar, the problems of uneven coloring and poor preview effect of the three-dimensional point cloud are solved, the problem of large difference of the color values of the points in the same point cloud area is avoided, and the visual coherence and authenticity of the color values of the point cloud are improved.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology and image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Art

[0002] In a laser Simultaneous Localization And Mapping (SLAM) scanning device, a lidar and a camera are usually mounted. Among them, the lidar is used to perform laser scanning on the surrounding environment and generate corresponding point clouds, and the camera is used to collect images of the surrounding environment, so as to color the point clouds generated by the lidar using the color values of each point in the images.

[0003] During the three-dimensional reconstruction process, the camera moves with the SLAM device. During the movement of the camera, due to environmental changes, the white balance and exposure during image acquisition by the camera also change, resulting in the problem that the acquired image sequence shows inconsistent colors and brightness. When using the image sequence for point cloud coloring, there may be a situation where the content captured in multiple images overlaps. If the colors and brightness of these multiple images are inconsistent, it will cause problems such as uneven coloring of the three-dimensional point cloud and poor preview effect. Summary of the Invention

[0004] In view of the above technical problems, embodiments of the present disclosure provide an image processing method, apparatus, device, and storage medium.

[0005] One aspect of the embodiments of the present disclosure provides an image processing method, including:

[0006] Determine the co-visibility relationship between the images based on the point cloud coordinate ranges corresponding to each image in the image sequence in the world coordinate system, where the point cloud coordinate ranges corresponding to the images with the co-visibility relationship have an intersection;

[0007] Construct an undirected graph corresponding to the image sequence with the images as nodes and the co-visibility relationship between the images as edges;

[0008] Generate a loss function of the undirected graph, where the function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values of the nodes. The color values include at least one of chromaticity values and brightness values, and the total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to an edge is the difference between the color values of the two nodes connected by the edge;

[0009] Iteratively solve the loss function to determine the gain coefficient of each node when the function value satisfies a preset constraint condition. The gain coefficient is used to characterize the degree to which the color value needs to be adjusted, and the preset constraint condition includes at least one of the function value being minimized and the function value being less than a target value;

[0010] Use the gain coefficient to perform image color processing on the image corresponding to the node.

[0011] Optionally, the loss function for generating the undirected graph includes:

[0012] For each pair of nodes connected by an edge, determine the common viewing area of each node based on the point cloud intersection. The common viewing area is the projection area of the point cloud intersection in the image corresponding to the node;

[0013] Determine the color value of the node as the mean of the color values of the pixel points in the common viewing area;

[0014] Generate the loss function of the undirected graph based on the color value of each node and the gain coefficient, where the gain coefficient is the variable to be solved in the loss function.

[0015] Optionally, before determining the color value of the node as the mean of the color values of the pixel points in the common viewing area, the method includes:

[0016] Determine the matching block of each pixel point based on a preset matching block size. The matching block is an image area centered on the pixel point with a size of the preset matching block size;

[0017] Determine the color value of the central pixel point of the matching block as the mean of the color values of the pixel points in the matching block.

[0018] Optionally, determining the color value of the node as the mean of the color values of the pixel points in the common viewing area includes:

[0019] Use the interquartile range method to determine the first pixel points in the common viewing area. The first pixel points include pixel points with a pixel color difference greater than the upper quartile. The pixel color difference is the difference between the color values of a common viewing pixel pair, and the common viewing pixel pair is a pair of pixel points in the corresponding images of the same point in the point cloud intersection;

[0020] Perform object recognition on the image. In response to the presence of a target object belonging to a preset object type, determine the second pixel points corresponding to the target object;

[0021] Exclude the first pixel points and the second pixel points in the common viewing area to obtain the target pixel points in the common viewing area;

[0022] Determine the mean value of the color values of the target pixel points as the color value of the node.

[0023] Optionally, generating the loss function of the undirected graph based on the color value of each node and the gain coefficient includes:

[0024] Determine the loss weight of the edge based on the number of pixel points in the co-visible area corresponding to the edge, and the loss weight is proportional to the number of pixel points;

[0025] Construct the loss function based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0026] Optionally, the function value of the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference between the color values of the two nodes corresponding to the edge after adjusting the color values based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regularization distances. The sum of the regularization distances is the sum of the regularization distances corresponding to all nodes in the undirected graph. The regularization distance is the distance between the gain coefficient and the preset parameter, and the preset parameter is a positive integer.

[0027] Optionally, iteratively solving the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraint conditions includes:

[0028] Using the gradient descent method, iteratively solve the loss function based on the preset constraint conditions to determine the gain coefficient of each node, where the preset constraint conditions further include at least one of the minimum gain coefficient and not all gain coefficients being 0.

[0029] Another aspect of the embodiments of the present disclosure provides an image processing apparatus, including:

[0030] A first determination module, configured to determine the co-visible relationship between the images based on the point cloud coordinate ranges corresponding to each image in the image sequence in the world coordinate system, where there is an intersection in the point cloud coordinate ranges corresponding to the images with the co-visible relationship;

[0031] A generation module, configured to construct an undirected graph corresponding to the image sequence with the images as nodes and the co-visible relationship between the images as edges;

[0032] A function construction module for generating a loss function of the undirected graph, where the function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values of the nodes. Among them, the color values include at least one of chromaticity values and brightness values, and the total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to the edge is the difference between the color values of the two nodes connected by the edge;

[0033] A second determination module for iteratively solving the loss function to determine the gain coefficient of each node when the function value satisfies a preset constraint condition. The gain coefficient is used to characterize the degree of adjustment to be made to the color value, and the preset constraint condition includes at least one of the function value being the smallest and the function value being less than a target value;

[0034] A processing module for performing image color processing on the image corresponding to the node by using the gain coefficient.

[0035] On the other hand, an embodiment of the present disclosure provides an electronic device, including:

[0036] A memory for storing a computer program;

[0037] A processor for executing the computer program stored in the memory, and when the computer program is executed, the method described in the above aspect is implemented.

[0038] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above aspect is implemented.

[0039] On the other hand, an embodiment of the present disclosure provides a computer program, including computer program instructions. When the computer program instructions are executed by a processor, the method described in the above aspect is implemented.

[0040] Based on the embodiments of the present disclosure, the co-visibility relationship between images is determined based on the corresponding point cloud coordinate ranges between the images, and an undirected graph is constructed. A corresponding loss function is generated based on the connected structure of the undirected graph. The function value corresponding to the loss function characterizes the total color difference of the undirected graph after adjusting the color values by using the gain coefficient. The color values of each node are constrained to be consistent through preset constraint conditions. By iteratively calculating the loss function to solve the gain coefficient corresponding to each node when the function value satisfies the preset constraint conditions, and using this gain coefficient to optimize the color values of the images. In the case where environmental changes cause the colors and brightness of the overlapping contents in multiple images to be inconsistent, the color values of the images can be processed so that the color values of the images with co-visibility relationships are similar, solving the problems of uneven coloring of the three-dimensional point cloud and poor preview effect, avoiding the problem of large differences in the color values of points in the same point cloud area, and improving the visual coherence and authenticity of the point cloud color values.

[0041] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0042] The drawings forming a part of the specification depict embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0043] With reference to the accompanying drawings, the present disclosure can be more clearly understood according to the following detailed description, where:

[0044] Figure 1 is a flowchart of an embodiment of the image processing method of the present disclosure;

[0045] Figure 2 is a schematic diagram of an undirected graph provided by an embodiment of the present disclosure;

[0046] Figure 3 is a flowchart of another embodiment of the image processing method of the present disclosure;

[0047] Figure 4 is a flowchart of another embodiment of the image processing method of the present disclosure;

[0048] Figure 5 is a schematic structural diagram of an embodiment of the image processing apparatus of the present disclosure;

[0049] Figure 6 is a schematic structural diagram of another embodiment of the image processing apparatus of the present disclosure;

[0050] Figure 7 is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. Detailed Embodiments

[0051] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0052] Those skilled in the art can understand that the terms "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.

[0053] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two, or more.

[0054] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, in the absence of clear limitations or contrary implications in the context, it can generally be understood as one or more.

[0055] In addition, the term "and / or" in the present disclosure is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the associated objects before and after.

[0056] It should also be understood that the descriptions of the various embodiments in the present disclosure emphasize the differences between the various embodiments, and their similarities can be referred to each other. For the sake of brevity, they will not be elaborated one by one.

[0057] At the same time, it should be understood that for the sake of description, the dimensions of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0058] The following description of at least one exemplary embodiment is actually merely illustrative and in no way a limitation on the present disclosure and its application or use.

[0059] The techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the said techniques, methods, and devices should be regarded as part of the specification.

[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0061] Figure 1 The flowchart of the image processing method provided for an exemplary embodiment of the present disclosure. The image processing method of the embodiments of the present disclosure can be implemented by an electronic device with image processing capabilities. For example, the electronic device may include, but is not limited to, a laser Simultaneous Localization And Mapping (SLAM) scanning device, a smart phone, a laptop computer, a desktop computer, and other devices. The present disclosure will be described by taking the SLAM scanning device as an example.

[0062] As Figure 1 shown, the method includes the following steps:

[0063] Step 101, determine the co-visibility relationship between images based on the point cloud coordinate range corresponding to each image in the image sequence in the world coordinate system.

[0064] Among them, there is an intersection in the point cloud coordinate ranges corresponding to the images with a co-visibility relationship.

[0065] In a possible implementation, the image sequence is obtained by a camera in a SLAM scanning device collecting images of a target scanning scene at a preset frequency. To avoid missing information collection, image collection is usually performed at a relatively high frequency, so that there is content overlap between two adjacent or multiple images. For example, the left half of the first image and the right half of the second image capture the same area of the target scanning scene. By determining the point cloud coordinate range corresponding to each image, it can be determined whether there is content overlap between the images, that is, whether there is a co-visibility relationship. Among them, the point cloud coordinate range corresponding to an image is the coordinate range of the point cloud projected within the image in the world coordinate system.

[0066] For the point cloud coordinate range of an image, it can be calculated using parameters such as the coordinates of each point in the point cloud in the world coordinate system, the camera pose, the lidar-camera extrinsic parameters, and the camera intrinsic parameters. Among them, the camera pose is used to represent the position and orientation of the camera in the world coordinate system, and the lidar-camera extrinsic parameters are used to represent the relative position and relative orientation of the lidar in the world coordinate system relative to the camera. Based on whether there is an intersection between the point cloud coordinate ranges corresponding to the images, the co-visibility relationship between the images is determined. For example, two images with the point cloud size of the intersection greater than the preset point cloud size can be determined as a pair of images with a co-visibility relationship.

[0067] Step 102: Construct an undirected graph corresponding to the image sequence with the images as nodes and the co-visibility relationship between the images as edges.

[0068] In a possible implementation, an undirected graph is constructed with the images as nodes and the co-visibility relationship between the images as edges. Each edge in this undirected graph connects the nodes corresponding to two images with a co-visibility relationship. Optionally, if there is an isolated node, that is, a node that is not connected to any other node by an edge, then this node can be removed from the undirected graph. The image corresponding to this node is not co-visible with any other image, so there is no need to adjust the color value, and the nodes in the undirected graph are connected to at least one of the other nodes.

[0069] Schematically, Figure 2 shows an undirected graph. As Figure 2 shown, in this undirected graph, there is a co-visibility relationship between the image corresponding to node 1 and the images corresponding to nodes 0, 2, 3, and 4, and the image of node 6 has a co-visibility relationship only with the image of node 3.

[0070] Step 103: Generate a loss function for the undirected graph.

[0071] Among them, the function value corresponding to the loss function is used to represent the total color difference of the undirected graph after adjusting the color value of the node. Among them, the color value includes at least one of the chromaticity value and the brightness value (for example, it can be the chromaticity value, or it can be the brightness value, or it can be a vector composed of the chromaticity value and the brightness value). The total color difference includes the sum of the color differences corresponding to each edge, and the color difference corresponding to the edge is the difference between the color values of the two nodes connected by the edge.

[0072] In a possible implementation manner, the gain coefficient of each node can be set as a variable to be solved or optimized, and the loss function of the undirected graph is constructed. The function value of this loss function is used to represent the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Therefore, by constraining this function value, the numerical value of the gain coefficient can be continuously adjusted to solve the ideal gain coefficient corresponding to each node.

[0073] Optionally, the color value can adopt the YCrCb value, that is, the YUV value, where Y is the brightness value, and U and V are the chromaticity values (used to describe hue and saturation).

[0074] Step 104: Iteratively solve the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraint condition.

[0075] Among them, the gain coefficient is used to represent the degree to which the color value is to be adjusted, and the preset constraint condition includes at least one of the function value being the smallest and the function value being less than the target value.

[0076] Optionally, the constraint conditions for solving the loss function include at least one of the function value being the smallest and the function value being less than the target value. During the iterative solution process of the loss function, the numerical value of the gain coefficient can be continuously changed and the corresponding function value can be calculated. When the function value corresponding to a set of gain coefficients satisfies the constraint condition, this set of gain coefficients is determined as the final solution result.

[0077] Step 105: Perform image color processing on the image corresponding to the node by using the gain coefficient.

[0078] The gain coefficient represents the degree to which the color value is to be adjusted. For example, if the gain coefficient obtained by solving for node A is 0.8, then the target color value of the image corresponding to node A can be calculated by multiplying the original color value of the image corresponding to node A by this gain coefficient 0.8, and the image data of the image corresponding to node A can be modified based on this target color value to obtain the image after image color processing.

[0079] Based on the embodiments of the present disclosure, the co - visibility relationship between images is determined based on the corresponding point - cloud coordinate ranges between the images, and an undirected graph is constructed. A corresponding loss function is generated based on the connected structure of the undirected graph. The function value corresponding to the loss function represents the total color difference of the undirected graph after adjusting the color values using the gain coefficient. By preset constraints, the color values of each node are constrained to be consistent. By iteratively calculating the loss function to solve for the gain coefficient corresponding to each node when the function value satisfies the preset constraints, the color values of the images are optimized using the gain coefficient. In the case where environmental changes cause inconsistent colors and brightness of overlapping content in multiple images, the color values of the images can be processed so that the color values of images with co - visibility relationships are similar, solving the problems of uneven coloring of the 3D point cloud and poor preview effects, avoiding the problem of large differences in the color values of points within the same point - cloud region, and improving the visual coherence and authenticity of the point - cloud color values.

[0080] In a possible implementation manner, since the embodiments of the present disclosure optimize the color differences in the co - visibility regions of different images to determine the degree of adjustment of the color values, for a pair of nodes with a co - visibility relationship, the mean value of the color values of the pixel points in the co - visibility region of the corresponding images can be determined as the color value of the image (node). As Figure 3 shown, step 103 may specifically include the following steps:

[0081] Step 103a, for each pair of nodes connected by an edge, based on the point - cloud intersection, determine the co - visibility region of each node. The co - visibility region is the projection region of the point - cloud intersection in the image corresponding to the node.

[0082] Optionally, after determining the point - cloud coordinate range corresponding to each image, for each pair of images with a co - visibility relationship (i.e., each pair of nodes connected by an edge), the co - visibility region of each image in the pair can be determined respectively based on the projection of the point - cloud intersection in the pair of images. Among them, for the same image, the co - visibility regions between it and different images may be located at different positions in the image. For example, the co - visibility region between image a and image b in image a is located in the left - hand part of image a, and the co - visibility region between image a and image c in image a is located in the right - hand part of image a.

[0083] Step 103b, determine the mean value of the color values of the pixel points in the co - visibility region as the color value of the node.

[0084] Optionally, for two nodes in a co - visibility relationship, the mean value of the color values of the pixel points in their respective co - visibility regions in the co - visibility relationship can be directly determined as their corresponding color values. The mean value of the color values is the ratio of the sum of the color values of all pixel points in the co - visibility region to the number of pixel points.

[0085] Optionally, considering the color value error and robustness of a single pixel, a local window, i.e., a matching block, can be used to determine the color value of the pixel, thereby improving the accuracy of the image color value and the gain coefficient. Before step 103b, the embodiments of the present disclosure may further include the following steps:

[0086] Determine the matching block for each pixel based on a preset matching block size; determine the mean value of the color values of the pixels within the matching block as the color value of the central pixel of the matching block.

[0087] Among them, the matching block is an image area centered on a pixel with a size of the preset matching block size.

[0088] For example, if the preset matching block size is 3*3, where 3 represents 3 pixels, then for any pixel in the co-visible area, the pixel and its surrounding 8 pixels are determined as the matching block corresponding to the pixel, and the mean value of the actual color values of the 9 pixels within the matching block is determined as the color value of the pixel.

[0089] Therefore, for step 103b, the color value of the pixel participating in the calculation of the image color value is the mean value of the actual color values of the pixels within the matching block corresponding to the pixel.

[0090] In a possible implementation manner, in order to further improve the accuracy of the color value of the node and avoid the interference of special pixels, as Figure 4 shown, step 103b may further include the following steps:

[0091] Step b1, using the interquartile range method, determine the first pixel in the co-visible area.

[0092] The first pixel includes pixels with a pixel color difference greater than the upper quartile. The pixel color difference is the difference between the color values of the co-visible pixel pair. The co-visible pixel pair refers to a pair of pixels in the corresponding images of the same point in the point cloud intersection.

[0093] In the embodiments of the present disclosure, the quartiles are obtained by arranging all the pixel color differences of the co-visible pixel pairs in the co-visible area in ascending order and dividing them into four equal parts. The pixel color differences at the three segmentation points include the first quartile (i.e., the lower quartile, located at the 25% position), the second quartile (i.e., the median), and the third quartile (i.e., the upper quartile, located at the 75% position). The pixels with a pixel color difference greater than the upper quartile are determined as the first pixels.

[0094] Step b2, perform object recognition on the image. In response to the presence of a target object belonging to a preset object type, determine the second pixel corresponding to the target object.

[0095] Among them, the preset object types may include, but are not limited to, object types with relatively low reliability of color values, such as glass, mirrors, screens, transient objects, etc. Optionally, the images corresponding to each node can be input into an object recognition model. The object recognition model outputs various object types included in the image and the image regions corresponding to each object type, and the pixel points within the image regions corresponding to the preset object types are determined as the second pixel points.

[0096] It is worth mentioning that there is no strict order of execution between Step 1 and Step 2.

[0097] Step b3: Remove the first pixel points and the second pixel points within the co-visible region to obtain the target pixel points within the co-visible region.

[0098] Step b4: Determine the mean value of the color values of the target pixel points as the color value of the node.

[0099] Filtering and removing the first pixel points can reduce the interference of abnormal pixel points in image acquisition on color value calculation. Filtering and removing the second pixel points can avoid the interference of pixel points with relatively low color value reliability on color value calculation. Obtaining the color value corresponding to the node based on the mean value of the color values of the retained target pixel points can improve the reliability of the color value corresponding to the node, thereby improving the accuracy of subsequent gain coefficient solution.

[0100] Step 103c: Generate a loss function of the undirected graph based on the color value and gain coefficient of each node.

[0101] Among them, the gain coefficient is the variable to be solved in the loss function.

[0102] Set the gain coefficient of each node as the variable to be solved or optimized, and construct a loss function of the undirected graph. The function value of this loss function is used to characterize the total color difference in the undirected graph after adjusting the color value of the corresponding node based on the gain coefficient of each node. Thus, by constraining this function value, the numerical value of the gain coefficient can be continuously adjusted to solve the ideal gain coefficient corresponding to each node.

[0103] Optionally, the strengths of different co-visible relationships are different. For node pairs with a larger co-visible region area, their co-visible relationship is stronger. For node pairs with a smaller co-visible region area, their co-visible relationship is weaker and the stability of the edge is lower. By reducing the influence degree of the color values of node pairs with a weaker co-visible relationship on the function value, the obtained gain coefficient can be made more suitable for the image processing requirements of this image sequence. Step 103c may include the following steps:

[0104] Determine the loss weight of the edge based on the number of pixel points in the co-visible region corresponding to the edge, and construct a loss function based on the loss weight of each edge, the color value and gain coefficient of each node. Among them, the loss weight is proportional to the number of pixel points.

[0105] Optionally, the value range of the loss weight can be preset, and the value of the loss weight is determined from this value range based on the number of pixel points in the co-visibility area. Schematically, the expression of the loss weight is as follows:

[0106] W <i,j> = min(1.0, M <i , j> / M mean ) (1)

[0107] Among them, there is a co-visibility relationship between nodes i and j, denoted as edge <i, j>, and W <i,j> is the loss weight of edge <i, j>, and M <i,j> is the number of pixel points in the co-visibility area corresponding to edge <i, j>, and M mean is the average value of the number of pixel points in the co-visibility areas corresponding to all edges in the undirected graph. As shown in formula (1), when the number of pixel points in the co-visibility area corresponding to edge <i, j> exceeds the average value, the value of edge <i, j> is 1, otherwise the value is the ratio of the number of pixel points in the co-visibility area to the average value.

[0108] Optionally, the function value in the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference between the color values of the two nodes corresponding to the edge after adjusting the color value based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regularization distances. The sum of the regularization distances is the total sum of the regularization distances corresponding to all nodes in the undirected graph. The regularization distance is the distance between the gain coefficient and the preset parameter, where the preset parameter is a positive integer.

[0109] Schematically, the expression of the loss function is as follows:

[0110]

[0111] i, j ∈ [0, N] and i ≠ j

[0112] Among them, L is the function value, W <i,j> is the loss weight of edge <i, j>, is the color value of node i in edge <i, j>, w i is the gain coefficient of node i, is the image color value of node j in edge <i, j>, w j is the gain coefficient of node j, is the regularization term, where λ is the regularization coefficient, and this regularization term can constrain that the gain coefficients w k are not all 0 to prevent the obtained gain coefficients from being meaningless.

[0113] Optionally, in the process of solving the above loss function to determine the gain coefficient, the gradient descent method can be used to iteratively solve the loss function based on the constraint conditions to determine the gain coefficient of each node. The preset constraint conditions further include at least one of the minimum gain coefficient and not all gain coefficients being 0. By constraining the minimum gain coefficient, the degree of adjustment of the color values of the image can be minimized, and the original display effect of the image can be retained as much as possible while making the color values of the co-view images similar.

[0114] Based on the embodiments of the present disclosure, the color values of pixel points are determined using the matching blocks, and the pixel points with large differences in color values and low reliability of color values in the co-view area are filtered. Then, based on the mean value of the color values of the pixel points in the co-view area, the image color value corresponding to the node is determined, which can reduce the influence of errors on the color values of the nodes, improve the robustness, and further improve the accuracy of the gain coefficient; based on the size of the co-view area corresponding to the edge, the loss weight of the edge is determined, which can reduce the influence degree of the edges with weak co-view relationships on the function value and improve the reliability of the gain coefficient.

[0115] Figure 5 The block diagram of an image processing apparatus provided by an exemplary embodiment of the present disclosure is shown. The image processing apparatus includes:

[0116] A first determination module 501, configured to determine the co-view relationship between images based on the point cloud coordinate ranges corresponding to each image in the image sequence in the world coordinate system, where there is an intersection in the point cloud coordinate ranges corresponding to the images with the co-view relationship;

[0117] A construction module 502, configured to construct an undirected graph corresponding to the image sequence with the images as nodes and the co-view relationship between the images determined by the first determination module 501 as edges;

[0118] A generation module 503, configured to generate a loss function of the undirected graph constructed by the construction module 502, and the function value of the loss function is used to characterize the total color difference of the undirected graph after adjusting the color values of the nodes. The color values include at least one of the chromaticity value and the luminance value, and the total color difference includes the sum of the color differences corresponding to each edge. The color difference corresponding to the edge is the difference between the color values of the two nodes connected by the edge;

[0119] A second determination module 504, configured to iteratively solve the loss function generated by the generation module 503 to determine the gain coefficient of each node when the function value satisfies the preset constraint conditions. The gain coefficient is used to characterize the degree of adjustment of the color value, and the preset constraint conditions include at least one of the minimum function value and the function value being less than the target value;

[0120] A processing module 505, configured to perform image color processing on the image corresponding to the node using the gain coefficient determined by the second determination module 504.

[0121] Optionally, in a possible implementation, the above-mentioned generation module 503 can also be used to:

[0122] For each pair of nodes connected by an edge, determine the co-visibility region of each node based on the point cloud intersection, where the co-visibility region is the projection region of the point cloud intersection in the image corresponding to the node;

[0123] Determine the average value of the color values of the pixel points in the co-visibility region as the color value of the node;

[0124] Generate a loss function of the undirected graph based on the color value and gain coefficient of each node, where the gain coefficient is a variable to be solved in the loss function.

[0125] Optionally, in a possible implementation, as Figure 6 shown, the image processing device further includes:

[0126] A third determination module 601, configured to determine a matching block for each pixel point based on a preset matching block size, where the matching block is an image region centered on the pixel point and with a size of the preset matching block size;

[0127] A fourth determination module 602, configured to determine the average value of the color values of the pixel points in the matching block as the color value of the pixel point at the center of the matching block.

[0128] Optionally, in a possible implementation, the above-mentioned generation module 503 can also be used to:

[0129] Adopt the interquartile range method to determine the first pixel points in the co-visibility region, where the first pixel points include pixel points with a pixel color difference greater than the upper quartile, and the pixel color difference is the difference between the color values of a co-visibility pixel pair, and the co-visibility pixel pair is a pair of pixel points in the corresponding images of the same point in the point cloud intersection;

[0130] Perform object recognition on the image, and in response to the presence of a target object belonging to a preset object type, determine the second pixel points corresponding to the target object;

[0131] Exclude the first pixel points and the second pixel points in the co-visibility region to obtain the target pixel points in the co-visibility region;

[0132] Determine the average value of the color values of the target pixel points as the color value of the node.

[0133] Optionally, in a possible implementation, the above-mentioned generation module 503 can also be used to:

[0134] Determine the loss weight of the edge based on the number of pixel points in the co-visibility region corresponding to the edge, and the loss weight is proportional to the number of pixel points;

[0135] Construct a loss function based on the loss weight of each edge, the color value of each node, and the gain coefficient.

[0136] Optionally, in a possible implementation, the function value in the loss function is the sum of the total color difference and the regularization term. The total color difference is the weighted sum of the squares of the color differences corresponding to each edge. The color difference is the difference between the color values of the two nodes corresponding to the edge after adjusting the color value based on the gain coefficient. The regularization term is the product of the regularization coefficient and the sum of the regularization distances. The sum of the regularization distances is the total sum of the regularization distances corresponding to all nodes in the undirected graph. The regularization distance is the distance between the gain coefficient and the preset parameter, and the preset parameter is a positive integer.

[0137] Optionally, in a possible implementation, the second determination module 504 can also be used for:

[0138] Use the gradient descent method to iteratively solve the loss function based on the preset constraint conditions to determine the gain coefficient of each node, where the preset constraint conditions also include at least one of the minimum gain coefficient and not all gain coefficients being 0.

[0139] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same, similar or corresponding parts among the embodiments, reference can be made to each other. Since the method, apparatus, system, and device embodiments basically correspond, reference can be made to the corresponding parts of the description for the relevant parts. The methods, apparatuses, systems, and devices of the embodiments of the present disclosure also correspond to each other in terms of specific implementation manners and beneficial technical effects. The relevant content can be referred to each other and will not be repeated.

[0140] In addition, the embodiments of the present disclosure also provide an electronic device, including:

[0141] A memory for storing a computer program;

[0142] A processor for executing the computer program stored in the memory, and when the computer program is executed, implementing the image processing method described in any one of the above embodiments of the present disclosure.

[0143] Figure 7 This is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. Next, refer to Figure 7 to describe the electronic device according to the embodiments of the present disclosure. The electronic device can be any one or both of the first device and the second device, or a stand-alone device independent of them. The stand-alone device can communicate with the first device and the second device to receive the input signals collected from them.

[0144] As Figure 7 shown, the electronic device includes one or more processors and a memory.

[0145] The processor can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0146] The memory can include one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage media, and the processor can run the program instructions to implement the image processing methods of various embodiments of the present disclosure described above and / or other desired functions.

[0147] In one example, the electronic device can further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0148] In addition, the input device can further include, for example, a keyboard, a mouse, and so on.

[0149] The output device can output various information to the outside, including the determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0150] Of course, for simplicity, Figure 7 only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device can further include any other appropriate components.

[0151] In addition to the above methods and devices, the embodiments of the present disclosure can also be a computer program product, which includes computer program instructions, and the computer program instructions, when run by the processor, cause the processor to execute the steps in the image processing methods according to various embodiments of the present disclosure described in the above part of this specification.

[0152] The computer program product may be written in any combination of one or more programming languages for executing the program code of the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium storing computer program instructions, which when run by a processor cause the processor to execute the steps in the image processing method according to various embodiments of the present disclosure described in the foregoing part of this specification.

[0154] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program may be stored in a computer-readable storage medium, and when executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other various media that can store program code.

[0156] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purposes of illustration and facilitating understanding, rather than limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0157] In the present specification, each embodiment is described in a progressive manner. The key point of each embodiment is the difference from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment.

[0158] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended words, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with each other, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with each other.

[0159] The methods and apparatuses of the present disclosure can be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is only for illustration. The steps of the method of the present disclosure are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers the recording medium storing the program for executing the method according to the present disclosure.

[0160] It should also be noted that in the devices, equipment, and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0161] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be very obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0162] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those of skill in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. An image processing method, characterized in that: include: Determining a common-view relationship between the images based on a point cloud coordinate range corresponding to each image of the image sequence in a world coordinate system, wherein the point cloud coordinate ranges corresponding to the images having the common-view relationship have an intersection; Constructing an undirected graph corresponding to the image sequence with the images as nodes and the co-viewing relationships between the images as edges; Generate a loss function of the undirected graph, wherein the function value of the loss function is used to represent the total color difference of the undirected graph after adjusting the color value of the node, wherein the color value includes at least one of a chromaticity value and a brightness value, and the total color difference includes the sum of the color difference values ​​corresponding to each edge, and the color difference value corresponding to the edge is the difference between the color values ​​of two nodes connected by the edge; Iteratively solving the loss function to determine a gain coefficient of each node when the function value satisfies a preset constraint condition, wherein the gain coefficient is used to characterize the degree to which the color value needs to be adjusted, and the preset constraint condition includes at least one of the function value being minimum and the function value being less than a target value; The gain coefficient is used to perform image color processing on the image corresponding to the node.

2. The method according to claim 1, characterized in that The loss function for generating the undirected graph includes: For each edge connecting two nodes, determine the common view area of ​​each node based on the point cloud intersection, where the common view area is the projection area of ​​the point cloud intersection in the image corresponding to the node; Determine the average of the color values ​​of the pixels in the common viewing area as the color value of the node; A loss function of the undirected graph is generated based on the color value of each node and the gain coefficient, wherein the gain coefficient is a variable to be solved in the loss function.

3. The method according to claim 2, characterized in that Before determining the mean value of the color values ​​of the pixels in the common viewing area as the color value of the node, the method includes: Determine a matching block for each pixel based on a preset matching block size, wherein the matching block is an image region centered on the pixel and having a size equal to the preset matching block size; The average value of the color values ​​of the pixels in the matching block is determined as the color value of the central pixel of the matching block.

4. The method according to claim 3, characterized in that The step of determining the average value of the color values ​​of the pixels in the common viewing area as the color value of the node includes: Using the interquartile range method, determine the first pixel point in the common viewing area, the first pixel point includes a pixel point whose pixel color difference is greater than the upper quartile, the pixel color difference is the difference in color values ​​of a common viewing pixel pair, and the common viewing pixel pair is a pair of pixel points of the same point in the point cloud intersection in the corresponding image; Performing object recognition on the image, and in response to the presence of a target object belonging to a preset object type, determining a second pixel point corresponding to the target object; Eliminating the first pixel point and the second pixel point in the common viewing area to obtain a target pixel point in the common viewing area; The average value of the color values ​​of the target pixel points is determined as the color value of the node.

5. The method according to any one of claims 2 to 4, characterized in that: The loss function of the undirected graph is generated based on the color value of each node and the gain coefficient, comprising: Determine a loss weight of the edge based on the number of pixels in the common view area corresponding to the edge, where the loss weight is proportional to the number of pixels; The loss function is constructed based on the loss weight of each edge, the color value of each node and the gain coefficient.

6. The method according to claim 5, characterized in that The function value of the loss function is the sum of the total color difference and the regularization term, the total color difference is the weighted sum of the squares of the color difference corresponding to each edge, the color difference is the difference in color values ​​of the two nodes corresponding to the edge after the color value is adjusted based on the gain coefficient, the regularization term is the product of the regularization coefficient and the sum of the regularization distances, the sum of the regularization distances is the sum of the regularization distances corresponding to all nodes in the undirected graph, the regularization distance is the distance between the gain coefficient and a preset parameter, and the preset parameter is a positive integer.

7. The method according to any one of claims 1 to 4, characterized in that: The iteratively solving the loss function to determine the gain coefficient of each node when the function value satisfies the preset constraint condition includes: The gradient descent method is used to iteratively solve the loss function based on the preset constraints to determine the gain coefficient of each node, wherein the preset constraints also include at least one of the gain coefficient being minimum and the gain coefficients not all being 0.

8. An image processing device, characterized in that: include: A first determination module is used to determine a common view relationship between the images based on a point cloud coordinate range corresponding to each image in the image sequence in a world coordinate system, wherein the point cloud coordinate ranges corresponding to the images having the common view relationship have an intersection; A construction module, used to construct an undirected graph corresponding to the image sequence with the images as nodes and the co-viewing relationships between the images as edges; A generating module, configured to generate a loss function of the undirected graph, wherein the function value of the loss function is used to represent the total color difference of the undirected graph after the color value of the node is adjusted, wherein the color value includes at least one of a chromaticity value and a brightness value, and the total color difference includes the sum of the color difference values ​​corresponding to each edge, and the color difference value corresponding to the edge is the difference between the color values ​​of two nodes connected by the edge; A second determination module is used to iteratively solve the loss function to determine a gain coefficient of each node when the function value satisfies a preset constraint condition, wherein the gain coefficient is used to characterize the degree to be adjusted of the color value, and the preset constraint condition includes at least one of the function value being minimum and the function value being less than a target value; The processing module is used to perform image color processing on the image corresponding to the node by using the gain coefficient.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images