Image processing method, device, apparatus and storage medium

Abstract

Embodiments of the present disclosure relate to an image processing method, device, equipment and storage medium, wherein the method comprises: obtaining a texture image of a target model, a scalar image corresponding to the texture image, and initial camera internal and external parameters corresponding to the texture image; determining an error corresponding to the initial camera internal and external parameters based on the texture image, the scalar image and the target model; and optimizing the initial camera internal and external parameters based on the error corresponding to the initial camera internal and external parameters. According to the embodiments of the present disclosure, the misalignment problem of the texture image and the target model can be improved, and the texture image is globally consistent with respect to the target model, thereby improving the texture fusion crack problem. Moreover, the embodiments of the present disclosure can reduce the influence of the light factor on the accuracy of error calculation, improve the error accuracy, and thereby the camera internal and external parameters optimized based on the error can better improve the misalignment problem of the texture image and the target model.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional scanning technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] A 3D scanner is a scientific instrument used to detect and analyze the shape (geometry) and appearance (such as texture images) of objects or environments in the real world. Currently, during the scanning process, due to inherent acquisition intervals (i.e., texture images and depth images are not acquired simultaneously), a slight misalignment always exists between the texture image and the model. This misalignment can lead to cracks in the subsequent texture blending effect.

[0003] Therefore, there is an urgent need for an image processing method that can solve the problem of texture fusion cracks. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, device and storage medium.

[0005] A first aspect of this disclosure provides an image processing method, the method comprising:

[0006] The target model's texture image, the scalar image corresponding to the texture image, and the initial camera intrinsic and extrinsic parameters corresponding to the texture image are obtained. The texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels.

[0007] The error corresponding to the initial camera intrinsic and extrinsic parameters is determined based on the texture image, the scalar image, and the target model;

[0008] The initial camera intrinsic and extrinsic parameters are optimized based on the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0009] A second aspect of this disclosure provides an image processing apparatus, the apparatus comprising:

[0010] The first acquisition module is used to acquire the texture image of the target model, the scalar image corresponding to the texture image, and the initial camera intrinsic and extrinsic parameters corresponding to the texture image. The texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels.

[0011] The determination module is used to determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image, the scalar image, and the target model;

[0012] An optimization module is used to optimize the initial camera intrinsic and extrinsic parameters based on the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0013] A third aspect of this disclosure provides an electronic device, the server comprising: a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of the first aspect described above.

[0014] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the method of the first aspect described above.

[0015] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0016] This embodiment of the disclosure can acquire a texture image of the target model, a scalar image corresponding to the texture image, and initial camera intrinsic and extrinsic parameters corresponding to the texture image. The texture image includes multiple first pixels and their corresponding texture values, and the scalar image includes multiple second pixels and their corresponding scalar values. The error corresponding to the initial camera intrinsic and extrinsic parameters is determined based on the texture image, scalar image, and target model. The initial camera intrinsic and extrinsic parameters are then optimized based on this error. It is evident that by employing the above technical solution, the camera intrinsic and extrinsic parameters (i.e., camera intrinsic and extrinsic parameters) corresponding to the texture image can be optimized, thereby improving the misalignment problem between the texture image and the target model. This ensures global consistency of the texture image relative to the target model, improving the texture fusion crack problem. Furthermore, since the above technical solution calculates the error corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image and a scalar image less affected by illumination (i.e., evaluating the global consistency of the texture image relative to the target model), the impact of illumination factors on the accuracy of error calculation can be reduced, improving error accuracy. This allows the error-optimized camera intrinsic and extrinsic parameters to better improve the misalignment problem between the texture image and the target model. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart of an image processing method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a flowchart of another image processing method provided in this embodiment of the disclosure;

[0021] Figure 3 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0023] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0024] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0025] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this disclosure. This method can be executed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0026] S110. Obtain the texture image of the target model, the scalar image corresponding to the texture image, and the initial camera intrinsic and extrinsic parameters corresponding to the texture image. The texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels.

[0027] Specifically, the target model is a model that has been scanned in three dimensions.

[0028] Specifically, the texture image of the target model is a texture image obtained by scanning the target model, and there are usually multiple texture images. Each texture image includes multiple first pixels and texture values ​​for each first pixel. The texture values ​​may include at least one of color values, grayscale values, brightness values, etc.

[0029] Specifically, each scalar image includes multiple second pixels and a corresponding scalar value for each second pixel. The multiple second pixels in the scalar image correspond one-to-one with the multiple first pixels in the corresponding texture image. Those skilled in the art should understand that scalar values ​​are less affected by illumination than texture values; therefore, the impact of illumination on scalar images is less than the impact of illumination on texture images.

[0030] There are several ways to obtain scalar images; the following are some typical examples.

[0031] Optionally, the first method for obtaining the scalar image is as follows: perform a Laplacian transform on the texture image to obtain the corresponding scalar image. Further, optionally, before performing the Laplacian transform on the texture image, noise reduction processing can be performed on the texture image to reduce noise interference.

[0032] Specifically, for each texture image, noise reduction is performed followed by Laplacian transformation to obtain the corresponding scalar image.

[0033] Optionally, a second method for obtaining the scalar image is as follows: obtain the tone image corresponding to the texture image; perform a Laplacian transform on the tone image to obtain the scalar image corresponding to the texture image. Further, optionally, before obtaining the tone image corresponding to the texture image, the texture image can be denoised to reduce noise interference.

[0034] Specifically, each tone image includes multiple third pixels and a tone value for each third pixel, and the multiple third pixels in the tone image correspond one-to-one with the multiple first pixels in the corresponding texture image. It should be noted that any possible method for obtaining tone images can be used to obtain the corresponding tone image based on the texture image, and this application does not limit this.

[0035] Specifically, for each texture image, noise reduction is performed before obtaining the tone image, and then the Laplacian transform is performed on the tone image to obtain the corresponding scalar image.

[0036] Optionally, a second method for obtaining the scalar image is as follows: perform line detection processing on the texture image to obtain the corresponding scalar image. Further, optionally, before performing line detection processing on the texture image, noise reduction processing can be performed on the texture image to reduce noise interference.

[0037] Specifically, for each texture image, noise reduction is performed followed by line detection to obtain the corresponding scalar image. At this point, the scalar value is used to indicate whether a pixel is located on a line segment.

[0038] It is understandable that the three methods described above for acquiring scalar images are simple, convenient, and streamlined, which helps reduce the implementation difficulty of acquiring scalar images and thus reduces the performance requirements of electronic devices. Of course, scalar images can also be acquired using AI (i.e., neural network models) and other methods, and this application does not limit this approach.

[0039] Specifically, the initial camera intrinsic and extrinsic parameters are the camera intrinsic and extrinsic parameters before optimization. The camera intrinsic parameters may include the camera focal length and the projection coordinates of the camera center on the pixel plane, while the camera extrinsic parameters may include the camera pose.

[0040] Understandably, since the initial camera intrinsics and extrinsic parameters are not the actual camera intrinsics and extrinsic parameters of the texture image, determining the mapping relationship between the first pixel in the texture image and the 3D point on the target model based on the initial camera intrinsics and extrinsic parameters will result in misalignment between the texture image and the target model. This will lead to texture fusion cracks when performing texture fusion based on these initial camera intrinsics and extrinsic parameters. Therefore, it is necessary to optimize the initial camera intrinsics and extrinsic parameters to make them closer to the actual camera intrinsics and extrinsic parameters.

[0041] S120. Determine the errors corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image, scalar image, and target model.

[0042] Specifically, the errors corresponding to the initial camera intrinsic and extrinsic parameters are used to characterize the degree of global consistency of the texture image of the target model relative to the target model; in other words, they are used to characterize the degree of misalignment of the texture image of the target model relative to the target model.

[0043] It should be noted that the scalar image mentioned here includes a scalar image acquired through at least one acquisition method. For example, it may include a scalar image acquired through the first acquisition method described above, and / or a scalar image acquired through the second acquisition method described above.

[0044] There are several ways to determine the errors corresponding to the initial camera intrinsic and extrinsic parameters. The following is an explanation of a typical example.

[0045] Optionally, S120 may include: determining a first sub-error corresponding to each target 3D point on the target model based on the texture image and the target model; determining a second sub-error corresponding to each target 3D point on the target model based on the scalar image and the target model; and determining the error corresponding to the initial camera intrinsic and extrinsic parameters based on the first and second sub-errors corresponding to each target 3D point on the target model.

[0046] Specifically, the target model includes multiple three-dimensional points, and the target three-dimensional points are three-dimensional points on the target model used to calculate the errors corresponding to the initial camera intrinsic and extrinsic parameters. For example, each three-dimensional point on the target model can be a target three-dimensional point, or a portion of the three-dimensional points can be randomly selected from multiple three-dimensional points on the target model as target three-dimensional points, but it is not limited to this.

[0047] Specifically, the first sub-error is used to characterize the degree of misalignment between the target 3D point and the first pixel in the texture image that has a mapping relationship with the target 3D point.

[0048] Specifically, the second sub-error is used to characterize the degree of misalignment between the target 3D point and the second pixel point in the scalar image that has a mapping relationship with the target 3D point.

[0049] Of course, AI or other methods can also be used to determine the errors corresponding to the initial camera intrinsic and extrinsic parameters, and this application does not limit this.

[0050] S130. Optimize the initial camera intrinsic and extrinsic parameters based on the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0051] Specifically, an iterative method can be used to optimize the initial camera intrinsic and extrinsic parameters. That is, if the error corresponding to the initial camera intrinsic and extrinsic parameters is greater than a preset error threshold, the initial camera intrinsic and extrinsic parameters are optimized, and the step of determining the error corresponding to the optimized initial camera intrinsic and extrinsic parameters is returned, until the error corresponding to the optimized initial camera intrinsic and extrinsic parameters is less than or equal to the preset error threshold. It should be noted that the specific value of the preset error threshold can be set by those skilled in the art according to actual circumstances, and this application does not limit it in this regard.

[0052] Of course, scalar images can also be obtained using AI or other methods, and this application does not limit this.

[0053] This embodiment optimizes the camera intrinsic and extrinsic parameters (i.e., camera intrinsic and extrinsic parameters) corresponding to the texture image, thereby improving the misalignment problem between the texture image and the target model. This ensures global consistency of the texture image relative to the target model, mitigating texture fusion cracks. Furthermore, since the above technical solution calculates the error corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image and a scalar image less affected by illumination (i.e., evaluating the global consistency of the texture image relative to the target model), it reduces the impact of illumination factors on the accuracy of error calculation, improves error accuracy, and thus allows the error-optimized camera intrinsic and extrinsic parameters to better improve the misalignment problem between the texture image and the target model.

[0054] Figure 2 This is a schematic flowchart of another image processing method provided in this disclosure. This disclosure optimizes the above embodiments and can be combined with various optional solutions from one or more of the above embodiments.

[0055] like Figure 2 As shown, the image processing method may include the following steps.

[0056] S210. Obtain the texture image of the target model, the scalar image corresponding to the texture image, and the initial camera intrinsic and extrinsic parameters corresponding to the texture image. The texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels.

[0057] Specifically, S210 is similar to S110, and there are no limitations on it.

[0058] S220. For each target 3D point on the target model, determine at least two first pixels corresponding to the target 3D point from the texture image based on the initial camera intrinsic and extrinsic parameters, and determine the first sub-error corresponding to the target 3D point based on the texture values ​​of the at least two first pixels.

[0059] Specifically, for each texture image, based on the initial camera intrinsic and extrinsic parameters corresponding to that texture image, the mapping relationship between the first pixel in the texture image and the 3D points on the target model can be determined, that is, determining which 3D point on the target model each first pixel in the texture image was scanned from. Thus, for each target 3D point on the target model, all first pixels that have a mapping relationship with that target 3D point can be determined from the texture image of the target model.

[0060] The "at least two first pixels" mentioned in S220 can be all first pixels that have a mapping relationship with the target three-dimensional point, or it can be a part of the first pixels among all the first pixels that have a mapping relationship with the target three-dimensional point, and there is no limitation on this.

[0061] Optionally, determining the first sub-error corresponding to the target 3D point based on the texture values ​​of at least two first pixels may include:

[0062] S221. For each pair of first pixels among at least two first pixels, determine the error corresponding to the first pixel pair based on the texture values ​​of the two first pixels in the first pixel pair.

[0063] Specifically, in the phrase "at least two first pixels", each pair of first pixels can form a pair of first pixels.

[0064] Optionally, S221 may include: for each first pixel pair, determining the error corresponding to the first pixel pair based on the difference between the texture values ​​of the two first pixels in the first pixel pair. For example, the difference between the texture values ​​of the two first pixels in the first pixel pair (or the absolute value of the difference, or the square of the absolute value of the difference) may be used as the error corresponding to the first pixel pair, but it is not limited to this.

[0065] S222. The errors corresponding to each first pixel point pair are weighted and summed to obtain the first sub-error corresponding to the target 3D point. The weight corresponding to the first pixel point pair is negatively correlated with the difference between the texture values ​​of the two first pixels in the first pixel point pair.

[0066] Of course, the errors corresponding to each first pixel point can also be summed to obtain the first sub-error corresponding to the target three-dimensional point. This application does not limit this.

[0067] It is understandable that the two first pixels in a first pixel pair have a mapping relationship with the same target 3D point. Theoretically, the texture values ​​of these two first pixels should be similar, that is, the difference between their texture values ​​should be small. When the difference is large, it may be that the texture value of one of the first pixels has been distorted due to the influence of lighting. In this application, by setting the weight corresponding to the first pixel pair to be negatively correlated with the difference between the texture values ​​of the two first pixels in the first pixel pair, the influence of the error corresponding to the low accuracy of the first pixel pair on the first sub-error can be reduced, that is, the influence of lighting on the first sub-error is reduced, the accuracy of the first sub-error is improved, and thus the accuracy of the error corresponding to the initial camera intrinsic and extrinsic parameters is improved. In this way, the initial camera intrinsic and extrinsic parameters can be better optimized, so that the final optimized initial camera intrinsic and extrinsic parameters are closer to the real camera intrinsic and extrinsic parameters.

[0068] S230. For each target 3D point on the target model, determine at least two second pixel points corresponding to the target 3D point from the scalar image, and determine the second sub-error corresponding to the target 3D point based on the scalar values ​​of the at least two second pixel points.

[0069] Specifically, as mentioned above, for each texture image, the mapping relationship between the first pixel in the texture image and the 3D point on the target model can be determined based on the initial camera intrinsic and extrinsic parameters corresponding to the texture image. For each texture image, there is a mapping relationship between multiple first pixels in the texture image and multiple second pixels in the corresponding scalar image. Therefore, for each scalar image, it can be determined which 3D point on the target model each second pixel in the scalar image is mapped to. Thus, for each target 3D point on the target model, all second pixels mapped to that target 3D point can be determined from the scalar image. The "at least two second pixels" mentioned in S230 can be all second pixels mapped to the target 3D point, or it can be a subset of all second pixels mapped to the target 3D point; there is no limitation on this.

[0070] Optionally, determining the second sub-error corresponding to the target 3D point based on the scalar values ​​of at least two second pixel points may include:

[0071] S231. For each pair of second pixel points in at least two second pixel points, determine the error corresponding to the second pixel point pair based on the scalar values ​​of the two second pixel points in the second pixel point pair.

[0072] Specifically, in the phrase "at least two second pixels", each pair of second pixels can form a pair of second pixels.

[0073] Optionally, S231 may include: for each pair of second pixel points, determining the error corresponding to the second pixel point pair based on the difference between the scalar values ​​of the two second pixel points in the pair. For example, the difference between the scalar values ​​of the two second pixel points in the pair (or the absolute value of the difference, or the square of the absolute value of the difference) may be used as the error corresponding to the second pixel point pair, but it is not limited to this.

[0074] S222. Sum the errors corresponding to each second pixel point to obtain the second sub-error corresponding to the target 3D point.

[0075] It should be noted that if the scalar image corresponding to the texture image includes scalar images obtained by at least two acquisition methods, then for each acquisition method, the second sub-error corresponding to the target 3D point under the scalar image obtained by that acquisition method is obtained through S230. Then, the second sub-errors corresponding to the target 3D point under the scalar images obtained by various acquisition methods are directly summed or weighted to obtain the final second sub-error corresponding to the target 3D point.

[0076] S240. For each target 3D point on the target model, determine the error corresponding to the target 3D point based on the first sub-error and the second sub-error corresponding to the target 3D point.

[0077] Specifically, the error corresponding to the target 3D point is used to characterize the degree of misalignment between the target 3D point and the following two types of pixels: the first pixel in the texture image that has a mapping relationship with the target 3D point, and the second pixel in the scalar image that has a mapping relationship with the target 3D point.

[0078] Optionally, S250 may include: weighting and summing the first sub-error and the second sub-error corresponding to the target three-dimensional point, or directly summing them to obtain the error corresponding to the target three-dimensional point.

[0079] S250. Determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the error corresponding to each target 3D point.

[0080] Optionally, S250 may include: weighting and summing the errors corresponding to each target 3D point or directly summing them to obtain the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0081] Optionally, before S250, the method further includes: performing feature point recognition and feature point matching on the texture image to obtain at least one feature point pair; and determining the error corresponding to the feature point pair based on the initial camera intrinsic and extrinsic parameters. In this case, S250 may include: determining the error corresponding to the initial camera intrinsic and extrinsic parameters based on the error corresponding to each target 3D point and the error corresponding to each feature point pair.

[0082] Specifically, feature points refer to relatively salient points in an image, such as contour points, bright spots in darker areas, dark spots in brighter areas, or points with large curvature at the edges of the image, but are not limited to these. Matching allows for a mapping relationship between the two first pixels (i.e., feature points) in a feature point pair and the same 3D point on the target model.

[0083] Local or global feature recognition can be performed on the texture image of the target model; this application does not limit this. Furthermore, feature point recognition and matching can be performed using methods such as ORB (Oriented Fast and Rotated BRIEF), SIFT, SuperPoint, and Kd-tree+rasac; this application also does not limit this.

[0084] Specifically, the error corresponding to the feature point pair is used to characterize the misalignment error between two 3D points that have a mapping relationship with the initial feature point pair.

[0085] Optionally, the error corresponding to the feature point pair is determined based on the initial camera intrinsic and extrinsic parameters, including: for each feature point pair, determining the two 3D points corresponding to the two first pixel points in the feature point pair when mapped onto the target model based on the initial camera intrinsic and extrinsic parameters, and determining the error corresponding to the feature point pair based on the distance between the two 3D points.

[0086] For example, the difference in distance between two 3D points that have a mapping relationship with a feature point pair (or the absolute value of the difference, or the square of the absolute value of the difference) can be used as the error corresponding to the feature point pair, but it is not limited to this.

[0087] Optionally, determining the error corresponding to the initial camera intrinsic and extrinsic parameters based on the error corresponding to each target 3D point and the error corresponding to each feature point pair may include: using the sum of the errors corresponding to each target 3D point and the errors corresponding to each feature point pair as the error corresponding to the initial camera intrinsic and extrinsic parameters; or, summing the errors corresponding to each target 3D point to obtain a first summed value, summing the errors corresponding to each feature point pair to obtain a second summed value, and weighting the first summed value and the second summed value to obtain the error corresponding to the initial camera intrinsic and extrinsic parameters.

[0088] It is understandable that the first pixel in a feature point pair is usually the point least affected by lighting interference, thus the error corresponding to the feature point pair is less affected by lighting. Taking the error corresponding to the feature point pair into account when calculating the error of the initial camera intrinsic and extrinsic parameters allows for a more comprehensive consideration of factors and reduces the impact of lighting interference on the error. This, in turn, helps to accelerate the optimization speed of the initial camera intrinsic and extrinsic parameters, reduce the influence of lighting, and improve the optimization accuracy of the initial camera intrinsic and extrinsic parameters.

[0089] S260. Optimize the initial camera intrinsic and extrinsic parameters based on the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0090] Specifically, S260 is similar to S130, and will not be described in detail here.

[0091] This embodiment of the present disclosure can determine the first sub-error corresponding to the target 3D point based on the texture image and the initial camera intrinsic and extrinsic parameters, and determine the second sub-error corresponding to the target 3D point based on the scalar image that is less affected by illumination and the initial camera intrinsic and extrinsic parameters. Then, based on the first sub-error and the second sub-error corresponding to the target 3D point, the error corresponding to the target 3D point is determined. Thus, based on the error corresponding to each target 3D point, the error corresponding to the initial camera intrinsic and extrinsic parameters is determined. This makes the determination of the initial camera intrinsic and extrinsic parameters simple and convenient, which helps to reduce the implementation difficulty of determining the initial camera intrinsic and extrinsic parameters, thereby reducing the performance requirements of electronic devices.

[0092] Figure 3This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. This image processing apparatus can be understood as the aforementioned electronic device or a functional module within the aforementioned electronic device. For example... Figure 3 As shown, the image processing apparatus 300 includes:

[0093] The first acquisition module 310 is used to acquire the texture image of the target model, the scalar image corresponding to the texture image, and the initial camera intrinsic and extrinsic parameters corresponding to the texture image. The texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels.

[0094] The first determining module 320 is used to determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image, the scalar image, and the target model.

[0095] The optimization module 330 is used to optimize the initial camera intrinsic and extrinsic parameters based on the errors corresponding to the initial camera intrinsic and extrinsic parameters.

[0096] In another embodiment of this disclosure, the first acquisition module 310 may include:

[0097] The first transformation submodule is used to perform a Laplacian transform on the texture image to obtain the scalar image corresponding to the texture image.

[0098] In yet another embodiment of this disclosure, the first acquisition module 310 may include:

[0099] The first acquisition submodule is used to acquire the tone image corresponding to the texture image;

[0100] The second transformation submodule is used to perform a Laplacian transform on the tone image to obtain the scalar image corresponding to the texture image.

[0101] In another embodiment of this disclosure, the first determining module 320 may include:

[0102] The first determining submodule is used to determine, for each target 3D point on the target model, at least two first pixels corresponding to the target 3D point from the texture image based on the initial camera intrinsic and extrinsic parameters, and to determine the first sub-error corresponding to the target 3D point based on the texture values ​​of the at least two first pixels.

[0103] The second determining submodule is used to determine at least two second pixel points corresponding to each target 3D point on the target model from the scalar image, and to determine the second sub-error corresponding to the target 3D point based on the scalar values ​​of the at least two second pixel points.

[0104] The third determining submodule is used to determine the error corresponding to the target three-dimensional point for each target three-dimensional point on the target model based on the first sub-error and the second sub-error corresponding to the target three-dimensional point.

[0105] The fourth determining submodule is used to determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the error corresponding to each of the target 3D points.

[0106] In another embodiment of this disclosure, the first determining submodule may include:

[0107] The first determining unit is configured to determine the error corresponding to the first pixel point pair based on the texture values ​​of the two first pixels in the first pixel point pair for each of the at least two first pixel point pairs;

[0108] The summation unit is used to sum the errors corresponding to each pair of first pixels with weights to obtain the first sub-error corresponding to the target three-dimensional point, wherein the weight corresponding to the pair of first pixels is negatively correlated with the difference between the texture values ​​of the two first pixels in the pair of first pixels.

[0109] In another embodiment of this disclosure, the device further includes:

[0110] The identification and matching module is used to identify and match feature points in the texture image to obtain at least one pair of feature points.

[0111] The second determining module is used to determine the error corresponding to the feature point pair based on the initial camera intrinsic and extrinsic parameters.

[0112] Specifically, the fourth determining submodule is used to determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the error corresponding to each of the target 3D points and the error corresponding to each of the feature point pairs.

[0113] In another embodiment of this disclosure, the second determining module is specifically used to, for each feature point pair, determine two three-dimensional points corresponding to the two first pixel points in the feature point pair when mapped onto the target model based on the initial camera intrinsic and extrinsic parameters, and determine the error corresponding to the feature point pair based on the distance between the two three-dimensional points.

[0114] The apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0115] This disclosure also provides an electronic device, which includes: a memory storing a computer program; and a processor for executing the computer program, wherein when the computer program is executed by the processor, it can implement the methods of any of the above embodiments.

[0116] Example, Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device 400 in the embodiments of this disclosure. The electronic device 400 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0117] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0118] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0119] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0120] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0121] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0122] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0123] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a texture image of a target model, a scalar image corresponding to the texture image, and initial camera intrinsic and extrinsic parameters corresponding to the texture image, wherein the texture image includes multiple first pixels and texture values ​​corresponding to the multiple first pixels, and the scalar image includes multiple second pixels and scalar values ​​corresponding to the multiple second pixels; determine the error corresponding to the initial camera intrinsic and extrinsic parameters based on the texture image, the scalar image, and the target model; and optimize the initial camera intrinsic and extrinsic parameters based on the error corresponding to the initial camera intrinsic and extrinsic parameters.

[0124] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0127] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0128] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a texture image of a target model, a scalar image corresponding to the texture image, and initial camera internal and external parameters corresponding to the texture image, wherein the texture image comprises a plurality of first pixels and texture values corresponding to the plurality of first pixels, and the scalar image comprises a plurality of second pixels and scalar values corresponding to the plurality of second pixels; for each target three-dimensional point on the target model, determining at least two first pixel points corresponding to the target three-dimensional point from the texture image based on the initial camera internal and external parameters, and determining a first sub-error corresponding to the target three-dimensional point based on texture values of the at least two first pixel points; for each target three-dimensional point on the target model, determining at least two second pixel points corresponding to the target three-dimensional point from the scalar image, and determining a second sub-error corresponding to the target three-dimensional point based on scalar values of the at least two second pixel points; for each target three-dimensional point on the target model, determining an error corresponding to the target three-dimensional point based on the first sub-error and the second sub-error corresponding to the target three-dimensional point; determining an error corresponding to the initial camera internal and external parameters based on errors corresponding to the target three-dimensional points; optimizing the initial camera internal and external parameters based on the error corresponding to the initial camera internal and external parameters.

2. The method of claim 1, wherein, The method comprises: performing Laplace transform on the texture image to obtain the scalar image corresponding to the texture image.

3. The method of claim 1, wherein, The method comprises: acquiring a hue image corresponding to the texture image; performing Laplace transform on the hue image to obtain the scalar image corresponding to the texture image.

4. The method of claim 1, wherein, The method comprises: for each first pixel pair in the at least two first pixel points, determining an error corresponding to the first pixel pair based on texture values of the two first pixels in the first pixel pair; weighting and adding errors corresponding to the first pixel pairs to obtain the first sub-error corresponding to the target three-dimensional point, wherein a weight corresponding to the first pixel pair is negatively related to a difference between the texture values of the two first pixels in the first pixel pair.

5. The method of claim 1, wherein, The method further comprises: performing feature point recognition and feature point matching on the texture image to obtain at least one feature point pair; determining an error corresponding to the feature point pair based on the initial camera internal and external parameters; The method comprises: determining the error corresponding to the initial camera internal and external parameters based on errors corresponding to the target three-dimensional points and errors corresponding to the feature point pairs.

6. The method of claim 5, wherein, The method comprises: for each feature point pair, determining two three-dimensional points corresponding to the two first pixels in the feature point pair when the two first pixels are mapped onto the target model based on the initial camera internal and external parameters, and determining an error corresponding to the feature point pair based on a distance between the two three-dimensional points.

7. An image processing apparatus characterized by comprising: The method comprises: The first acquisition module is configured to acquire a texture image of a target model, a scalar image corresponding to the texture image, and initial camera internal and external parameters corresponding to the texture image, wherein the texture image comprises a plurality of first pixels and texture values corresponding to the plurality of first pixels, and the scalar image comprises a plurality of second pixels and scalar values corresponding to the plurality of second pixels; The determination module is configured to, for each target three-dimensional point on the target model, determine at least two first pixel points corresponding to the target three-dimensional point from the texture image based on the initial camera internal and external parameters, and determine a first sub-error corresponding to the target three-dimensional point based on texture values of the at least two first pixel points; For each target three-dimensional point on the target model, determine at least two second pixel points corresponding to the target three-dimensional point from the scalar image, and determine a second sub-error corresponding to the target three-dimensional point based on scalar values of the at least two second pixel points; For each target three-dimensional point on the target model, determine a first sub-error corresponding to the target three-dimensional point based on the first sub-error and the second sub-error corresponding to the target three-dimensional point; Determine an error corresponding to the initial camera internal and external parameters based on errors corresponding to the target three-dimensional points; The optimization module is configured to optimize the initial camera internal and external parameters based on the error corresponding to the initial camera internal and external parameters.

8. An electronic device, comprising: The processor and the memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method of any one of claims 1-6. The storage medium stores a computer program, and when the computer program is executed by the processor, the method of any one of claims 1-6 is implemented.

9. A computer-readable storage medium, characterized in that, ​

Images