Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, electronic device, and storage medium, wherein the method includes: obtaining an initial intrinsic parameter value of a camera; and, in response to a change in the camera-object distance, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance. The present disclosure compensates the camera intrinsic parameter in real time in conjunction with the change in object distance, effectively resolving issues such as random changes in camera intrinsic parameters caused by autofocus. The updated intrinsic parameter value after compensation is more accurate, thereby making subsequent positioning results more precise.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer vision, visual-inertial positioning systems (VISs) using cameras and inertial measurement units (IMUs) have become a research hotspot and are widely used in various fields such as drones, autonomous vehicles, robotics, and augmented reality. Computer vision primarily extracts information from images by simulating human vision, and ultimately achieves actual positioning through processing and calculation of this image information. Accurate camera intrinsic parameters are essential for stable and robust visual positioning.

[0003] In the related art, camera intrinsic parameters can be obtained mainly through offline methods. In this offline method, the camera's intrinsic parameters are usually calibrated in advance, and the camera intrinsic parameters remain unchanged during the entire system operation. However, the accuracy of offline camera intrinsic parameter calibration depends on the accuracy of the calibration plate used. High-precision calibration plates are difficult to meet the needs of more users. In addition, the fixed intrinsic parameters determined by this offline method are difficult to adapt to relevant scenarios with visual changes, such as autofocus, which will adversely affect positioning accuracy. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide an image processing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0006] Get the initial intrinsic parameter value of the camera;

[0007] In response to a change in the camera-object distance, an updated intrinsic parameter value after intrinsic parameter compensation is performed on the camera is determined based on the initial intrinsic parameter value and the changed camera-object distance.

[0008] Using the above image processing method, once the initial intrinsic parameter values ​​of the camera are obtained, if the camera-object distance changes, the camera can be compensated for the intrinsic parameters based on the initial intrinsic parameter values ​​and the changed camera-object distance, thereby determining a more accurate updated intrinsic parameter value. The present disclosure compensates for the camera intrinsic parameters in real time based on changes in object distance, effectively resolving issues such as random changes in camera intrinsic parameters caused by autofocus. The compensated updated intrinsic parameter values ​​are more accurate, thereby resulting in more precise subsequent positioning results.

[0009] In a possible implementation, obtaining an initial intrinsic parameter value of the camera includes:

[0010] Obtain multiple frames of target images captured by the camera set on the mobile device;

[0011] An initial intrinsic parameter value of the camera is determined based on the multiple frames of target images.

[0012] In a possible implementation, determining the initial intrinsic parameter value of the camera based on the multiple frames of target images includes:

[0013] For any two adjacent target image frames in the multiple target image frames, constructing an error function for evaluating positioning accuracy based on a correspondence between an image coordinate system and a world coordinate system;

[0014] The error function constructed is minimized based on the camera intrinsic parameters to obtain the initial intrinsic parameter values ​​of the camera.

[0015] Here, the error function constructed by combining the correspondence between the image coordinate system and the world coordinate system can determine the initial intrinsic parameter value. The operation is simple, and the determined initial intrinsic parameter value is relatively accurate.

[0016] In a possible implementation, constructing an error function for evaluating positioning accuracy based on the correspondence between the image coordinate system and the world coordinate system includes:

[0017] For a first image point included in a first target image of the two target image frames, determining three-dimensional point representation information after projection of the first pixel point based on a correspondence between a first image coordinate system of the first target image frame and a world coordinate system of the three-dimensional point;

[0018] Determining second image point representation information after the three-dimensional point is projected based on the three-dimensional point representation information after the first pixel point is projected and a correspondence between a second image coordinate system of the second frame target image and a world coordinate system of the three-dimensional point;

[0019] A difference operation is performed between the second image point representation information and the value of the second image point to obtain an error function for evaluating positioning accuracy.

[0020] Here, based on the reprojection method, an error function can be constructed to determine the initial intrinsic parameter value of the camera. The error function is used to evaluate the accuracy of positioning. The positioning accuracy increases with the decrease of the error function value. The higher the accuracy, the more accurate the camera intrinsic parameter is to a certain extent.

[0021] In a possible implementation, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance includes:

[0022] In the case where the camera-object distance changes, the updated intrinsic parameter value is calculated based on the initial intrinsic parameter value and the changed camera-object distance through a preset linear constraint relationship.

[0023] Here, the camera intrinsic parameters can be compensated based on the above linear constraint relationship, and the updated intrinsic parameter values ​​obtained by compensation are more accurate.

[0024] In a possible implementation, calculating the updated intrinsic parameter value using a preset linear constraint relationship includes:

[0025] Obtaining a first correspondence between the camera image distance, the camera focal length, and the camera object distance, and a second correspondence between the camera image distances within the camera;

[0026] Determining a linear constraint relationship among camera intrinsic parameters, camera focal length, and camera object distance based on the first corresponding relationship and the second corresponding relationship;

[0027] The initial intrinsic parameter value and the changed camera-object distance are used as input parameters of the linear constraint relationship to calculate and obtain an updated intrinsic parameter value.

[0028] In one possible implementation, the method further includes:

[0029] The mobile device is positioned based on the updated internal parameter value to obtain a positioning result.

[0030] In a possible implementation, positioning the mobile device based on the updated internal parameter value to obtain a positioning result includes:

[0031] The mobile device is positioned based on the updated intrinsic parameter value and the target image currently captured by the camera to obtain a positioning result.

[0032] In a possible implementation, when multiple initial intrinsic parameter values ​​are determined, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter values ​​and the changed camera-object distance includes:

[0033] Performing sliding window filtering based on the multiple initial intrinsic parameter values ​​to obtain processed initial intrinsic parameter values;

[0034] An updated intrinsic parameter value after intrinsic parameter compensation is performed on the camera is determined based on the processed initial intrinsic parameter value and the changed camera-object distance.

[0035] Here, the initial intrinsic parameter value may be processed based on sliding window filtering, so that the processed initial intrinsic parameter value is more accurate, and further the compensated updated intrinsic parameter value is also more accurate.

[0036] In one possible implementation, the method further includes:

[0037] In response to the statistical value of the initial intrinsic parameter value corresponding to the current sliding window meeting the preset condition, an online calibration mode of the camera intrinsic parameter is started.

[0038] Here, online calibration is performed only after the statistical values ​​of the initial internal parameter values ​​are evaluated and found to meet the preset conditions, thereby effectively avoiding calibration errors caused by extreme data and effectively improving subsequent positioning accuracy.

[0039] In a second aspect, the embodiments of the present disclosure further provide an image processing device, including:

[0040] The acquisition module is used to obtain the initial intrinsic parameter values ​​of the camera;

[0041] An updating module is configured to determine, in response to a change in the camera-object distance, an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance.

[0042] In a third aspect, an embodiment of the present disclosure further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method as described in the first aspect and any of its various embodiments are performed.

[0043] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the image processing method as described in the first aspect and any of its various embodiments.

[0044] For a description of the effects of the above-mentioned image processing device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned image processing method, which will not be repeated here.

[0045] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0047] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0048] Figure 2 A schematic diagram illustrating a specific method of coordinate system conversion in the image processing method provided by an embodiment of the present disclosure is shown;

[0049] Figure 3 A schematic diagram of an image processing device provided by an embodiment of the present disclosure is shown;

[0050] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0053] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0054] Research has found that in related technologies, camera intrinsic parameters can be obtained primarily through offline methods. In this offline method, the camera's intrinsic parameters are typically calibrated in advance, and the camera's intrinsic parameters remain unchanged throughout the system's operation. However, the accuracy of offline camera intrinsic calibration depends on the accuracy of the calibration plate used. High-precision calibration plates are difficult to meet the needs of more users. Furthermore, the fixed intrinsic parameters determined by this offline method are difficult to adapt to scenes with visual changes, such as autofocus, which can negatively impact positioning accuracy.

[0055] Another related art method provides an online intrinsic parameter calibration method. This method assigns an initial intrinsic parameter value and treats the camera intrinsic parameter as the system's state variable. The camera intrinsic parameter is then corrected in real time based on current observation information. However, due to issues such as camera focal length variations and insufficient observations, convergence can be slow, and the solution obtained through nonlinear optimization may not be optimal. Consequently, the camera intrinsic parameters corrected using observation information may not be the actual values ​​of the current system.

[0056] Based on the above research, the present disclosure provides an image processing method, device, electronic device and storage medium for real-time compensation of camera intrinsic parameters in combination with changes in object distance, so as to achieve real-time and accurate intrinsic parameter compensation.

[0057] To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, such as a terminal device, a server, or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a personal digital assistant (PDA), a handheld device, a computing device, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0058] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S102, wherein:

[0059] S101: Obtain the initial intrinsic parameter value of the camera;

[0060] S102: In response to a change in the camera-object distance, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance.

[0061] To facilitate understanding of the image processing method provided by the embodiments of the present disclosure, the following first describes the application scenarios of the method. The above-described image processing method can be primarily applied to the field of visual positioning technology. Specifically, by determining updated intrinsic parameter values, more stable and robust device positioning can be achieved. In practical applications, this can be used for positioning mobile devices such as drones, unmanned vehicles, and robots.

[0062] Considering that related technologies, whether determining camera intrinsic parameters offline or online, cannot achieve real-time and accurate intrinsic parameter compensation, the disclosed embodiments provide a solution for implementing intrinsic parameter compensation in conjunction with changes in object distance, thereby providing more accurate and effective camera intrinsic parameters and providing excellent data support for subsequent precise positioning.

[0063] Before implementing intrinsic parameter compensation, the initial intrinsic parameter value of the camera can be determined. This initial intrinsic parameter value can be determined based on an offline or online method. In addition, the embodiments of the present disclosure can determine the relevant initial intrinsic parameter value based on image processing methods. For example, it can be determined based on multiple frames of target images captured by the camera. The related processing of the multiple frames of target images for the same target object can better reflect the camera intrinsic parameter, so the determined initial intrinsic parameter value is also more accurate.

[0064] Among them, the camera in the embodiment of the present disclosure can be set on a mobile device. For different application scenarios, the mobile device here is different. For example, it can be various mobile devices such as unmanned vehicles and drones.

[0065] The multi-frame target images here can be continuously captured by a camera, or can be obtained by sampling the continuously captured multi-frame images. In order to achieve real-time and accurate positioning, continuous multi-frame target images can be used here.

[0066] For multiple consecutive frames of target images, the initial camera intrinsic parameter values ​​can be determined by the correspondence between the image coordinate system and the world coordinate system. Considering that the camera intrinsic parameters will change in situations such as camera autofocus, the disclosed embodiments can perform intrinsic parameter compensation on the initial intrinsic parameter values. The updated intrinsic parameter values ​​obtained from this compensation can be used as positioning reference data for the next frame of image.

[0067] In the embodiment of the present disclosure, during the process of performing intrinsic parameter compensation, the main consideration is the situation where the camera-object distance changes. This is mainly due to the autofocus method in the related art, in which the camera focal length itself does not change, and autofocus is mostly achieved by changing the object distance. With the change of the object distance, the initial intrinsic parameter value can be well compensated, so that the updated intrinsic parameter value determined is more accurate.

[0068] Here, the initial internal parameter value is used as the basic data for internal parameter compensation and can be determined as follows:

[0069] Step 1: for any two adjacent target frames in the multi-frame target image, construct an error function for evaluating positioning accuracy based on the correspondence between the image coordinate system and the world coordinate system;

[0070] Step 2: Minimize the error function constructed based on the camera's intrinsic parameters to obtain the initial intrinsic parameter values ​​of the camera.

[0071] Here, the initial intrinsic parameter values ​​are determined by constructing a functional relationship between an error function used to evaluate positioning accuracy and the camera intrinsic parameters. Specifically, the initial intrinsic parameter values ​​are obtained by solving the constructed error function for the minimum value of the relevant camera intrinsic parameters. This is primarily because positioning accuracy is directly affected by the relevant camera intrinsic parameters. The more the determined camera intrinsic parameters are consistent with the actual values, the higher the positioning accuracy. The relevant error function is precisely constructed as an evaluation function based on positioning accuracy. With the camera intrinsic parameters as the independent variable, a relatively accurate initial intrinsic parameter value can be determined by minimizing the error function.

[0072] The error function can be determined as follows:

[0073] Step 1: for a first image point included in a first target image of the two target image frames, determining three-dimensional point representation information after projection of the first pixel point based on a correspondence between a first image coordinate system of the first target image and a world coordinate system of the three-dimensional point;

[0074] Step 2: Determine second image point representation information after the three-dimensional point projection based on the three-dimensional point representation information after the first pixel point is projected and the correspondence between the second image coordinate system of the second frame target image and the world coordinate system of the three-dimensional point;

[0075] Step 3: Perform a difference operation on the second image point representation information and the value of the second image point to obtain an error function for evaluating positioning accuracy.

[0076] Here, the three-dimensional points seen in the first image coordinate system of the first frame target image can be transformed into the second image coordinate system of the second frame target image. The transformed second image point representation information and the value of the relevant second image point on the second frame target image can be used to construct an error function. The smaller the value of the error function, the higher the positioning accuracy. Conversely, the smaller the value of the error function, the lower the positioning accuracy.

[0077] Among them, the world coordinate system here is an absolute coordinate system. Before the image coordinate system is established, the coordinates of all points on the screen are determined by the origin of the coordinate system. The three-dimensional points here can correspond to points on the world coordinate system, for example, they can be point cloud points in point cloud data.

[0078] In order to explain the construction process of the above error function, we can combine Figure 2 And the specific formula is further described.

[0079] The corresponding relationship between the image coordinate system and the world coordinate system is as follows:

[0080]

[0081] Where [μ, υ] is the image coordinate, [X, Y, Z] is the three-dimensional position of a certain three-dimensional point (such as a landmark point), [f x , f y , c x , c y ] is the camera internal parameter. The above formula is simplified as follows:

[0082]

[0083] In the above formula, F is a function. Its physical meaning is to project a three-dimensional point [X, Y, Z] into a two-dimensional image point [μ, υ] according to the camera model, where the brackets contain some relevant parameters. -1 It is a reverse process of F, which can reversely project the two-dimensional image point [μ, υ] into a point [X, Y, Z].

[0084] like Figure 2 As shown, P is a three-dimensional point observed by a camera at different locations C1 and C2. p1 and p2 are the corresponding observation information. p2 is the corresponding observation information. 21 is the relative pose [direction and displacement] of the two places mentioned above. The physical meaning is to transform the point P seen in the C1 coordinate system (the first image coordinate system) to the C2 coordinate system (the second image coordinate system). In summary, the error function can be constructed:

[0085] in,

[0086] By solving the error function, the camera's initial intrinsic parameter values ​​can be obtained. This allows for real-time intrinsic parameter compensation based on the initial intrinsic parameter values ​​and the changed camera-object distance. Specifically, as the camera-object distance changes, the initial intrinsic parameter values, the camera's focal length, and the changed camera-object distance are input into a pre-set linear constraint relationship to obtain updated intrinsic parameter values ​​for the camera after intrinsic parameter compensation.

[0087] Here, a linear constraint relationship between the camera intrinsic parameters, camera focal length, and camera object distance can be constructed in advance. Furthermore, the initial intrinsic parameter value and the changed camera object distance can be used as input parameters of the linear constraint relationship to calculate the updated intrinsic parameter value. The linear constraint relationship can be determined according to the following steps:

[0088] Step 1: obtaining a first correspondence between the camera image distance, the camera focal length, and the camera object distance, and a second correspondence between the camera image distances within the camera;

[0089] Step 2: Based on the first corresponding relationship and the second corresponding relationship, determine the linear constraint relationship between the camera intrinsic parameters, the camera focal length, and the camera object distance.

[0090] Here, the camera image distance, camera intersection, and camera object distance have a first corresponding relationship, and the camera intrinsic parameter and camera image distance have a second corresponding relationship. It can be seen that the camera image distance can be used as an intermediate parameter. In this way, when the two corresponding relationships (the first corresponding relationship and the second corresponding relationship) are approximated, the linear constraint relationship between the camera intrinsic parameters, camera focal length, and camera object distance can be obtained.

[0091] In order to further illustrate the process of constructing the above linear constraint relationship, the following can be explained with reference to specific formulas.

[0092] Considering that the focal length does not change on cameras that do not support optical zoom, the change from blur to clarity is due to the camera making fine adjustments based on the object distance and image distance to make the image clear, which is what is called focusing. Since the image distance has changed, the image in this state will appear slightly enlarged or reduced. When the image distance changes, the camera's field of view (FOV) will also change. If the image width and height remain unchanged, the camera's intrinsic parameter f x 、f y There will be slight changes. Here, the linear constraint relationship can be explained in combination with the above description.

[0093] According to the first correspondence between the camera image distance v, the camera object distance u, and the camera focal length f:

[0094]

[0095] And the camera internal parameter f x 、f y The second correspondence between and the camera image distance v:

[0096]

[0097] Among them, image w and image h Represents the width and height of the image, L w and L h Indicates the size of the camera's photosensitive component.

[0098] According to the above two corresponding relationships and the corresponding approximate processing, the linear constraint relationship between the camera object distance u and the camera focal length f can be determined:

[0099]

[0100] in, Represents the initial intrinsic parameter value of the camera, f and u represent the camera focal length and camera object distance, and the camera object distance changes during the focusing process.

[0101] The linear relationship above shows that the camera-object distance is linearly related to the camera intrinsic parameters. Using the above formula, once the focal length and object distance are known, the camera intrinsic parameters can be compensated in real time.

[0102] The image processing method provided by the disclosed embodiments enables real-time positioning of mobile devices based on updated internal parameter values ​​obtained through real-time internal parameter compensation. Because the updated internal parameter values ​​are more accurate, the resulting positioning results are also more precise, facilitating various device control operations. For example, if the distance to the vehicle ahead is relatively close based on the position of the unmanned vehicle, vehicle deceleration operations such as braking can be implemented.

[0103] In specific applications, the mobile device can be positioned based on the updated intrinsic parameter value and the target image currently captured by the camera. That is, the updated intrinsic parameter value determined by the previous frame of the target image can be applied to the target image captured by the next frame, and the target image analysis method can be used to achieve accurate positioning of the relevant device.

[0104] In order to further improve the accuracy of the internal parameter compensation, before performing the internal parameter compensation, a plurality of initial internal parameter values ​​may be first filtered based on a sliding window filtering method, and then updated using the processed initial internal parameter values.

[0105] In specific applications, before performing sliding window filtering, we can first obtain a set of data, for example, 5 initial intrinsic parameter values, remove obviously invalid data by sorting by size, and then specify a slider with a width of 4 to slide from left to right, calculate the difference between the rightmost and leftmost ends of the slider, which should be less than the preset threshold (otherwise discarded), find the 4 data with the smallest difference and then calculate the average output. In this way, the processed initial intrinsic parameter value can be a value that avoids the influence of extreme data and is more in line with the actual situation.

[0106] Considering the critical role of accurate camera intrinsic parameters for precise positioning, the online calibration mode of the camera intrinsic parameters can be enabled only when the camera intrinsic parameters are set to meet specific response conditions. In this way, dynamically switching whether to enable online calibration will help improve the overall accuracy of the positioning system.

[0107] Among them, the relevant response conditions can be determined based on the statistical values ​​of the initial intrinsic parameter values ​​corresponding to the current sliding window. For example, it can be whether the variance of the initial intrinsic parameter values ​​corresponding to the current sliding window is less than a preset threshold; for example, it can be whether the mean of the initial intrinsic parameter values ​​corresponding to the current sliding window is greater than a preset threshold.

[0108] The relevant statistical values ​​in the embodiments of the present disclosure can indirectly reflect the current positioning accuracy. When the positioning accuracy is poor, the error function value determined at this time is large. In this case, it is not beneficial to perform internal parameter calibration. Based on this, the embodiments of the present disclosure can use the evaluation of statistical values ​​to realize dynamic switching of the online positioning function, thereby improving the positioning accuracy.

[0109] It should be noted that, when it is determined that the relevant initial internal parameter values ​​do not meet the preset conditions, calibration can be performed using the internal parameter values ​​obtained by offline calibration.

[0110] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0111] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0112] Reference Figure 3 FIG. 1 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure, the device comprising: an acquisition module 301 and an update module 302; wherein,

[0113] An acquisition module 301 is used to obtain the initial intrinsic parameter values ​​of the camera;

[0114] The updating module 302 is configured to determine, in response to a change in the camera-object distance, an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance.

[0115] Using the above-described image processing device, once the initial intrinsic parameter values ​​of the camera are acquired, if the camera-object distance changes, the camera can be compensated for the intrinsic parameters based on the initial intrinsic parameter values ​​and the changed camera-object distance, thereby determining a more accurate updated intrinsic parameter value. The present disclosure performs real-time compensation of the camera intrinsic parameters in conjunction with changes in object distance, effectively resolving issues such as random changes in camera intrinsic parameters caused by autofocus. The compensated updated intrinsic parameter values ​​are more accurate, thereby resulting in more precise subsequent positioning results.

[0116] In a possible implementation, the acquisition module 301 is configured to acquire initial intrinsic parameter values ​​of the camera according to the following steps, including:

[0117] Obtain multiple frames of target images captured by the camera set on the mobile device;

[0118] The initial intrinsic parameter values ​​of the camera are determined based on multiple frames of target images.

[0119] In a possible implementation, the acquisition module 301 is configured to determine the initial intrinsic parameter values ​​of the camera based on the multiple frames of target images according to the following steps:

[0120] For any two adjacent target frames in the multi-frame target image, an error function for evaluating positioning accuracy is constructed based on the correspondence between the image coordinate system and the world coordinate system;

[0121] The constructed error function is minimized based on the camera intrinsic parameters to obtain the initial intrinsic parameter values ​​of the camera.

[0122] In a possible implementation, the acquisition module 301 is configured to construct an error function for evaluating positioning accuracy based on the correspondence between the image coordinate system and the world coordinate system according to the following steps:

[0123] For a first image point included in a first target image of the two target image frames, determining three-dimensional point representation information after projection of the first pixel point based on a correspondence between a first image coordinate system of the first target image and a world coordinate system of the three-dimensional point;

[0124] Determining second image point representation information after the three-dimensional point projection based on the three-dimensional point representation information after the first pixel point is projected and a correspondence between the second image coordinate system of the second frame target image and the world coordinate system of the three-dimensional point;

[0125] A difference operation is performed between the second image point representation information and the value of the second image point to obtain an error function for evaluating positioning accuracy.

[0126] In one possible implementation, the updating module 302 is configured to determine an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance according to the following steps:

[0127] When the camera-object distance changes, the intrinsic parameter value is updated based on the initial intrinsic parameter value and the changed camera-object distance through the preset linear constraint relationship.

[0128] In a possible implementation, the updating module 302 is configured to calculate the updated intrinsic parameter value through a preset linear constraint relationship according to the following steps:

[0129] Obtaining a first correspondence between the camera image distance, the camera focal length, and the camera object distance, and a second correspondence between the camera image distances within the camera;

[0130] Determine a linear constraint relationship between camera intrinsic parameters, camera focal length, and camera object distance based on the first corresponding relationship and the second corresponding relationship;

[0131] The initial intrinsic parameter value and the changed camera-object distance are used as input parameters of the linear constraint relationship to calculate and obtain an updated intrinsic parameter value.

[0132] In a possible implementation, the above device further includes:

[0133] The positioning module 303 is configured to position the mobile device based on the updated internal parameter value to obtain a positioning result.

[0134] In a possible implementation, the positioning module 303 is configured to locate the mobile device based on the updated internal parameter value according to the following steps to obtain a positioning result:

[0135] The mobile device is positioned based on the updated intrinsic parameter value and the target image currently captured by the camera to obtain a positioning result.

[0136] In one possible implementation, when multiple initial intrinsic parameter values ​​are determined, the updating module 302 is configured to determine an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance according to the following steps:

[0137] Perform sliding window filtering based on multiple initial internal parameter values ​​to obtain processed initial internal parameter values;

[0138] An updated intrinsic parameter value after intrinsic parameter compensation is performed on the camera is determined based on the processed initial intrinsic parameter value and the changed camera-object distance.

[0139] In a possible implementation, the above device further includes:

[0140] The enabling module 304 is configured to enable an online calibration mode of the camera intrinsic parameters in response to the statistical value of the initial intrinsic parameter value corresponding to the current sliding window meeting a preset condition.

[0141] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0142] The present disclosure also provides an electronic device, such as Figure 4 FIG. 4 is a schematic diagram of the electronic device structure provided by an embodiment of the present disclosure, including: a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions (e.g., Figure 3 When the electronic device is running, the processor 401 communicates with the memory 402 via the bus 403. When the machine-readable instructions are executed by the processor 401, the following processing is performed:

[0143] Get the initial intrinsic parameter value of the camera;

[0144] In response to a change in the camera-object distance, an updated intrinsic parameter value after intrinsic parameter compensation is performed on the camera is determined based on the initial intrinsic parameter value and the changed camera-object distance.

[0145] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the image processing method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0146] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0147] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Get the initial intrinsic parameter value of the camera; In response to a change in the camera-object distance, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance; The determining, based on the initial intrinsic parameter value and the changed camera-object distance, an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera includes: Based on the initial intrinsic parameter value and the changed camera-object distance, the updated intrinsic parameter value is calculated through a preset linear constraint relationship, wherein the linear constraint relationship is used to characterize the relationship between the camera intrinsic parameter, the camera focal length, and the camera-object distance; The step of calculating the updated intrinsic parameter value based on the initial intrinsic parameter value and the changed camera-object distance through a preset linear constraint relationship includes: Obtaining a first correspondence between the camera image distance, the camera focal length, and the camera object distance, and a second correspondence between the camera image distances within the camera; Determining a linear constraint relationship among camera intrinsic parameters, camera focal length, and camera object distance based on the first corresponding relationship and the second corresponding relationship; The initial intrinsic parameter value and the changed camera-object distance are used as input parameters of the linear constraint relationship to calculate and obtain an updated intrinsic parameter value.

2. The method according to claim 1, characterized in that The obtaining of the initial intrinsic parameter value of the camera includes: Obtain multiple frames of target images captured by the camera set on the mobile device; An initial intrinsic parameter value of the camera is determined based on the multiple frames of target images.

3. The method according to claim 2, characterized in that The determining of the initial intrinsic parameter value of the camera based on the multiple frames of target images includes: For any two adjacent target image frames in the multiple target image frames, constructing an error function for evaluating positioning accuracy based on a correspondence between an image coordinate system and a world coordinate system; The error function constructed is minimized based on the camera intrinsic parameters to obtain the initial intrinsic parameter values ​​of the camera.

4. The method according to claim 3, characterized in that The error function for evaluating positioning accuracy is constructed based on the correspondence between the image coordinate system and the world coordinate system, including: For a first image point included in a first target image of the two target image frames, determining three-dimensional point representation information after projection of the first pixel point based on a correspondence between a first image coordinate system of the first target image frame and a world coordinate system of the three-dimensional point; Determining second image point representation information after the three-dimensional point is projected based on the three-dimensional point representation information after the first pixel point is projected and a correspondence between a second image coordinate system of the second frame target image and a world coordinate system of the three-dimensional point; A difference operation is performed between the second image point representation information and the value of the second image point to obtain an error function for evaluating positioning accuracy.

5. The method according to claim 2, characterized in that The method further comprises: The mobile device is positioned based on the updated internal parameter value to obtain a positioning result.

6. The method according to claim 5, characterized in that Positioning the mobile device based on the updated internal parameter value to obtain a positioning result includes: The mobile device is positioned based on the updated intrinsic parameter value and the target image currently captured by the camera to obtain a positioning result.

7. The method according to any one of claims 1 to 6, characterized in that: In a case where a plurality of initial intrinsic parameter values ​​are determined, determining an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter values ​​and the changed camera-object distance includes: Performing sliding window filtering based on the multiple initial intrinsic parameter values ​​to obtain processed initial intrinsic parameter values; An updated intrinsic parameter value after intrinsic parameter compensation is performed on the camera is determined based on the processed initial intrinsic parameter value and the changed camera-object distance.

8. The method according to claim 7, characterized in that The method further comprises: In response to the statistical value of the initial intrinsic parameter value corresponding to the current sliding window meeting the preset condition, an online calibration mode of the camera intrinsic parameter is started.

9. An image processing device, characterized in that: include: The acquisition module is used to obtain the initial intrinsic parameter values ​​of the camera; an updating module, configured to determine, in response to a change in the camera-object distance, an updated intrinsic parameter value after performing intrinsic parameter compensation on the camera based on the initial intrinsic parameter value and the changed camera-object distance; The update module is specifically used for: Based on the initial intrinsic parameter value and the changed camera-object distance, the updated intrinsic parameter value is calculated through a preset linear constraint relationship, wherein the linear constraint relationship is used to characterize the relationship between the camera intrinsic parameter, the camera focal length, and the camera-object distance; When the updating module calculates the updated intrinsic parameter value based on the initial intrinsic parameter value and the changed camera-object distance through a preset linear constraint relationship, it is specifically used to: Obtaining a first correspondence between the camera image distance, the camera focal length, and the camera object distance, and a second correspondence between the camera image distances within the camera; Determining a linear constraint relationship among camera intrinsic parameters, camera focal length, and camera object distance based on the first corresponding relationship and the second corresponding relationship; The initial intrinsic parameter value and the changed camera-object distance are used as input parameters of the linear constraint relationship to calculate and obtain an updated intrinsic parameter value.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the image processing method according to any one of claims 1 to 8.

Images