Image motion error processing method, system, equipment and medium

Abstract

The invention relates to the technical field of image processing, and discloses an image motion error processing method, system and device and a medium, and the method comprises the steps: determining a real position and an expected position through the position, corresponding to a selected region in wide-angle imaging, of a long-focus picture in the wide-angle imaging, thereby determining the image motion error of all long-focus regions in the wide-angle imaging; the image motion error is converted into the azimuth angle and the pitch angle of the visual axis under the coordinate system of the two-dimensional pointing mirror, so that the rotation degree required for adjusting the image motion error currently is determined, the image motion error is effectively reduced, image distortion is avoided, complex calculation is not needed, the problem of difficulty in accurate positioning of a visual field expansion system is effectively solved, and the visual field effect is also improved. And fatigue and easy damage are caused by frequent use.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, system, device, and medium for processing image shift error. Background Art

[0002] Expanding the field of view while maintaining high resolution is crucial for various applications. However, vision sensing systems have fundamental limitations in terms of clarity and field of view, and are restricted by the size and resolution of image sensors.

[0003] Existing technologies use image super-resolution methods to improve the resolution of imaging pictures. Mainstream methods are mainly based on image post-processing technology to generate high-resolution images after imaging. Inevitably, some features in the original image are lost during this process, resulting in image distortion. At the same time, image post-processing technology requires a large amount of computing power and is costly. Secondly, these technologies often only solve the resolution problem in one aspect and have extremely limited improvement in the field of view. Summary of the Invention

[0004] In view of this, the present invention provides a method, system, device, and medium for processing image shift error, which solves the technical problems that existing technologies inevitably lose some features in the original image, resulting in image distortion. At the same time, image post-processing technology requires a large amount of computing power and is costly. Secondly, these technologies often only solve the resolution problem in one aspect and have extremely limited improvement in the field of view.

[0005] The first aspect of the present invention provides a method for processing image shift error, including:

[0006] Determine the real position and the expected position according to the position of the telephoto picture within the wide-angle imaging corresponding to the selected area within the wide-angle imaging and the origin position of the telephoto picture;

[0007] Determine the image shift error of all telephoto areas within the wide-angle imaging according to the real position and the expected position;

[0008] Determine the image shift amount of the wide-angle imaging according to the image shift error and the focal length ratio; wherein, the focal length ratio is the ratio of the wide-angle focal length to the telephoto focal length;

[0009] Determine the azimuth angle and the pitch angle of the optical axis in the two-dimensional pointing mirror coordinate system according to the image shift amount of the wide-angle imaging and the wide-angle geometric size;

[0010] Determine the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the azimuth angle and the pitch angle;

[0011] Determine the rotation degrees required to adjust the image shift error currently according to the rotation degrees of the azimuth axis and elevation axis of the two-dimensional pointing mirror, as well as the current rotation degrees.

[0012] Preferably, the determining of the true position and the expected position according to the position of the telephoto picture in the wide-angle imaging corresponding to the selected area in the wide-angle imaging and the origin position of the telephoto picture includes:

[0013] Determine the telephoto pictures corresponding to the respective selected areas according to multiple selected areas in the wide-angle imaging picture collected by the wide-angle camera;

[0014] Establish rectangular coordinate systems for the wide-angle imaging picture and each of the telephoto pictures respectively; wherein, the rectangular coordinate system of the wide-angle imaging picture is established with the upper left corner of the picture as the origin, and the rectangular coordinate system of the telephoto picture is established with the center point of the picture as the origin;

[0015] Determine the expected position according to the origin of the rectangular coordinate system of each of the telephoto pictures;

[0016] Determine the true position according to the position in each of the telephoto pictures corresponding to the selected area in the wide-angle imaging picture.

[0017] Preferably, the determining of the true position according to the position in each of the telephoto pictures corresponding to the selected area in the wide-angle imaging picture includes:

[0018] Mark the position of the selected area in the wide-angle imaging picture as target feature points, and extract the color features of the target feature points;

[0019] Convert the telephoto picture into the HSV color space, and determine the position in the telephoto picture that is consistent with the color features of the target feature points according to the color features, so as to determine the true position.

[0020] Preferably, the image shift amount of the wide-angle imaging includes a horizontal axis image shift amount and a vertical axis image shift amount;

[0021] The determining of the line-of-sight azimuth angle and elevation angle in the two-dimensional pointing mirror coordinate system according to the image shift amount of the wide-angle imaging and the wide-angle geometric size includes:

[0022] Determine the line-of-sight azimuth angle in the two-dimensional pointing mirror coordinate system according to the initial coordinates of the selected area in the wide-angle imaging, the horizontal axis image shift amount, the center coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging, and the wide-angle focal length;

[0023] Determine the pitch angle in the two-dimensional pointing mirror coordinate system based on the initial coordinates of the selected area within the wide-angle imaging, the longitudinal axis image shift amount, the central coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging, and the wide-angle focal length.

[0024] Preferably, the determining the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the line-of-sight azimuth angle and the pitch angle includes:

[0025] Determine the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the line-of-sight azimuth angle and the pitch angle through a degree conversion formula; wherein, the degree conversion formula is:

[0026]

[0027]

[0028] In the formula, A and B respectively represent the line-of-sight azimuth angle and the pitch angle in the two-dimensional pointing mirror coordinate system, and α and β are the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror.

[0029] Preferably, the determining the rotation degrees required for current adjustment of the image shift error according to the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror, and the current rotation degrees includes:

[0030] Determine the initial rotation degrees required for current adjustment of the image shift error according to the difference between the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror and the current rotation degrees;

[0031] Obtain the rotation degrees required for current adjustment of the image shift error by dividing the initial rotation degrees by the preset maximum deflection of the optical angle of the two-dimensional pointing mirror.

[0032] Preferably, the method further includes:

[0033] Construct a training data set according to the pixel coordinates corresponding to the real position and the rotation degrees required for current adjustment of the image shift error corresponding to the pixel coordinates;

[0034] Train a radial basis function neural network through the training data set to obtain an initial rotation degrees prediction model;

[0035] Optimize the network parameters of the initial rotation degrees prediction model through a test data set to obtain an optimized rotation degrees prediction model; wherein, the test data set includes pixel coordinate test samples corresponding to the real position and rotation degree test samples required for current adjustment of the image shift error corresponding to the pixel coordinate test samples.

[0036] In a second aspect, the present invention further provides an image motion error processing system, including:

[0037] A position determination module, configured to determine a true position and an expected position according to the position of a long - focal - length picture within a wide - angle imaging corresponding to a selected area within the wide - angle imaging, and the origin position of the long - focal - length picture;

[0038] An image motion error determination module, configured to determine the image motion errors of all long - focal - length areas within the wide - angle imaging according to the true position and the expected position;

[0039] An image motion amount determination module, configured to determine the image motion amount of the wide - angle imaging according to the image motion error and the focal length ratio; wherein, the focal length ratio is the ratio of the wide - angle focal length to the long - focal - length focal length;

[0040] An angle determination module, configured to determine the azimuth angle and the pitch angle of the optical axis in the two - dimensional pointing mirror coordinate system according to the image motion amount of the wide - angle imaging and the wide - angle geometric size;

[0041] A rotation degree determination module, configured to determine the rotation degrees of the azimuth axis and the pitch axis of the two - dimensional pointing mirror according to the azimuth angle and the pitch angle of the optical axis;

[0042] A rotation degree adjustment module, configured to determine the rotation degrees required for currently adjusting the image motion error according to the rotation degrees of the azimuth axis and the pitch axis of the two - dimensional pointing mirror, and the current rotation degrees.

[0043] In a third aspect, the present invention further provides an electronic device, including a memory and a processor. When a computer program stored in the memory is executed by the processor, the processor is caused to execute the steps of the image motion error processing method as described in the first aspect.

[0044] In a fourth aspect, the present invention further provides a computer - readable storage medium, on which a computer program is stored. When the computer program is executed, the steps of the image motion error processing method as described in the first aspect are implemented.

[0045] As can be seen from the above technical solutions, the present invention determines the true position and the expected position by the position of the long - focal - length picture within the wide - angle imaging corresponding to the selected area within the wide - angle imaging, thereby determining the image motion errors of all long - focal - length areas within the wide - angle imaging. By converting the image motion error into the azimuth angle and the pitch angle of the optical axis in the two - dimensional pointing mirror coordinate system, the rotation degrees required for currently adjusting the image motion error are determined, effectively reducing the image motion error, avoiding image distortion, without complex calculations, effectively solving the problem of difficult precise positioning faced by the field - of - view expansion system, and also improving the field - of - view effect and the problem of easy fatigue and damage during frequent use. Description of the Drawings

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0047] Figure 1 An application environment for an image motion error processing method provided by an embodiment of the present invention;

[0048] Figure 2 A flowchart of an image motion error processing method provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the conversion between a wide-angle coordinate system and the visual axis of a two-dimensional pointing mirror provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of an error function fitting curve under the azimuth axis provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of an error function fitting curve under the pitch axis provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the structure of an image motion error processing system provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0054] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0055] The image motion error processing method provided by the embodiments of the present application can be used for fitting image motion errors in a field of view expansion system. The field of view expansion system includes a wide-angle camera, a telephoto camera, and a two-dimensional pointing mirror with a pan-tilt head and a controller. Both the wide-angle camera and the telephoto camera are equipped with liquid lenses that can quickly zoom. The wide-angle camera has the characteristics of a large field of view but low resolution, and the telephoto camera has the characteristics of high resolution but a small field of view. The two-dimensional pointing mirror can rotate at a speed of milliseconds. The working principle of this system is that a picture with a large field of view but low resolution is taken by the wide-angle camera, and then the two-dimensional pointing mirror controlled by the motor can rotate in the horizontal and pitch directions to control the optical path entering the telephoto camera, so that the field of view of the telephoto camera can cover the entire field of view of the wide-angle camera, and the interesting target in the field of view can be quickly aligned.

[0056] Although the field of view expansion system has the advantages of fast response and high-definition imaging, there are still the following deficiencies: The two-dimensional pointing mirror can rotate quickly, but there is a small image motion error in the swing accuracy, the camera uses a rough calibration method such as the nine-point calibration method, there is a certain image motion error, and there is an image motion error caused by the non-coaxiality of the wide-angle camera and the two-dimensional pointing mirror. These errors are all reflected in the imaging of the telephoto camera. For example, the red dot is the target point corresponding to the wide-angle camera, but after being adjusted by the two-dimensional pointing mirror, there is an image motion error and it deviates from the expected imaging center.

[0057] Based on this, the image motion error processing method provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 101 communicates with the server 102 through the network. The data storage system can store the data that the server 102 needs to process. The data storage system can be integrated on the server 102, or can be placed on the cloud or other network servers. The terminal 101 or the server 102 determines the real position and the expected position according to the position of the telephoto picture in the wide-angle imaging corresponding to the selected area in the wide-angle imaging and the origin position of the telephoto picture; determines the image motion errors of all telephoto areas in the wide-angle imaging according to the real position and the expected position; determines the image motion amount of the wide-angle imaging according to the image motion error and the focal length ratio, where the focal length ratio is the ratio of the wide-angle focal length to the telephoto focal length; determines the azimuth angle and pitch angle of the optical axis in the coordinate system of the two-dimensional pointing mirror according to the image motion amount of the wide-angle imaging and the wide-angle geometric size; determines the rotation degrees of the azimuth axis and pitch axis of the two-dimensional pointing mirror according to the azimuth angle and pitch angle; determines the rotation degrees required to adjust the current image motion error according to the rotation degrees of the azimuth axis and pitch axis of the two-dimensional pointing mirror and the current rotation degrees.

[0058] The terminal 101 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, etc.

[0059] The server 102 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0060] As Figure 2 shown, an embodiment of the present application provides a method for processing image shift error. Taking the method applied to Figure 1 the terminal 101 or the server 102 in

[0061] as an example, it includes the following steps S1 to S6. Among them:

[0062] Since it is difficult to analyze and solve the cause of each image shift error in isolation from a single angle. An embodiment of the present application takes a global large-field low-resolution image captured by a wide-angle camera as a prior, and then selects N*N regions from the wide-angle imaging to capture corresponding long-focus images. Through the wide-angle imaging picture captured by the wide-angle camera, the selected regions are identified, and these selected regions are matched with the long-focus images taken by the long-focus camera.

[0063] Among them, the origin position of the long-focus image is used as a reference point to determine the expected position. The real position is the actual position corresponding to the selected region in the wide-angle imaging picture in the long-focus image. By comparing the real position and the expected position, the magnitude and direction of the image shift error can be quantified.

[0064] Step S2, determine the image shift error of all long-focus regions in the wide-angle imaging according to the real position and the expected position.

[0065] Among them, by calculating the coordinate offset difference between the real position and the expected position, the image shift error of each long-focus region is obtained. These error values reflect the image shift error generated during the rotation of the two-dimensional steering mirror, the image shift error caused by camera calibration, and the image shift error caused by the non-coaxiality of the wide-angle camera and the two-dimensional steering mirror.

[0066] Step S3, determine the image shift amount of the wide-angle imaging according to the image shift error and the focal length ratio; where the focal length ratio is the ratio of the wide-angle focal length to the long-focus focal length.

[0067] Among them, the image shift error is scaled by the focal length ratio to obtain the image shift amount of the wide-angle imaging relative to the long-focus imaging. Since the focal lengths of the wide-angle camera and the long-focus camera are different, it is necessary to convert the image shift error of the long-focus region into the image shift amount under the wide-angle imaging for subsequent processing. The calculation of the focal length ratio can be solved through the known values of the wide-angle focal length and the long-focus focal length, so as to ensure the accuracy of the image shift amount.

[0068] Specifically, according to the image shift error and the focal length ratio, the image shift amount for wide-angle imaging is determined as follows:

[0069]

[0070]

[0071] In the formula, X and Y are the horizontal-axis image shift amount and the vertical-axis image shift amount in the wide-angle coordinate system respectively, is the focal length of the wide-angle camera, is the focal length of the telephoto camera, and x and y are the image shift error in the horizontal-axis direction and the image shift error in the vertical-axis direction respectively.

[0072] Step S4: Determine the azimuth angle and elevation angle of the optical axis in the two-dimensional steering mirror coordinate system according to the image shift amount of wide-angle imaging and the wide-angle geometric size.

[0073] Among them, the image shift amount of wide-angle imaging includes the horizontal-axis image shift amount and the vertical-axis image shift amount;

[0074] Among them, determining the azimuth angle and elevation angle of the optical axis in the two-dimensional steering mirror coordinate system according to the image shift amount of wide-angle imaging and the wide-angle geometric size includes:

[0075] Step S401: Determine the azimuth angle of the optical axis in the two-dimensional steering mirror coordinate system according to the initial coordinates of the selected area in wide-angle imaging, the horizontal-axis image shift amount, the center coordinates of the wide-angle pixel coordinate system of wide-angle imaging, the pixel size of wide-angle imaging, and the wide-angle focal length.

[0076] Step S402: Determine the elevation angle of the optical axis in the two-dimensional steering mirror coordinate system according to the initial coordinates of the selected area in wide-angle imaging, the vertical-axis image shift amount, the center coordinates of the wide-angle pixel coordinate system of wide-angle imaging, the pixel size of wide-angle imaging, and the wide-angle focal length.

[0077] Among them, as Figure 3 shown in the conversion process of the wide-angle coordinate system and the two-dimensional steering mirror optical axis, it is converted into the corresponding azimuth angle and elevation angle of the two-dimensional steering mirror coordinate system by using the following formula:

[0078]

[0079]

[0080] In the formula, X1 and Y1 are the initial abscissa and initial ordinate of the N*N area selected in the wide-angle camera, , are the central abscissa and central ordinate of the wide-angle pixel coordinate system, k is the pixel size of the wide-angle camera, and A and B are the azimuth angle and elevation angle of the optical axis of the two-dimensional steering mirror respectively.

[0081] Step S5: Determine the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror according to the azimuth angle and the elevation angle of the line of sight.

[0082] Among them, through a pre-set correspondence table or conversion formula between the rotation degrees and the azimuth angle and the elevation angle of the line of sight, the calculated azimuth angle and elevation angle of the line of sight are converted into the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror, so as to ensure that the two-dimensional pointing mirror can be accurately rotated to the expected position, which is the key to eliminating the image motion error.

[0083] Among them, the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror are determined according to the azimuth angle and the elevation angle of the line of sight through a degree conversion formula; among them, the degree conversion formula is:

[0084]

[0085]

[0086] In the formula, A and B respectively represent the azimuth angle and the elevation angle of the line of sight in the coordinate system of the two-dimensional pointing mirror, and α and β are the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror.

[0087] Step S6: Determine the rotation degrees required to adjust the current image motion error according to the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror, and the current rotation degrees.

[0088] Among them, by calculating the difference between the target rotation degrees required for the azimuth axis and the elevation axis of the two-dimensional pointing mirror and the current rotation degrees, the rotation degrees that need to be adjusted are determined, which is the key to accurately controlling the rotation of the two-dimensional pointing mirror to achieve image motion error correction. It ensures that the two-dimensional pointing mirror can be quickly and accurately adjusted in place, thereby effectively reducing or eliminating the image motion error and improving the imaging quality and system performance.

[0089] It should be noted that in the embodiment of the present application, the true position and the expected position are determined by the position of the long-focus picture in the wide-angle imaging corresponding to the selected area in the wide-angle imaging, so as to determine the image motion error of all long-focus areas in the wide-angle imaging. The image motion error is converted into the azimuth angle and the elevation angle of the line of sight in the coordinate system of the two-dimensional pointing mirror, so as to determine the rotation degrees required to adjust the current image motion error, effectively reducing the image motion error, avoiding image distortion, without complex calculations, effectively solving the problem of difficult precise positioning faced by the field of view expansion system, and also improving the field of view effect and the problem of easy fatigue and damage during frequent use.

[0090] In some embodiments, determining the true position and the expected position according to the position of the long-focus picture in the wide-angle imaging corresponding to the selected area in the wide-angle imaging, and the origin position of the long-focus picture, includes:

[0091] Step S101: Determine the telephoto pictures corresponding to each selected area based on multiple selected areas in the wide-angle imaging pictures captured by the wide-angle camera.

[0092] Among them, the wide-angle imaging pictures are captured by the wide-angle camera, and the telephoto pictures corresponding to each selected area in the wide-angle imaging pictures are determined through the selected areas in the wide-angle imaging pictures.

[0093] Step S102: Establish a rectangular coordinate system for the wide-angle imaging picture and each telephoto picture respectively; among them, the rectangular coordinate system of the wide-angle imaging picture is established with the upper left corner of the picture as the origin, and the rectangular coordinate system of the telephoto picture is established with the center point of the picture as the origin.

[0094] Among them, establishing a coordinate system for the two groups of pictures and taking the center point of the telephoto picture as the origin to establish the rectangular coordinate system of the telephoto picture helps to more accurately describe the position of the target point in the telephoto picture. By establishing the rectangular coordinate systems of the wide-angle imaging picture and the telephoto picture, the coordinate offset difference, that is, the image shift error, between the real position and the expected position can be conveniently calculated.

[0095] Step S103: Determine the expected position according to the origin of the rectangular coordinate system of each telephoto picture.

[0096] Among them, taking the origin of the rectangular coordinate system of each telephoto picture as the expected position, and the expected position refers to the position where the telephoto picture should be located in the corresponding selected area of the wide-angle imaging picture without image shift error. By setting the origin position of the telephoto picture as the expected position, a reference point can be provided for the subsequent calculation of the image shift error.

[0097] Step S104: Determine the real position according to the position in each telephoto picture corresponding to the selected area in the wide-angle imaging picture.

[0098] Among them, by matching the target points in the telephoto picture with the positions of the selected areas in the wide-angle imaging picture, the actual position of the telephoto picture in the wide-angle imaging picture, that is, the real position, is determined. The real position reflects the actual imaging position of the telephoto picture after being affected by the image shift error.

[0099] In some embodiments, in order to more quickly determine the real position, an image algorithm is used to extract color features for assisting in statistically determining the real position. Therefore, determining the real position according to the position in each telephoto picture corresponding to the selected area in the wide-angle imaging picture includes:

[0100] Step S1041: Mark the positions of the selected areas in the wide-angle imaging picture as target feature points, and extract the color features of the target feature points;

[0101] Step S1042: Convert the telephoto picture to the HSV color space, determine the position in the telephoto picture that is consistent with the color feature of the target feature point according to the color feature, and determine the true position.

[0102] Exemplarily, for a telephoto picture, when it is collected, a red dot is automatically marked at the true position T. When we count data, convert the telephoto picture from the RGB color space to the HSV color space. Because in the HSV color space, the similarity and difference between colors can be more easily measured and analyzed, this characteristic makes the HSV color space more advantageous in color feature extraction. Among them, the color feature of the target feature point includes brightness, saturation, and hue. The conversion processes of brightness, saturation, and hue are as follows:

[0103] Among them, the brightness V represents the brightness of the color, expressed as:

[0104]

[0105] In the formula, R, G, and B are the red component, green component, and blue component respectively.

[0106] The saturation S represents the purity or depth of the color, expressed as:

[0107]

[0108] The hue represents the basic attribute of the color and is an angular measure, expressed as:

[0109]

[0110] .

[0111] In some embodiments, according to the rotation degrees of the azimuth axis and elevation axis of the two-dimensional pointing mirror, and the current rotation degree, determine the rotation degree required to adjust the current image shift error, including:

[0112] Step S601: According to the rotation degrees of the azimuth axis and elevation axis of the two-dimensional pointing mirror, and the difference between the current rotation degrees, determine the initial rotation degree required to adjust the current image shift error.

[0113] Among them, since there will be an initial rotation degree during the process that the telephoto lens can see the target due to the swing of the two-dimensional pointing mirror, this initial rotation degree is used as the rotation degree corresponding to the expected position, that is, the current rotation degree.

[0114] By calculating the difference between the rotation degrees of the azimuth axis and elevation axis of the two-dimensional pointing mirror and the current rotation degree, determine the initial rotation degree required to adjust the current image shift error.

[0115] Step S602: Obtain the rotation degree required for current adjustment of image shift error by presetting the maximum deflection of the optical angle of the two-dimensional pointing mirror for the initial rotation degree.

[0116] Among them, due to the physical characteristics of the two-dimensional pointing mirror itself, a coordinate system can be defined to calibrate the internal optical feedback mechanism and associate it with the physical position of the two-dimensional pointing mirror. This coordinate system is a Cartesian coordinate system with an X-axis and a Y-axis. The X-axis is perpendicular to the cable extending from the lens, and the Y-axis is parallel to the cable. The values on the axis have no unit and are defined by the maximum deflection of the optical angle of the mirror (i.e., 50°). The conversion relationship between the rotation degrees of the azimuth angle and the pitch angle and the coordinate values along the x-axis is as follows:

[0117]

[0118] In the formula, R i (i = α, β) is the rotation component adjusted by the two-dimensional pointing mirror control system, (i = α, β) are the initial rotation degrees of the azimuth axis and the pitch axis respectively.

[0119] Among them, according to R i The rotation component can control the rotation of the two-dimensional pointing mirror to adjust the image shift error.

[0120] In some embodiments, in order to quickly adjust the image shift error, this method further includes:

[0121] Step S701: Construct a training dataset according to the pixel coordinates corresponding to the real position and the rotation degree required for current adjustment of image shift error corresponding to the pixel coordinates.

[0122] Among them, the mapping relationship is determined by the pixel coordinates corresponding to the real position and the rotation degree required for current adjustment of image shift error obtained through the foregoing steps S1~S6, and a training dataset is constructed based on the mapping relationship.

[0123] Step S702: Train a radial basis function neural network with the training dataset to obtain an initial rotation degree prediction model.

[0124] Among them, the radial basis function neural network learns the complex mapping relationship between the real position and the required rotation degree through the training dataset. This model can receive the pixel coordinates of the real position as input and quickly output the predicted rotation degree, thus greatly accelerating the adjustment process of the image shift error.

[0125] The Radial Basis Function (RBF) neural network is a feedforward neural network that consists of an input layer, a hidden layer, and an output layer. According to different methods of determining the centers of the radial basis functions, the RBF neural network has different learning strategies. We adopt the method of randomly selecting fixed centers. The radial basis function is denoted as:

[0126]

[0127] In the formula, is the radial basis function, are the input and output of the radial basis function respectively.

[0128] In the embodiments of this application, the Gaussian function is selected as the basis function. The activation function is:

[0129]

[0130] In the formula, is the activation function, is the i-th input sample, is the center of the i-th hidden layer radial basis function, is the field width of the i-th hidden layer.

[0131] In the RBF network with randomly selected fixed centers, the parameters to be trained are: the centers of the basis functions in the hidden layer; the standard deviations of the basis functions in the hidden layer; the weights between the hidden layer and the output layer.

[0132] In the method of randomly selecting fixed centers, both the centers and standard deviations of the basis functions are fixed. The only parameter to be trained is the weight between the hidden layer and the output layer. The centers of the hidden layer basis functions are randomly selected from the input sample data and remain fixed. The standard deviation of the central basis function is determined as follows:

[0133]

[0134] In the formula, is the maximum distance between the selected centers. n is the number of hidden function nodes. This is to prevent the radial basis function from being too sharp or too flat. After determining the centers and standard deviations, the basis function is obtained:

[0135]

[0136] In the formula, is the basis function.

[0137] The pseudo-inverse method is used to solve the output weights. Assume Y = is the expected output, which is the expected output value of the k-th input vector at the i-th output node. W = Let \(w_{ij}\) be the weight from the \(i\)-th hidden node to the \(j\)-th output node. Then the output weight matrix \(W\) can be obtained by the following formula:

[0138]

[0139] where \(G = Substitute the training data into the above formula and the output weight matrix \(W\) to solve, and the predicted value is obtained:

[0140]

[0141] where is the weight from the \(i\)-th node in the hidden layer to the \(j\)-th node in the output layer, \(h\) is the number of nodes, and is the output.

[0142] The error function can be fitted through the image shift error, respectively as Figures 4 to 5 .

[0143] Step S703: Optimize the network parameters of the initial rotation degree prediction model through the test data set to obtain the optimized rotation degree prediction model; wherein, the test data set includes the pixel coordinate test samples corresponding to the real position, and the rotation degree test samples corresponding to the current adjusted image shift error of the pixel coordinate test samples.

[0144] Among them, in order to evaluate the goodness of the fitting effect, this patent introduces the coefficient of determination as an evaluation index. The coefficient of determination (Goodness of Fit) refers to the fitting degree of the regression curve to the observed values. The statistic for measuring the goodness of fit is the coefficient of determination (also known as the determination coefficient) \(R^{2}\). The maximum value of \(R^{2}\) is 1. The closer the value of \(R^{2}\) is to 1, the better the fitting degree of the regression curve to the observed values; conversely, the worse the fitting degree of the regression curve to the observed values. Its calculation formula is as follows:

[0145]

[0146] where is the data to be fitted, the mean of is . According to the above steps, the image shift error \((x,y)\) on the telephoto image, the image shift amount \((X,Y)\) on the wide-angle camera image, and the rotation amounts \((α,β)\) of the azimuth axis and elevation axis of the two-dimensional pointing mirror obtained in the above experimental process can be input into different angular representations of the fitting error function of the above RBF neural network respectively to verify the effectiveness and robustness of the network fitting.

[0147] When it is verified that the fitting effect of the network is poor, then , , thereby optimizing the rotation degree prediction model.

[0148] Through the above experiments and experimental results. The embodiment of the present application has completed the research and analysis of the image shift error caused by the two-dimensional pointing mirror in the field of view expansion system, and has effectively reduced the error. The field of view expansion system has the ability to align the target more accurately and quickly, meeting the application of more production and security fields. At the same time, the embodiment of the present application is aimed at the problem that the prior art cannot take into account both large field of view and high resolution at the same time when expanding the field of view, and can make full use of the advantages of wide-angle lenses and telephoto lenses to achieve the characteristics of taking into account both large field of view and high resolution at the same time. However, when using a two-dimensional pointing lens for field of view expansion, there is an image shift error, which makes the telephoto camera unable to align the target. The embodiment of the present application analyzes the image shift characteristics of the field of view expansion system during imaging from the perspective of imaging results, and uses a neural network to fit the image shift error characteristics of the field of view expansion system, and evaluates the fitting effect from multiple angles to verify the effectiveness and robustness of the fitting method.

[0149] Based on the same inventive concept, an embodiment of the present application also provides an image motion error processing system for implementing the above-mentioned image motion error processing method.

[0150] The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above method. Therefore, the specific limitations in one or more image motion error processing system embodiments provided below can be referred to the limitations on the image motion error processing method above and will not be repeated here.

[0151] like Figure 6 As shown, an embodiment of the present application provides an image motion error processing system, comprising:

[0152] A position determination module 100, for determining a real position and an expected position according to a position of a telephoto picture in a wide-angle image corresponding to a selected area in the wide-angle image and an origin position of the telephoto picture;

[0153] An image motion error determination module 200 is used to determine the image motion errors of all telephoto areas in the wide-angle imaging according to the actual position and the expected position;

[0154] The image motion determination module 300 is used to determine the image motion of wide-angle imaging according to the image motion error and the focal length ratio; wherein the focal length ratio is the ratio of the wide-angle focal length to the telephoto focal length;

[0155] Angle determination module 400, used to determine the azimuth angle and elevation angle of the visual axis in the two-dimensional pointing mirror coordinate system according to the image shift and wide-angle geometric dimensions of the wide-angle imaging;

[0156] The rotation degree determination module 500 is configured to determine the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror according to the visual axis azimuth angle and the elevation angle;

[0157] The rotation degree adjustment module 600 is configured to determine the rotation degrees required to adjust the current image shift error according to the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror and the current rotation degrees.

[0158] In some embodiments, the position determination module 100 is specifically configured to:

[0159] Determine the corresponding telephoto pictures for each selected area in the wide-angle imaging picture collected by the wide-angle camera;

[0160] Establish a rectangular coordinate system for the wide-angle imaging picture and each telephoto picture respectively; among them, the rectangular coordinate system of the wide-angle imaging picture is established with the upper left corner of the picture as the origin, and the rectangular coordinate system of the telephoto picture is established with the center point of the picture as the origin;

[0161] Determine the expected position according to the origin of the rectangular coordinate system of each telephoto picture;

[0162] Determine the actual position according to the position in each telephoto picture corresponding to the selected area in the wide-angle imaging picture.

[0163] In some embodiments, determining the actual position according to the position in each telephoto picture corresponding to the selected area in the wide-angle imaging picture includes:

[0164] Mark the position of the selected area in the wide-angle imaging picture as a target feature point, and extract the color feature of the target feature point;

[0165] Convert the telephoto picture to the HSV color space, and determine the position in the telephoto picture that is consistent with the color feature of the target feature point according to the color feature, so as to determine the actual position.

[0166] In some embodiments, the image shift amount of the wide-angle imaging includes a horizontal axis image shift amount and a vertical axis image shift amount;

[0167] The angle determination module 400 is configured to:

[0168] Determine the visual axis azimuth angle in the two-dimensional pointing mirror coordinate system according to the initial coordinates of the selected area in the wide-angle imaging, the horizontal axis image shift amount, the center coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging, and the wide-angle focal length;

[0169] Determine the elevation angle in the two-dimensional pointing mirror coordinate system according to the initial coordinates of the selected area in the wide-angle imaging, the vertical axis image shift amount, the center coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging, and the wide-angle focal length.

[0170] In some embodiments, a rotation degree determination module 500 is configured to:

[0171] Determine the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the line-of-sight azimuth angle and the pitch angle through a degree conversion formula; wherein, the degree conversion formula is:

[0172]

[0173]

[0174] In the formula, A and B respectively represent the line-of-sight azimuth angle and the pitch angle in the two-dimensional pointing mirror coordinate system, and α and β are the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror.

[0175] In some embodiments, a rotation degree adjustment module 600 is configured to:

[0176] Determine the initial rotation degree required to adjust the current image shift error according to the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror and the difference between the current rotation degrees;

[0177] Obtain the rotation degree required to adjust the current image shift error by dividing the initial rotation degree by the maximum deflection preset for the optical angle of the two-dimensional pointing mirror.

[0178] In some embodiments, the system further includes: a prediction model training module, configured to:

[0179] Construct a training data set according to the pixel coordinates corresponding to the real position and the rotation degrees required to adjust the current image shift error corresponding to the pixel coordinates;

[0180] Train a radial basis function neural network through the training data set to obtain an initial rotation degree prediction model;

[0181] Optimize the network parameters of the initial rotation degree prediction model through a test data set to obtain an optimized rotation degree prediction model; wherein, the test data set includes pixel coordinate test samples corresponding to the real position and rotation degree test samples required to adjust the current image shift error corresponding to the pixel coordinate test samples.

[0182] As Figure 7 shown, an embodiment of the present application further provides an electronic device. The electronic device 10 includes a memory 20 and a processor 30. When a computer program stored in the memory 20 is executed by the processor 30, the processor 30 is caused to execute the steps of the image shift error processing method in the above embodiments.

[0183] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the steps of the image shift error processing method in the above embodiments are implemented.

[0184] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system, electronic device, and computer storage medium can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0185] It should be noted that the terms "including" and "having" in the specification and claims of the present invention and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0186] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless clearly stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0187] In several embodiments provided by the present invention, it should be understood that the disclosed system, electronic device, computer storage medium, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, indirect couplings or communication connections of devices or units, and can be in electrical, mechanical, or other forms.

[0188] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (English full name: Read-Only Memory, English abbreviation: ROM), random access memories (English full name: Random Access Memory, English abbreviation: RAM), magnetic disks or optical discs and other various media that can store program codes.

[0191] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A method for processing image motion errors, characterized in that: include: Determine the actual position and the expected position according to the position of the telephoto picture in the wide-angle image corresponding to the selected area in the wide-angle image and the origin position of the telephoto picture; Determining image shift errors of all telephoto areas within the wide-angle imaging according to the actual position and the expected position; Determining the image shift amount of wide-angle imaging according to the image shift error and the focal length ratio; wherein the focal length ratio is the ratio of the wide-angle focal length to the telephoto focal length; Determining the azimuth and elevation angles of the viewing axis in a two-dimensional pointing mirror coordinate system according to the image shift and wide-angle geometric dimensions of the wide-angle imaging; Determining the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the visual axis azimuth angle and the pitch angle; The rotation degree required for currently adjusting the image shift error is determined according to the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror and the current rotation degree.

2. The image motion error processing method according to claim 1, characterized in that: The determining of the actual position and the expected position according to the position of the telephoto picture in the wide-angle image corresponding to the selected area in the wide-angle image and the origin position of the telephoto picture comprises: Determine, according to a plurality of selected areas in the wide-angle imaging picture collected by the wide-angle camera, a telephoto picture corresponding to each of the selected areas; Establishing rectangular coordinate systems for the wide-angle imaging picture and each of the telephoto pictures respectively; wherein the rectangular coordinate system of the wide-angle imaging picture is established with the upper left corner of the picture as the origin, and the rectangular coordinate system of the telephoto picture is established with the center point of the picture as the origin; Determining an expected position according to the origin of the rectangular coordinate system of each of the telephoto images; The real position is determined according to the position of each of the telephoto pictures corresponding to the selected area in the wide-angle imaging picture.

3. The image motion error processing method according to claim 2, characterized in that: Determining the real position according to the position of each of the telephoto pictures corresponding to the selected area in the wide-angle imaging picture includes: Marking the position of the selected area in the wide-angle imaging picture as a target feature point, and extracting the color features of the target feature point; The telephoto picture is converted into the HSV color space, and a position consistent with the color feature of the target feature point is determined in the telephoto picture according to the color feature to determine the real position.

4. The image motion error processing method according to claim 1, characterized in that: The image shift of the wide-angle imaging includes a horizontal axis image shift and a vertical axis image shift; Determining the visual axis azimuth and elevation angle in a two-dimensional pointing mirror coordinate system according to the image shift and wide-angle geometric dimensions of the wide-angle imaging includes: Determine the visual axis azimuth in the two-dimensional pointing mirror coordinate system according to the initial coordinates of the selected area in the wide-angle imaging, the transverse axis image shift, the central coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging and the wide-angle focal length; The pitch angle in the two-dimensional pointing mirror coordinate system is determined according to the initial coordinates of the selected area in the wide-angle imaging, the longitudinal axis image shift, the central coordinates of the wide-angle pixel coordinate system of the wide-angle imaging, the pixel size of the wide-angle imaging and the wide-angle focal length.

5. The image motion error processing method according to claim 1, characterized in that: Determining the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the visual axis azimuth angle and the pitch angle includes: The rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror are determined by a degree conversion formula according to the visual axis azimuth angle and the pitch angle; wherein the degree conversion formula is: Where A and B represent the azimuth and elevation angles of the line of sight in the two-dimensional pointing mirror coordinate system, respectively; α and β are the rotation degrees of the azimuth and elevation axes of the two-dimensional pointing mirror.

6. The image motion error processing method according to claim 1, characterized in that: Determining the rotation degree required for adjusting the image shift error according to the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror and the current rotation degree includes: Determining an initial rotation degree required for adjusting the image shift error according to the rotation degrees of the azimuth axis and the elevation axis of the two-dimensional pointing mirror and the difference between the current rotation degrees; The maximum deflection preset by the optical angle of the two-dimensional pointing mirror is adjusted to the initial rotation degree to obtain the rotation degree currently required for adjusting the image shift error.

7. The image motion error processing method according to claim 1, characterized in that: Also includes: Constructing a training data set according to the pixel coordinates corresponding to the real position and the rotation degree currently required to adjust the image motion error corresponding to the pixel coordinates; The radial basis function neural network is trained by using the training data set to obtain an initial rotation degree prediction model; The network parameters of the initial rotation degree prediction model are optimized through a test data set to obtain an optimized rotation degree prediction model; wherein the test data set includes a pixel coordinate test sample corresponding to the real position, and a rotation degree test sample corresponding to the pixel coordinate test sample and required for currently adjusting the image motion error.

8. An image motion error processing system, characterized in that: include: A position determination module, for determining a real position and an expected position according to a position of a telephoto picture in a wide-angle image corresponding to a selected area in the wide-angle image and an origin position of the telephoto picture; An image motion error determination module, used to determine the image motion errors of all telephoto areas in the wide-angle imaging according to the actual position and the expected position; An image motion determination module, used to determine the image motion of wide-angle imaging according to the image motion error and the focal length ratio; wherein the focal length ratio is the ratio of the wide-angle focal length to the telephoto focal length; An angle determination module, used to determine the azimuth angle and the elevation angle of the visual axis in the two-dimensional pointing mirror coordinate system according to the image shift and the wide-angle geometric dimensions of the wide-angle imaging; A rotation degree determination module, used to determine the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror according to the visual axis azimuth angle and the pitch angle; The rotation degree adjustment module is used to determine the rotation degree required for adjusting the image shift error according to the rotation degrees of the azimuth axis and the pitch axis of the two-dimensional pointing mirror and the current rotation degree.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the image motion error processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the image motion error processing method as described in any one of claims 1 to 7 are implemented.

Images