Image processing method, device and medium

Abstract

The embodiment of the present invention provides an image processing method and device, and a device for image processing, wherein the method specifically comprises: receiving a sequence frame image acquired by an image acquisition device; performing a first image registration on the sequence frame image according to an image feature method to obtain a corresponding registration result; if the registration result is a registration failure, performing a second image registration on the sequence frame image according to an image template method. The embodiment of the present invention can improve the accuracy of image registration.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image processing method and apparatus, and a machine-readable medium. Background Art

[0002] As a learning tool that can realize self-service word lookup, translation, and oral English learning, the dictionary pen has been gradually applied to learning scenarios in recent years, and can provide a fast word lookup experience in learning scenarios where mobile phones are restricted.

[0003] The dictionary pen first uses the pen tip camera to scan the text, and then outputs a spliced line of text images for image recognition. Since the field of view of the pen tip camera is relatively narrow, the scanned text images are usually sequential frame images; therefore, in order to obtain a complete line of text images, image stitching technology is required.

[0004] In order to obtain better stitching effects, the current image stitching technology usually first performs image preprocessing on the sequential frame images, and then stitches the preprocessed sequential frame images. The above image preprocessing uses operations such as binarization and dilation to estimate the tilt angle of the text in the sequential frame images, and uses the estimation results to correct the sequential frame images; in this way, the above correction can overcome the text distortion problem in the sequential frame images, and send the corrected sequential frame images to the image stitching process, which can improve the flatness of the stitched images.

[0005] The inventors found in the process of implementing the embodiments of the present invention that in the current image stitching technology, in backgrounds such as white background with black characters or black background with white characters, the text and the background are clearly distinguishable, and better preprocessing effects and image stitching effects can be obtained; however, in complex backgrounds such as picture books, the text and the background are not clearly distinguishable, and both the obtained preprocessing results and image stitching effects are poor.

[0006] Moreover, the current stitching technology usually includes: image registration and image fusion. The commonly used image registration method is usually the method of image features. However, this method depends on the texture information of the image. If the texture information of the image is rich, more image features can be extracted; if there are many blanks or less texture information in the image, few image feature points can be extracted, and the accuracy of image registration is low. For example, in the case of a large blank or many simple-structured characters (such as the character "one") in the text image, the method of image features will obtain poor image registration results. Summary of the Invention

[0007] In view of the above problems, embodiments of the present invention are proposed to provide an image processing method, an image processing apparatus, and an apparatus for image processing that can overcome the above problems or at least partially solve the above problems, and the embodiments of the present invention can improve the accuracy of image registration.

[0008] In order to solve the above problems, the present invention discloses an image processing method, comprising:

[0009] Receiving a sequence of frame images acquired by an image acquisition device;

[0010] Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0011] If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0012] On the other hand, an embodiment of the present invention discloses an image processing device, including:

[0013] A receiving module, used for receiving a sequence of frame images collected by an image acquisition device;

[0014] A first image registration module, used for performing first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0015] The second image registration module is used to perform second image registration on the sequence frame images according to an image template method if the registration result is a registration failure.

[0016] In another aspect, an embodiment of the present invention discloses an apparatus for image processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0017] Receiving a sequence of frame images acquired by an image acquisition device;

[0018] Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0019] If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0020] The embodiment of the present invention further discloses one or more machine-readable media, characterized in that instructions are stored thereon, which, when executed by one or more processors, enable the device to execute the aforementioned method.

[0021] The embodiments of the present invention include the following advantages:

[0022] The embodiment of the present invention directly performs image registration on the sequence frame images collected by the image acquisition device. Since the embodiment of the present invention does not perform pre-processing operations such as binarization and dilation on the sequence frame images before image registration, the embodiment of the present invention can overcome the problem of inaccurate image registration caused by inaccurate pre-processing effects in complex backgrounds such as picture books, and improve the accuracy of image registration.

[0023] Furthermore, the embodiment of the present application combines the image feature method and the image template method to perform image registration on the frame images. Specifically, firstly, according to the image feature method, the first image registration is performed on the sequence frame images to obtain the corresponding registration result; then, when the registration result is a registration failure, according to the image template method, the second image registration is performed on the sequence frame images.

[0024] Since a failed registration result may be obtained when there are large blank spaces or more characters with simple structures in the text image, the embodiment of the present invention performs a second image registration on the sequence frame images based on the image template method in this case, which can improve the accuracy of image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flowchart of the steps of a first embodiment of an image processing method of the present invention;

[0026] Figure 2 is a flowchart of a method for correcting a text image according to an embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of pairing upper and lower key points corresponding to a text image in an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a corrected image according to an embodiment of the present invention;

[0029] Figure 5 It is a schematic diagram of a text image, a rectified image, and a mapping relationship between a source point and a target point according to an embodiment of the present invention;

[0030] Figure 6 is a flowchart of the steps of Embodiment 3 of an image processing method of the present invention;

[0031] Figure 7 It is a schematic diagram of a prediction process of the pairing of the first, last, upper and lower key points in an embodiment of the present invention;

[0032] Figure 8 is a structural block diagram of an image processing device of the present invention;

[0033] Fig. 9is a block diagram of an apparatus 1300 for image processing according to the present invention; and

[0034] Fig.10 It is a structural schematic diagram of a server of the present invention. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The embodiment of the present invention can be used to process sequence frame images. The embodiment of the present invention can accurately register the sequence frame images to obtain accurate registration parameters. Moreover, the embodiment of the present invention can apply accurate configuration parameters to the splicing of sequence frame images to improve the splicing accuracy of the sequence frame images.

[0037] Image stitching technology specifically includes: image registration and image fusion. Image registration is the process of matching and superimposing at least two frames of images acquired at different times, different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.). The result of image registration may include: motion transformation parameters between two adjacent frames of images. Image fusion is to synthesize at least two frames of images according to the motion transformation parameters between the images to obtain a complete image.

[0038] The embodiment of the present invention does not limit the application scenario corresponding to the sequence frame image. The application scenario can be any scenario that can generate a sequence frame image, such as a learning assistance scenario of a dictionary pen, or a photo translation scenario of a mobile phone. Taking the dictionary pen as an example, a user can scan a text with the dictionary pen to obtain information such as the pronunciation, interpretation, and translation of the text.

[0039] An embodiment of the present invention provides an image processing solution, which specifically includes: receiving a sequence frame image collected by an image acquisition device; performing a first image registration on the above sequence frame images according to an image feature method to obtain a corresponding registration result; if the above registration result is a registration failure, performing a second image registration on the above sequence frame images according to an image template method.

[0040] The embodiment of the present invention directly performs image registration on the sequence frame images collected by the image acquisition device. Since the embodiment of the present invention does not perform pre-processing operations such as binarization and dilation on the sequence frame images before image registration, the embodiment of the present invention can overcome the problem of inaccurate image registration caused by inaccurate pre-processing effects in complex backgrounds such as picture books, and improve the accuracy of image registration.

[0041] In addition, in order to solve the technical problem that the image feature method will obtain poor image registration results when there are large blank spaces or many characters with simple structures in the text image, the embodiment of the present application combines the image feature method and the image template method to perform image registration on the frame images. Specifically, firstly, according to the image feature method, the first image registration is performed on the sequence frame images to obtain the corresponding registration result; then, when the registration result is a registration failure, the second image registration is performed on the sequence frame images according to the image template method.

[0042] Since a failed registration result may be obtained when there are large blank spaces or more characters with simple structures in the text image, the embodiment of the present invention performs a second image registration on the sequence frame images based on the image template method in this case, which can improve the accuracy of image registration.

[0043] The image processing method provided by the embodiment of the present invention can be applied to an application environment corresponding to a client and a server. The client and the server are located in a wired or wireless network, and the client and the server exchange data through the wired or wireless network.

[0044] Optionally, the client can be run on a terminal, which specifically includes but is not limited to: a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a car computer, a desktop computer, a set-top box, a smart TV, a wearable device, a learning terminal such as a dictionary pen, etc. The client can correspond to a website or an APP (Application).

[0045] According to one embodiment, a client may receive a sequence frame image captured by an image acquisition device; perform a first image registration on the sequence frame image based on an image feature method to obtain a corresponding registration result; and if the registration result is a registration failure, perform a second image registration on the sequence frame image based on an image template method.

[0046] According to another embodiment, the server may receive a sequence frame image captured by an image acquisition device from a client; perform a first image registration on the sequence frame image based on an image feature method to obtain a corresponding registration result; and if the registration result is a registration failure, perform a second image registration on the sequence frame image based on an image template method.

[0047] Method Example 1

[0048] Reference Figure 1 , shows a flowchart of the steps of an image processing method embodiment 1 of the present invention, and the method may specifically include the following steps:

[0049] Step 101, receiving a sequence of frame images acquired by an image acquisition device;

[0050] Step 102: Perform first image registration on the sequence frame images according to the image feature method to obtain corresponding registration results;

[0051] Step 103: If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0052] Figure 1 The method embodiment shown may be executed by a client and / or a server. It is to be understood that the embodiment of the present invention does not limit the specific execution subject of the method embodiment.

[0053] In step 101, the terminal may have an image acquisition device built in or externally, and collects a sequence of frame images through the image acquisition device. The sequence of frame images may include: at least two frames of images. It can be understood that the embodiment of the present invention does not limit the specific acquisition method of the sequence of frame images.

[0054] In order to improve the accuracy of image registration of the sequence frame images, the embodiment of the present invention combines step 102 and step 103 to perform image registration on the sequence frame images.

[0055] The image feature method first extracts image features, then generates feature descriptors, and finally matches the features of the two frames of images according to the similarity of the feature descriptors. Image features may include but are not limited to: points, lines (edges), regions (surfaces), etc., and may also include but are not limited to: local features and global features.

[0056] Image features may include: texture features. When there are large blank spaces or more characters with simple structures in the text image, the image feature method is likely to obtain poor image registration results. Therefore, the embodiment of the present application first performs a first image registration on the sequence frame images according to the image feature method to obtain a corresponding registration result; then, when the registration result is a registration failure, a second image registration is performed on the sequence frame images according to the image template method to overcome the problems of the image feature method and thereby improve the accuracy of image registration.

[0057] According to an embodiment, the registration result can be determined based on the matching degree between the image features. For example, if the matching degree between the image features of two adjacent frame images is less than a first threshold, the registration result is a registration failure. Alternatively, if the average matching degree between the image features of all adjacent frame images is less than a second threshold, the registration result is a registration failure, etc.

[0058] According to another embodiment, the above-mentioned first image registration of the sequence frame images specifically includes: performing feature matching on two adjacent frame images in the sequence frame images to obtain a feature matching pair; clustering the feature matching pairs to obtain corresponding clustering results; if the clustering results do not meet preset conditions, the obtained registration result includes a registration failure.

[0059] The embodiment of the present invention determines the registration result based on the clustering result of the feature matching pair. The clustering result can characterize the overall matching situation of the sequence frame images in terms of image features, and thus can be used as a basis for determining the registration result.

[0060] Optionally, the above preset conditions specifically include: the number of feature matching pairs contained in any clustering category is greater than the number threshold. When the preset conditions are not met, the number of feature matching pairs contained in all clustering categories is not greater than the number threshold, which can indicate that the overall matching of the sequence frame images in terms of image features does not meet expectations. In this case, image registration will result in a lower accuracy rate. Therefore, in this case, the registration result can be set as registration failure, so that another registration method can be used for registration.

[0061] In practical applications, the number of feature matching pairs contained in all clustering categories can be compared with the number threshold, respectively. Alternatively, the target clustering category containing the largest number of feature matching pairs can be first determined, and then the number of feature matching pairs contained in the target clustering category can be compared with the number threshold.

[0062] In an optional embodiment of the present invention, the first image registration of the sequence frame images may further include: if the clustering result meets the preset conditions, a registration result of successful configuration may be obtained. In this case, the motion transformation parameters corresponding to a single feature matching pair may be determined based on the statistical results of the motion transformation parameters corresponding to the multiple feature matching pairs contained in the sequence frame images; the motion transformation parameters may be used for further image fusion.

[0063] If the clustering result meets the preset conditions, it can indicate that the overall matching of the sequence frame images in terms of image features can meet expectations. In this case, image registration will achieve a higher accuracy. The embodiment of the present invention can perform the first image registration using the image feature method to obtain the corresponding motion transformation parameters.

[0064] The statistical results of the motion transformation parameters corresponding to the multiple feature matching pairs may include: the average of the motion transformation parameters corresponding to the multiple feature matching pairs, etc. The statistical results are used as the motion transformation parameters corresponding to the single feature matching pair, so as to fuse the two frames of images corresponding to the single feature matching pair according to the motion transformation parameters corresponding to the single feature matching pair.

[0065] In an application example of the present invention, it is assumed that the image frame sequence is {I1, I2, …, I n}, the method based on image features, performing first image registration on the sequence frame images, may specifically include the following steps:

[0066] Step A1: extract features from the above sequence frame images to obtain image I i image features; i can be a natural number greater than 0, and n can be a natural number greater than 1;

[0067] Step A2: image I i-1 and image I i Perform feature matching on the image features to obtain a feature matching pair;

[0068] The feature matching pair can be represented by the displacement vector p j =(dx j ,dy j ) is used to represent the distance between feature matching pairs, and d(p i ,p j )=|dx i -dx j |+|dy i -dy j | to indicate.

[0069] Step A3: clustering all feature matching pairs to obtain multiple cluster categories;

[0070] Step A4: for the target cluster category containing the most feature matching pairs, the number m of feature matching pairs contained in the target cluster category is compared with the number threshold T. If m<T, it is considered that the registration has failed, otherwise it is considered that the registration has succeeded.

[0071] If the registration is successful, the image I is determined based on the mean of the displacement vectors of all feature matching pairs. i-1 and image I i The relative displacement (x i ,y i ), which can be expressed by formula (1):

[0072]

[0073] In step 103, if the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0074] The image template method can use correlation operations and other methods to calculate the correlation value to find the best matching position. Template matching is to find a sub-image that matches the template image in another frame based on a known template image.

[0075] Specifically in the embodiment of the present invention, the image I i As a template image, in image I i-1 Total Search and Image I i The matching sub-image.

[0076] In an optional embodiment of the present invention, the above-mentioned second image registration of the sequence frame images specifically includes: determining the matching degree of the corresponding overlapping parts of two adjacent frame images in the sequence frame images; and determining the motion transformation parameters corresponding to the two adjacent frame images based on the motion transformation parameters corresponding to the matching degree.

[0077] Optionally, the image I i In the image I i-1 Slide on the image frame, determine the matching degree of the overlapping parts of the two frames in the case of real-time displacement vector, and record the displacement vector (x i ,y i ), as image I i-1 and image I i relative displacement.

[0078] It can be understood that the above displacement vector is only an example of motion transformation parameters. In fact, those skilled in the art can adopt other motion transformation parameters, such as image rotation parameters, image scaling parameters, etc. according to actual application requirements. It can be understood that the embodiments of the present invention do not limit the specific motion transformation parameters.

[0079] The embodiment of the present invention may output the motion transformation parameters obtained in step 102 or step 103, or may use the motion transformation parameters obtained in step 102 or step 103 in image processing scenarios such as image fusion.

[0080] In an optional embodiment of the present invention, the method may further include: performing image fusion on the sequence frame images according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

[0081] Image fusion is the process of synthesizing at least two frames of images according to the motion transformation parameters between the images to obtain a complete image.

[0082] In an optional embodiment of the present invention, during the image fusion process, the first frame image in the sequence of frame images can be used as a reference frame, and the other frame images in the sequence of frame images except the first frame image can be transformed into the coordinate system of the first frame image. i The translation vector relative to image I1 can be expressed by formula (2):

[0083]

[0084] Optionally, in the process of image fusion of two adjacent frames of images, the pixel average of the overlapping parts of the two frames of images is taken.

[0085] In summary, the image processing method of the embodiment of the present invention directly performs image registration on the sequence frame images collected by the image acquisition device. Since the embodiment of the present invention does not perform pre-processing operations such as binarization and dilation on the sequence frame images before image registration, the embodiment of the present invention can overcome the problem of inaccurate image registration caused by inaccurate pre-processing effects in complex backgrounds such as picture books, and improve the accuracy of image registration.

[0086] Furthermore, the embodiment of the present application combines the image feature method and the image template method to perform image registration on the frame images. Specifically, firstly, according to the image feature method, the first image registration is performed on the sequence frame images to obtain the corresponding registration result; then, when the registration result is a registration failure, according to the image template method, the second image registration is performed on the sequence frame images.

[0087] Since a failed registration result may be obtained when there are large blank spaces or more characters with simple structures in the text image, the embodiment of the present invention performs a second image registration on the sequence frame images based on the image template method in this case, which can improve the accuracy of image registration.

[0088] Method Example 2

[0089] The embodiment of the present invention does not perform preprocessing operations such as binarization and dilation on the sequence frame images before image registration, and directly performs image registration on the sequence frame images collected by the image acquisition device; this may cause problems such as distorted text in the text image obtained by image fusion.

[0090] Method Embodiment 2 Correcting the text image based on the pairing of upper and lower key points can overcome the problem of distorted text in the text image to a certain extent. On this basis, using the corrected image for text recognition can improve the accuracy of text recognition.

[0091] Reference Figure 2, shows a flowchart of a text image correction method of the present invention, the method may specifically include the following steps:

[0092] Step 201: determining an upper and lower key point pair corresponding to a text line in a text image; the upper and lower key point pair specifically includes: an upper key point located above the text line, and a lower key point located below the text line; a line connecting the upper key point and the lower key point is perpendicular to a preset direction;

[0093] Step 202: segment the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0094] Step 203: Determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points;

[0095] Step 204: Map the segmented text image according to the mapping relationship to obtain a corresponding segmented corrected image.

[0096] Figure 2 The method embodiment shown may be executed by a client and / or a server. It is to be understood that the embodiment of the present invention does not limit the specific execution subject of the method embodiment.

[0097] In step 201, the text lines in the text image may be obtained by performing text detection on the text image. Text detection may be used to detect text in a text image, and the position of the text in the text image may be obtained, such as the area of ​​the text in the text image.

[0098] The above-mentioned upper and lower key point pairing specifically includes: an upper key point located above the text line, and a lower key point located below the text line, which can reflect the trend of the text line.

[0099] Reference Figure 3 , shows a schematic diagram of the pairing of upper and lower key points corresponding to a text image according to an embodiment of the present invention. In which, the upper key point and the lower key point can be set respectively at the upper and lower parts of the text line in the text image.

[0100] Optionally, the line between the upper key point and the lower key point is perpendicular to a preset direction. The preset direction may be a target direction for correction, so that the text image is corrected according to the preset direction, and the corresponding correction target may be: to make the obtained corrected image match the preset direction. For example, the preset direction may be a horizontal direction corresponding to the text image, or a horizontal direction corresponding to the first character in a text line. Of course, the embodiment of the present invention does not limit the specific preset direction. Assuming that an XOY coordinate system is established according to a preset direction, and the preset direction corresponds to the X-axis direction, the upper key point and the lower key point may correspond to the same X-coordinate.

[0101] In the embodiment of the present invention, optionally, the interval between two adjacent upper and lower key point pairs (hereinafter referred to as adjacent pairs) is a preset interval. Specifically, the interval between two adjacent upper key points is a preset interval, and the interval between two adjacent lower key points is a preset interval. Different adjacent pairs may correspond to the same or different preset intervals.

[0102] In an optional embodiment of the present invention, the above-mentioned determination of the upper and lower key point pairs corresponding to the text lines in the text image specifically includes: determining the detection results of the upper and lower key point pairs corresponding to the text lines in the text image based on a detection model; the detection model can be trained based on text image samples and their corresponding annotated upper and lower key point pairs.

[0103] The above detection result may include: information such as the number and position of upper and lower key point pairs corresponding to the text line; wherein different adjacent pairs may correspond to the same preset interval.

[0104] In an optional embodiment of the present application, the mathematical model can be trained based on the training data to obtain the data analyzer detection model. The above training can be the embodiment of the present invention can obtain the typesetting information corresponding to the web page content samples before and after typesetting, and the actual scanning data of a large amount of typesetting change information as training data, and train the training data so that the data analyzer detection model has the recognition ability of the typesetting upper and lower key point pairing of the web page of the preset web page category. The data analyzer detection model can characterize the mapping relationship between the input data (original content text image) and the output data (information of the typesetting content upper and lower key point pairing).

[0105] In the embodiment of the present invention, the positions of the upper and lower key point pairs obtained by the detection model are accurate and closely distributed, and can match the trend of the text line. In addition, the detection model is not limited by the background of the text image or the type of text content. Therefore, the above detection model also has good robustness in complex backgrounds or mixed Chinese and English scenes, and can improve the accuracy of the detection results.

[0106] Mathematical model is a scientific or engineering model constructed using mathematical logic methods and mathematical language. Mathematical model is a mathematical structure that is expressed in a general or approximate manner using mathematical language with reference to the characteristics or quantitative dependencies of a certain system of things. This mathematical structure is a relational structure depicted with the help of mathematical symbols. Mathematical model can be one or a group of algebraic equations, differential equations, difference equations, integral equations or statistical equations and their combinations, which quantitatively or qualitatively describe the mutual relationship or causal relationship between the variables of the system. In addition to mathematical models described by equations, there are also models described by other mathematical tools, such as algebra, geometry, topology, mathematical logic, etc. Among them, mathematical models describe the behavior and characteristics of the system rather than the actual structure of the system. Among them, machine learning, deep learning methods and other methods can be used to train mathematical models. Machine learning methods may include: linear regression, decision tree, random forest, etc., and deep learning methods may include: convolutional neural networks (CNN), long short-term memory networks (LSTM), gated recurrent units (GRU), etc.

[0107] In an optional embodiment of the present invention, to address the problem of possible deviation in the pairing of the first and last upper and lower key points in the detection results, the pairing of the first and last upper and lower key points in the detection results can be removed to improve the accuracy of the pairing of the upper and lower key points in the detection results.

[0108] Further optionally, if the first and last upper and lower key point pairs in the detection results are directly removed, the upper and lower key point pairs may fail to cover the first and last text in the text image, resulting in a low coverage rate of the upper and lower key point pairs for the text. In order to improve the coverage rate of the upper and lower key point pairs for the text, an embodiment of the present invention can determine the prediction results of the first and last upper and lower key point pairs based on the non-first and last upper and lower key point pairs in the detection results, and determine the upper and lower key point pairs corresponding to the text lines in the text image based on the non-first and last upper and lower key point pairs in the detection results and the prediction results.

[0109] The embodiment of the present invention can calculate the position of the head and tail upper and lower key point pairing as a prediction result based on the position information of the non-head and tail upper and lower key point pairing adjacent to the head and tail upper and lower key point pairing.

[0110] The embodiment of the present invention can calculate the position of the head and tail upper and lower key point pairing as a prediction result based on the position information of the non-head and tail upper and lower key point pairing adjacent to the head and tail upper and lower key point pairing.

[0111] According to an embodiment, the position of the head and tail upper and lower key point pairing can be determined based on the slope of the straight line between two adjacent non-head and tail upper and lower key point pairs of the head and tail upper and lower key point pairing and the interval between two adjacent non-head and tail upper and lower key point pairs.

[0112] by Figure 2 Take the text image shown as an example, refer to Figure 7 , which shows a schematic diagram of a prediction process of the pairing of the first, last, upper and lower key points in an embodiment of the present invention.

[0113] Assuming that the text image includes n (n is a natural number greater than 1) upper and lower key point pairs, the last several upper and lower key point pairs are represented as: the (n-2)th upper and lower key point pair, the (n-1)th upper and lower key point pair, and the nth upper and lower key point pair. Assuming that the nth upper and lower key point pair is removed, the position of the nth upper and lower key point pair can be predicted based on the slope and interval of the straight line between the position of the (n-1)th upper and lower key point pair and the (n-2)th upper and lower key point pair, as follows: Figure 7 As shown in 701.

[0114] According to another embodiment, the position of the head and tail up and down key point pairing can be determined based on the slope of the straight line between two adjacent non-head and tail up and down key point pairs of the head and tail up and down key point pairing and the image boundary. This embodiment can be applicable to the case where the interval between two adjacent non-head and tail up and down key point pairs exceeds the image boundary.

[0115] by Figure 7 As an example, assume that the upper and lower key point pairs are numbered in order from front to back to obtain the first upper and lower key point pairing, the second upper and lower key point pairing, the third upper and lower key point pairing, etc. Assuming that the first upper and lower key point pairing is removed, the position of the first upper and lower key point pairing is first predicted based on the slope and interval of the straight line between the second upper and lower key point pairing and the third upper and lower key point pairing. It can be seen that the prediction result exceeds the image boundary, so in this case, the position of the first upper and lower key point pairing can be predicted based on the above slope and image boundary, such as Figure 7 As shown in 702.

[0116] In step 202, the text image can be segmented based on the pairing of the upper and lower key points to obtain a segmented text image. When the field of view of the image acquisition device is so narrow that the complete text image cannot be directly captured, it is usually necessary to splice the sequence frame images of the image acquisition device to obtain a complete text image. The segmentation of the embodiment of the present invention can make the segmented correction image more compatible with the frame image captured by the image acquisition device, and on this basis, the accuracy of the correction of the text image can be further improved.

[0117] According to an embodiment, the number of segmented text images may be a preset value, and those skilled in the art may determine the preset value based on prior knowledge.

[0118] According to another embodiment, the number of segmented text images can be obtained based on the width of the text image and the field of view width of the image acquisition device; in this way, adaptive segmentation based on the field of view width of the image acquisition device can be achieved, thereby improving the matching degree between the segmented text image and the frame image of the image acquisition device.

[0119] In an optional embodiment of the present invention, formula (3) shows the process of determining the number of segmented text images and the process of determining the number of upper and lower key point pairs included in the segmented text images.

[0120]

[0121] Wherein, w represents the width of the text image, block_thres represents the field of view width of the image acquisition device, and with the restriction that the number of segmented text images is at least 2, team_num can represent the number of segmented text images.

[0122] num_p represents the number of upper and lower key point pairs included in the text image, and point_num represents the number of upper and lower key point pairs included in the segmented text image.

[0123] This embodiment of the present invention sets the number of segmented text images to be at least 2, and the number of upper and lower key point pairs included in the segmented text image to be at least 2, which can improve the correction speed and correction accuracy of the text image. Of course, in other embodiments of the present invention, the number of segmented text images can be less than 2, and the number of upper and lower key point pairs included in the segmented text image can be less than 2.

[0124] In step 203, the segmented text image is corrected based on the pairing of the upper and lower key points to obtain a corresponding segmented corrected image.

[0125] The embodiment of the present invention corrects the text image according to the preset direction, and the corresponding correction target may be: making the obtained corrected image match the preset direction.

[0126] In an optional embodiment of the present invention, the method may further include: stitching the segmented rectified images to obtain a rectified image. In the embodiment of the present invention, the segmented rectified images may be stitched according to the position order of the segmented rectified images corresponding to the segmented text images to obtain a rectified image. Figure 4 , which shows a schematic diagram of a corrected image according to an embodiment of the present invention, which can be used for Figure 3 The text image shown is rectified.

[0127] In an optional embodiment of the present invention, the above-mentioned correction of the segmented text image specifically includes: determining the mapping relationship between the source point and the target point corresponding to the segmented text image based on the pairing of the upper and lower key points; and mapping the segmented text image based on the mapping relationship to obtain a corresponding segmented corrected image.

[0128] The source point can represent the pixel point in the segmented text image, and the target point can represent the pixel point in the segmented rectified image.

[0129] In an optional embodiment of the present invention, the above-mentioned determination of the mapping relationship between the source point and the target point corresponding to the segmented text image may specifically include: determining the target area corresponding to the segmented text image based on the height between the upper and lower key point pairs, a preset direction, and the upper and lower key point pairs included in the segmented text image; determining the mapping relationship between the source point and the target point corresponding to the segmented text image based on the upper and lower key point pairs corresponding to the target area and the upper and lower key point pairs corresponding to the segmented text image.

[0130] Reference Figure 5 , which shows a schematic diagram of a text image, a rectified image, and a mapping relationship between a source point and a target point according to an embodiment of the present invention.

[0131] The width of the corrected image may be the width difference of the first and last upper and lower key point pairs, and the height of the corrected image may be the height difference of the first upper and lower key point pairs. Of course, the embodiment of the present invention does not limit the width of the corrected image and the height of the corrected image. For example, the width of the corrected image may be equal to the width of the text image. Alternatively, the height of the corrected image may be the average of the height differences of all upper and lower key point pairs, etc.

[0132] The embodiment of the present invention can pair the first upper and lower key points included in the segmented text image as the source point. In addition, the Y coordinate of the source point of the segmented text image can be adjusted to obtain the target point. For a segmented text image, in the corresponding segmented rectified image, the upper key point corresponds to the same Y coordinate, and the lower key point corresponds to the same Y coordinate.

[0133] The height of the target area can be the height of the corrected image, and the width of the target area can be obtained based on the width difference of the upper and lower key point pairs included in the segmented text image. The upper and lower key point pairs corresponding to the target area can correspond to the target point, and the upper and lower key point pairs corresponding to the segmented text image can correspond to the source point.

[0134] After the source point and the target point corresponding to the segmented text image are determined, the mapping relationship between the source point and the target point can be determined according to the positions of the source point and the target point corresponding to the segmented text image.

[0135] Optionally, the mapping relationship between the source point and the target point may be represented by a mapping matrix. Parameters of the mapping matrix may include: at least one of a rotation parameter, a scaling parameter, and a displacement parameter.

[0136] For example, formula (4) shows an example of a mapping matrix.

[0137]

[0138] Among them (b x , b y ) represents the displacement parameter, parameter a ij Represent the rotation parameters and scaling parameters respectively.

[0139] Optionally, the mapping of the segmented text image specifically includes: using a mapping matrix to perform a mapping transformation on the segmented text image. The mapping transformation process can be expressed by formula (3).

[0140]

[0141] Among them, (x, y) represents the coordinates of the target point, and (X, Y) represents the coordinates of the source point.

[0142] After obtaining the segmented rectified images, the segmented rectified images can be sequentially spliced ​​into the corresponding segmented text images according to the position order of the segmented rectified images. Figure 5 On the rectified image, we can obtain Figure 4 The corrected image is shown. Figure 4 It can be seen that by applying the technical solution of the embodiment of the present invention, the originally distorted text can be corrected to nearly horizontal text.

[0143] The embodiment of the present invention can use the corrected image obtained in step 203 for text recognition to obtain text information in the corrected image. The text recognition method may include: OCR (Optical Character Recognition) method, etc. It can be understood that the embodiment of the present invention does not limit the specific text recognition method.

[0144] In summary, the image processing method of the embodiment of the present invention corrects the text image based on the pairing of upper and lower key points. Among them, the above-mentioned pairing of upper and lower key points specifically includes: an upper key point located above the text line, and a lower key point located below the text line. Since the above-mentioned pairing of upper and lower key points can characterize the trend of the text line, the embodiment of the present invention can correct the text image based on the trend of the text line, and is not limited to the background of the text image or the type of text content. Therefore, the embodiment of the present invention has good applicability to complex backgrounds such as picture books or mixed Chinese and English scenes, and further, the embodiment of the present invention can improve the accuracy of text image correction in complex backgrounds such as picture books or mixed Chinese and English scenes.

[0145] Furthermore, the embodiment of the present invention segments the text image, and corrects the obtained segmented text image based on the pairing of the upper and lower key points. The above segmentation-based image correction can improve the correction accuracy of the text image.

[0146] In addition, when the field of view of the image acquisition device is too narrow to directly capture a complete text image, it is usually necessary to splice the sequence frame images of the image acquisition device to obtain a complete text image. The segmentation of the embodiment of the present invention can make the segmented correction image more compatible with the frame image captured by the image acquisition device, and on this basis, the accuracy of text image correction can be further improved.

[0147] Method Example 3

[0148] Reference Figure 6 , shows a flowchart of the steps of a third embodiment of an image processing method of the present invention, which may specifically include the following steps:

[0149] Step 601, receiving a sequence of frame images acquired by an image acquisition device;

[0150] Step 602: Perform first image registration on the sequence frame images according to the image feature method to obtain corresponding registration results;

[0151] Step 603: If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method;

[0152] Step 604: performing image fusion on the sequence frame images according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image;

[0153] Step 605: Determine the upper and lower key point pairs corresponding to the text lines in the text image;

[0154] The pairing of the upper and lower key points specifically includes: an upper key point located above the text line, and a lower key point located below the text line; a line connecting the upper key point and the lower key point is perpendicular to a preset direction;

[0155] Step 606: segment the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0156] Step 607: correct the segmented text image according to the pairing of the upper and lower key points to obtain a corresponding segmented corrected image;

[0157] Step 608: stitching the above segmented corrected images to obtain a corrected image;

[0158] Step 609: Perform text recognition on the corrected image to obtain corresponding text information.

[0159] The image processing method of the embodiment of the present invention can be applied to scenarios such as dictionary pens to process sequence frame images captured by the dictionary pen and finally obtain text information corresponding to the frame images, thereby providing functions such as word search and translation based on the text information.

[0160] The image processing method may include: an image stitching process, an image correction process and a text recognition process to obtain text information in the source image.

[0161] Among them, steps 601 to 604 correspond to the image stitching process, steps 605 to 608 correspond to the image correction process, and step 609 corresponds to the text recognition process.

[0162] It should be noted that, for the method embodiments, for the sake of simplicity of description, they are all described as a series of motion action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the described motion actions, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the motion actions involved are not necessarily required by the embodiments of the present invention.

[0163] Device Embodiment

[0164] Reference Figure 8 , shows a structural block diagram of an embodiment of an image processing device of the present invention, which may specifically include:

[0165] The receiving module 801 is used to receive the sequence frame images collected by the image acquisition device;

[0166] A first image registration module 802 is used to perform first image registration on the above sequence frame images according to an image feature method to obtain a corresponding registration result;

[0167] The second image registration module 803 is used to perform second image registration on the sequence frame images according to the image template method if the registration result is a registration failure.

[0168] Optionally, the first image registration module 801 may include:

[0169] A feature matching module, used for performing feature matching on two adjacent frame images in the above sequence of frame images to obtain a feature matching pair;

[0170] A clustering module is used to cluster the above feature matching pairs to obtain corresponding clustering results;

[0171] A registration result determination module, configured to obtain a registration result including registration failure if the clustering result does not meet a preset condition;

[0172] The above-mentioned preconditions may include:

[0173] The number of feature matching pairs contained in any cluster category is greater than the number threshold.

[0174] Optionally, the first image registration module 801 may further include:

[0175] The first motion transformation parameter determination module is used to determine the motion transformation parameter corresponding to a single feature matching pair according to the statistical results of the motion transformation parameters corresponding to multiple feature matching pairs contained in the sequence frame images if the above clustering result meets the preset conditions.

[0176] Optionally, the second image registration module 802 may include:

[0177] A matching degree determination module, used to determine the matching degree of the corresponding overlapping parts of two adjacent frame images in the above sequence of frame images;

[0178] The second motion transformation parameter determination module is used to determine the motion transformation parameters corresponding to two adjacent frame images according to the motion transformation parameters corresponding to the above matching degree.

[0179] Optionally, the above device may further include:

[0180] The image fusion module is used to perform image fusion on the sequence frame images according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

[0181] Optionally, the above device may further include:

[0182] The upper and lower key point pairing determination module is used to determine the upper and lower key point pairing corresponding to the text line in the text image; the upper and lower key point pairing may include: an upper key point located above the text line, and a lower key point located below the text line; the line between the upper key point and the lower key point is perpendicular to a preset direction;

[0183] A segmentation module, used for segmenting the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0184] A mapping determination module, used to determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points;

[0185] The mapping module is used to map the segmented text image according to the mapping relationship to obtain a corresponding segmented corrected image.

[0186] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0189] Fig. 9 1 is a block diagram of an apparatus 1300 for image processing according to an exemplary embodiment. For example, the apparatus 1300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0190] Reference Fig. 9 , the device 1300 may include one or more of the following components: a processing component 1302 , a memory 1304 , a power component 1306 , a multimedia component 1308 , an audio component 1310 , an input / output (I / O) interface 1312 , a sensor component 1314 , and a communication component 1316 .

[0191] The processing component 1302 generally controls the overall operation of the device 1300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1302 may include one or more modules to facilitate interaction between the processing component 1302 and other components. For example, the processing component 1302 may include a multimedia module to facilitate interaction between the multimedia component 1308 and the processing component 1302.

[0192] The memory 1304 is configured to store various types of data to support operations on the device 1300. Examples of such data include instructions for any application or method operating on the device 1300, contact data, phone book data, messages, pictures, videos, etc. The memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0193] The power supply component 1306 provides power to the various components of the device 1300. The power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1300.

[0194] The multimedia component 1308 includes a screen that provides an output interface between the device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1308 includes a front camera and / or a rear camera. When the device 1300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0195] The audio component 1310 is configured to output and / or input audio signals. For example, the audio component 1310 includes a microphone (MIC), and when the device 1300 is in an operation mode, such as a call mode, a recording mode, and a voice data processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1304 or sent via the communication component 1316. In some embodiments, the audio component 1310 also includes a speaker for outputting audio signals.

[0196] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0197] The sensor assembly 1314 includes one or more sensors for providing various aspects of the status assessment of the device 1300. For example, the sensor assembly 1314 can detect the open / closed state of the device 1300, the relative positioning of components, such as the display and keypad of the device 1300, the sensor assembly 1314 can also detect the position change of the device 1300 or a component of the device 1300, the presence or absence of user contact with the device 1300, the orientation or acceleration / deceleration of the device 1300, and the temperature change of the device 1300. The sensor assembly 1314 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1314 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1314 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0198] The communication component 1316 is configured to facilitate wired or wireless communication between the device 1300 and other devices. The device 1300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency data processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0199] In an exemplary embodiment, the device 1300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned methods.

[0200] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, and the instructions can be executed by the processor 1320 of the device 1300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0201] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute an image processing method, the method comprising: receiving a sequence frame image acquired by an image acquisition device; performing a first image registration on the sequence frame image according to an image feature method to obtain a corresponding registration result; if the registration result is a registration failure, performing a second image registration on the sequence frame image according to an image template method.

[0202] Fig.10 1 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0203] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or one or more operating systems 1941, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0204] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0205] It should be understood that the present invention is not limited to the precise structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0207] The embodiment of the present invention discloses A1, an image processing method, comprising:

[0208] Receiving a sequence of frame images acquired by an image acquisition device;

[0209] Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0210] If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0211] A2. According to the method described in A1, performing first image registration on the sequence of frame images comprises:

[0212] Performing feature matching on two adjacent frame images in the sequence of frame images to obtain a feature matching pair;

[0213] Clustering the feature matching pairs to obtain corresponding clustering results;

[0214] If the clustering result does not meet the preset conditions, the obtained registration result includes registration failure;

[0215] The preset conditions include:

[0216] The number of feature matching pairs contained in any cluster category is greater than the number threshold.

[0217] A3. The method according to A2, wherein the first image registration of the sequence frame images further comprises:

[0218] If the clustering result meets the preset condition, the motion transformation parameter corresponding to a single feature matching pair is determined according to the statistical result of the motion transformation parameters corresponding to a plurality of feature matching pairs included in the sequence frame images.

[0219] A4. According to the method described in A1, performing second image registration on the sequence frame images comprises:

[0220] Determining the matching degree of corresponding overlapping parts of two adjacent frame images in the sequence of frame images;

[0221] According to the motion transformation parameters corresponding to the matching degree, the motion transformation parameters corresponding to two adjacent frame images are determined.

[0222] A5. The method according to any one of A1 to A4, further comprising:

[0223] The sequence frame images are subjected to image fusion according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

[0224] A6. The method according to A5, further comprising:

[0225] Determine an upper and lower key point pair corresponding to a text line in a text image; the upper and lower key point pair includes: an upper key point located above the text line, and a lower key point located below the text line; a line connecting the upper key point and the lower key point is perpendicular to a preset direction;

[0226] Segmenting the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0227] Determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points;

[0228] According to the mapping relationship, the segmented text image is mapped to obtain a corresponding segmented rectified image.

[0229] The embodiment of the present invention discloses B7, an image processing device, comprising:

[0230] A receiving module, used for receiving a sequence of frame images collected by an image acquisition device;

[0231] A first image registration module, used for performing first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0232] The second image registration module is used to perform second image registration on the sequence frame images according to an image template method if the registration result is a registration failure.

[0233] B8. According to the device described in B7, the first image registration module includes:

[0234] A feature matching module, used for performing feature matching on two adjacent frame images in the sequence of frame images to obtain a feature matching pair;

[0235] A clustering module, used for clustering the feature matching pairs to obtain corresponding clustering results;

[0236] A registration result determination module, configured to obtain a registration result including registration failure if the clustering result does not meet a preset condition;

[0237] The preset conditions include:

[0238] The number of feature matching pairs contained in any cluster category is greater than the number threshold.

[0239] B9. According to the device described in B8, the first image registration module also includes:

[0240] The first motion transformation parameter determination module is used to determine the motion transformation parameter corresponding to a single feature matching pair according to the statistical results of the motion transformation parameters corresponding to multiple feature matching pairs contained in the sequence frame images if the clustering result meets the preset conditions.

[0241] B10. According to the device described in B7, the second image registration module comprises:

[0242] A matching degree determination module, used to determine the matching degree of the corresponding overlapping parts of two adjacent frame images in the sequence of frame images;

[0243] The second motion transformation parameter determination module is used to determine the motion transformation parameters corresponding to two adjacent frame images according to the motion transformation parameters corresponding to the matching degree.

[0244] B11. The device according to any one of B7 to B10, further comprising:

[0245] The image fusion module is used to perform image fusion on the sequence frame images according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

[0246] B12. The device according to B11, further comprising:

[0247] The upper and lower key point pairing determination module is used to determine the upper and lower key point pairing corresponding to the text line in the text image; the upper and lower key point pairing includes: an upper key point located above the text line, and a lower key point located below the text line; the line between the upper key point and the lower key point is perpendicular to a preset direction;

[0248] A segmentation module, used for segmenting the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0249] A mapping determination module, used to determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points;

[0250] A mapping module is used to map the segmented text image according to the mapping relationship to obtain a corresponding segmented corrected image.

[0251] The embodiment of the present invention discloses C13, a device for image processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0252] Receiving a sequence of frame images acquired by an image acquisition device;

[0253] Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result;

[0254] If the registration result is a registration failure, a second image registration is performed on the sequence frame images according to an image template method.

[0255] C14. According to the device described in C13, the first image registration of the sequence frame images comprises:

[0256] Performing feature matching on two adjacent frame images in the sequence of frame images to obtain a feature matching pair;

[0257] Clustering the feature matching pairs to obtain corresponding clustering results;

[0258] If the clustering result does not meet the preset conditions, the obtained registration result includes registration failure;

[0259] The preset conditions include:

[0260] The number of feature matching pairs contained in any cluster category is greater than the number threshold.

[0261] C15. The device according to C14, wherein the performing first image registration on the sequence of frame images further comprises:

[0262] If the clustering result meets the preset condition, the motion transformation parameter corresponding to a single feature matching pair is determined according to the statistical result of the motion transformation parameters corresponding to a plurality of feature matching pairs included in the sequence frame images.

[0263] C16. According to the device described in C13, the performing second image registration on the sequence frame images comprises:

[0264] Determining the matching degree of corresponding overlapping parts of two adjacent frame images in the sequence of frame images;

[0265] According to the motion transformation parameters corresponding to the matching degree, the motion transformation parameters corresponding to two adjacent frame images are determined.

[0266] C17. The device according to any one of C13 to C16, wherein the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:

[0267] The sequence frame images are subjected to image fusion according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

[0268] C18. The device according to C17, wherein the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:

[0269] Determine an upper and lower key point pair corresponding to a text line in a text image; the upper and lower key point pair includes: an upper key point located above the text line, and a lower key point located below the text line; a line connecting the upper key point and the lower key point is perpendicular to a preset direction;

[0270] Segmenting the text image according to the pairing of the upper and lower key points to obtain a segmented text image;

[0271] Determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points;

[0272] According to the mapping relationship, the segmented text image is mapped to obtain a corresponding segmented rectified image.

[0273] An embodiment of the present invention discloses D19, one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause the device to execute one or more of the methods described in A1 to A6.

[0274] The above is a detailed introduction to an image processing method, an image processing device and a device for image processing provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Receiving a sequence of frame images acquired by an image acquisition device, and directly performing image registration on the sequence of frame images acquired by the image acquisition device; Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result; If the registration result is a registration failure, performing a second image registration on the sequence frame images according to an image template method; Wherein, performing second image registration on the sequence frame images includes: Determining the matching degree of corresponding overlapping parts of two adjacent frame images in the sequence of frame images; According to the motion transformation parameters corresponding to the matching degree, the motion transformation parameters corresponding to two adjacent frame images are determined.

2. The method according to claim 1, characterized in that The performing first image registration on the sequence frame images comprises: Performing feature matching on two adjacent frame images in the sequence of frame images to obtain a feature matching pair; Clustering the feature matching pairs to obtain corresponding clustering results; If the clustering result does not meet the preset conditions, the obtained registration result includes registration failure; The preset conditions include: The number of feature matching pairs contained in any cluster category is greater than the number threshold.

3. The method according to claim 2, characterized in that The performing first image registration on the sequence frame images further includes: If the clustering result meets the preset condition, the motion transformation parameter corresponding to a single feature matching pair is determined according to the statistical result of the motion transformation parameters corresponding to a plurality of feature matching pairs included in the sequence frame images.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The sequence frame images are subjected to image fusion according to the motion transformation parameters obtained by the first image registration or the second image registration to obtain a text image.

5. The method according to claim 4, characterized in that The method further comprises: Determine an upper and lower key point pair corresponding to a text line in a text image; the upper and lower key point pair includes: an upper key point located above the text line, and a lower key point located below the text line; a line connecting the upper key point and the lower key point is perpendicular to a preset direction; Segmenting the text image according to the pairing of the upper and lower key points to obtain a segmented text image; Determine the mapping relationship between the source point and the target point corresponding to the segmented text image according to the pairing of the upper and lower key points; According to the mapping relationship, the segmented text image is mapped to obtain a corresponding segmented rectified image.

6. An image processing device, characterized in that: include: A receiving module, used for receiving a sequence of frame images acquired by an image acquisition device, and directly performing image registration on the sequence of frame images acquired by the image acquisition device; A first image registration module, used for performing first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result; A second image registration module, configured to perform a second image registration on the sequence frame images according to an image template method if the registration result is a registration failure; Wherein, the second image registration module comprises: A matching degree determination module, used to determine the matching degree of the corresponding overlapping parts of two adjacent frame images in the sequence of frame images; The second motion transformation parameter determination module is used to determine the motion transformation parameters corresponding to two adjacent frame images according to the motion transformation parameters corresponding to the matching degree.

7. The device according to claim 6, characterized in that The first image registration module comprises: A feature matching module, used for performing feature matching on two adjacent frame images in the sequence of frame images to obtain a feature matching pair; A clustering module, used for clustering the feature matching pairs to obtain corresponding clustering results; A registration result determination module, configured to obtain a registration result including registration failure if the clustering result does not meet a preset condition; The preset conditions include: The number of feature matching pairs contained in any cluster category is greater than the number threshold.

8. A device for image processing, characterized in that: The system comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: Receiving a sequence of frame images acquired by an image acquisition device, and directly performing image registration on the sequence of frame images acquired by the image acquisition device; Performing a first image registration on the sequence frame images according to an image feature method to obtain a corresponding registration result; If the registration result is a registration failure, performing a second image registration on the sequence frame images according to an image template method; Wherein, performing second image registration on the sequence frame images includes: Determining the matching degree of corresponding overlapping parts of two adjacent frame images in the sequence of frame images; According to the motion transformation parameters corresponding to the matching degree, the motion transformation parameters corresponding to two adjacent frame images are determined.

9. One or more machine-readable media, characterized in that Instructions are stored thereon which, when executed by one or more processors, cause the apparatus to perform the method as claimed in one or more of claims 1 to 5.

Images