Image processing method and device, chip and electronic device

Abstract

The embodiment of the present application discloses an image processing method and device, a chip and an electronic device, the method comprising: acquiring a first frame image and a second frame image, the first frame image being located before the second frame image in time sequence; performing edge line segment detection on the first frame image to obtain a first line segment detection image, and performing edge line segment detection on the second frame image to obtain a second line segment detection image; performing image affine transformation on the first line segment detection image to obtain a third line segment detection image; performing time sequence filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image; performing quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image, thereby reducing the instability caused by random disturbances and noise in the scanning process by performing time sequence filtering on the line segment detection images of the two frames of images in time sequence, thereby facilitating improving the accuracy, precision and stability of the quadrilateral image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method and device, a chip and an electronic device. Background Art

[0002] In daily life or office, people need to scan (photograph / capture, etc.) a large number of documents, business cards, certificates, materials, posters and other objects into images to store them in electronic devices such as mobile phones or computers.

[0003] Since the edge contour of the scanned object is usually a quadrilateral, but due to reasons such as the non-fixed scanning position, the quadrilateral in the scanned image is often transformed into an inclined irregular quadrilateral through perspective, resulting in the image being tilted and distorted, which is not conducive to subsequent use such as reading and printing. Therefore, it is necessary to correct the quadrilateral in the scanned image.

[0004] In addition, before extracting a quadrilateral from a scanned image, it is usually necessary to first perform edge line segment detection on the image to obtain line segments in the image, and then find the best quadrilateral based on the line segments.

[0005] However, the detected line segments may be incomplete, missing or redundant. For example, only a portion of the edge contour of an object in an image is detected, resulting in missing line segments, while the contours of other parts of the image are misdetected, resulting in redundant line segments, making it impossible to accurately correct the quadrilateral. Summary of the invention

[0006] The first aspect is an image processing method of the present application, comprising:

[0007] Acquire a first frame image and a second frame image, wherein the first frame image is located before the second frame image in time sequence;

[0008] Performing edge line segment detection on the first frame image to obtain a first line segment detection image, and performing edge line segment detection on the second frame image to obtain a second line segment detection image;

[0009] Performing image affine transformation on the first line segment detection image to obtain a third line segment detection image;

[0010] Performing time-series filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image;

[0011] Perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

[0012] A second aspect is an image processing device of the present application, comprising:

[0013] A frame image acquisition unit, used to acquire a first frame image and a second frame image, wherein the first frame image is a frame image that is located before the second frame image in time sequence;

[0014] An edge line segment detection unit, configured to perform edge line segment detection on the first frame image to obtain a first line segment detection image, and perform edge line segment detection on the second frame image to obtain a second line segment detection image;

[0015] An image affine transformation unit, configured to perform image affine transformation on the first line segment detection image to obtain a third line segment detection image;

[0016] a temporal filtering unit, configured to perform temporal filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image;

[0017] The quadrilateral detection unit is used to perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

[0018] The third aspect is a chip of the present application, comprising a processor, which, when executed, implements the steps in the method designed in the first aspect.

[0019] The fourth aspect is an electronic device of the present application, comprising a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps in the method designed in the first aspect above.

[0020] The fifth aspect is a computer-readable storage medium of the present application, wherein the computer-readable storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the method designed in the first aspect are implemented.

[0021] The sixth aspect is a computer program product of the present application, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps in the method designed in the first aspect above.

[0022] It can be seen that in order to avoid the situation where the line segments detected by the edge line segments are incomplete, missing or redundant, which will lead to inaccurate detected quadrilaterals, the embodiment of the present application utilizes the characteristics of noise. By performing time series filtering on the line segment detection images of the two frames of images before and after, the instability caused by random disturbances and noise in the scanning (shooting / capturing, etc.) process is reduced, and the line segment detection image is stabilized on the time axis, which is beneficial to improve the accuracy and precision of the quadrilateral, and further beneficial to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0024] Figure 1 is a schematic diagram of the architecture of an image processing system according to an embodiment of the present application;

[0025] Figure 2 is a structural schematic diagram of an image processing method according to an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of a line segment detection and noise reduction processing method according to an embodiment of the present application;

[0027] Figure 4 It is a structural schematic diagram of a homography matrix fusion calculation in an embodiment of the present application;

[0028] Figure 5 is a flowchart of an image processing method according to an embodiment of the present application;

[0029] Figure 6 It is a block diagram of the functional units of an image processing device according to an embodiment of the present application;

[0030] Figure 7 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to better understand the technical solution of the present application by those skilled in the art, the technical solution in the embodiments of the present application is described below in conjunction with the drawings in the embodiments of the present application. It is obvious that the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. With respect to the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0032] It should be understood that the terms "first", "second", etc. involved in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, software, product, or device that includes a series of steps or units is not limited to the listed steps or units, but also includes steps or units that are not listed, or also includes other steps or units inherent to these processes, methods, products, or devices.

[0033] The "embodiment" involved in the embodiments of the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In the embodiments of the present application, "at least one" refers to one or more, and "a plurality" refers to two or more.

[0035] The "and / or" in the embodiments of the present application describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural. The character " / " can indicate that the previous and next associated objects are in an "or" relationship. In addition, the symbol " / " can also represent a division sign, that is, performing a division operation.

[0036] In the embodiments of the present application, "at least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0037] In the embodiments of the present application, "scan" can be expressed as the same meaning or interpretation as "capture", "photograph", etc., such as scanning an object, capturing an object, photographing an object, etc. have the same meaning, and there is no specific limitation on this.

[0038] The "connection" in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and there is no specific limitation on this.

[0039] In the process of scanning (shooting / capturing, etc.) documents, business cards, certificates, materials, posters and other objects into images, since the scanned objects usually have quadrilaterals, but due to the non-fixed scanning position and other reasons, the quadrilaterals in the scanned images are often perspectively transformed into tilted irregular quadrilaterals, resulting in tilt, distortion and other phenomena in the images, which is not conducive to subsequent use such as reading and printing. Therefore, it is necessary to correct the quadrilaterals in the scanned images.

[0040] In order to realize image-based plane scanning, it is usually necessary to extract the range of a quadrilateral from the image, so as to calculate a transformation matrix for correction based on the range of the quadrilateral, and then the target quadrilateral can be corrected based on the transformation matrix.

[0041] In addition, before extracting a quadrilateral from a scanned image, it is usually necessary to first perform edge line segment detection on the image to obtain line segments in the image, and then find the best quadrilateral based on the line segments.

[0042] However, the detected line segments may be incomplete, missing or redundant. For example, only a portion of the edge contour of an object in an image is detected, resulting in missing line segments, while the contours of other parts of the image are misdetected, resulting in redundant line segments, making it impossible to accurately correct the quadrilateral.

[0043] At the same time, in the process of scanning documents, business cards, certificates, materials, posters and other objects into images, they may be affected by disturbances and noise, resulting in random disturbances and noise in the scanned images, affecting edge segment detection.

[0044] Based on this, in order to deal with the situation where the line segments detected by edge line segments are incomplete, missing or redundant, the embodiment of the present application utilizes the characteristics of noise to perform time series filtering on the line segment detection images of the two frames of images before and after, thereby reducing the instability caused by random disturbances and noise in the scanning (shooting / capturing, etc.) process, and achieving stabilization of the line segment detection image on the time axis, which is beneficial to improve the accuracy and precision of the quadrilateral, and further beneficial to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning.

[0045] The technical solutions and related concepts involved in the embodiments of the present application are described in detail below.

[0046] 1. Electronic equipment

[0047] The electronic device of the embodiment of the present application may be a handheld device, a vehicle-mounted device, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, a projection device, a projector, or other devices connected to a wireless modem, or may be various specific forms of user equipment (UE), terminal device, terminal, mobile terminal, mobile phone, smart screen, smart TV, smart watch, laptop computer, smart speaker, camera, game controller, microphone, station (STA), access point (AP), mobile station (MS), personal digital assistant (PDA), personal computer (PC) or relay device, etc.

[0048] For example, the electronic device may be a wearable device. The wearable device may also be referred to as an intelligent wearable device, which is a general term for intelligent devices that use wearable technology to intelligently design and develop daily wearables, such as smart glasses, smart gloves, smart watches, smart bracelets for monitoring various specific features, smart jewelry, etc. The wearable device is a portable device that can be worn directly on the body or integrated into the user's clothing or accessories. The wearable device can be equipped with not only a dedicated hardware architecture, but also a dedicated software architecture for data interaction, cloud interaction, etc. The wearable intelligent device can achieve complete or partial functions without relying on other intelligent devices.

[0049] 1) Hardware and software structure of electronic equipment

[0050] ①Processor

[0051] In an embodiment of the present application, the electronic device may include a processor.

[0052] The processor can be used to run or load an operating system, which can be an Android operating system, an RTOS (real-time operating system) operating system, a UNIX operating system, a Linux operating system, a DOS operating system, a Windows operating system, a Mac operating system, etc.

[0053] The processor can be regarded as a complete system on chip (SOC).

[0054] The processor may include one or more processing units. For example, the processor may include at least one of a central processing unit (CPU), an application processor (AP), a microcontroller unit (MCU), a single chip microcomputer (SCM), a single chip microcomputer, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor, a neural-network processing unit (NPU), etc. Among them, different processing units may be separated or integrated together.

[0055] The processor may also include a memory for storing instructions and data.

[0056] For example, the processor may call a program stored in the memory to run an operating system.

[0057] For another example, the memory in the processor can store or cache instructions that the processor has just used or circulated. If the processor needs to use the instruction or data again, it can be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processor and improving system efficiency.

[0058] For another example, the memory in the processor can also store or cache data, and synchronize or transmit the data to other processors for execution. The memory in the processor can be a high-speed cache memory.

[0059] The processor may include one or more communication interfaces. The communication interface may include at least one of a serial peripheral interface (SPI), an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, a universal serial bus (USB) interface, etc.

[0060] ②Sensor components

[0061] In an embodiment of the present application, the electronic device may include a sensing component.

[0062] The sensing component may be a sensor.

[0063] The sensor module components may include inertial sensors (such as inertial motion units (IMUs)), pressure sensors, gyroscope sensors, air pressure sensors, magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, ultra-wideband UWB sensors, near-field communication NFC sensors, laser sensors and / or visible light sensors, etc.

[0064] ③Display components

[0065] In an embodiment of the present application, an electronic device may include a display component. The display component may be used to display a user interface, user interface elements and features, user-selectable controls, various displayable objects, and the like.

[0066] The display component may be a display screen, a touch screen, etc.

[0067] The display component may include a display panel, wherein the display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a quantum dot light-emitting diode (QLED), etc.

[0068] It should be noted that the electronic device can realize the display function through a GPU, a display component, and a processor. Among them, the GPU can be used to perform mathematical and geometric calculations and perform graphic rendering. In addition, the GPU can be a microprocessor for image processing and connect the display component and the processor. The processor may include one or more GPUs, which execute program instructions to generate or change display information.

[0069] ④Camera component

[0070] In an embodiment of the present application, the electronic device may include a camera component.

[0071] The camera component may be a camera or a camera module, which is used to capture (shoot / scan, etc.) static / dynamic images or videos.

[0072] The camera assembly may include a lens, a photosensitive element, etc., and the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.

[0073] Therefore, the object can generate an optical image through the lens and project it onto the photosensitive element. The photosensitive element can convert the light signal in the optical image into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0074] It should be noted that electronic devices can realize functions such as capturing (shooting / scanning, etc.) images through ISP, DSP, camera components, video codecs, GPU, display components and processors.

[0075] ISP can be used to process the data fed back by the camera component. For example, when taking a photo, the shutter is opened first, and then the light is transmitted through the lens of the camera component to the photosensitive element of the camera component to convert the light signal into an electrical signal, and finally the electrical signal is transmitted to the ISP through the photosensitive element to be converted into a digital image.

[0076] In addition, ISP can also perform algorithm optimization on image noise, brightness, and skin color.

[0077] ISP can also optimize parameters such as exposure and color temperature of the shooting scene.

[0078] In some possible examples, the ISP and / or DSP may be provided in the camera assembly.

[0079] ⑤Input driver

[0080] In an embodiment of the present application, the electronic device may include an input driver to process various inputs from a user operating the electronic device.

[0081] For example, when the display screen is a touch screen, the input driver can be operated to detect and process various touch inputs and / or touch events. Among them, the touch input or touch event on the touch screen can simultaneously indicate the area of ​​interest and start scanning the object (such as a document). The object can be displayed on the touch screen as a preview of the image to be scanned, and the touch event at a specific location on the touch screen indicates the image that should be scanned.

[0082] 2. Frame image

[0083] It should be noted that when an electronic device is used to perform image-based plane scanning (photographing / capturing, etc.) of an object such as a document, business card, certificate, information, or poster, multiple frames of images may be generated on a time axis. That is, each frame of image has a sequential order in time sequence.

[0084] For example, a user may aim a camera component of an electronic device at an object (such as a certificate) and use a button, touch, or other suitable input to initiate scanning (shooting / capturing, etc.) of the object so as to capture an image of the object. When the user initiates scanning (shooting / capturing, etc.), a scanning operation is performed to capture multiple frames of images of the object. The image capture may initiate various processing of the captured images to create one or more scanned documents for the captured multiple frames of images.

[0085] In addition, a frame image may also be referred to as a frame picture, an image frame, a picture frame, etc., without any specific limitation.

[0086] In order to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning (shooting / capturing, etc.), the embodiment of the present application needs to perform image processing on the two frames of images that are in the order of time. Therefore, in order to facilitate distinction, the two frames of images that are in the order of time (or adjacent) can be the "first frame of image" and the "second frame of image", and the "first frame of image" is the frame image that is located before the "second frame of image" in the order of time.

[0087] Of course, the “first frame image” and the “second frame image” may also be described using other terms, and there is no specific limitation to this.

[0088] In addition, the “second frame image” can be understood as the current frame image, and the “first frame image” can be understood as the N (eg, 1) frame images before the current frame image in time sequence.

[0089] In the following description, the embodiment of the present application will be explained by taking the "second frame image" as the nth frame image or the current frame image, and the "first frame image" as the n-1th frame image or the previous frame image of the current frame image.

[0090] 3. An image processing system

[0091] In order to improve the accuracy, precision and stability of the scanned quadrilateral image, an embodiment of the present application proposes an image processing system.

[0092] For example, Figure 1 As shown, the image processing system 10 may include at least one of the following:

[0093] The input module 110 may be used to input multiple frames of images (including the n-1th frame image and the nth frame image) into the temporal filtering line segment detection module in a time sequence;

[0094] The line segment detection and noise reduction processing module 120 may be used to perform edge line segment detection on the n-1th frame image and the nth frame image to obtain a line segment detection image, and perform noise reduction processing on the line segment detection image to obtain a more stable line segment detection image;

[0095] The quadrilateral detection module 130 may be used to detect quadrilaterals in a line segment detection image obtained by performing line segment detection and noise reduction processing on the n-1th frame image and the n-th frame image, and / or detect quadrilaterals in a line segment detection image obtained by performing line segment detection and noise reduction processing on the n-2th frame image and the n-1th frame image;

[0096] The stabilization filtering module 140 may be used to perform stabilization filtering on the first quadrilateral image and the second quadrilateral image to obtain a final quadrilateral image (for the sake of distinction, the quadrilateral image may be referred to as a target quadrilateral image or other terms);

[0097] The output module 150 may be used to output the final quadrilateral image to a preview screen or store it.

[0098] It should be noted that the modules included in the image processing system 10 may be divided according to functions, two or more functions may be integrated into one module, and may be hardware modules and / or software modules that execute the corresponding functions. In other words, the modules may be implemented in the form of hardware or software.

[0099] In addition, the modules included in the image processing system 10 can also be regarded as units, that is, the modules can be units, and there is no specific limitation on this. At the same time, the division of modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation. Each module is described in detail below.

[0100] (1) Line segment detection and noise reduction processing module 120

[0101] The line segment detection and noise reduction processing module 120 may include at least one of the following: a feature point detection and tracking module, a feature point matching module, an edge line segment detection module, an image affine transformation module, and a temporal filtering module.

[0102] 1) Feature point detection and tracking module

[0103] The feature point detection and tracking module can be used to detect and track feature points on the n-1th frame image and the nth frame image.

[0104] In addition, the feature point detection and tracking module can also be used to estimate the position of specific points on the n-1th frame image and the nth frame image, and save the position of the feature points so as to track the feature points on subsequent frame images.

[0105] ① Feature point detection

[0106] It should be noted that the feature point detection can be implemented by a feature point detection algorithm, which may include a hash corner point detection algorithm, a SIFT algorithm, an ORB algorithm, a SURF algorithm, an AKAZE algorithm, a BRISK algorithm, and the like.

[0107] In some possible implementations, the n-1th frame image and the nth frame image can be divided into blocks to obtain N*N image blocks of equal size and non-overlapping, where the value of N can be 2, 4, 8 or 16, etc., and then feature point detection is performed on each image block to obtain feature points.

[0108] ② Feature point tracking

[0109] It should be noted that after performing feature point detection on the n-1th frame image, feature points can be tracked in the nth frame image by feature point tracking without performing feature point detection again, so as to improve processing efficiency.

[0110] In some possible implementations, feature point tracking may be implemented by a feature point tracking algorithm, wherein the feature point tracking algorithm includes an optical flow method.

[0111] For example, the feature points between two consecutive (or adjacent) frames of images are tracked using the Lukas-Kanade optical flow method.

[0112] In addition, when the feature points are tracked in the frame image, the number of feature points obtained by tracking may become smaller and smaller, the obtained feature points may become more and more concentrated, or it may be necessary to track in multiple frame images. In this regard, the embodiment of the present application can be specifically implemented as follows:

[0113] If the number of feature points detected by the n-1th frame image (nth frame image) after feature point tracking is too small (for example, the number of feature points is less than a preset threshold) or the feature points are too concentrated (for example, the distance between feature points is less than a preset threshold), the feature point detection can be performed again on the n-1th frame image (nth frame image) to ensure that there are always enough feature points available to avoid affecting the subsequent feature point matching and improve the accuracy of image processing;

[0114] If the feature points detected by the n-1th frame image (nth frame image) are too concentrated after feature point tracking (for example, the distance between the feature points is less than a preset threshold), the feature point detection can be performed again on the n-1th frame image (nth frame image) to ensure that there are always enough available feature points to avoid affecting the subsequent feature point matching and improve the accuracy of image processing;

[0115] If the number of frame images required to track the feature points detected by the n-1th frame image (nth frame image) is too large (such as the number is greater than a preset threshold), the feature point detection can be performed again on the n-1th frame image (nth frame image) to ensure that there are always enough feature points available to avoid affecting the subsequent feature point matching and improve the accuracy of image processing.

[0116] 2) Feature point matching module

[0117] The feature point matching module can be used to perform feature point matching on the feature points detected in the n-1th frame image and the nth frame image to obtain a matching result.

[0118] It should be noted that the matching result can be understood as the corresponding positional relationship of each feature point in the n-1th frame image in the nth frame image.

[0119] Feature point matching can be understood as matching the feature points detected in the n-1th frame image and the nth frame image respectively, and finding the corresponding relationship between the n-1th frame image and the nth frame image, which can be achieved by methods such as RANSAC.

[0120] In addition, feature point matching can also be understood as searching and matching the feature points of the n-1th frame image to the nth frame image to obtain the best motion vector (MV). The search and matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, etc.

[0121] 3) Edge segment detection module

[0122] The edge line segment detection module can be used to perform edge line segment detection on the n-1th frame image to obtain a line segment detection image (for the sake of distinction, the line segment detection image can be called a "first line segment detection image" or other terms), and to perform edge line segment detection on the nth frame image to obtain a line segment detection image (for the sake of distinction, the line segment detection image can be called a "second line segment detection image" or other terms).

[0123] It should be noted that according to computer vision theory, the human eye recognizes an object based on its edge contours. Similarly, a computer vision system needs to imitate human vision to recognize an object in an image based on the edge contour features of the object.

[0124] Since edge contours can be composed of edge segments, and edge segments have simple geometric features and good geometric resolution, edge segments can be a way to describe edge contour features.

[0125] Based on this, the embodiment of the present application can perform edge line segment detection on the n-1th frame image and the nth frame image to obtain an edge line segment image.

[0126] In addition, edge segmentation is performed on the input n-1th frame image and the nth frame image to obtain the edge contour features of the object in each frame image. Generally, edge segment detection is used as a pre-processing for detecting quadrilaterals in object detection. Canny algorithm can be used, or CNN-based algorithms (such as HED) can be used to obtain better results.

[0127] In some possible implementations, edge segment detection may be implemented by an edge segment detection algorithm, which may include Roberts algorithm, Prewitt algorithm, Sobel algorithm, Canny algorithm, Laplacian algorithm, Hough Transform, and the like.

[0128] 4) Image affine transformation module

[0129] The image affine transformation module can be used to perform image affine transformation on the edge line segment image of the n-1th frame image to obtain a new edge line segment image (for ease of distinction, the new line segment detection image can also be called the "third line segment detection image" or other terms).

[0130] It should be noted that image warping can be understood as mapping each pixel in the image to a new position according to a certain rule (such as spatial coordinate transformation). Therefore, the embodiment of the present application can perform image affine transformation on the edge segment image of the n-1th frame image, thereby obtaining the image of the edge segment image of the n-1th frame image at the current viewing angle.

[0131] In addition, image affine transformation can also be understood as perspective transformation.

[0132] In some possible implementations, the image affine transformation may include a homography transformation.

[0133] For example, the image affine transformation module may include a homography matrix calculation module and a homography transformation module.

[0134] The homography matrix calculation module can be used to calculate the homography matrix according to the feature point matching results, so as to obtain the perspective transformation relationship between the n-1th frame image and the nth frame image;

[0135] The homography transformation module can be used to perform homography transformation on the line segment detection image of the n-1th frame image using the homography matrix, so as to obtain a line segment detection image overlapping with the nth frame image (ie, the "third line segment detection image").

[0136] In some possible implementations, the homography matrix can be calculated using algorithms such as RANSAC (Random Sample Consensus), PROSAC (Progressive Sampling Consensus), minimum median method, and least squares method.

[0137] In some possible implementations, the homography matrix calculation module can also add the input of sensors such as IMU (Inertial Motion Unit), that is, the data information (such as movement information) obtained by the IMU and other sensors and the feature point matching results (such as movement information such as motion vectors obtained by feature point matching) are fused to calculate the best homography matrix. Among them, the fusion algorithm can use algorithms such as EKF and MSCKF.

[0138] 5) Timing filter module

[0139] The temporal filtering module can be used to perform temporal filtering on the second line segment detection image and the third line segment detection image to obtain a new line segment detection image (for ease of distinction, the new line segment detection image may also be referred to as a "fourth line segment detection image" or other terms).

[0140] It can be understood that the embodiment of the present application can perform time-series filtering on the line segment detection image of the nth frame image and the line segment detection image of the n-1th frame image after the image affine transformation (such as homography transformation), so as to realize the fusion of the line segment detection image of the nth frame image and the line segment detection image of the n-1th frame image after the image affine transformation. The fused line segment detection image is more complete and can avoid noise interference.

[0141] It should be noted that scanning devices, such as complementary metal-oxide-semiconductor (CMOS) sensors and charge-coupled device (CCD) sensors, may be affected by disturbances and noise in the process of scanning documents, business cards, certificates, materials, posters and other objects into images, resulting in random disturbances and noise in the frame images. Therefore, image noise reduction technology is needed to remove the random disturbances and noise.

[0142] Image denoising methods can be divided into spatial domain, frequency domain, wavelet domain, time domain, space-time domain, etc. according to different processing domains. Among them, denoising methods between different processing domains may overlap, or one denoising method may involve multiple processing domains.

[0143] For example, a frequency domain filtering denoising method may be used in the time domain or spatiotemporal domain denoising method, that is, after converting the frame image to the frequency domain through Fourier transform, time domain filtering or spatiotemporal domain filtering is used for denoising.

[0144] Spatial domain filtering is to directly perform algebraic operations on pixel values ​​of frame images in the video stream, and only consider the correlation of frame images in the spatial domain.

[0145] Frequency domain filtering is to convert the frame image into the frequency domain through Fourier transform, and then attenuate the frequency representing the noise to retain the original information in the frame image to the greatest extent.

[0146] Wavelet domain filtering is to convert the frame image into the time-frequency domain and then perform noise reduction.

[0147] Time domain filtering is a noise reduction filtering method that considers the correlation of frame images in the time dimension. It has simple calculation, high efficiency, and does not introduce spatial blur.

[0148] Based on this, the timing filtering in the embodiment of the present application can also be understood as a time domain filtering.

[0149] In some possible implementations, temporal filtering can be implemented by a temporal filtering algorithm, wherein the temporal filtering algorithm can adopt a calculation method of weighted average of pixel values, a calculation method of maximum pixel values, a calculation method of minimum pixel values, a calculation method of linear superposition of pixel values, etc.

[0150] For example, the pixel value range is (0, 255), the pixel value of the second line segment detection image is X, and the pixel value of the third line segment detection image is Y. Therefore, after the second line segment detection image and the third line segment detection image are temporally filtered, the pixel value X and the pixel value Y are fused to obtain the pixel value Z. The pixel value Z can exist as follows:

[0151] Z = min(X,Y); or,

[0152] Z = max(X,Y); or,

[0153] Z=a*X+b*Y, a+b=1; a represents a preset weight (such as 0.4), and b represents a preset weight (such as 0.6).

[0154] (2) Quadrilateral detection module 130

[0155] The quadrilateral detection module 130 can be used to perform quadrilateral detection on the line segment detection image obtained by performing line segment detection and noise reduction processing on the n-1th frame image and the nth frame image to obtain a quadrilateral image (for the sake of distinction, the quadrilateral image may also be referred to as a "first quadrilateral image" or other terms), and to perform quadrilateral detection on the line segment detection image obtained by performing line segment detection and noise reduction processing on the n-2th frame image and the n-1th frame image to obtain a quadrilateral image (for the sake of distinction, the quadrilateral image may also be referred to as a "second quadrilateral image" or other terms).

[0156] In some possible implementations, quadrilateral detection may be implemented by a quadrilateral detection algorithm, wherein the quadrilateral detection algorithm may include a Hough transform algorithm, a random Hough transform algorithm, a least squares straight line fitting algorithm, a RANSAC algorithm, and the like.

[0157] It should be noted that there are multiple line segments in the line segment detection image obtained by edge line segment detection. However, multiple line segments are used as inputs of the quadrilateral detection algorithm, and the final detected quadrilateral image is obtained through the steps of merging and deleting the initial input line segments, initial quadrilateral detection, and final quadrilateral generation. The quadrilateral image can be used for feature correspondence of multiple frame images, and can also be used for recognition, classification, and detection of image objects.

[0158] (3) Stable filter module 140

[0159] The stabilization filtering module 140 may be used to perform stabilization filtering on the first quadrilateral image and the second quadrilateral image to obtain a final quadrilateral image.

[0160] It should be noted that the object of the stabilization filter is the quadrilateral image obtained by detection, and its function is to make the display of the detection result more stable.

[0161] 4. An image processing method

[0162] In combination with the content in “3. An image processing system” above, an embodiment of the present application proposes an image processing method.

[0163] For example, Figure 2 The image processing method may include the following steps:

[0164] 1) Input frame image

[0165] It should be noted that the multiple frames of images (including the n-2th frame of image, the n-1th frame of image and the nth frame of image) are input in time sequence.

[0166] It can be understood that the n-2th frame image is the previous frame image of the n-1th frame image, the n-1th frame image is the previous frame image of the nth frame image, and the nth frame image may be the current frame image.

[0167] In addition, inputting a frame image may also be understood as acquiring a frame image, and there is no specific limitation on this.

[0168] 2) Line segment detection and noise reduction

[0169] It should be noted that the line segment detection and noise reduction processing may include at least one of the following: feature point detection and tracking, feature point matching, edge line segment detection, homography matrix calculation, homography transformation, and time series filtering.

[0170] In addition, the embodiment of the present application needs to perform line segment detection and noise reduction processing on the n-1th frame image and the nth frame image, and also needs to perform line segment detection and noise reduction processing on the n-2th frame image and the n-1th frame image. The embodiment of the present application is specifically described below by taking the line segment detection and noise reduction processing on the n-1th frame image and the nth frame image as an example, and the line segment detection and noise reduction processing between the n-2th frame image and the n-1th frame image can be understood in the same way.

[0171] ① Line segment detection and noise reduction processing of the n-1th frame image and the nth frame image

[0172] For example, Figure 3 As shown, there are the following implementations:

[0173] Feature point detection and tracking

[0174] In a specific implementation, feature point detection and tracking are performed on the n-1th frame image 310 to obtain a feature point detection result 3101 , and feature point detection and tracking are performed on the nth frame image 320 to obtain a feature point detection result 3201 .

[0175] It should be noted that the feature points in the detected frame image are used for feature point detection, and feature point detection and tracking are used to ensure that there are always enough available feature points.

[0176] Feature point matching

[0177] In specific implementation, feature point matching is performed on the feature point detection result 3101 and the feature point detection result 3201 to obtain a feature point matching result.

[0178] It should be noted that feature matching is performed on the feature points detected in the n-1th frame image 310 and the feature points detected in the nth frame image 320, that is, the corresponding position relationship of each feature point in the n-1th frame image 310 in the nth frame image 320 is found. Therefore, the feature point matching result can be the corresponding relationship of the feature points.

[0179] Edge segment detection

[0180] In a specific implementation, edge line segment detection is performed on the n-1th frame image 310 to obtain a line segment detection image 3102 , and edge line segment detection is performed on the nth frame image 320 to obtain a line segment detection image 3202 .

[0181] It should be noted that edge line segmentation is performed on the input n-1th frame image 310 and the nth frame image 320 to obtain edge contour features of the object in each frame image. Generally, edge line segment detection is used as a pre-processing for detecting quadrilaterals in object detection, and the Canny algorithm or a CNN-based algorithm (such as HED) can be used to obtain better results.

[0182] However, it is difficult for edge segment detection to ensure that the edges of objects (such as documents) can be completely detected. Edge segments are often incomplete or not detected. In addition, due to interference from lines in the background or texture, the detected segments are missing or redundant, which greatly interferes with subsequent quadrilateral detection and makes it impossible to accurately complete the detection of objects.

[0183] Based on this, the embodiment of the present application introduces timing filtering, thereby improving accuracy, precision and stability through timing filtering.

[0184] Homography matrix calculation

[0185] In specific implementation, the homography matrix is ​​determined according to the feature point matching results.

[0186] It should be noted that the homography matrix representing the overall perspective transformation relationship between the (n-1)th frame image 310 and the (n)th frame image 320 is calculated according to the correspondence relationship of the feature points.

[0187] Homography transformation

[0188] In specific implementation, the homography matrix is ​​used to perform homography transformation on the line segment detection image 3102 to obtain the line segment detection image 3103 .

[0189] It should be noted that the line segment detection image of the n-1th frame image 310 is homographically transformed using the calculated homography matrix, thereby obtaining a line segment detection image 3103 overlapping with the nth frame image 320, which is beneficial to improving accuracy, precision and stability.

[0190] Timing filtering

[0191] In a specific implementation, the line segment detection image 3103 and the line segment detection image 3202 are subjected to time series filtering to obtain the line segment detection image 3203 .

[0192] It should be noted that the line segment detection image 3103 and the line segment detection image 3202 are mixed to eliminate noise interference, so that the mixed line segment detection image 3203 is more complete.

[0193] Assume that the image pixel range is (0, 255), the pixel value of the line segment detection image 3103 is X, the pixel value of the line segment detection image 3202 is Y, and the pixel value X and the pixel value Y are mixed to obtain the pixel value Z, where the pixel value Z can exist as follows:

[0194] Z = min(X,Y); or,

[0195] Z = max(X,Y); or,

[0196] Z=a*X+b*Y, a+b=1; a represents a preset weight (such as 0.4), and b represents a preset weight (such as 0.6).

[0197] Homography matrix calculation

[0198] ② Homography matrix fusion calculation

[0199] In the above-mentioned homography matrix calculation, the embodiment of the present application can also add the input of sensors such as IMU, that is, the data information (such as movement information) obtained by the IMU and other sensors is fused with the feature point matching results (such as movement information such as motion vectors obtained by feature point matching) to calculate the best homography matrix, such as Figure 4 As shown in the figure, the fusion algorithm can adopt EKF, MSCKF and other algorithms.

[0200] 3) Quadrilateral detection

[0201] It should be noted that the embodiments of the present application can perform quadrilateral detection on the line segment detection image obtained by performing line segment detection and noise reduction processing on the n-1th frame image and the nth frame image to obtain a quadrilateral image, and perform quadrilateral detection on the line segment detection image obtained by performing line segment detection and noise reduction processing on the n-2th frame image and the n-1th frame image to obtain a quadrilateral image, and find the best quadrilateral therefrom.

[0202] 4) Stable filtering

[0203] It should be noted that the embodiment of the present application can perform stability filtering on the first quadrilateral image and the second quadrilateral image to obtain the final quadrilateral image (i.e., the target quadrilateral image). The stable filtering can make the quadrilateral in the frame image more stable, which is beneficial to improving the stability of the image obtained by scanning.

[0204] 5) Output quadrilateral image

[0205] Finally, the embodiment of the present application can output the final quadrilateral image.

[0206] 5. An exemplary description of an image processing method

[0207] In combination with the above description, an image processing method according to an embodiment of the present application is illustrated below.

[0208] like Figure 5 As shown, Figure 5 This is a flowchart of an image processing method according to an embodiment of the present application, which can be applied to electronic devices, chips, chip modules, image processing systems, image processing devices, etc., and can specifically include the following steps:

[0209] S510 , acquiring a first frame image and a second frame image, wherein the first frame image is located before the second frame image in time sequence.

[0210] It should be noted that the relevant descriptions of the “first frame image” and the “second frame image” can be found in the above description and will not be repeated here.

[0211] In some possible implementations, the first frame image and the second frame image may be obtained by scanning (photographing / capturing, etc.) by a camera component of the electronic device.

[0212] S520: Perform edge line segment detection on the first frame image to obtain a first line segment detection image, and perform edge line segment detection on the second frame image to obtain a second line segment detection image.

[0213] It should be noted that the “edge segment detection” can be found in the above description in detail and will not be repeated here.

[0214] S530: Perform image affine transformation on the first line segment detection image to obtain a third line segment detection image.

[0215] It should be noted that the relevant description of “image affine transformation” can be found in the above description and will not be repeated here.

[0216] Therefore, the embodiment of the present application can perform an image affine transformation on the edge line segment image of the first frame image (i.e., the first line segment detection image), so that the first line segment detection image is subjected to the same affine transformation as the second line segment detection image, thereby obtaining the second line segment detection image of the first line segment detection image at the current viewing angle of the second frame image. The image affine transformation can also be understood as a perspective transformation.

[0217] In some possible implementations, the image affine transformation may include a homography transformation, wherein the homography transformation may be determined by the first frame image and the second frame image, and may be pre-configured, pre-stored, or pre-set, and no specific limitation is imposed on this.

[0218] For example, if the homography transformation is determined by the first frame image and the second frame image, it can be specifically implemented as follows:

[0219] A homography matrix between the first frame image and the second frame image is obtained; and the first line segment detection image is homographically transformed using the homography matrix to obtain a third line segment detection image.

[0220] It can be seen that the image affine transformation of the first line segment detection image is achieved through the homography matrix determined by the first frame image and the second frame image.

[0221] In addition, the homography matrix determined by the first frame image and the second frame image can be determined based on the feature point matching result between the first frame image and the second frame image, and can be pre-configured, pre-stored, and pre-set, and there is no specific limitation on this.

[0222] For example, if the homography matrix is ​​determined based on the feature point matching result between the first frame image and the second frame image, it can be specifically implemented as follows:

[0223] Obtaining the homography matrix between the first frame image and the second frame image may include the following steps: performing feature point detection and tracking on the first frame image, and performing feature point detection and tracking on the second frame image; performing feature point matching on the feature points detected in the first frame image and the feature points detected in the second frame image to obtain feature point matching results; and determining the homography matrix based on the feature point matching results.

[0224] It can be seen that the homography matrix is ​​determined by detecting and tracking the feature points of the first frame image and the second frame image, and thus it is easy to implement.

[0225] In addition, in order to ensure that the best / more accurate homography matrix is ​​calculated, the embodiment of the present application can add the input of sensors such as IMU, so it can be specifically implemented as follows:

[0226] The data information obtained by the IMU is obtained, and the homography matrix is ​​determined according to the feature point matching result and the data information.

[0227] In addition, in tracking feature points using feature points detected by frame images, in order to ensure that there are always enough available feature points to avoid affecting subsequent feature point matching and improve the accuracy of image processing, the embodiment of the present application can re-detect and track feature points based on feature point tracking, which can be specifically implemented as follows:

[0228] If the first condition is met, the feature point detection and tracking are performed again on the first frame image, and the first condition is one of the following: the number of feature points obtained after the feature point tracking of the feature points detected in the first frame image is less than the first preset threshold, the distance between the feature points obtained after the feature point tracking of the feature points detected in the first frame image is less than the second preset threshold, and the number of frame images required to track the feature points detected in the first frame image is greater than the third preset threshold; or,

[0229] If the second condition is met, feature point detection and tracking are performed again on the second frame image, and the second condition is one of the following: the number of feature points obtained after feature point tracking of the feature points detected in the second frame image is less than the first preset threshold, the distance between the feature points obtained after feature point tracking of the feature points detected in the second frame image is less than the second preset threshold, and the number of frame images required to track the feature points detected in the second frame image is greater than the third preset threshold.

[0230] It should be noted that the "first condition" and the "second condition" are only for the convenience of distinction, and other terms may also be used without specific limitation.

[0231] Similarly, the “first preset threshold”, “second preset threshold” and “third preset threshold” are only for the convenience of distinction, and other terms may also be used without specific limitation.

[0232] S540: Perform temporal filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image.

[0233] It should be noted that the relevant description of “timing filtering” can be found in the above description and will not be repeated here.

[0234] In order to realize timing filtering, it can be implemented as follows:

[0235] The pixel values ​​in the second line segment detection image and the pixel values ​​in the third line segment detection image are calculated to obtain a fourth line segment detection image, and the calculation processing includes one of the following: weighted average calculation, maximum value calculation, and minimum value calculation.

[0236] S550: Perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

[0237] It should be noted that the relevant instructions for “quadrilateral detection” can be found in the above description and will not be repeated here.

[0238] In addition, in order to further improve the stability of the image obtained by scanning, the embodiment of the present application also needs to perform a stable filtering, which can be specifically implemented as follows:

[0239] A second quadrilateral image is acquired, where the second quadrilateral image is determined by a first frame image and a third frame image, and the third frame image is located before the first frame image in time sequence; and stable filtering is performed on the first quadrilateral image and the second quadrilateral image to obtain a target quadrilateral image.

[0240] It should be noted that the second quadrilateral image may be executed according to the above-mentioned step of acquiring the “first quadrilateral image”, and may be pre-configured, pre-stored, or pre-set, and there is no specific limitation on this.

[0241] For example, combining the above Figure 2 , according to the above steps of obtaining the "first quadrilateral image", the second quadrilateral image is obtained, which can be specifically implemented as follows:

[0242] Acquire a third frame image; perform edge line segment detection on the third frame image to obtain a fifth line segment detection image; perform image affine transformation on the fifth line segment detection image to obtain a sixth line segment detection image; perform time series filtering on the first line segment detection image and the sixth line segment detection image to obtain a seventh line segment detection image; perform quadrilateral detection on the seventh line segment detection image to obtain a second quadrilateral image.

[0243] It should be noted that the specific relevant implementation methods are detailed in the above description and will not be repeated here.

[0244] It can be seen that in order to avoid the situation where the line segments detected by the edge line segments are incomplete, missing or redundant, which will lead to inaccurate detected quadrilaterals, the embodiment of the present application utilizes the characteristics of noise. By performing time series filtering on the line segment detection images of the two frames of images before and after, the instability caused by random disturbances and noise in the scanning (shooting / capturing, etc.) process is reduced, and the line segment detection image is stabilized on the time axis, which is beneficial to improve the accuracy and precision of the quadrilateral, and further beneficial to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning.

[0245] 6. Exemplary description of an image processing device

[0246] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process of the method side. It is understandable that in order to realize the above functions, it is necessary to introduce the hardware structure and / or software module corresponding to each function. Those skilled in the art should know that, in combination with the methods, functions, modules, units or steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain method, function, module, unit or step is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described methods, functions, modules, units or steps for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0247] The embodiment of the present application can introduce the division of functional units / modules according to the above method embodiment. For example, each function can be divided into each functional unit / module, or two or more functions can be integrated into one functional unit / module. The above integrated functional unit / module can be implemented in hardware or in software program. It should be noted that the division of functional units / modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.

[0248] In the case of an integrated unit, Figure 6 The image processing device 600 comprises: a frame image acquisition unit 610 , an edge segment detection unit 620 , an image affine transformation unit 630 , a temporal filtering unit 640 and a quadrilateral detection unit 650 .

[0249] It should be noted that the frame image acquisition unit 610 may be a module / unit for acquiring or processing frame images, etc., and no specific limitation is imposed on this. In addition, the frame image acquisition unit 610 may be the input module 110 mentioned above.

[0250] The edge line segment detection unit 620 may be a module / unit for processing frame images, etc., and is not specifically limited thereto. In addition, the edge line segment detection unit 620 may be the line segment detection and noise reduction processing module 120 described above.

[0251] The image affine transformation unit 630 may be a module / unit for processing a frame image, etc., and is not specifically limited thereto. In addition, the image affine transformation unit 630 may be the line segment detection and noise reduction processing module 120 described above.

[0252] The temporal filtering unit 640 may be a module / unit for processing frame images, etc., and is not specifically limited thereto. In addition, the temporal filtering unit 640 may be the line segment detection and noise reduction processing module 120 described above.

[0253] The quadrilateral detection unit 650 may be a module / unit for processing frame images, etc., and is not specifically limited thereto. In addition, the quadrilateral detection unit 650 may be the quadrilateral detection module 130 described above.

[0254] In some possible implementations, the frame image acquisition unit 610, the edge segment detection unit 620, the image affine transformation unit 630, the temporal filtering unit 640 and the quadrilateral detection unit 650 may be separated from each other or integrated into the same unit.

[0255] In some possible implementations, the frame image acquisition unit 610, the edge segment detection unit 620, the image affine transformation unit 630, the temporal filtering unit 640 and the quadrilateral detection unit 650 can be integrated in the processing unit. The processing unit can be a processor or a controller, such as a central processing unit (CPU), a GPU, a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processing unit can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0256] In some possible implementations, the image processing apparatus 600 may further include a storage unit for storing computer programs or instructions executed by the image processing apparatus 600. The storage unit may be a memory.

[0257] In some possible implementations, the image processing device 600 may be a chip / chip module / processor / electronic device / operating system.

[0258] In specific implementation, the frame image acquisition unit 610, the edge segment detection unit 620, the image affine transformation unit 630, the temporal filtering unit 640 and the quadrilateral detection unit 650 are used to perform the steps described in the above method embodiment.

[0259] The frame image acquisition unit 610 is used to acquire a first frame image and a second frame image, where the first frame image is a frame image that is located before the second frame image in time sequence;

[0260] An edge line segment detection unit 620, configured to perform edge line segment detection on the first frame image to obtain a first line segment detection image, and perform edge line segment detection on the second frame image to obtain a second line segment detection image;

[0261] An image affine transformation unit 630, configured to perform an image affine transformation on the first line segment detection image to obtain a third line segment detection image;

[0262] A temporal filtering unit 640, configured to perform temporal filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image;

[0263] The quadrilateral detection unit 650 is configured to perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

[0264] It can be seen that in order to avoid the situation where the line segments detected by the edge line segments are incomplete, missing or redundant, which will lead to inaccurate detected quadrilaterals, the embodiment of the present application utilizes the characteristics of noise. By performing time series filtering on the line segment detection images of the two frames of images before and after, the instability caused by random disturbances and noise in the scanning (shooting / capturing, etc.) process is reduced, and the line segment detection image is stabilized on the time axis, which is beneficial to improve the accuracy and precision of the quadrilateral, and further beneficial to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning.

[0265] It should be noted that the specific implementation of each operation performed by the image processing device 600 can refer to the corresponding description of the above method embodiment, which will not be repeated here.

[0266] In some possible implementations, the image affine transformation includes a homography transformation; in performing an image affine transformation on the first line segment detection image to obtain a third line segment detection image, the image affine transformation unit 630 is used to: obtain a homography matrix between the first frame image and the second frame image; and perform a homography transformation on the first line segment detection image using the homography matrix to obtain a third line segment detection image.

[0267] In some possible implementations, in terms of obtaining the homography matrix between the first frame image and the second frame image, the image affine transformation unit 630 is used to: perform feature point detection and tracking on the first frame image, and perform feature point detection and tracking on the second frame image; perform feature point matching on the feature points detected in the first frame image and the feature points detected in the second frame image to obtain feature point matching results; and determine the homography matrix based on the feature point matching results.

[0268] In some possible implementations, the image processing apparatus 600 may further include:

[0269] A data information acquisition unit, used to acquire data information obtained by the inertial motion unit;

[0270] In terms of determining the homography matrix according to the feature point matching results, the image affine transformation unit 630 is used to determine the homography matrix according to the feature point matching results and the data information.

[0271] In some possible implementations, the image processing apparatus 600 may further include:

[0272] A repeated detection and tracking unit is used to re-detect and track feature points on the first frame image if a first condition is met, and the first condition is one of the following: the number of feature points obtained after tracking the feature points detected in the first frame image is less than a first preset threshold, the distance between the feature points obtained after tracking the feature points detected in the first frame image is less than a second preset threshold, and the number of frame images required to track the feature points detected in the first frame image is greater than a third preset threshold; or,

[0273] If the second condition is met, feature point detection and tracking are performed again on the second frame image, and the second condition is one of the following: the number of feature points detected in the second frame image after feature point tracking is less than a first preset threshold, the distance between the feature points detected in the second frame image after feature point tracking is less than a second preset threshold, and the number of frame images required to track the feature points detected in the second frame image is greater than a third preset threshold.

[0274] In some possible implementations, in performing temporal filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image, the temporal filtering unit 640 is used to: perform calculation processing on the pixel values ​​in the second line segment detection image and the pixel values ​​in the third line segment detection image to obtain the fourth line segment detection image, and the calculation processing includes one of the following: calculation of weighted average, calculation of maximum value, and calculation of minimum value.

[0275] In some possible implementations, after performing quadrilateral detection on the fourth line segment detection image to obtain the first quadrilateral image, the image processing device 600 may further include:

[0276] A quadrilateral image acquisition unit, used for acquiring a second quadrilateral image, wherein the second quadrilateral image is determined by the first frame image and the third frame image, and the third frame image is located before the first frame image in terms of time sequence;

[0277] The stable filtering unit is used to perform stable filtering on the first quadrilateral image and the second quadrilateral image to obtain a target quadrilateral image.

[0278] 7. Exemplary description of an electronic device

[0279] The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 The electronic device 700 includes a processor 710 , a memory 720 , and at least one communication bus for connecting the processor 710 and the memory 720 .

[0280] The processor 710 may be one or more central processing units (CPUs). In the case where the processor 710 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The memory 720 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 720 is used to store computer programs or instructions.

[0281] The electronic device 700 further includes a communication interface, which is used to receive and send data.

[0282] The processor 710 in the electronic device 700 is used to execute the computer program or instruction 721 stored in the memory 720 to implement the following steps:

[0283] Acquire a first frame image and a second frame image, wherein the first frame image is located before the second frame image in time sequence;

[0284] Performing edge line segment detection on the first frame image to obtain a first line segment detection image, and performing edge line segment detection on the second frame image to obtain a second line segment detection image;

[0285] Performing image affine transformation on the first line segment detection image to obtain a third line segment detection image;

[0286] Performing time-series filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image;

[0287] Perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

[0288] It can be seen that in order to avoid the situation where the line segments detected by the edge line segments are incomplete, missing or redundant, which will lead to inaccurate detected quadrilaterals, the embodiment of the present application utilizes the characteristics of noise. By performing time series filtering on the line segment detection images of the two frames of images before and after, the instability caused by random disturbances and noise in the scanning (shooting / capturing, etc.) process is reduced, and the line segment detection image is stabilized on the time axis, which is beneficial to improve the accuracy and precision of the quadrilateral, and further beneficial to improve the accuracy, precision and stability of the quadrilateral image obtained by scanning.

[0289] It should be noted that the specific implementation of each operation performed by the electronic device 700 can refer to the corresponding description of the above method embodiment, which will not be repeated here.

[0290] 8. Other exemplary explanations

[0291] An embodiment of the present application also provides a chip, which includes a processor, and the processor is used to implement the steps described in the above embodiment when executed.

[0292] An embodiment of the present application also provides a computer-readable storage medium, wherein a computer program or instructions are stored on the computer-readable storage medium, and when the computer program or instructions are executed by a processor, the steps described in the above embodiment are implemented.

[0293] The present application also provides a computer program product, including a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps described in the above embodiment. Exemplarily, the computer program product may be a software installation package.

[0294] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required by the embodiments of the present application.

[0295] In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0296] Those skilled in the art should be aware that the methods, steps or functions of related modules / units described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, or it can be implemented by a processor executing a computer program instruction. Wherein, the computer program product includes at least one computer program instruction, and the computer program instruction can be composed of corresponding software modules, and the software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, register, hard disk, mobile hard disk, read-only compact disk (CD-ROM) or any other form of storage medium known in the art. The computer program instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instruction can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (eg, an SSD), etc.

[0297] The modules / units included in the devices or products described in the above embodiments may be software modules / units, hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for each device or product applied to or integrated in a chip, each module / unit included therein may be implemented in the form of hardware such as circuits; or, a portion of the modules / units included therein may be implemented in the form of a software program, which runs on a processor integrated inside the chip, while a portion of the modules / units of another portion (if any) may be implemented in the form of hardware such as circuits. The same is true for each device or product applied to or integrated in a chip module, or each device or product applied to or integrated in a terminal.

[0298] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present application in detail. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first frame image and a second frame image, wherein the first frame image is located before the second frame image in time sequence; Performing edge line segment detection on the first frame image to obtain a first line segment detection image, and performing edge line segment detection on the second frame image to obtain a second line segment detection image; Performing image affine transformation on the first line segment detection image to obtain a third line segment detection image; The second line segment detection image and the third line segment detection image are subjected to time series filtering to obtain a fourth line segment detection image, specifically, by utilizing the characteristics of noise and performing time series filtering on the line segment detection images of two frames of images before and after in time series, so as to reduce the instability caused by random disturbance and noise in the scanning process; Perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

2. The method according to claim 1, characterized in that The image affine transformation includes homography transformation; The performing image affine transformation on the first line segment detection image to obtain a third line segment detection image includes: Acquire a homography matrix between the first frame image and the second frame image; The homography matrix is ​​used to perform the homography transformation on the first line segment detection image to obtain the third line segment detection image.

3. The method according to claim 2, characterized in that The acquiring a homography matrix between the first frame image and the second frame image includes: Performing feature point detection and tracking on the first frame image, and performing feature point detection and tracking on the second frame image; Performing feature point matching on the feature points detected in the first frame image and the feature points detected in the second frame image to obtain a feature point matching result; The homography matrix is ​​determined according to the feature point matching result.

4. The method according to claim 3, characterized in that Also includes: Obtain data information obtained by the inertial motion unit; The determining the homography matrix according to the feature point matching result includes: The homography matrix is ​​determined according to the feature point matching result and the data information.

5. The method according to claim 3, characterized in that: Also includes: If the first condition is met, re-detecting and tracking the feature points of the first frame image, the first condition being one of the following: the number of feature points obtained after tracking the feature points detected by the first frame image is less than a first preset threshold, the distance between the feature points obtained after tracking the feature points detected by the first frame image is less than a second preset threshold, and the number of frame images required to track the feature points detected by the first frame image is greater than a third preset threshold; or, If the second condition is met, feature point detection and tracking are performed again on the second frame image, and the second condition is one of the following: the number of feature points obtained after feature point tracking of the feature points detected in the second frame image is less than the first preset threshold, the distance between the feature points obtained after feature point tracking of the feature points detected in the second frame image is less than the second preset threshold, and the number of frame images required to track the feature points detected in the second frame image is greater than the third preset threshold.

6. The method according to claim 1, characterized in that The performing temporal filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image includes: The pixel values ​​in the second line segment detection image and the pixel values ​​in the third line segment detection image are calculated to obtain the fourth line segment detection image, and the calculation processing includes one of the following: weighted average calculation, maximum value calculation, and minimum value calculation.

7. The method according to any one of claims 1 to 6, characterized in that: After performing quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image, the method further includes: Acquire a second quadrilateral image, where the second quadrilateral image is determined by the first frame image and a third frame image, and the third frame image is located before the first frame image in time sequence; The first quadrilateral image and the second quadrilateral image are subjected to stable filtering to obtain a target quadrilateral image.

8. An image processing device, characterized in that: include: A frame image acquisition unit, used to acquire a first frame image and a second frame image, wherein the first frame image is a frame image that is located before the second frame image in time sequence; An edge line segment detection unit, configured to perform edge line segment detection on the first frame image to obtain a first line segment detection image, and perform edge line segment detection on the second frame image to obtain a second line segment detection image; An image affine transformation unit, configured to perform image affine transformation on the first line segment detection image to obtain a third line segment detection image; A time sequence filtering unit is used to perform time sequence filtering on the second line segment detection image and the third line segment detection image to obtain a fourth line segment detection image, specifically: utilizing the characteristics of noise, by performing time sequence filtering on the line segment detection images of two frames of images before and after in time sequence, so as to reduce the instability caused by random disturbance and noise in the scanning process; The quadrilateral detection unit is used to perform quadrilateral detection on the fourth line segment detection image to obtain a first quadrilateral image.

9. A chip, characterized in that: The method comprises a processor, wherein the processor executes the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 7.

Images