Image processing method and device, electronic equipment and storage medium

Abstract

The application provides an image processing method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: identifying a face region in a to-be-processed image, wherein the to-be-processed image is obtained by image acquisition on a certificate; determining a certificate region in the to-be-processed image according to the relative position and proportion between the face region and the certificate region, wherein the certificate region comprises the face region; performing segmentation processing on the to-be-processed image according to the certificate region to obtain a certificate image; obtaining a feature of the certificate image, performing quality detection processing based on the feature of the certificate image to obtain a detection result representing whether the certificate image is qualified. The application can accurately extract a qualified certificate image from the acquired image.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Image recognition technology is an important field of artificial intelligence. It refers to the technology of object recognition in images to identify targets and objects of various patterns. Currently, image recognition technology is becoming increasingly mature and is widely used in the recognition of faces and other objects.

[0003] In scenarios such as opening a bank account, it is necessary to scan and upload photos of documents, and the original documents must be photographed. At this time, the mobile phone will have a front-end guided detection to check whether the document is within the photo area, whether the size is appropriate, and whether the ID card is complete, etc., to obtain a document image that meets the requirements.

[0004] Among related technologies, traditional edge detection algorithms are greatly affected by background interference in photos, resulting in unstable performance, difficult parameter adjustment, inability to identify document types, and lack of quality checks such as photocopies, copies, and integrity checks. This makes it easy to collect non-compliant document images, leading to low efficiency. Furthermore, it is still difficult to accurately distinguish high-fidelity qualified images captured by high refresh rate and high-resolution imaging devices. Summary of the Invention

[0005] This application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can accurately extract qualified document images from acquired images.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an image processing method, including:

[0008] Identify facial regions in an image to be processed, wherein the image to be processed is obtained by image acquisition of an identification document;

[0009] The document region in the image to be processed is determined based on the relative position and proportion between the face region and the document region, wherein the document region includes the face region;

[0010] The image to be processed is segmented according to the document area to obtain the document image;

[0011] The features of the document image are obtained, and quality detection processing is performed based on the features of the document image to obtain a detection result characterizing whether the document image is qualified.

[0012] This application provides an image processing apparatus, including:

[0013] An image recognition module is used to identify facial regions in an image to be processed, wherein the image to be processed is obtained by image acquisition of an identification document;

[0014] An image detection module is used to determine the document region in the image to be processed based on the relative position and proportion between the face region and the preset document region, wherein the document region includes the face region;

[0015] The image segmentation module is used to segment the image to be processed according to the document area to obtain the document image;

[0016] The image quality detection module is used to acquire the features of the document image, perform quality detection processing based on the features of the document image, and obtain a detection result that characterizes whether the document image is qualified.

[0017] In the above scheme, the device further includes: a document type recognition module, used to perform document type recognition processing on the image to be processed to obtain the type of document in the image to be processed; and to query a mapping table according to the type to obtain the relative position and proportion between the face region and the document region in the image to be processed, wherein the mapping table includes the relative position and proportion between the face region and the document region in different types of document images.

[0018] In the above scheme, the relative position between the face region and the document region includes the relative position between the first center point of the face region and the second center point of the document region; the ratio between the face region and the document region includes a first ratio of the width of the face region to the width of the document region, and a second ratio of the height of the face region to the height of the document region; the type recognition module is further configured to move the first center point of the face region according to the relative position between the first center point of the face region and the second center point of the document region, so as to move to the second center point of the document region; using the second center point of the document region as a reference, the width of the face region is extended according to the first ratio, and the height of the face region is extended according to the second ratio to obtain the document region.

[0019] In the above scheme, the quality detection includes integrity detection; the image quality detection module is further used to acquire the content features of the document image; perform integrity classification processing on the document image based on the content features of the document image to obtain an integrity classification detection result; when the integrity classification detection result indicates that the document image is a complete image, the document image is determined to be a qualified image; when the integrity classification detection result indicates that the document image is an incomplete image, the document image is determined to be a unqualified image.

[0020] In the above scheme, the content features of the document image include the number of keywords, location information, and document type; the image quality detection module is also used to obtain document type rules corresponding to the document type; according to the document type rules, and based on the obtained number of keywords and location information, the document image is classified to obtain a classification detection result, wherein the preset document type rules include the number of keywords and location information corresponding to different document types.

[0021] In the above scheme, the quality detection includes photocopy detection; the image quality detection module is further used to acquire the texture features of the document image; based on the texture features, a binary classification model is invoked to perform photocopy classification processing on the document image to obtain a photocopy classification detection result; when the photocopy classification detection result indicates that the document image is a non-photocopy image, the document image is determined to be a qualified image; when the photocopy classification detection result indicates that the document image is a photocopy image, the document image is determined to be a non-qualified image.

[0022] In the above scheme, the quality detection includes sharpness detection; the image quality detection module is further used to acquire the visual features of the document image, wherein the visual features include at least one of the following: color information, brightness; based on the visual features, the document image is determined to undergo sharpness detection to obtain the sharpness detection result; when the sharpness detection result indicates that the document image is a sharp image, the document image is determined to be a qualified image; when the sharpness detection result indicates that the document image is a non-sharp image, the document image is determined to be a non-qualified image.

[0023] In the above scheme, the device further includes: a counterfeit document identification module, used to perform image acquisition and processing on the document from different angles to obtain multiple images to be processed including the document; to obtain image features of anti-counterfeiting marks in each image to be processed; to perform counterfeit document classification processing on the images to be processed according to the image features of the anti-counterfeiting marks to obtain a counterfeit document classification result; when the counterfeit document classification result indicates that the document is not a counterfeit document, it is determined that the process of identifying the face region in any of the images to be processed will be performed.

[0024] This application provides an electronic device, including:

[0025] Memory, used to store executable instructions;

[0026] The processor, when executing executable instructions stored in the memory, implements the image processing method provided in the embodiments of this application.

[0027] This application provides a computer-readable storage medium storing executable instructions for implementing the image processing method provided in this application when executed by a processor.

[0028] This application provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method provided in this application.

[0029] The embodiments of this application have the following beneficial effects:

[0030] By identifying the face region from the image to be processed, and efficiently and accurately determining the document region based on the relative position and proportion between the face region and the document region, a preliminary qualified document region is identified. This preliminary qualified document region can then be further identified by combining quality detection, resulting in a more accurate and efficient determination that the image to be processed is a qualified document image. Compared with related technologies using traditional edge detection algorithms, this approach eliminates background interference in the image to be processed, resulting in more stable performance. Furthermore, by adding quality detection, accurate differentiation of the image to be processed is achieved. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0033] Figure 3 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0034] Figure 4 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram illustrating the movement of the intersection point of the lines connecting the diagonals of the face region provided in this application embodiment;

[0036] Figures 6A-6B This is a schematic flowchart of the image recognition method provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram of an image recognition method provided in an embodiment of this application;

[0038] Figure 8 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0039] Figure 9 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0040] Figure 10 This is a schematic flowchart of an image recognition method provided in an embodiment of this application;

[0041] Figure 11 This is a schematic flowchart of an image recognition method provided in an embodiment of this application; Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0046] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0047] 1) A qualified image is an image that has passed quality inspection and is deemed qualified in terms of its reproducibility, completeness, brightness, and clarity.

[0048] 2) Image segmentation process: Image segmentation process is the technique and process of dividing an image into several specific regions with unique properties and extracting the target of interest.

[0049] 3) Face detection refers to using a certain strategy to search any given image to determine whether it contains a face, and if so, returning the position, size, and pose of the face.

[0050] The applicant discovered that in order to obtain qualified document images, the relevant technologies typically use traditional edge detection algorithms that are greatly affected by the background of the photo, resulting in unstable performance, difficult parameter adjustment, inability to identify document types, and lack of quality checks such as photographs, photocopies, and integrity checks, making it easy to collect non-compliant document images.

[0051] This application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can accurately identify images of qualified documents.

[0052] First, the image processing system provided in the embodiments of this application will be described, see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image processing system 100 provided in the embodiments of this application. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using a wireless link.

[0053] In some embodiments, terminal 400 may be a laptop, tablet, desktop computer, smartphone, dedicated messaging device, portable gaming device, smart speaker, smartwatch, etc., but is not limited thereto. Server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Network 300 may be a wide area network (WAN) or a local area network (LAN), or a combination of both. Terminal 400 and server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0054] Terminal 400 is used to send an image recognition request carrying an image to be recognized to server 200, so as to request server 200 to recognize whether the image to be recognized is a qualified image.

[0055] Server 200 is configured to: parse the face region from the image to be recognized and processed from the image recognition request; determine the document region in the image to be processed based on the relative position and proportion between the face region and a preset document region; segment the image to be processed based on the document region to obtain a document image; acquire the features of the document image; perform quality detection processing based on the features of the document image to obtain a detection result characterizing whether the document image is qualified; and return the detection result to terminal 400; terminal 400 is also configured to output the detection result of whether the image to be recognized is a qualified image.

[0056] In some embodiments, an image recognition client 410 is provided on the terminal 400. The user selects an image to be recognized based on the qualified image recognition client 410 and triggers an image recognition command based on the selected image. The qualified image recognition client 410 responds to the image recognition command by sending an image recognition request carrying the image to be recognized to the server. The server parses the face region in the image to be recognized from the image recognition request, wherein the image to be recognized is obtained by image acquisition of an ID card. The server determines the ID card region in the image to be recognized based on the relative position and proportion between the face region and the preset ID card region. The server segments the image to be recognized based on the ID card region to obtain an ID card image. The server acquires the features of the ID card image and performs quality detection processing based on the features of the ID card image to obtain a detection result characterizing whether the ID card image is qualified. The server returns the classification result to the qualified image recognition client 410, and the qualified image recognition client 410 outputs the classification result indicating whether the image to be recognized is a qualified image.

[0057] It should be noted that the image processing method provided in this application embodiment can be implemented by the terminal and the server in combination, or it can be implemented by the terminal independently.

[0058] The following describes an exemplary application of the image processing method provided in the embodiments of this application when the electronic device is a terminal.

[0059] For example, with Figure 1Taking terminal 400 as an example, terminal 400 is equipped with an image recognition client 410. The user selects an image to be recognized based on the image recognition client 410 and triggers an image recognition command based on the selected image. The image recognition client 410 responds to the image recognition command and parses the face region in the image to be recognized from the image recognition request. The image to be recognized is obtained by image acquisition of an ID card. The ID card region in the image to be recognized is determined according to the relative position and proportion between the face region and the preset ID card region. The image to be recognized is segmented according to the ID card region to obtain an ID card image. The features of the ID card image are obtained, and quality detection processing is performed based on the features of the ID card image to obtain a detection result characterizing whether the ID card image is qualified. The classification result is returned to the image recognition client 410, and the image recognition client 410 outputs the classification result of whether the image to be recognized is an image.

[0060] See Figure 2 , Figure 2 This is a schematic diagram of the electronic device 500 provided in an embodiment of this application. In practical applications, the electronic device 500 can be implemented as follows: Figure 1 Terminal 400 or server 200 in the middle, with electronic devices as Figure 1 Taking the server 200 shown as an example, an electronic device implementing the image processing method of this application embodiment will be described. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0061] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0064] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0065] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0066] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0067] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0068] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0069] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0070] In some embodiments, the user interface, presentation module, and output processing device described above may be omitted from the server's specifications.

[0071] In some embodiments, the image processing apparatus provided in this application can be implemented in software. Figure 2 An image processing apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: image recognition module 5551, feature detection module 5552, feature segmentation module 5553, image quality detection module 5554, document type recognition module 5555, and counterfeit document recognition module 5556. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0072] In other embodiments, the image processing apparatus provided in this application can be implemented in hardware. As an example, the image processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0073] See Figure 3 , Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this application, which will be combined with... Figure 3 The steps shown are explained.

[0074] In step 101, the face region in the image to be processed is identified, wherein the image to be processed is obtained by image acquisition of the document.

[0075] Here, the documents can be any common documents in real life, such as ID cards, passports, bank cards, driver's licenses, etc.

[0076] It should be noted that in the embodiments of this application, the image to be processed can be a specific area of ​​an image that has been cropped, or the image to be processed can be the original image captured by the camera, or it can be an image stored in the photo album of the user's electronic device. The face region is the area in the image to be processed that includes the face. The image acquisition can be real-time or non-real-time. The face region can be identified by obtaining the coordinates of the face bounding box through a face detection algorithm.

[0077] As an example, face detection algorithms include face detection algorithms based on active contour models, which can use face contours as the main features to describe faces. The active contour model method can model and extract the characteristics of deformed contours of arbitrary shapes. It requires collecting a large number of samples for learning and training to obtain the coordinates of the face bounding box.

[0078] In step 102, the document region in the image to be processed is determined based on the relative position and proportion between the face region and the document region, wherein the document region includes the face region.

[0079] In some embodiments, the document region in the image to be processed is determined based on the face region in a specific type of document. This specific type of document can be any one of an ID card, passport, bank card, or driver's license. For example, when only driver's license images are recognized, to determine whether the scanned document meets the requirements, the face region is directly determined based on the relative position and proportion between the face region in the driver's license image and the document region. The document region is determined based on the relative position between the intersection of the diagonals of the face region and the intersection of the diagonals of the document region, as well as the width and length proportion between the face region and the document region in the driver's license.

[0080] In other embodiments, the image to be processed is subjected to document type recognition processing to obtain the type of document in the image to be processed; according to the type, a mapping table is consulted to obtain the relative position and proportion between the face region and the document region in the image to be processed, wherein the mapping table includes the relative position and proportion between the face region and the document region in different types of document images.

[0081] As an example, depending on the type of document, the document area can be for ID cards, passports, bank cards, driver's licenses, etc. Since different types of documents have fixed sizes, the relative positions and proportions between the face area and the document area are fixed. Because different types of document photos have relatively fixed size requirements, based on experience, the positions and proportions corresponding to the face area and document area are preset for different documents and stored in a type lookup mapping table. For example, when the document type is an ID card, a point is determined in the face area, and the coordinates P1 of this point represent the position point of the face area. A corresponding point is determined in the document area, and the coordinates P2 of this point represent the position point of the document area. Coordinates P1 and P2 are stored in the type lookup mapping table. The area of ​​the face area is denoted as R1, and the area of ​​the document area is denoted as R2. Areas R1 and R2 are stored in a class lookup mapping table. By representing the relative positional relationship between the face area and document area of ​​different types of documents using point coordinates and storing their area relationship in the type lookup mapping table, it is possible to achieve processing compatible with various documents.

[0082] In some implementations, the type of document in the image to be processed is obtained through image recognition. For example, a document type classification model can be called, or the document type can be manually indicated by the user, or the document type that needs to be processed in the specific business scenario (such as real-name authentication) can be read to determine the document type.

[0083] It should be noted that the document type classification model can be various types of neural network models, such as convolutional neural networks, deep convolutional neural networks, fully connected neural networks, etc. The labels of the training samples in the document type classification model can be different types of document labels. The server can obtain labeled training images from the database, or it can obtain labeled training images by manual annotation, or it can obtain labeled training images by other methods. This application embodiment does not limit the scope of the invention.

[0084] As an example, taking the document type classification model as an image classification model based on convolutional neural networks, the training samples of the convolutional neural network model include normal document images. The labeled data of the training samples include the document size and keyword information in the normal document images. The document size and keyword information are extracted from the normal document images. The image features of each image in the first training set are directly extracted through the convolutional layer of the convolutional neural network model. The extracted image features are then sequentially input into the fully connected layer (used to connect the image features into a vector) and the pooling layer (used for average pooling or max pooling) of the convolutional neural network model to determine the prediction result corresponding to each image. The prediction result is then optimized through a loss function.

[0085] It should be noted that the loss function uses the error between the predicted image type and the corresponding label as a discrepancy factor, and minimizes this discrepancy factor. The types of loss functions can include Mean Squared Error (MSE), Hinge Loss Function (HLF), and Cross Entropy, etc.

[0086] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the method for creating an index table provided in this application embodiment. Step 102 can be achieved through... Figure 4 Steps 1021-1022 are implemented.

[0087] In step 1021, the first center point of the face region is moved relative to the second center point of the document region, so as to the second center point of the document region.

[0088] In some embodiments, Figure 5 The example illustrates the relative position and proportion between the face area and the document area when the identification document is an ID card. Figure 5 Midpoint A is the intersection of the diagonals of the document area, and B is the intersection of the diagonals of the face area. Point A is the origin of the coordinate system, the coordinates of B are (x, y), and the coordinates of A are (0, 0). The relative positional relationship between the face area and the document area can be represented by the coordinates between the points determined by the face area and the document area. It should be noted that, for example, the length of the ID card area is 85.6 mm and the width is 54 mm, while the length of the face area is 26 mm and the width is 37 mm. Therefore, the length ratio and width ratio of the ID card area to the face area are fixed. Thus, when the document type is determined, the length ratio of the document area to the face area is fixed for the same document type, and the width ratio of the document area to the face area is also fixed.

[0089] As an example, the first center point of the face region can be the intersection of the diagonals of the face region, and the second center point of the ID card region can be the intersection of the diagonals of the ID card region. The relative positions of the intersection points of the face region and the ID card region diagonals are then adjusted (based on the coordinate relationship between the two points). Figures 6A-6B This is a schematic diagram illustrating the movement of the intersection point of the lines connecting the diagonals of the face region according to an embodiment of the present invention. Figure 6A This shows a diagram before the intersection of the face regions moves. Figure 6B This displays a diagram showing the movement of the intersection point of the face region. This is based on the assumption that the face region and the document region have been determined. Figure 6AAs shown in Figure 601, the face region before movement is defined. Based on the defined face region, the intersection point A is marked according to its diagonals. The intersection point B of the diagonals of the document region is determined, and the coordinates of A and B are obtained. Then, based on the coordinate information of points A and B, the intersection point B of the line connecting the diagonals of the face region is moved to the intersection point A of the line connecting the diagonals of the document region.

[0090] In step 1022, taking the second center point of the document area as a reference, the width of the face area is extended according to the first ratio, and the height of the face area is extended according to the second ratio to obtain the document area.

[0091] As an example, such as Figure 6B The result is 602, which is the moved face region; and the width and length of the face region are enlarged according to the width and length ratio of the face region and the document region after the movement, to obtain the document region 603.

[0092] In step 103, the image to be processed is segmented according to the document area to obtain the document image.

[0093] In some embodiments, the image to be processed is segmented based on the document region obtained from the face region to obtain a document image.

[0094] As an example, segmentation can be achieved using a semantic segmentation model. Here, the type of semantic segmentation model can be a U-net convolutional neural network, a fully convolutional neural network (FCN), a deep neural network, etc., and this embodiment does not limit the specific type.

[0095] In practical implementation, taking the Fully Convolutional Neural Network (FCN) model as an example, after several stacked convolutional and pooling layers for semantic feature extraction and downsampling, a semantic feature segmentation result map corresponding to the original image can be obtained. Then, bilinear interpolation and other methods are used to upsample the length and width of the semantic feature segmentation result map to the size of the original image to achieve pixel-level end-to-end semantic segmentation and obtain the corresponding semantic segmentation map, which includes different regions representing different categories.

[0096] For example, using the semantic segmentation model in this application embodiment, semantic features are extracted from the image to be processed, and semantic recognition is performed on the image to be processed to obtain a semantic segmentation map smaller than the original image size. The semantic segmentation map includes at least one semantic region and an identification region. The at least one semantic region obtained from the semantic recognition processing is taken as the identification region. It should be noted that in this application embodiment, the target region has already been obtained in the image to be processed based on the determined identification region, so it is not necessary to use an object detection algorithm to determine the identification region in the image to be processed. This avoids the problem of traditional edge detection algorithms being greatly affected by the background of the photo and having very unstable results. Based on the multi-object detection algorithm, the instability problem of traditional edge algorithms is overcome, and the accuracy and stability of identification border detection are improved.

[0097] In step 104, the features of the document image are obtained, and quality detection processing is performed based on the features of the document image to obtain a detection result characterizing whether the document image is qualified.

[0098] It should be noted that in practical applications, different processing methods can be used to perform quality detection and combined processing on the images to be recognized, which will be explained below.

[0099] In some embodiments, see Figure 7 , Figure 7 This is a flowchart illustrating a quality inspection process, including integrity testing, provided in an embodiment of this application. Step 104 can be achieved through... Figure 7 Steps 1041A-1044A are implemented in this process. Quality inspection includes integrity inspection; correspondingly, in step 104, features of the document image are acquired, and quality inspection processing is performed based on these features to obtain a detection result characterizing whether the document image is qualified. This can be achieved through the following technical solution: In step 1041A, content features of the document image are acquired; in step 1042A, integrity classification processing is performed on the document image based on its content features to obtain an integrity classification detection result; in step 1043A, when the integrity classification detection result indicates that the document image is a complete image, the document image is determined to be a qualified image; in step 1044A, when the integrity classification detection result indicates that the document image is an incomplete image, the document image is determined to be a unqualified image.

[0100] As an example, content features can be the number of keywords or the location information of keywords. Document type rules are used to determine the completeness of document images. Document type rules can include the number of keywords and the location information of keywords in different types of documents. Based on the number and location information of keywords in the document image, the document image is classified according to the document type rules to obtain the completeness classification detection result. For example, when detecting a driver's license, keywords can be name, gender, nationality, and date of birth. Keyword location information can be the specific arrangement of each keyword on the driver's license, such as the interval between the name and gender. When missing keywords are detected, or when the number and location information of keywords differ from the document type rules, the document can be determined to be incomplete.

[0101] In some embodiments, see Figure 8 , Figure 8 This is a flowchart illustrating a quality inspection process, including integrity testing, provided in an embodiment of this application. Step 104 can be achieved through... Figure 8 Steps 1041B to 1042B are implemented. In step 1041B, the document type rule corresponding to the document type is obtained; in step 1042B, the document image is classified according to the document type rule and based on the number and location information of the obtained keywords to obtain the classification detection result. The preset document type rule includes the number and location information of keywords corresponding to different document types.

[0102] As an example, the corresponding document type rules are determined based on the obtained document type, and keyword detection is performed on the acquired document image to obtain the positional relationships of multiple keywords in the document image. Specifically, a multi-object detection model is used to perform multi-keyword detection on the document image to obtain the positional relationships of multiple keywords in the document image, where the positional relationships of keywords can include the coordinates of the keywords. In image processing, the positional relationships of keywords represent the combination or contextual relationships of multiple keywords within a fixed region, where the positional relationships represent the combination relationships of the keyword context and surrounding neighborhood. The document image is then classified based on the detected positional relationships and number of keywords to obtain the classification detection results.

[0103] It should be noted that multi-target recognition can be achieved using various types of neural network models, such as convolutional neural networks, deep convolutional neural networks, and fully connected neural networks. This application does not limit the specific type of model used in its embodiments.

[0104] In practical implementation, taking a fully convolutional neural network as an example, the two fully connected layers are replaced with convolutional layers, and four more convolutional layers are added to construct the network structure. The outputs of five different convolutional layers are convolved with two 3x3 convolutional kernels. One outputs confidence for classification, generating 21 confidence points for each default box; the other outputs localization for regression, generating four coordinate values ​​(x, y, w, h) for each default box. Additionally, these five convolutional layers pass through a priorBox layer to generate default boxes (generating coordinates). The number of default boxes in each of the five convolutional layers is given. Finally, the results of the first three calculations are statistically analyzed, and the identified multi-target points are selected based on these results, thus achieving fast multi-target detection speed and high detection accuracy.

[0105] In some embodiments, see Figure 9 , Figure 9 This is a flowchart illustrating a quality inspection process, including integrity testing, provided in an embodiment of this application. Step 104 can be achieved through... Figure 9 Steps 1041C to 1044C are implemented as follows: In step 1041C, the texture features of the document image are obtained; in step 1042C, a binary classification model is called based on the texture features to classify the document image as a copy, and a copy classification detection result is obtained; in step 1043C, when the copy classification detection result indicates that the document image is a non-copy image, the document image is determined to be a qualified image; in step 1044C, when the copy classification detection result indicates that the document image is a copy image, the document image is determined to be an unqualified image.

[0106] As an example, the texture features corresponding to the document image are extracted according to the convolutional neural network model for document classification. Based on the obtained texture features, the probability that the document image is a document copy is determined by a binary classification model. The binary classification model can be a fully connected layer structure. The fully connected layer structure will output the probabilities of the two classification results, and the larger probability will be used as the classification detection result of document copy.

[0107] In some embodiments, visual features of an ID card image are acquired, wherein the visual features include at least one of the following: color information and brightness; based on the visual features, a sharpness detection is performed on the ID card image to obtain a sharpness detection result; when the sharpness detection result indicates that the ID card image is a sharp image, the ID card image is determined to be a qualified image; when the sharpness detection result indicates that the ID card image is an unclear image, the ID card image is determined to be an unqualified image.

[0108] As an example, perform histogram equalization on the document image; convert the three primary color channels of the document image into HSV channels; take the V channel for brightness processing to obtain a new V channel; merge the new V channel with the H channel and the S channel, and convert back to the three primary color channels, and calculate the average brightness of the document image; and calculate the image sharpness through the Laplacian operator according to the average brightness and preset a threshold, and determine the document image according to the preset threshold.

[0109] It should be noted that in the embodiments of the present application, various combinations of quality detection methods can be used for the document image to screen the qualified document images for the image to be processed. For example, after performing integrity classification processing on the document image, duplicate detection, copy detection, brightness and sharpness detection can be performed in sequence, and all passes are considered qualified.

[0110] In addition, it should be noted that the combination form of quality detection for different types of images is determined according to the specific application scenario. The higher the security factor required by the application scenario, the more types of detection are required. The security factor is positively correlated with the types of detection required. For example, when collecting an image of an ID card, since the ID card information requires high precision and integrity in the corresponding image collection, the identification of the ID card image to be processed needs to pass integrity classification processing, duplicate detection, copy detection, brightness and sharpness detection, and all passes are considered qualified.

[0111] Next, an exemplary application of the embodiments of the present application in a practical application scenario will be described. The image processing method provided by the embodiment will be described as being implemented on a mobile device.

[0112] See Figure 10 , Figure 10 is a flowchart of an image processing method provided by an embodiment of the present application, and will be described in conjunction with the steps shown in Figure 10 .

[0113] Step 401, anti-counterfeiting detection of the original ID card.

[0114] Here, before the mobile device inputs the image to be recognized into the trained autoencoder model, it is necessary to use a detection model to detect the anti-counterfeiting points on the document, such as the Great Wall laser pattern on the Chinese ID card, the laser change of the driving license, etc., to determine whether it is a genuine document. Among them, for the laser transformation on the document, a short video needs to be collected at the pitch angle, that is, continuously change the angle to collect multi-angle document images, and then extract features through a multi-dimensional CNN network, and comprehensively judge whether it is an original document through multiple images. There will be no laser transformation for non-original documents.

[0115] Compared to existing related technologies, traditional convolutional networks take an image as input, extract features through a CNN network, and then pass it to a binary classifier to obtain the classification result. Multidimensional CNNs take multiple images of documents from different angles as input, then extract features through a multidimensional CNN network and pass them to a classifier to determine whether the document is an original.

[0116] Step 402: Receive the image captured by the terminal device through the camera, wherein the ID card is placed in the photo area.

[0117] Step 403: Based on a face detection algorithm (e.g., RetinaFace), detect the face on the document and obtain the coordinates of the face bounding box.

[0118] Step 404: Based on the face frame and the relative position and size ratio of the face area and the document area, scale to obtain the document frame.

[0119] Step 405: Segment the document image based on the detected document border.

[0120] Based on the position and proportion of the document frame within the photo, it is determined whether the scanned document meets the requirements. If the document image segmented according to the document frame exceeds the photo boundary, it indicates that the scanned document does not meet the requirements.

[0121] Step 406: Perform quality inspection on the segmented document image.

[0122] See Figure 11 , Figure 11 This is a flowchart illustrating a quality inspection process, including integrity testing, provided in an embodiment of this application. Step 406 can be achieved through... Figure 11 Steps 4061 to 4064 are implemented.

[0123] Step 4061, Reproduction Detection. Wavelet transform is used to extract moiré patterns from the segmented document images, and a CNN classifier model is used to determine whether the image is a reproduction.

[0124] Step 4062, Copy Detection. Features are extracted from the segmented document images using a convolutional neural network model, and a binary classification model is used to determine whether the image is a copy.

[0125] Step 4063, Integrity Detection. A multi-object detection model is used on the segmented document image to obtain the coordinates of each key point, and the image's integrity is determined by comparing these coordinates with the preset rules and conditions for the corresponding document.

[0126] In some embodiments, based on the number and location of keywords detected by the target detection model and the corresponding rules and conditions preset on the document, it is determined whether the document is complete and clear; if it is unclear or obscured, the key points will be undetectable or incomplete.

[0127] As an example, multi-object detection models such as YOLO, SSD, and Mask R-CNN are used to simultaneously detect the document border, document type, and key points on the document (key points that constitute keywords, equivalent to detecting whether the document contains preset keywords), obtaining the coordinates of each key point. For example, the ID card area, the portrait on it, and the keywords such as name, gender, birth date, ethnicity, address, and citizen identification number are detected as targets, and the image is judged to be complete according to the preset rules and conditions of the corresponding document.

[0128] Step 4064, Brightness and Sharpness Detection. The average brightness of the segmented document image is calculated based on the HSV color space; the image sharpness is calculated using the Laplacian operator; and a preset threshold for image sharpness is used to determine whether the conditions are met.

[0129] Step 407: If the photo meets the preset rules and conditions for the document, it is sent to the backend for image collection and other processing, and the process ends.

[0130] In this embodiment, a multi-target detection algorithm is used to detect the border of the target document and segment the document image. Next, it performs photocopy detection, copy detection, integrity detection, brightness and sharpness detection, and determines whether it meets the requirements based on preset conditions and rules. In the image quality detection, the multi-target detection algorithm overcomes the instability of traditional edge detection algorithms, greatly improving the accuracy and stability of offline document border detection. Furthermore, the algorithm uses keywords to determine the clarity and integrity of the document image, which is more reliable than traditional clarity and brightness detection algorithms. This ensures that compliant images are obtained for photocopying, copying, integrity, brightness, and sharpness detection, significantly improving the quality of document photo acquisition and avoiding non-compliant photos.

[0131] The following description continues to illustrate the exemplary structure of the image processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the image processing device 555 in the memory 550 may include: an image recognition module 5551, used to recognize the face region in the image to be processed, wherein the image to be processed is obtained by image acquisition of an ID card; an image detection module 5552, used to determine the ID card region in the image to be processed based on the relative position and proportion between the face region and the ID card region, wherein the ID card region includes the face region; an image segmentation module 5553, used to segment the image to be processed based on the ID card region to obtain an ID card image; and an image quality detection module 5554, used to acquire the features of the ID card image, perform quality detection processing based on the features of the ID card image, and obtain a detection result characterizing whether the ID card image is qualified.

[0132] In some embodiments, the apparatus further includes: a document type recognition module 5555, configured to perform document type recognition processing on the image to be processed to obtain the type of document in the image to be processed; and to obtain the relative position and proportion between the face region and the document region in the image to be processed according to a type lookup mapping table, wherein the mapping table includes the relative position and proportion between the face region and the document region in different types of document images.

[0133] In some embodiments, the relative position between the face region and the document region includes the relative position between the first center point of the face region and the second center point of the document region, and the ratio between the face region and the document region includes a first ratio of the width of the face region to the width of the document region, and a second ratio of the height of the face region to the height of the document region; the type recognition module 5555 is further configured to move the first center point of the face region according to the relative position between the first center point of the face region and the second center point of the document region, so as to move to the second center point of the document region; and based on the second center point of the document region, extend the width of the face region according to the first ratio and extend the height of the face region according to the second ratio to obtain the document region.

[0134] In some embodiments, quality detection includes integrity detection; the image quality detection module 5554 is further configured to acquire content features of the document image; perform integrity classification processing on the document image based on the content features of the document image to obtain an integrity classification detection result; when the integrity classification detection result indicates that the document image is a complete image, the document image is determined to be a qualified image; when the integrity classification detection result indicates that the document image is an incomplete image, the document image is determined to be a unqualified image.

[0135] In some embodiments, the content features of the document image include the number of keywords, location information, and document type; the image quality detection module 5554 is further used to obtain document type rules corresponding to the document type; according to the document type rules, and based on the obtained number of keywords and location information, the document image is classified to obtain a classification detection result, wherein the preset document type rules include the number of keywords and location information corresponding to different document types.

[0136] In some embodiments, quality detection includes copy detection; the image quality detection module 5554 is further configured to acquire the texture features of the document image; call a binary classification model based on the texture features to perform copy classification processing on the document image, and obtain a copy classification detection result; when the copy classification detection result indicates that the document image is a non-copy image, the document image is determined to be a qualified image; when the copy classification detection result indicates that the document image is a copy image, the document image is determined to be a non-qualified image.

[0137] In some embodiments, the quality detection includes sharpness detection; the image quality detection module 5554 is further configured to acquire visual features of the document image, wherein the visual features include at least one of the following: color information, brightness; determine the sharpness of the document image based on the visual features to obtain a sharpness detection result; when the sharpness detection result indicates that the document image is a sharp image, determine that the document image is a qualified image; when the sharpness detection result indicates that the document image is a non-sharp image, determine that the document image is a non-qualified image.

[0138] In some embodiments, the apparatus further includes: a counterfeit document identification module 5556, configured to perform image acquisition processing on the document from different angles to obtain multiple images to be processed including the document; acquire image features of anti-counterfeiting marks in each image to be processed; perform counterfeit document classification processing on the images to be processed according to the image features of the anti-counterfeiting marks to obtain a counterfeit document classification result; when the counterfeit document classification result indicates that the document is not a counterfeit document, determine that the processing of identifying the face region in any image to be processed will be performed.

[0139] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0140] This application provides a computer program product, including a computer program, characterized in that the computer program, when executed by a processor, implements the image processing method provided in this application.

[0141] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and when executed by a processor, the processor will execute the image processing method provided in this application.

[0142] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0143] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0144] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0145] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0146] In summary, the technical solutions of the embodiments of the present invention have the following beneficial effects:

[0147] 1) Based on a multi-target detection algorithm, the instability problem of traditional edge detection algorithms is overcome, which greatly improves the accuracy and stability of offline document border detection on mobile devices.

[0148] 2) Judging sharpness and completeness based on keywords (key points) is more reliable than traditional sharpness and brightness detection algorithms, and can guarantee that the image meets the requirements.

[0149] 3) Conducting inspections on photocopies, completeness, brightness, and clarity greatly improves the quality of ID photo collection and avoids non-compliant photos.

[0150] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method includes: Identify facial regions in an image to be processed, wherein the image to be processed is obtained by image acquisition of an identification document; The document region in the image to be processed is determined based on the relative position and proportion between the face region and the document region, wherein the document region includes the face region; The image to be processed is segmented according to the document area to obtain the document image; The features of the document image are obtained, and a quality detection process is performed based on the features of the document image to obtain a detection result characterizing whether the document image is qualified. Before determining the document region in the image to be processed based on the relative position and proportion between the face region and the document region, the method further includes: Perform document type recognition processing on the image to be processed to obtain the type of the document in the image to be processed; According to the type query mapping table, the relative position and proportion between the face region and the document region in the image to be processed are obtained, wherein the mapping table includes the relative position and proportion between the face region and the document region in different types of document images; The relative position between the face region and the document region includes the relative position between the first center point of the face region and the second center point of the document region. The ratio between the face region and the document region includes a first ratio of the width of the face region to the width of the document region, and a second ratio of the height of the face region to the height of the document region. Determining the document region in the image to be processed based on the relative position and proportion between the face region and the document region includes: The first center point of the face region is moved relative to the second center point of the document region, so as to the second center point of the document region. Using the second center point of the document area as a reference, the width of the face area is extended according to the first ratio, and the height of the face area is extended according to the second ratio to obtain the document area.

2. The method according to claim 1, characterized in that, The quality inspection includes integrity inspection; the process of acquiring features of the document image, performing quality inspection processing based on the features of the document image, and obtaining a detection result characterizing whether the document image is qualified includes: Obtain the content features of the document image; Based on the content features of the document image, integrity classification processing is performed on the document image to obtain integrity classification detection results; When the integrity classification detection result indicates that the document image is a complete image, the document image is determined to be a qualified image; When the integrity classification detection result indicates that the document image is an incomplete image, the document image is determined to be an unqualified image.

3. The method according to claim 2, characterized in that, The content features of the document image include the number of keywords, location information, and document type; The process of performing integrity classification on the document image based on the content features to obtain classification detection results includes: Obtain the document type rule corresponding to the document type; According to the document type rules, and based on the number of keywords and location information obtained, the document image is classified to obtain a classification detection result. The document type rules include the number of keywords and location information corresponding to different document types.

4. The method according to claim 1, characterized in that, The quality inspection includes photocopy inspection; The process of acquiring features of the document image, performing quality detection processing based on the features of the document image, and obtaining a detection result characterizing whether the document image is qualified includes: Obtain the texture features of the document image; Based on the texture features, a binary classification model is invoked to classify the document image as a photocopy, thereby obtaining the photocopy classification detection result. When the photocopy classification detection result indicates that the document image is a non-photocopy image, the document image is determined to be a qualified image; When the photocopy classification detection result indicates that the document image is a photocopy image, the document image is determined to be an unqualified image.

5. The method according to claim 1, characterized in that, The quality inspection includes sharpness inspection; The process of acquiring features of the document image, performing quality detection processing based on the features of the document image, and obtaining a detection result characterizing whether the document image is qualified includes: The visual features of the document image are obtained, wherein the visual features include at least one of the following: color information and brightness. Based on the aforementioned visual features, the clarity of the document image is determined and a clarity detection result is obtained. When the clarity detection result indicates that the document image is a clear image, the document image is determined to be a qualified image; When the clarity detection result indicates that the document image is not clear, the document image is determined to be an unqualified image.

6. The method according to claim 1, characterized in that, Before identifying face regions in the image to be processed, the method further includes: The document is subjected to image acquisition and processing from different angles to obtain multiple images of the document to be processed. Obtain the image features of the anti-counterfeiting mark in each of the images to be processed; Based on the image features of the anti-counterfeiting mark, the image to be processed is classified as a counterfeit document to obtain the counterfeit document classification result. When the forged document classification result indicates that the document is not a forged document, it is determined that the process of identifying the face region in any of the images to be processed will be performed.

7. An image processing apparatus, characterized in that, The image processing apparatus is used to implement the method according to any one of claims 1 to 6, the apparatus comprising: An image recognition module is used to identify facial regions in an image to be processed, wherein the image to be processed is obtained by image acquisition of an identification document; An image detection module is used to determine the document region in the image to be processed based on the relative position and proportion between the face region and the document region, wherein the document region includes the face region; The image segmentation module is used to segment the image to be processed according to the document area to obtain the document image; The image quality detection module is used to acquire the features of the document image, perform quality detection processing based on the features of the document image, and obtain a detection result that characterizes whether the document image is qualified.

8. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the image processing method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1 to 6.

Images