Image processing method and apparatus, electronic device, computer-readable storage medium and computer program product

Abstract

The embodiments of the present application provides an image processing method and apparatus, an electronic device, a computer-readable storage medium and a computer program product, which are at least applied in the field of cloud technology or image recognition. The method comprises: acquiring a first original image and a second original image; performing image cropping on the first original image to obtain a first cropped image; performing image cropping on the second original image to obtain a second cropped image; performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image; performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image; and performing distortion space alignment on the first distortion-corrected image and the second distortion-corrected image, so as to obtain spatially aligned images containing a target object.

Description

Image processing method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410273571.6 and application date of March 11, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of the Internet, and is related to, but not limited to, an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0004] Dual-camera image acquisition devices are now common in smart devices, particularly for biometric information collection and recognition. For example, they are widely used in face and palm scanning scenarios. Accurate biometric recognition requires combining the image information captured by both cameras. However, due to issues such as the positioning and assembly of the camera modules' external structures or errors introduced during camera capture, the images captured by both cameras can become misaligned.

[0005] In related technologies, when aligning images captured by dual cameras, for example, in face-scanning scenarios, spatial alignment is usually performed on the color image, infrared image, and depth image; while in traditional palm-scanning scenarios, the color image and infrared image are usually aligned at fixed positions, or the two images are dynamically aligned based on the depth distance of the proximity sensor (PSensor).

[0006] However, the image alignment methods in related technologies all have problems such as poor alignment effect and small effective field of view (FOV), which leads to a small range of available distances for palm swiping. Summary of the Invention

[0007] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can be applied at least in the field of cloud technology or image recognition, and can optimize the effective FOV of the image acquisition device while ensuring the image alignment effect.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] An embodiment of the present application provides an image processing method, which is executed by an electronic device and includes: acquiring an original image pair, the original image pair including a first original image acquired by a first type of image acquisition device and a second original image acquired by a second type of image acquisition device; both the first original image and the second original image contain a target object; performing image cropping on the first original image to obtain a first cropped image; performing image cropping on the second original image to obtain a second cropped image; performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image; performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image; and performing distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

[0010] An embodiment of the present application provides an image processing device, including: an acquisition module, configured to acquire an original image pair, wherein the original image pair includes a first original image acquired by a first type of image acquisition device and a second original image acquired by a second type of image acquisition device; the first original image and the second original image both contain a target object; an image cropping module, configured to perform image cropping on the first original image to obtain a first cropped image; and perform image cropping on the second original image to obtain a second cropped image; a distortion correction module, configured to perform distortion correction processing on the first cropped image to obtain a first distortion-corrected image; and perform distortion correction processing on the second cropped image to obtain a second distortion-corrected image; and a spatial alignment module, configured to perform distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

[0011] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned image processing method when executing the executable instructions stored in the memory.

[0012] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein a processor of an electronic device reads the executable instructions from the computer-readable storage medium and implements the above-mentioned image processing method when executing the executable instructions.

[0013] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned image processing method.

[0014] The embodiments of the present application have the following beneficial effects:

[0015] When spatially aligning the original image pair, the electronic device sequentially performs effective image cropping and distortion correction processing on the first original image, and also sequentially performs effective image cropping and distortion correction processing on the second original image, thereby obtaining a first distortion-corrected image and a second distortion-corrected image after distortion correction, respectively. Subsequently, the first distortion-corrected image and the second distortion-corrected image are subjected to distortion spatial alignment to obtain a spatially aligned image containing the target object. Since the first distortion-corrected image and the second distortion-corrected image are subjected to distortion spatial alignment, the distortion spatial alignment simultaneously distorts the first distortion-corrected image and the second distortion-corrected image, thereby ensuring that non-intersecting portions of the first and second original images are not excessively cropped during image cropping. In other words, the effective FOV of the original image pair is not cropped. Therefore, the method of the embodiments of the present application can optimize the effective FOV between the original image pairs captured by the first and second image acquisition devices while ensuring the alignment of the original image pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a schematic diagram of an effective FOV and an independent FOV provided in an embodiment of the present application;

[0017] FIG2 is a schematic diagram of an optional architecture of an image processing system provided in an embodiment of the present application;

[0018] FIG3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0019] FIG4 is a schematic diagram of an optional flow chart of an image processing method provided in an embodiment of the present application;

[0020] FIG5 is another optional flowchart of the image processing method provided in an embodiment of the present application;

[0021] FIG6 is a schematic diagram of an implementation flow of image processing provided in an embodiment of the present application;

[0022] FIG7 is a schematic diagram showing an implementation principle of image processing using a center alignment method according to an embodiment of the present application;

[0023] FIG8 is a schematic diagram of another implementation flow of the image processing method provided in an embodiment of the present application;

[0024] FIG9 is a schematic diagram of another optional flow chart of the image processing method provided in an embodiment of the present application;

[0025] FIG10 is a flow chart of a method for implementing distortion correction processing provided in an embodiment of the present application;

[0026] FIG11 is a schematic diagram of a conventional process for obtaining an effective FOV within a palm brushing area according to an embodiment of the present application;

[0027] FIG12 is a flow chart of a method for optimizing the effective FOV of a camera palm brushing within a palm brushing field provided by an embodiment of the present application;

[0028] FIG13a is a schematic diagram showing a result of using a full-size FOV resolution with an unconventional ratio in an effective image cropping stage according to an embodiment of the present application;

[0029] FIG13 b is a schematic diagram showing another result of using a full-size FOV resolution with an unconventional ratio in the effective image cropping stage provided by an embodiment of the present application;

[0030] FIG13c is a schematic diagram showing another result of using a full-size FOV resolution with an unconventional ratio in the effective image cropping stage provided by an embodiment of the present application;

[0031] FIG14 is a schematic diagram illustrating the problem of uneven vertical alignment of cameras provided by an embodiment of the present application;

[0032] FIG15 is a schematic diagram showing the result of random deviation of the camera provided in an embodiment of the present application;

[0033] FIG16 is a schematic diagram of a lateral loss analysis provided in an embodiment of the present application;

[0034] FIG17 is a schematic diagram comparing alignment to a color image and alignment to the center provided in an embodiment of the present application;

[0035] FIG18 is a schematic diagram comparing the results of alignment in different directions provided by an embodiment of the present application;

[0036] FIG19 is a schematic diagram of an arc region for distortion correction provided by an embodiment of the present application;

[0037] FIG20 is a schematic diagram showing the comparison results between the normal process provided by an embodiment of the present application and direct spatial alignment based on the distortion-corrected original image;

[0038] FIG21 a is a schematic diagram of the overall flow of the image processing method before optimization provided in an embodiment of the present application;

[0039] Figure 21b is a schematic diagram of the overall flow of the image processing method for optimizing FOV provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0043] Before explaining the image processing method provided in the embodiment of the present application, the professional terms involved in the embodiment of the present application are first explained.

[0044] (1) Palm region detection: This refers to the process of using target detection technology to locate the finger joints and accurately identify the position and boundaries of the palm from the captured image, thereby extracting the palm print area image based on the identified position and boundaries. The palm detection model used in palm region detection can adopt an encoder-decoder architecture, combined with cross-layer connections to extract multi-scale features, thereby performing palm region detection. This encoder-decoder architecture can effectively handle the complexity and diversity of palms, including different sizes, occlusions, and self-occlusions. After detecting the palm region, the key point detection model will further analyze the palm region and predict 21 2.5D hand key points. These key points include the wrist, the base of each finger, and the fingertips. The key point detection model can not only detect the position of the key points, but also distinguish between the left and right hands and provide confidence information.

[0045] (2) Palmprint recognition: It refers to identifying the identity information of different users by analyzing the lines, wrinkles, and fine textures on the palm of the hand based on the image of the palmprint area. Palmprint recognition technology can be implemented based on the following methods: based on point and line features (extracting point and line features in the palmprint, such as ridge ends, bifurcation points, etc.), based on texture (using Gabor filtering, wavelet transform and other methods to analyze the texture features of the palmprint), based on subspace (extracting features through principal component analysis or linear discriminant analysis), feature fusion (combining multiple feature extraction methods to improve the accuracy and robustness of recognition). Palmprint recognition technology is widely used in the fields of identity authentication and identification, including but not limited to the following application scenarios: access control systems, attendance systems, financial fields, and information security fields.

[0046] (3) Cross-device registration and identification: refers to the registration and identification on two devices with significant differences, such as registration on a mobile phone and identification on an IoT device. Cross-device registration and identification technology has important applications in the fields of multi-device collaboration, identity authentication, and personalized services. The core of cross-device registration and identification technology lies in how to achieve consistent identity authentication on different devices while ensuring data security and privacy. Methods of cross-device registration and identification technology include: device identifier (authentication on different devices by generating a unique device identifier), biometric identification (authentication on different devices using biometrics (such as fingerprints, palm prints, facial recognition, etc.), and radio frequency fingerprint identification (authentication using the device's radio frequency fingerprint, which generates a unique fingerprint based on the device's hardware characteristics). Cross-device registration and identification technology can be applied to at least the following scenarios: smart home (through a unified device identification system, seamless collaboration between smart devices is achieved), identity authentication (authentication on different devices to improve user experience and security), and data sharing (efficient data sharing between devices to improve the efficiency of multi-device collaboration).

[0047] (4) Color map: refers to a color map of natural light image captured by a color sensor, which usually contains information from three color channels: red (R), green (G), and blue (B). In face / palm payment, it is generally used for: face / palm optimization and comparative recognition. Among them, for face / palm optimization, the color map system can quickly identify and locate the area of ​​the face or palm, thereby selecting the best recognition area. In addition, since the color map provides rich texture and color information, it helps to improve the accuracy and reliability of recognition. Comparative recognition means that in face or palm payment, the color map is used to compare with the pre-registered image to verify the identity of the user; since the color map can provide detailed visual information, it helps the system to authenticate the identity more accurately and reduce the possibility of misidentification. For example, in the face payment scenario, the color map is used to quickly locate the face area and compare it with the registered image to complete the identity authentication; for another example, in the palm payment, the color map is used to identify the texture and shape of the palm and compare it with the registered palm image to ensure the security of the payment.

[0048] (5) Infrared image: refers to an infrared image captured by an infrared sensor that uses infrared light to image the surrounding area. It is generally used for liveness detection in face-scanning / palm-scanning payment. Infrared images capture infrared rays radiated from the surface of the human body to generate images with depth information. Compared with visible light images, infrared images can better reflect the three-dimensional structure and subtle changes of the face or palm, such as tiny bumps on the skin surface and changes in blood flow. Infrared images are insensitive to lighting conditions and can work normally at night or in low-light environments. At the same time, they can effectively distinguish real faces from forgeries such as photos and videos. In face-scanning and palm-scanning payment, infrared images are used to detect whether the user is a real live person, thereby improving the security of payment.

[0049] (6) Optimization: refers to selecting a group of color images, depth images, or infrared images that meet the preconditions of the liveness detection and contrast recognition algorithms (in the face scanning scenario: color image, depth image, or infrared image; in the palm scanning scenario: color image or infrared image). Among them, the color image can be optimized based on indicators such as the angle, size, centering of the face / palm, and the clarity of the color image; the infrared image can be optimized based on indicators such as the brightness of the infrared image; and the depth image can be optimized based on indicators such as the completeness of the depth image (face scanning) and other indicators.

[0050] (7) Effective FOV: Field of view (FOV) refers to the maximum range that a lens can capture. FOV is usually expressed in degrees and is divided into horizontal field of view, vertical field of view, and diagonal field of view. In a multi-camera system, the effective FOV refers to the intersection of the fields of view of the color camera and the infrared camera (such as area 101 in Figure 1), that is, the overlap of the fields of view of the two cameras. This area is the common area that both cameras can capture. In the field of liveness detection, the infrared camera is used to detect liveness, while the color camera is used to capture color images of faces or palms. The effective FOV ensures that the two cameras can capture the same area at the same time, thereby achieving accurate liveness detection. In comparative recognition, the image within the effective FOV is used to compare with the registered image to ensure the accuracy and security of recognition. In a multimodal recognition system, the image data within the effective FOV can be fused and processed to improve the accuracy and reliability of recognition.

[0051] (8) Independent FOV: refers to the non-intersecting parts of the field of view of the color camera and the infrared camera (such as area 102 and area 103 in Figure 1). This part of the field of view is unique to each camera and cannot be covered by the other camera. In a multimodal imaging system, an independent FOV can be used to expand the field of view of the system. For example, in some video detection systems, the color camera and the infrared camera may cover different areas respectively. By combining the independent FOVs of the two, a wider video detection range can be achieved. In some special environments, such as at night or in low light conditions, the independent FOV of the infrared camera can provide additional detection capabilities, while the color camera is used for detection under normal lighting conditions.

[0052] (9) Black edges: These are the areas in the images captured by the color camera and the infrared camera that are filled with black pixels due to spatial alignment, where there is no object in the image. These black edges usually appear at the edge of the image and are caused by the incomplete overlap of the fields of view of the two cameras or the design of the imaging system. For example, the fields of view of the color camera and the infrared camera may not completely overlap, resulting in some areas not being captured by both cameras at the same time, thus forming black edges in the image; or, in order to ensure image quality within the effective FOV, the imaging system may be designed to leave a certain amount of space at the edge, which is filled with black pixels to form black edges; or, in a multi-camera system, in order to achieve spatial alignment of the image, the image may be cropped or padded, resulting in the appearance of black edges.

[0053] (10) Spatial alignment: In a multi-camera system, since the color camera and the infrared camera are not at the same point in the horizontal direction, the relative positions of the palms in the two images are different. The spatial alignment process is to align the relative positions of the palms in the two images through translation or other geometric transformations for further processing and analysis.

[0054] The embodiment of the present application provides an image processing method, which uses an unconventional ratio (such as full-size resolution) to perform effective image cropping (Crop), aligns to the middle of two images during spatial alignment, and uses the arc black edge method in the distorted original image to optimize the effective FOV of the camera in the palm brushing area. Specifically, in the image processing method provided by the embodiment of the present application, first, the electronic device obtains an original image pair, and the original image pair includes a first original image containing a target object acquired by a first type of image acquisition device and a second original image containing a target object acquired by a second type of image acquisition device; then, the first original image is cropped to obtain a first cropped image; the second original image is cropped to obtain a second cropped image; then, the first cropped image is subjected to distortion correction processing to obtain a first distortion-corrected image; the second cropped image is subjected to distortion correction processing to obtain a second distortion-corrected image; finally, the first distortion-corrected image and the second distortion-corrected image are subjected to distortion spatial alignment to obtain a spatially aligned image containing the target object. Since the first distortion-corrected image and the second distortion-corrected image are spatially aligned, the distortion of the first distortion-corrected image and the second distortion-corrected image are simultaneously spatially aligned through the distortion spatial alignment. This ensures that the non-intersecting portions of the first original image and the second original image are not excessively cropped during image cropping. In other words, the effective FOV of the original image pair is not cropped. Therefore, the method of the embodiment of the present application can optimize the effective FOV between the original image pairs captured by the first type of image acquisition device and the second type of image acquisition device while ensuring the alignment of the original image pairs.

[0055] Here, first, an exemplary application of the image processing device of the embodiment of the present application is described, and the image processing device is an electronic device for implementing the image processing method. In one implementation, the image processing device (i.e., electronic device) provided by the embodiment of the present application can be implemented as a terminal or as a server. In one implementation, the image processing device provided by the embodiment of the present application can be implemented as a palm-swiping device, a face-swiping device, a laptop computer, a tablet computer, a desktop computer, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device, and any terminal with an identity information recognition function or an image data processing function; in another implementation, the image processing device provided by the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application. Below, an exemplary application when the image processing device is implemented as a server will be described.

[0056] In the case where the image processing device is implemented as a server, the image processing device is included in the image processing system. Referring to Figure 2, Figure 2 is an optional architectural diagram of the image processing system provided in an embodiment of the present application. The image processing system 10 in the embodiment of the present application includes at least a terminal 100, a network 200 and a server 300. The terminal 100 can be a palm-swiping device or a face-swiping device. Here, the palm-swiping device is taken as an example for explanation. A palm-swiping application is installed on the palm-swiping device, and the palm-swiping application can identify the palm prints of the user collected by the palm-swiping device, thereby realizing the identification of the user's identity information. In the embodiment of the present application, the server 300 can be a server of the palm-swiping application. The server 300 can constitute the image processing device in the embodiment of the present application, that is, the image processing method in the embodiment of the present application is implemented through the server 300. The terminal 100 is connected to the server 300 via the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two.

[0057] In an embodiment of the present application, the palm-scanning device includes at least a first type of image acquisition device and a second type of image acquisition device. For example, the first type of image acquisition device may be an infrared camera, and the second type of image acquisition device may be a color camera. Referring to FIG2 , when a user swipes their palm using the palm-scanning device, the first type of image acquisition device on the palm-scanning device captures a first original image containing the user's palm print (i.e., the target object), and the second type of image acquisition device on the palm-scanning device captures a second original image containing the user's palm print. The terminal 100 then encapsulates the first and second original images into an image processing request and sends the image processing request to the server 300 via the network 200. Subsequently, in response to the image processing request, server 300 crops the first original image to obtain a first cropped image, and crops the second original image to obtain a second cropped image. Next, server 300 performs distortion correction processing on the first cropped image to obtain a first distortion-corrected image, and on the second cropped image to obtain a second distortion-corrected image. Finally, server 300 performs distortion spatial alignment on the first and second distortion-corrected images to obtain a spatially aligned image containing the user's palm print. After obtaining the spatially aligned image, server 300 may return the spatially aligned image to the terminal for display, or may identify the user based on the spatially aligned image to obtain the user's identity information, and then perform corresponding service processing based on the identity information. For example, for a palm-swipe card-in service, after the user swipes their palm and passes identity identification and verification, a card-in record for the user is generated. For a palm-swipe payment service, after the user swipes their palm and passes identity identification and verification, the payment process for the bound account corresponding to the user's identity information is executed.

[0058] In some embodiments, the steps in the image processing method can also be performed by the terminal 100, that is, the palm-swiping device spatially aligns the first original image and the second original image that have been captured. The palm-swiping device crops the first original image to obtain a first cropped image, and crops the second original image to obtain a second cropped image. The palm-swiping device then performs distortion correction processing on the first cropped image to obtain a first distortion-corrected image, and performs distortion correction processing on the second cropped image to obtain a second distortion-corrected image. Finally, the palm-swiping device spatially aligns the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the user's palm print.

[0059] The image processing method provided in the embodiment of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 can be a cloud server. The cloud server can perform image cropping on the first original image to obtain a first cropped image, and can perform image cropping on the second original image to obtain a second cropped image. Alternatively, the cloud server can perform distortion correction processing on the first cropped image to obtain a first distortion-corrected image, and can perform distortion correction processing on the second cropped image to obtain a second distortion-corrected image. Alternatively, the cloud server can perform distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image.

[0060] In some embodiments, there may also be a cloud storage, and the original image pair may be stored in the cloud storage, or the first cropped image and the second cropped image may be stored in the cloud storage, or the first distortion-corrected image and the second distortion-corrected image may be stored in the cloud storage, or the final spatially aligned image may be stored in the cloud storage. In this way, when performing actual business processing, the spatially aligned image can be directly obtained from the cloud storage to implement the corresponding business processing flow.

[0061] The image processing device (i.e., electronic device) described above is further described below. FIG3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device shown in FIG3 may be an image processing device, which includes: at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the image processing device are coupled together via a bus system 340. It is understandable that the bus system 340 is used to implement connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 340 in FIG3.

[0062] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor. The user interface 330 includes one or more output devices 331 that enable the presentation of media content, and one or more input devices 332. The memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 350 optionally includes one or more storage devices physically remote from the processor 310. The memory 350 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 350 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, memory 350 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0063] The operating system 351 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320. Exemplary network interfaces 320 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB); an input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.

[0064] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. FIG3 shows an image processing apparatus 354 stored in a memory 350. The image processing apparatus 354 may be an image processing apparatus in an electronic device. The image processing apparatus 354 may be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 3541, an image cropping module 3542, a distortion correction module 3543, and a spatial alignment module 3544. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0065] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0066] The image processing methods provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the image processing methods in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0067] The image processing method provided in the embodiments of the present application can be applied to various business scenarios, such as liveness detection, contrast recognition, image fusion, and motion coordination detection, as illustrated below.

[0068] In liveness detection applications, the image processing method provided in the embodiment of the present application can be used to align the positions of palms or faces in infrared images and color images to ensure spatial consistency between the two. Then, it is determined whether the biometric features in front of the camera are real living beings, rather than photos, videos, masks, etc. Key features such as skin texture, blood flow, and micro-expression changes can be extracted from the spatially aligned images, and the extracted features can be analyzed using deep learning models or traditional image processing methods to determine whether they are real living beings. For example, in financial services, liveness detection can be used to ensure that users are real people when opening accounts, conducting transactions, or verifying their identities, thereby enhancing account security.

[0069] In contrast recognition applications, the image processing methods provided in the embodiments of this application can be used to generate spatially aligned images. These images can then be used for contrast recognition to verify the user's identity. By comparing the spatially aligned images with registered images in a database, it can be determined whether they represent the same person. For example, in smart access control systems, contrast recognition technology can be used to prevent unauthorized access by using photos or masks.

[0070] In image fusion applications, spatially aligned images can be fused using methods such as weighted averaging and maximum fusion. Features can then be extracted from the fused image for identification or classification. For example, in multimodal imaging systems, image fusion can improve image resolution and contrast, thereby increasing recognition accuracy.

[0071] In motion detection applications, motion detection is a liveness detection method that verifies the user's presence by requiring them to perform specific actions (such as blinking, opening their mouth, shaking their head, etc.) and combining it with facial landmark positioning technology. For example, in mobile phone face unlocking and payment, motion detection technology can effectively prevent forged methods such as photos and videos from unlocking the phone or making payments, protecting user privacy and property security.

[0072] FIG4 is a schematic flow chart of an optional image processing method provided in an embodiment of the present application. The following will be described in conjunction with the steps shown in FIG4 . As shown in FIG4 , the image processing method is described as an example in which the execution subject is a server. The method includes the following steps S101 to S104:

[0073] Step S101: obtaining an original image pair.

[0074] Here, the original image pair includes a first original image containing the target object acquired by a first type of image acquisition device and a second original image containing the target object acquired by a second type of image acquisition device.

[0075] The first type of image acquisition device and the second type of image acquisition device can be cameras located on the biometric acquisition device. The first type of image acquisition device is used to capture a first original image of the target object, and the second type of image acquisition device is used to capture a second original image of the target object. In other words, the biometric acquisition device is a dual-camera acquisition device capable of simultaneously capturing two images of the target object. For example, the biometric acquisition device can be a palm-scanning device in an access control system. The first type of image acquisition device can be an infrared camera located on the palm-scanning device, and the second type of image acquisition device can be a color camera located on the palm-scanning device. The infrared camera captures an infrared image containing the user's palm prints, and the color camera captures a color image containing the user's palm prints.

[0076] In the embodiment of the present application, since the biometric acquisition device simultaneously captures two images of the target object, and due to factors such as the design error of the camera and the camera parameters, the position of the target object in the two images will deviate. When using the two images captured by the biometric acquisition device for subsequent business processing, in order to improve the business processing accuracy and reduce the position deviation between the two images, it is necessary to first spatially align the two images to obtain a spatially aligned image, and then perform subsequent business processing based on the spatially aligned image.

[0077] Step S102 , cropping the first original image to obtain a first cropped image; and cropping the second original image to obtain a second cropped image.

[0078] Due to differences in camera viewing angles and the distance between the target object and the camera, blank areas of a certain size may appear at the edges of the first and second original images. These blank areas are areas surrounding the target object but not containing it. Blank areas are areas of white or other colors within an image that do not contain the target object.

[0079] In some embodiments, blank areas at the edges of the first and second original images can be detected using any of the following methods: grayscale conversion, binarization, or contour detection. Grayscale conversion and binarization involve converting the image to grayscale and then using thresholding to convert it to binary. Contour detection involves using an objective function (e.g., OpenCV's findContours function) to detect contours within the image and find the largest contour.

[0080] In an embodiment of the present application, after the first original image and the second original image are captured and the blank areas at the edges of the first original image and the second original image are detected, the first original image and the second original image can be cropped first, and the image cropping process can be considered as an effective image cropping process. Effective image cropping refers to cropping the image to retain the effective areas in the image that are related to a specific business and deleting the invalid areas that are not related to the specific business. For the palm-swiping field, when performing effective image cropping on the first original image and the second original image, the blank areas in the first original image and the second original image can be cropped off to retain as much area containing the target object as possible in the middle of the first original image and the second original image.

[0081] In an embodiment of the present application, a specific cropping ratio can be pre-set, and then the first original image and the second original image are effectively cropped according to the specific cropping ratio, thereby obtaining the first cropped image and the second cropped image respectively. During implementation, the specific cropping ratio can be an unconventional ratio, where a conventional ratio can be, for example, a 16:9 ratio or a 4:3 ratio, and an unconventional ratio is a ratio other than a conventional ratio, for example, a full-size ratio, i.e., a 1:1 ratio.

[0082] For example, in an embodiment of the present application, the first and second original images can be cropped using a full-size ratio to obtain first and second cropped images, respectively. This allows for the complete preservation of all valid information in the first and second original images, namely, all information about the target object, thereby preserving the entire FOV of the first and second original images.

[0083] Step S103 , performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image; performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image.

[0084] In image processing, distortion refers to the image deformation caused by technical or physical factors during camera capture. Light entering the camera through a lens is a delicate and complex process. As light passes through the lens, it experiences loss and refraction, resulting in a certain degree of distortion in every optical lens. This distortion can degrade image quality and affect image recognition, analysis, and application. Therefore, distortion correction is necessary.

[0085] Generally speaking, image distortion is divided into two main categories: radial distortion and tangential distortion. Radial distortion is caused by the optical properties of the lens and mainly manifests as magnification or reduction of the center and edges of the image. Radial distortion includes barrel distortion and pincushion distortion. Tangential distortion is caused by assembly errors between the lens and the imaging plane and mainly manifests as distortion of the image in certain directions. Tangential distortion is described by the following two parameters: k1 (the first parameter representing tangential distortion) and k2 (the second parameter representing tangential distortion).

[0086] The goal of the distortion correction processing in the embodiment of the present application is to correct radial distortion, that is, the types of image distortion in the embodiment of the present application mainly include pincushion distortion and barrel distortion. In pincushion distortion, the edges of the image are more reduced than the center, causing the image to look like a pincushion. Pincushion distortion generally occurs in telephoto lenses, or the telephoto end of a zoom lens, and the original straight line of the target object will shrink toward the middle, like a pillow. In barrel distortion, the edges of the image are more magnified than the center, causing the image to look like a barrel. Barrel distortion is a distortion phenomenon caused by the physical properties of the lens and the structure of the lens group in the camera lens, in which the imaging picture appears to be barrel-shaped and expanded. It generally occurs at the wide-angle end, or the wide-angle end of a zoom lens, and the straight lines at the edge of the picture expand outward to form a barrel shape. Barrel distortion is like sticking paper on a ball, and pincushion distortion is like sticking paper close to a bowl.

[0087] Distortion correction includes at least one of the following methods: geometric correction, correction plate correction, adaptive filtering, and convolution kernel correction. The geometric correction method is a distortion correction method based on camera intrinsic and extrinsic parameters. This method calculates the camera's internal and external parameters to estimate the transformation matrix required for distortion correction. During implementation, the camera can be calibrated first, by repeatedly photographing a specific calibration object to obtain the camera's internal and external parameters. These parameters are then used to perform distortion correction. The advantage of the geometric correction method is that it achieves good correction results and relatively high accuracy. The correction plate correction method is a simple and effective distortion correction method. The principle of the correction plate correction method is to first capture an image of a correction plate of known shape, then measure the shape of the correction plate within the image, and finally use the measurement results to perform distortion correction. The advantage of the correction plate correction method is that it is simple to implement, requiring only a correction plate of known shape, and it also offers relatively high correction accuracy. The method based on adaptive filtering is a method that filters based on the local features of the image. The idea of ​​the method based on adaptive filtering is to determine the degree of distortion based on the local features of the image and filter the local features of the image to achieve the purpose of distortion correction. The advantage of the method based on adaptive filtering is that it can adapt to different types and degrees of distortion, and can perform distortion correction processing without calibration objects. The correction method based on the convolution kernel is a method based on the transformation matrix. The idea of ​​the correction method based on the convolution kernel is to transform the image through the convolution kernel to achieve distortion correction. The convolution kernel is usually a square matrix, in which each element in the matrix corresponds to a coefficient in the transformation matrix. The advantage of the correction method based on the convolution kernel is that it is simple to implement. It only needs to calculate the convolution kernel and the transformation matrix. Moreover, the correction method based on the convolution kernel can adapt to a variety of distortion types and degrees.

[0088] Step S104 : performing distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

[0089] Here, distortion spatial alignment includes two processing processes: spatial alignment and image cropping. When performing distortion spatial alignment, spatial alignment and image cropping are performed simultaneously, that is, spatial alignment and image cropping are performed synchronously. Since the first and second cropped images are subjected to distortion correction processing respectively, and the first and second distortion-corrected images are obtained accordingly, the first and second distortion-corrected images are correspondingly deformed images. Therefore, when cropping the deformed images, the effective FOV in the first and second original images may be cropped. However, if the image cropping and spatial alignment processes are performed simultaneously on the first and second distortion-corrected images, cropping of the effective FOV in the first and second original images can be avoided during spatial alignment, thereby ensuring that the spatially aligned images retain more of the effective FOV in the first and second original images.

[0090] In an image processing method provided by an embodiment of the present application, when performing spatial alignment of a pair of original images, an electronic device sequentially performs effective image cropping and distortion correction processing on a first original image, and also sequentially performs effective image cropping and distortion correction processing on a second original image, thereby obtaining a first distortion-corrected image and a second distortion-corrected image after distortion correction, respectively. Subsequently, the first distortion-corrected image and the second distortion-corrected image are subjected to distortion spatial alignment to obtain a spatially aligned image containing a target object. Since the first and second distortion-corrected images are subjected to distortion spatial alignment, the distortion spatial alignment simultaneously distorts and spatially aligns the first and second distortion-corrected images. This ensures that non-intersecting portions of the first and second original images are not excessively cropped during image cropping, that is, the effective FOV of the original image pair is not cropped. Therefore, the method of the embodiment of the present application optimizes the effective FOV between the original image pairs captured by the first and second image acquisition devices while ensuring alignment of the original image pairs.

[0091] Below, taking a palm-swipe device as an example, an application scenario of the image processing method provided by the embodiment of the present application is described. The embodiment of the present application can be applied to at least any of the following exemplary scenarios:

[0092] Scenario 1: A palm-scanning device can be used for clocking in and out. The image processing system includes at least the palm-scanning device and the backend server of the clocking-in system. The palm-scanning device can be a dual-camera device with a first-type image acquisition device and a second-type image acquisition device. The first-type image acquisition device is used to capture infrared images, and the second-type image acquisition device is used to capture color images. When a user clocks in and out with a palm-scanning device, the palm-scanning device captures both an infrared image and a color image containing the user's palm print. The user's identity can be identified based on the infrared and color images.

[0093] To ensure the accuracy of identity recognition, the infrared image and the color image can be spatially aligned to obtain a spatially aligned image containing the user's palm print, which can then be used to identify the user. When spatially aligning the infrared image and the color image, the image processing method provided in the embodiments of the present application can be used to implement it.

[0094] Scenario 2: The palm-swiping device can be a collection device for implementing the palm-swiping payment service, and the image processing system includes at least the palm-swiping device and the backend server of the payment application corresponding to the palm-swiping payment service. For example, the payment application can be run on the palm-swiping device, and the palm-swiping device can also collect the user's palm print image, so as to identify the user based on the collected palm print image, determine the user's payment account according to the identification result, and deduct money from the payment account in the payment application. For another example, the palm-swiping device can also be connected to a terminal, on which the payment application can be run. The palm-swiping device collects the user's palm print image and sends the palm print image to the terminal. The terminal sends the user's palm print image to the backend server of the payment application through the payment application running. After the user is identified by the backend server of the payment application, the user's payment account is determined according to the identification result, and the payment account is deducted in the payment application. Among them, the palm-scanning device can be a device with dual cameras, which include a first type of image acquisition device and a second type of image acquisition device. The first type of image acquisition device is used to acquire infrared images, and the second type of image acquisition device is used to acquire color images. That is to say, the palm print images acquired by the palm-scanning device include infrared images and color images.

[0095] Similarly, to ensure the accuracy of identity recognition, the infrared image and the color image can be spatially aligned to obtain a spatially aligned image containing the user's palm print, and the user's identity can be identified using this spatially aligned image. When spatially aligning the infrared image and the color image, the image processing method provided in the embodiments of the present application can be used to implement it.

[0096] The image processing method provided by the embodiment of the present application will be described below using the above scenario 1 as an example. FIG5 is another optional flow chart of the image processing method provided by the embodiment of the present application. As shown in FIG5 , the method includes the following steps S201 to S211:

[0097] In step S201, the palm-swiping device captures a first original image containing a target object through a first type of image acquisition device, and captures a second original image containing the target object through a second type of image acquisition device.

[0098] Here, the first original image and the second original image constitute an original image pair. When the first original image and the second original image are acquired, the first type of image acquisition device and the second type of image acquisition device acquire the images simultaneously.

[0099] In step S202 , the palm-swiping device encapsulates the first original image and the second original image as an original image pair to be aligned into an identity authentication request.

[0100] Here, when encapsulating the first original image and the second original image as the original image pair to be aligned into the identity authentication request, you can first create a request object and create an identity authentication request object to store the image data and related metadata. Then, perform image serialization to serialize the image data of the first original image and the second original image into a format suitable for transmission, such as Base64 encoding. Finally, add the serialized image data to the identity authentication request object, and you can also add other necessary metadata, such as a timestamp and the device ID of the palm-swiping device.

[0101] Step S203: The palm-swiping device sends an identity authentication request to the server.

[0102] Here, the palm-swiping device may have a network module, and the identity authentication request may be sent to the server through the network module. In some embodiments, the palm-swiping device may also be connected to a terminal, and an application for a specific service (such as an identity authentication service) may be running on the terminal. After collecting the first original image and the second original image, the palm-swiping device may send the first original image and the second original image to the terminal, and the terminal may run the application to encapsulate the first original image and the second original image as the original image pair to be aligned into the identity authentication request, and the terminal may send the identity authentication request to the server. In some embodiments, the terminal may use protocols such as HTTP or Web Socket to send the identity authentication request.

[0103] Step S204: The server obtains the first original image and the second original image in the original image pair in response to the identity authentication request.

[0104] In the embodiment of the present application, the first and second original images may be images exhibiting barrel distortion. Barrel distortion is a distortion phenomenon caused by the physical properties of the camera lens and the structure of the lens assembly, resulting in the image appearing barrel-shaped. In barrel distortion, straight lines at the edges of the image expand outward, forming a barrel shape.

[0105] In step S205 , the server crops the first original image using a specific cropping ratio to obtain a first cropped image, and crops the second original image using a specific cropping ratio to obtain a second cropped image.

[0106] The crop ratio refers to the size ratio of the cropped image to the original image. For example, a crop ratio of 0.8 means the width and height of the cropped image are 80% of the original image, respectively. Cropping a first cropped image from a first original image based on a specific crop ratio may involve cropping a target region from the first original image. Cropping methods can include center cropping, edge cropping, or cropping based on a specific region.

[0107] In some embodiments, the specific cropping ratio may be an unconventional ratio; using the specific cropping ratio to perform effective image cropping on the first original image and the second original image respectively can be achieved in the following manner: using an unconventional ratio to perform full-size effective image cropping on the first original image to obtain a first cropped image, and using an unconventional ratio to perform full-size effective image cropping on the second original image to obtain a second cropped image.

[0108] In the embodiment of the present application, due to the use of an unconventional ratio, the first original image and the second original image are respectively cropped to full-size effective images, and the first original image and the second original image are images with barrel distortion, so the first cropped image and the second cropped image obtained after the effective image cropping are also images with barrel distortion. Moreover, by using a specific cropping ratio to crop the first original image and the second original image, the blank areas in the image can be effectively removed, and the core part of the target object can be retained. This method is widely used in the image preprocessing stage in palm-brush devices, which helps to improve image quality and processing efficiency. By reasonably selecting the cropping ratio, it can be ensured that the cropped image meets the needs of actual applications.

[0109] In step S206 , the server performs distortion correction processing on the first cropped image with barrel distortion to obtain a first initial corrected image with a pincushion shape, and performs distortion correction processing on the second cropped image with barrel distortion to obtain a second initial corrected image with a pincushion shape.

[0110] In an embodiment of the present application, any correction method can be used to perform distortion correction processing on the first cropped image and the second cropped image with barrel distortion, respectively, wherein the correction methods include: geometric correction method, correction plate correction method, adaptive filtering-based method and convolution kernel-based correction method.

[0111] After the first cropped image and the second cropped image with barrel distortion are respectively subjected to distortion correction processing, the resulting first initial corrected image and the second initial corrected image will have a pillow shape, that is, the left and right sides of the first initial corrected image and the second initial corrected image will shrink toward the middle, like a pillow.

[0112] In step S207, the server uses black pixels to fill two arc regions in the first initial corrected image having a pincushion shape to obtain a first distortion-corrected image, and uses black pixels to fill two arc regions in the second initial corrected image having a pincushion shape to obtain a second distortion-corrected image.

[0113] Here, the first distortion-corrected image and the second distortion-corrected image respectively include: two arc-shaped black-bordered areas formed after the first cropped image with barrel distortion is subjected to distortion correction processing, and two arc-shaped black-bordered areas formed after the second cropped image with barrel distortion is subjected to distortion correction processing. The two arc-shaped areas refer to the inwardly concave areas respectively possessed by the first initial corrected image and the second initial corrected image formed with a pillow shape. That is to say, in palm swipe recognition, after the first cropped image is subjected to distortion correction processing to obtain the first initial corrected image, and the second cropped image is subjected to distortion correction processing to obtain the second initial corrected image, it is usually necessary to fill in the blank areas generated by the distortion correction. These blank areas are usually in the shape of arcs and are located at the edges of the first initial corrected image and the second initial corrected image.

[0114] When filling the arc area, the two concave areas may be filled with black pixels, and the concave areas filled with black pixels are the arc black edge areas.

[0115] It should be noted that the embodiment of the present application uses the distortion correction process as an example to illustrate only the inward distortion correction of the left and right edges of the first and second cropped images with barrel distortion. In this case, two symmetrical left and right circular arc regions are formed. After filling these two circular arc regions with black pixels, two circular arc black-border regions are formed. Of course, in other embodiments, the distortion correction process can also be performed by correcting the inward distortion of the top, bottom, left, and right edges of the first and second cropped images with barrel distortion. In this way, two symmetrical circular arc regions and two symmetrical left and right circular arc regions are formed, for a total of four circular arc regions. After filling these four circular arc regions with black pixels, four circular arc black-border regions are formed.

[0116] In step S208 , the server performs spatial alignment on the first distortion-corrected image and the second distortion-corrected image using a center-alignment method to obtain a spatially aligned overlapping image.

[0117] In some embodiments, referring to FIG. 6 , FIG. 6 illustrates a process of spatially aligning the first distortion-corrected image and the second distortion-corrected image using a center-alignment method in step S208 , which can be implemented by following steps S2081 to S2084 :

[0118] Step S2081 : determining a first object center line of the target object in the first distortion-corrected image, and determining a second object center line of the target object in the second distortion-corrected image.

[0119] Here, a first object centerline of the target object in the first distortion-corrected image having the arc-shaped black border region and a second object centerline of the target object in the second distortion-corrected image having the arc-shaped black border region can be determined. The first object centerline is a line passing through the center point of the target object in the first distortion-corrected image and parallel to the left and right sides of the first distortion-corrected image having the arc-shaped black border region; the second object centerline is a line passing through the center point of the target object in the second distortion-corrected image and parallel to the left and right sides of the second distortion-corrected image having the arc-shaped black border region.

[0120] Step S2082: Determine the relative positional relationship between the first distortion-corrected image and the second distortion-corrected image.

[0121] Here, the relative positional relationship between the first distortion-corrected image having the arc-shaped black border region and the second distortion-corrected image having the arc-shaped black border region can be determined. The relative positional relationship refers to the left-right relative relationship between the first distortion-corrected image having the arc-shaped black border region and the second distortion-corrected image having the arc-shaped black border region. In other words, it determines which of the two images, the first distortion-corrected image having the arc-shaped black border region and the second distortion-corrected image having the arc-shaped black border region, is located relatively to the left and which is located relatively to the right.

[0122] Step S2083: Determine a first distance between a first image edge of the first distortion-corrected image and a center line of the first object based on the relative position relationship, and determine a second distance between a second image edge of the second distortion-corrected image and a center line of the second object.

[0123] Here, the first image edge is an edge away from the second distortion-corrected image determined based on the relative position relationship, and the second image edge is an edge away from the first distortion-corrected image determined based on the relative position relationship.

[0124] Step S2084: spatially aligning the first distortion-corrected image and the second distortion-corrected image based on the first distance and the second distance to obtain a spatially aligned overlapping image.

[0125] Here, a first distortion-corrected image with an arc-shaped black border region and a second distortion-corrected image with an arc-shaped black border region can be spatially aligned to obtain a spatially aligned overlapping image. During implementation, a first movement direction and a first movement distance of the first distortion-corrected image with an arc-shaped black border region can be calculated based on a first distance and a second distance, respectively. A second movement direction and a second movement distance of the second distortion-corrected image with an arc-shaped black border region can also be calculated based on the first distance and the second distance. Then, the first distortion-corrected image with an arc-shaped black border region is moved according to the first movement direction and the first movement distance, while the second distortion-corrected image with an arc-shaped black border region is simultaneously moved according to the second movement direction and the second movement distance. This completes the spatial alignment of the first distortion-corrected image with an arc-shaped black border region and the second distortion-corrected image with an arc-shaped black border region, obtaining a spatially aligned overlapping image. The spatially aligned overlapping image is the image of the overlapping region of the first distortion-corrected image with an arc-shaped black border region and the second distortion-corrected image with an arc-shaped black border region after both the first distortion-corrected image with an arc-shaped black border region and the second distortion-corrected image with an arc-shaped black border region have been moved. The image of this overlapping region constitutes the spatially aligned image.

[0126] In an embodiment of the present application, spatial alignment of a first distortion-corrected image having an arc-shaped black border area and a second distortion-corrected image having an arc-shaped black border area is performed based on a first distance and a second distance. This can be achieved in the following manner: first, based on the relative position relationship, a first alignment direction (i.e., a first movement direction) of the first distortion-corrected image having an arc-shaped black border area and a second alignment direction (i.e., a second movement direction) of the second distortion-corrected image having an arc-shaped black border area are determined; wherein the first alignment direction is opposite to the second alignment direction; then, based on the first distance and the second distance, a first movement distance of the first distortion-corrected image having an arc-shaped black border area is determined, and, based on the first distance and the second distance, a second movement distance of the second distortion-corrected image having an arc-shaped black border area is determined; finally, the first distortion-corrected image having an arc-shaped black border area is moved along the first alignment direction according to the first movement distance, and the second distortion-corrected image having an arc-shaped black border area is moved along the second alignment direction according to the second movement distance, to obtain overlapping images after spatial alignment. That is to say, after determining the first distance and the second distance, it is also necessary to calculate the first moving distance of the first distortion-corrected image with the arc black edge area based on the first distance and the second distance, and calculate the second moving distance of the second distortion-corrected image with the arc black edge area, and then use the first moving distance and the second moving distance to implement the spatial alignment process.

[0127] As shown in Figure 7, it is a schematic diagram of an implementation principle of image processing using a center alignment method provided by an embodiment of the present application. When performing spatial alignment, the two images (i.e., the first distortion-corrected image and the second distortion-corrected image) are moved and aligned in the left and right directions. Among them, the target object is the palm 700 in Figure 7, the solid-line frame corresponds to the first distortion-corrected image with a circular arc black border area, the dotted-line frame corresponds to the second distortion-corrected image with a circular arc black border area, and the circular arc black border area filled with black pixels is not shown in Figure 7. Assuming that the widths of the two images are the same and known, both are D, Figure 7 shows the result after spatial alignment. Therefore, the center line of the first object and the center line of the second object overlap in Figure 7, which is represented as the first object center line L (including the overlapping second object center line). The relative positional relationship between the first distortion-corrected image with the arc-shaped black border region and the second distortion-corrected image with the arc-shaped black border region is as follows: the first distortion-corrected image with the arc-shaped black border region is located to the left of the second distortion-corrected image with the arc-shaped black border region. Therefore, during spatial alignment, the first distortion-corrected image with the arc-shaped black border region should be moved to the right, and the second distortion-corrected image with the arc-shaped black border region should be moved to the left. Furthermore, based on the relative positional relationship, the first image edge L1 of the first distortion-corrected image with the arc-shaped black border region and the second image edge L2 of the second distortion-corrected image with the arc-shaped black border region can be determined, and the first distance d1 between the first image edge L1 of the first distortion-corrected image with the arc-shaped black border region and the first object centerline L can be calculated, as well as the second distance d2 between the second image edge L2 of the second distortion-corrected image with the arc-shaped black border region and the second object centerline L can be calculated.

[0128] After obtaining the first distance d1 and the second distance d2, the first distortion-corrected image with the arc-shaped black border area and the second distortion-corrected image with the arc-shaped black border area are spatially aligned based on the first distance d1 and the second distance d2. When performing spatial alignment, the length of d22 in Figure 7 can be calculated to be equal to D-d1, and the length of d11 in Figure 7 can be calculated to be equal to D-d2. After calculating d11 and d22, the first movement distance d1-d11 for the first distortion-corrected image with the arc-shaped black border area to the right and the second movement distance d2-d22 for the second distortion-corrected image with the arc-shaped black border area to the left can be calculated. In this way, the first distortion-corrected image with the arc-shaped black border area and the second distortion-corrected image with the arc-shaped black border area can be spatially aligned based on the calculated first movement distance d1-d11 and second movement distance d2-d22, respectively, to obtain the final spatially aligned overlapping image, i.e., the area 701 with the cross-hatching line in Figure 7.

[0129] In other embodiments, referring to FIG. 8 , FIG. 8 illustrates a process of spatially aligning the first distortion-corrected image and the second distortion-corrected image using a center-alignment method in step S208 , which may also be implemented by the following steps S2085 to S2086 :

[0130] Step S2085: Determine a first center line of the first distortion-corrected image and a second center line of the second distortion-corrected image.

[0131] Since the palm is usually located in the middle of the image when swiping the palm (even if it is not exactly in the middle, the deviation to either side will not be too large), spatial alignment is performed with the first center line of the first distortion-corrected image with the circular black border area and the second center line of the second distortion-corrected image with the circular black border area as the alignment targets.

[0132] Here, the first center line is the central axis of the first distortion-corrected image of the entire arc black-border area, and the first center line is parallel to the left and right sides of the first distortion-corrected image of the arc black-border area; the second center line is the central axis of the second distortion-corrected image of the entire arc black-border area, and the second center line is parallel to the left and right sides of the second distortion-corrected image of the arc black-border area.

[0133] Step S2086: spatially aligning the first distortion-corrected image and the second distortion-corrected image based on the first center line and the second center line to obtain a spatially aligned overlapping image.

[0134] Here, based on the first and second centerlines, spatial alignment is performed on the first distortion-corrected image having a circular black border region and the second distortion-corrected image having a circular black border region. Either the first or second centerline can be used as the target centerline, with the target centerline serving as the centerline of the final aligned overlapping image for spatial alignment of the first and second distortion-corrected images. Alternatively, a parallel line between the first and second centerlines can be used as the target centerline. For example, the vertical distance between the first and second centerlines can be determined, with half of the vertical distance used as the centerline shift value. The first centerline can then be shifted toward the second centerline according to the centerline shift value, or the second centerline can be shifted toward the first centerline according to the centerline shift value, with the line segment at the final end point of the shift being used as the target centerline. After determining the target centerline, the target centerline can be used as the centerline of the final aligned overlapping image for spatial alignment of the first and second distortion-corrected images.

[0135] When performing spatial alignment, the first distortion-corrected image and the second distortion-corrected image can be moved left and right respectively until the first center line of the first distortion-corrected image and the second center line of the second distortion-corrected image overlap with the target center line, thereby obtaining the final overlapping image after spatial alignment, see area 701 with a section line in Figure 7.

[0136] In step S209 , the server crops the spatially aligned overlapping images to obtain a spatially aligned image containing the target object.

[0137] Here, image cropping can be performed by cropping overlapping images after spatial alignment to obtain a spatially aligned image containing the target object. As shown in FIG7 , the area outside of region 701 having a crosshatch line can be cropped, leaving only region 701 having a crosshatch line. The image formed by region 701 is used as the spatially aligned image containing the target object.

[0138] In step S210 , the server performs identity recognition on the user based on the spatially aligned images to obtain an identity recognition result.

[0139] In an embodiment of the present application, a pre-trained identity recognition model can be used for identity recognition. For the identity recognition model, a sample image can be obtained first, and the sample image can be pre-processed. For example, the pre-processing steps include: grayscale (converting the sample image into a grayscale image), denoising (using methods such as Gaussian blur to remove noise in the sample image), and binarization (converting the sample image into a binary image for subsequent processing). Then, feature extraction is performed on the pre-processed sample image, and the features of the target object are extracted from the pre-processed sample image. Texture features can be extracted using a Gabor filter, or key points and descriptors can be extracted from the pre-processed sample image using an oriented fast feature detection and rotation BRIEF (ORB, Oriented FAST and Rotated BRIEF) feature extraction algorithm. After feature extraction, the extracted features can be used to train a machine learning model, such as a support vector machine (SVM) or a deep learning model (such as VGG16), to train an identity recognition model. Afterwards, the trained identity recognition model is used to perform identity recognition on the spatially aligned image.

[0140] Step S211: The server processes the current service based on the identity recognition result.

[0141] Here, the server processes the current business based on the identity recognition result. It can process various different businesses. The following examples illustrate:

[0142] In financial services, palm scanning devices capture a user's palm image. Palm recognition technology then identifies and compares the user's palm features. If authentication is successful, the system grants payment authorization, allowing the user to proceed. In banks and financial institutions, palm recognition can be used to log in and operate user accounts, ensuring account security and convenient operations.

[0143] In access control and security applications, for example, in businesses, schools, or community settings, palm recognition technology can be used in access control systems to verify and record the identities of people entering and exiting, improving security management. For another example, palm recognition technology can detect and compare facial information in real time, quickly identifying suspicious individuals and enhancing safety.

[0144] In intelligent transportation services, for example, palm recognition can be used to quickly verify passenger identities in public transportation scenarios such as subways and buses, enabling contactless payment and access, and improving travel efficiency. Another example is palm recognition, which can be used to quickly verify passenger identities at airport security checks and railway ticket inspections, reducing queuing times and improving travel efficiency.

[0145] In healthcare, for example, palm recognition can be used to quickly collect patient identity information during hospital registration and check-in, reducing waiting times and improving the efficiency of medical services. Another example is palm recognition, which can be used to quickly verify patient identity during medical insurance reimbursement and settlement, ensuring the safe use of medical insurance funds.

[0146] In the education sector, for example, during school enrollment and student status management, palm swipe recognition can be used to quickly collect and manage student identity information, improving the efficiency and accuracy of information collection. Another example is during the registration process for various exams, palm swipe recognition can be used to quickly verify the identity of candidates.

[0147] In smart retail, for example, in unmanned retail scenarios, palm recognition can be used to quickly verify customer identities, enabling contactless shopping and payment, and enhancing the shopping experience. Another example is that in retail enterprises, palm recognition can be used for member authentication and management, providing personalized services.

[0148] In government service businesses, for example, in the government's online service platform, palm recognition can be used by citizens to handle household registration, passport applications, social security inquiries and other services online, ensuring that the identity of citizens handling government services is authentic and valid.

[0149] In smart office applications, for example, in corporate attendance management, palm swipe recognition can be used to quickly verify employee identities, implement automated attendance, and improve management efficiency. Another example is a conference room reservation system, where palm swipe recognition can be used to quickly verify user identities and automatically reserve and use conference rooms.

[0150] The image processing method provided in the embodiment of the present application simultaneously performs image cropping and spatial alignment for a first original image and a second original image having barrel distortion, thereby ensuring that, during image cropping, non-intersecting portions of the first and second original images are not excessively cropped. In other words, the effective FOV of the original image pair is not cropped. Therefore, the method of the embodiment of the present application can optimize the effective FOV between the original images captured by the first and second image acquisition devices in the field of identity recognition while ensuring the alignment of the original image pair.

[0151] In some embodiments, the first and second original images may also be images exhibiting pincushion distortion. The image processing method provided by the present embodiment is described below using the first and second original images exhibiting pincushion distortion as an example. As shown in FIG9 , the method includes the following steps S301 to S311.

[0152] It should be noted that the implementation process of steps S301 to S311 is the same as that of steps S201 to S211. The only difference is that steps S301 to S311 perform spatial alignment processing on the first original image and the second original image with pincushion distortion, and steps S201 to S211 perform spatial alignment processing on the first original image and the second original image with barrel distortion. Therefore, the embodiment of the present application will not explain the repeated steps in detail. For specific explanations, please refer to the explanations of the corresponding steps in steps S201 to S211.

[0153] Step S301: The palm-swiping device captures a first original image containing a target object through a first type of image acquisition device, and captures a second original image containing the target object through a second type of image acquisition device.

[0154] Step S302: The palm-swiping device encapsulates the first original image and the second original image as an original image pair to be aligned into an identity authentication request.

[0155] Step S303: The palm-swiping device sends an identity authentication request to the server.

[0156] Step S304: The server obtains the first original image and the second original image in the original image pair in response to the identity authentication request.

[0157] In step S305 , the server crops the first original image using a specific cropping ratio to obtain a first cropped image, and also crops the second original image using a specific cropping ratio to obtain a second cropped image.

[0158] The first original image and the second original image exhibit pincushion distortion; accordingly, the first cropped image and the second cropped image also exhibit pincushion distortion. Pincushion distortion refers to the phenomenon in which the originally straight lines of the target object shrink toward the center, like a pillow.

[0159] In step S306 , the server performs distortion correction processing on the first cropped image with pincushion distortion to obtain a first initial corrected image with a barrel shape, and performs distortion correction processing on the second cropped image with pincushion distortion to obtain a second initial corrected image with a barrel shape.

[0160] In an embodiment of the present application, any correction method can be used to perform distortion correction processing on the first cropped image and the second cropped image with pincushion distortion, respectively, wherein the correction methods include: geometric correction method, correction plate correction method, adaptive filtering-based method and convolution kernel-based correction method.

[0161] After the distortion correction processing is performed on the first cropped image and the second cropped image with pincushion distortion, the resulting first and second initial corrected images will have a barrel shape. In other words, the left and right sides of the first and second initial corrected images will expand outward like a bucket.

[0162] In step S307, the server uses black pixels to fill the four arc regions in the first initial corrected image having a barrel shape to obtain a first distortion corrected image, and uses black pixels to fill the four arc regions in the second initial corrected image having a barrel shape to obtain a second distortion corrected image.

[0163] Here, the first distortion-corrected image and the second distortion-corrected image, respectively, include: four arc-shaped black-bordered regions formed after distortion correction processing is performed on the first cropped image with pincushion distortion, and four arc-shaped black-bordered regions formed after distortion correction processing is performed on the second cropped image with pincushion distortion. The four arc-shaped regions refer to the arc-shaped regions at the four corners of the barrel-shaped image formed after distortion correction processing is performed on the first cropped image with pincushion distortion and the second cropped image with pincushion distortion, respectively. That is, after distortion correction processing is performed on the first cropped image with pincushion distortion and the second cropped image with pincushion distortion to form a barrel-shaped image, a rectangle can be circumscribed around the barrel-shaped image. In this case, there will be four arc-shaped regions located outside the barrel-shaped image and inside the circumscribed rectangle. That is, in palm swipe recognition, after distortion correction processing is performed on the first cropped image to obtain the first initial corrected image and on the second cropped image to obtain the second initial corrected image, it is generally necessary to fill the blank regions at the four corners resulting from the distortion correction. These blank areas are generally in the shape of arcs and are located at the four corners of the first initial corrected image and the second initial corrected image.

[0164] When filling the arc area, the four arc areas may be filled with black pixels, and the arc area filled with black pixels is the arc black edge area mentioned above.

[0165] It should be noted that the embodiment of the present application uses the distortion correction process as an example to illustrate only the correction of the outward distortion of the left and right edges of the first cropped image and the second cropped image with pincushion distortion. In this case, two pairs of symmetrical arc regions (i.e., four arc regions) are formed. After these two pairs of arc regions are filled with black pixels, four arc regions with black edges are formed. Of course, in other embodiments, the distortion correction process can also be performed on the four outward distortion edges of the top, bottom, left, and right edges of the first cropped image and the second cropped image with pincushion distortion. In this way, two symmetrical pairs of arc regions and two symmetrical pairs of arc regions are formed, totaling four pairs of arc regions, i.e., a total of eight arc regions. After these eight arc regions are filled with black pixels, eight arc regions with black edges are formed.

[0166] In step S308 , the server spatially aligns the first distortion-corrected image and the second distortion-corrected image using a center-alignment method to obtain a spatially aligned overlapping image.

[0167] It should be noted that the implementation process of spatially aligning the first distortion-corrected image and the second distortion-corrected image in step S308 is the same as the implementation process in the above-mentioned step S208, and will not be repeated in this embodiment of the application.

[0168] In step S309 , the server crops the spatially aligned overlapping images to obtain a spatially aligned image containing the target object.

[0169] In step S310 , the server performs identity recognition on the user based on the spatially aligned images to obtain an identity recognition result.

[0170] Step S311: The server processes the current service based on the identity recognition result.

[0171] The image processing method provided in the embodiments of the present application simultaneously performs distorted image cropping and spatial alignment for first and second original images with pincushion distortion, thereby ensuring that, during distorted image cropping, non-intersecting portions of the first and second original images are not excessively cropped. In other words, the effective FOV of the original image pair is not cropped. Therefore, the method of the embodiments of the present application can optimize the effective FOV between original images captured by the first and second image acquisition devices in the field of identity recognition while ensuring alignment of the original image pair.

[0172] Based on the above embodiments, the present application further provides a method for implementing distortion correction processing, wherein the first cropped image with barrel distortion and the second cropped image with barrel distortion are respectively subjected to distortion correction processing, or the first cropped image with pincushion distortion and the second cropped image with pincushion distortion are respectively subjected to distortion correction processing, both of which can be performed using the implementation method provided in the present application. Figure 10 is a flow chart of the implementation method for distortion correction processing provided in the present application. As shown in Figure 10, the method includes the following steps S401 to S403:

[0173] Step S401: The server obtains a first device internal parameter of a first image acquisition device and a second device internal parameter of a second image acquisition device.

[0174] In some embodiments, the server obtains the first device internal parameter of the first image acquisition device and the second device internal parameter of the second image acquisition device, which can be achieved in the following manner: first, the server obtains a preset camera calibration standard value; then, using a preset calibration fixture, based on the camera calibration standard value, independently calibrates the first image acquisition device to obtain the first device internal parameter of the first image acquisition device, and, based on the camera calibration standard value, independently calibrates the second image acquisition device to obtain the second device internal parameter of the second image acquisition device.

[0175] The camera calibration standard values ​​are preset values. These typically include calibration board parameters, such as the number of checkerboard corner points and the actual physical dimensions of the squares. These parameters are used in the subsequent calibration process. The calibration fixture is typically a calibration board with a checkerboard pattern. The checkerboard corner points are used as feature points during calibration to calculate the camera's intrinsic parameters.

[0176] When the first image acquisition device (or the second image acquisition device) is independently calibrated based on the camera calibration standard value, a calibration image can be first acquired, and the first image acquisition device (or the second image acquisition device) can be used to capture images of the calibration plate from different angles and distances to ensure that the calibration plate covers the entire field of view of the camera and to capture images from multiple angles to obtain sufficient data. Then, the acquired image is processed to extract the corner point information of the chessboard. For example, the findChessboardCorners function provided by OpenCV can be used to automatically detect the corner points of the chessboard. Afterwards, the calibrateCamera function of OpenCV is used to calculate the internal parameters of the first device (i.e., the internal parameters of the first image acquisition device, or the internal parameters of the second device) based on the extracted corner point information and the actual size of the calibration plate. The internal parameters include focal length, principal point coordinates, and distortion coefficients.

[0177] In the embodiment of the present application, the first type of image acquisition device may be an infrared camera, and the second type of image acquisition device may be a color camera; accordingly, the first original image is an infrared image, and the second original image is a color image.

[0178] In step S402, the server obtains a first distortion coefficient of the first image acquisition device from the first device internal parameter, and uses the first distortion coefficient to perform distortion correction processing on the first cropped image to obtain a first distortion-corrected image.

[0179] Here, the first distortion coefficient refers to a coefficient that changes the image when the first image acquisition device acquires the image. When performing distortion correction processing, a distortion correction coefficient for the first image acquisition device can be determined based on the first distortion coefficient. The distortion correction coefficient has the same value as the first distortion coefficient and has an opposite sign. That is, if the first distortion correction coefficient is +0.5 (indicating that the image acquired by the first image acquisition device is distorted 0.5 times toward a square), the corresponding distortion correction coefficient is -0.5 (indicating that during distortion correction processing, the distortion needs to be corrected 0.5 times in the negative direction).

[0180] In step S403, the server obtains a second distortion coefficient of the second image acquisition device from the internal parameters of the second device, and uses the second distortion coefficient to perform distortion correction processing on the second cropped image to obtain a second distortion-corrected image.

[0181] Here, the second distortion coefficient refers to a coefficient that changes the image when the second image acquisition device acquires the image. During distortion correction processing, a distortion correction coefficient for the second image acquisition device can be determined based on the second distortion coefficient. The distortion correction coefficient has the same magnitude as the second distortion coefficient and an opposite sign.

[0182] In the embodiments of the present application, a calibration jig is used to independently calibrate the first and second image acquisition devices based on preset camera calibration standards, thereby obtaining device intrinsic parameters for the first and second image acquisition devices. These device intrinsic parameters can be used for subsequent image distortion correction and identity recognition, ensuring image quality and recognition accuracy.

[0183] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0184] In the face-scanning scenario, since the related art performs spatial alignment on the color image, infrared image, and depth image, the palm-scanning scenario does not care about the horizontal effective FOV, but is more concerned with the vertical effective FOV. In the palm-scanning scenario, the color image and infrared image are aligned according to a fixed position. The disadvantages of this fixed-position alignment are: on the one hand, the spatial alignment effect is poor, and the palm is only aligned at the selected calibration position (for example, at 8 cm (CM) calibration, only the palm at 8 cm is aligned); on the other hand, the horizontal effective FOV is small, resulting in a small range of available distances for palm-scanning and a poor user experience. Alternatively, in the palm-scanning field, the two images are dynamically aligned based on the depth distance of the PSensor. Although this method can achieve dynamic palm alignment (alignment within 5 cm to 12 cm of the palm), the horizontal effective FOV is small, resulting in a small range of available distances for palm-scanning and a poor user experience.

[0185] Based on at least one of the above-mentioned problems existing in the related art, an embodiment of the present application provides an image processing method, which optimizes the horizontal effective FOV of the camera in the palm-brushing area by using effective image cropping (Crop) with full-size resolution, aligning to the middle of the two images during spatial alignment, and using the arc black edges in the barrel-distorted original image.

[0186] Figure 11 is a schematic diagram of a process for obtaining an effective FOV in a palm-brushing field provided by an embodiment of the present application. The process for obtaining an effective FOV is an image processing method. For the infrared image 111 and the color image 112 (corresponding to the above-mentioned first original image and the second original image) in the original image, effective image cropping of the original image is performed in sequence to obtain the original cropped image (including the infrared cropped image and the color cropped image, corresponding to the above-mentioned first cropped image and the second cropped image); then, the original cropped image is subjected to distortion correction processing to obtain a distortion-corrected image (including the infrared distortion-corrected image and the color distortion-corrected image, corresponding to the above-mentioned first distortion-corrected image and the second distortion-corrected image); then, the distortion-corrected image is cropped again to obtain a distortion-corrected cropped image (including the infrared distortion-corrected cropped image and the color distortion-corrected cropped image). Finally, the two distortion-corrected cropped images are spatially aligned to obtain a final spatially aligned image, that is, a final effective FOV image.

[0187] Based on this spatial alignment method, an embodiment of the present application further provides a method for optimizing the effective FOV of a camera palm swiping within a palm swiping field. This method for optimizing the effective FOV of a camera palm swiping is also a spatial alignment method, as shown in FIG12 , including the following steps S501 to S505:

[0188] Step S501 : The server performs image cropping using a full-size FOV resolution with an unconventional ratio in an effective image cropping phase.

[0189] In the embodiment of the present application, as shown in FIG13a, effective image cropping (Crop) of a conventional ratio (e.g., a 16:9 ratio or a 4:3 ratio) is not performed during the effective image cropping and distortion correction stages of the original image, that is, the process in FIG13a is not used for spatial alignment, otherwise FOV loss will result. As shown in FIG13a, effective image cropping of a 16:9 ratio is performed during the effective image cropping and distortion correction stages of the original image, resulting in a loss of both horizontal and vertical FOV in the final spatially aligned image (field of view of 81°×99.6°) relative to the original image (field of view of 92°×116°).

[0190] Furthermore, in this embodiment of the present application, effective image cropping to a 16:9 ratio is not performed during the distortion correction phase, i.e., spatial alignment is not performed using the process in FIG13b , which would otherwise result in the aforementioned FOV loss. As shown in FIG13b , effective image cropping to a 16:9 ratio is performed during the distortion correction phase of the original image, resulting in a lateral FOV loss in the final spatially aligned image (with a field of view of 81°×116°) relative to the original image (with a field of view of 92°×116°).

[0191] In the embodiment of the present application, effective image cropping can be performed according to the full-size FOV resolution of the unconventional ratio, that is, spatial alignment is performed using the process in Figure 13c, so that the full-size FOV can be obtained. As shown in Figure 13c, effective image cropping of the full size of the unconventional ratio is performed in both the effective image cropping and distortion correction stages of the original image, thereby ensuring that the final spatially aligned image (field of view of 92°×116°) has no FOV loss relative to the original image (field of view of 92°×116°).

[0192] Step S502: Calibrate each camera separately in the distortion correction stage.

[0193] As shown in Figure 14, during factory production, due to factors such as design tolerances, stress, torque, glue properties, and the gluing process, the two cameras (such as camera 141 and camera 142 in Figure 14) may not be aligned in the vertical direction. As shown on the right side of Figure 14, the vertical field of view 143 of camera 141 is not completely aligned with the vertical field of view 144 of camera 142. Typically, these uneven problems are calibrated using the gold machine calibration method. After calibration, a set of parameters is adapted to all devices (i.e., infrared cameras and color cameras). Although this method is simple, it is prone to random deviations, as shown in Figure 15. In addition, a 14° buffer space is set in the upper layer based on the empirical value of online samples, which also reduces the available FOV. Due to the large randomness of the deviation of each device, a more reasonable solution should be to calibrate each device separately. That is, in the embodiment of the present application, a method of calibrating each camera separately is adopted during the distortion correction stage. This can provide a more accurate coordinate system for the upper layer, thereby removing the buffer space limitation. During the implementation process, a set of calibration fixtures (i.e., preset calibration fixtures) and calibration standards (for example, a set of camera calibration standard values) can be designed to calibrate each camera, thereby obtaining the device internal parameters of each camera used for distortion correction processing in the distortion correction stage, and then the original image can be distorted based on these device internal parameters.

[0194] Step S503: In the spatial alignment phase, the server performs spatial alignment on the two distortion-corrected images by aligning them toward the center.

[0195] As shown in Figure 16, it is a schematic diagram of the lateral loss analysis provided by the embodiment of the present application. For infrared cameras and color cameras, if the fields of view of the two cameras (such as the field of view 161 of the infrared camera and the field of view 162 of the color camera in Figure 16) do not intersect, FOV will be lost. However, the available intersection area (i.e., the area where the field of view 161 of the infrared camera and the field of view 162 of the color camera intersect) changes with the distance between the palm and the camera. The closer the distance, the smaller the intersection and the greater the FOV loss, and the farther the distance, the greater the intersection and the smaller the FOV loss. The relative position of the palm in the color image and the infrared image is different, and spatial alignment is required, which will also cause FOV loss.

[0196] The usage distance of traditional scenes such as faces is basically more than 30 cm, and the FOV has almost no effect at long distances, so it is basically aligned with the color image, as shown in Figure 17, which shows a comparison diagram of alignment with the color image and alignment with the center. However, palm swiping is usually a close-up scene (the distance between the palm and the camera is about 5 cm to 8 cm). If it is aligned with a certain picture, the FOV loss problem will be more obvious (for example, aligning with the color image will cause more loss of the non-intersection part of the infrared image). Therefore, the coping strategy provided by the embodiment of the present application is: both pictures are aligned to the middle, because the palm has the particularity of "the lens of the color camera and the infrared camera is the same" and "the palm is a plane", so it is feasible to align the palm to the middle.

[0197] In step S504 , during the spatial alignment phase, the server performs image distortion correction and spatial alignment together, and uses distortion correction to correct the arc black edges of the barrel image.

[0198] Here, after aligning toward the center, we found that the blank black edges of the color image correspond exactly to the non-intersecting portions of the infrared image. Similarly, the blank black edges of the infrared image correspond exactly to the non-intersecting portions of the color image. If we can optimize the blank black edges of each image, we can optimize the effective FOV. As shown in Figure 18, Figure 18 (a) shows the result of aligning toward the color image, Figure 18 (b) shows the result of aligning toward the infrared image, and Figure 18 (c) shows the result of aligning the two images toward the center. It can be seen that aligning toward the center optimizes the effective FOV of both images.

[0199] In the embodiment of the present application, after optimizing the blank black borders of the two images, the effective FOV can be optimized. It is also found that the blank black borders of the two images correspond exactly to the arc area of ​​the distortion correction, such as the arc area 1901 in Figure 19; except for a small black border 1902 in the middle of the arc area (i.e., the arc black border area), the rest of the arc area is a useful FOV; if the blank black borders of the two images can be optimized, the effective FOV can be optimized, as shown in Figure 20, which is a schematic diagram of the comparison results of the normal process 201 and the direct spatial alignment 202 based on the distortion-corrected original image. Therefore, after exploring the direction of optimizing the blank black borders of the two images, the following conclusion can be drawn: under the normal process, the blank black borders of the two images directly lose FOV; if the "distortion correction processing and spatial alignment" are combined, the full effective FOV can be obtained, with only a small loss of FOV at the arc.

[0200] In summary, the present invention further provides an overall flow of an image processing method for optimizing FOV, as shown in Figure 21b , where the flow in Figure 21a is the overall flow of the image processing method before optimization. The image processing method for optimizing FOV combines distortion correction with spatial alignment, so that a nearly complete FOV is maintained after spatial alignment.

[0201] Through the above steps, effective FOV optimization can be completed.

[0202] The image processing method provided in the embodiments of this application can optimize the effective FOV from 76.8° x 106° in related art methods to 95° x 116°. Furthermore, the palm-brushing distance can be optimized from 6.5-15 cm to 5-15 cm. In other words, a larger effective FOV can be achieved at a closer palm-brushing distance, effectively increasing the effective FOV in the palm-brushing range.

[0203] It is understandable that in the embodiments of the present application, if the content involves user information, such as the user's palm prints and other information, and if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0204] The following continues to describe an exemplary structure of the image processing device 354 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 3, the image processing device 354 includes: an acquisition module, configured to acquire an original image pair, wherein the original image pair includes a first original image acquired by a first type of image acquisition device and a second original image acquired by a second type of image acquisition device; the first original image and the second original image both contain a target object; an image cropping module, configured to perform image cropping on the first original image to obtain a first cropped image; and to perform image cropping on the second original image to obtain a second cropped image; a distortion correction module, configured to perform distortion correction processing on the first cropped image to obtain a first distortion-corrected image; and to perform distortion correction processing on the second cropped image to obtain a second distortion-corrected image; and a spatial alignment module, configured to perform distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

[0205] In some embodiments, the first original image is an image with barrel distortion; the distortion correction module is further configured to: perform distortion correction processing on the first cropped image with the barrel distortion to obtain a first initial corrected image with a pincushion shape; fill the arc area in the first initial corrected image with black pixels to obtain the first distortion corrected image; wherein the first distortion corrected image includes two arc black-edge areas formed after the distortion correction processing is performed on the first cropped image.

[0206] In some embodiments, the second original image is an image with barrel distortion; the distortion correction module is further configured to: perform distortion correction processing on the second cropped image with the barrel distortion to obtain a second initial corrected image with a pincushion shape; and fill the arc area in the second initial corrected image with black pixels to obtain the second distortion corrected image; wherein the second distortion corrected image includes two arc black-edge areas formed after the distortion correction processing is performed on the second cropped image.

[0207] In some embodiments, the distortion spatial alignment includes spatial alignment and image cropping; the spatial alignment module is further configured to: adopt a center alignment method to spatially align the first distortion-corrected image and the second distortion-corrected image to obtain an overlapping image after spatial alignment; and perform the image cropping on the overlapping image after spatial alignment to obtain a spatially aligned image containing the target object.

[0208] In some embodiments, the spatial alignment module is further configured to: determine a first object centerline of the target object in the first distortion-corrected image, and determine a second object centerline of the target object in the second distortion-corrected image; determine a relative positional relationship between the first distortion-corrected image and the second distortion-corrected image; based on the relative positional relationship, determine a first distance between a first image edge of the first distortion-corrected image and the first object centerline, and a second distance between a second image edge of the second distortion-corrected image and the second object centerline; wherein the first image edge is an edge away from the second distortion-corrected image determined based on the relative positional relationship, and the second image edge is an edge away from the first distortion-corrected image determined based on the relative positional relationship; based on the first distance and the second distance, spatially align the first distortion-corrected image and the second distortion-corrected image to obtain an overlapping image after spatial alignment.

[0209] In some embodiments, the spatial alignment module is further configured to: determine a first alignment direction of the first distortion-corrected image and a second alignment direction of the second distortion-corrected image based on the relative position relationship; wherein the first alignment direction is opposite to the second alignment direction; determine a first movement distance of the first distortion-corrected image based on the first distance and the second distance; determine a second movement distance of the second distortion-corrected image based on the first distance and the second distance; move the first distortion-corrected image along the first alignment direction according to the first movement distance, and move the second distortion-corrected image along the second alignment direction according to the second movement distance to obtain the overlapping image after spatial alignment.

[0210] In some embodiments, the spatial alignment module is further configured to: determine a first center line of the first distortion-corrected image and a second center line of the second distortion-corrected image; based on the first center line and the second center line, spatially align the first distortion-corrected image with the circular arc black edge area and the second distortion-corrected image with the circular arc black edge area to obtain an overlapping image after spatial alignment.

[0211] In some embodiments, the first original image is an image with pincushion distortion; the distortion correction module is further configured to: perform distortion correction processing on the first cropped image with the pincushion distortion to obtain a first initial corrected image with a barrel shape; and fill the arc area in the first initial corrected image with black pixels to obtain the first distortion corrected image; wherein the first distortion corrected image includes four arc black-edge areas formed after the distortion correction processing is performed on the first cropped image.

[0212] In some embodiments, the second original image is an image with pincushion distortion; the distortion correction module is further configured to: perform distortion correction processing on the second cropped image with the pincushion distortion to obtain a second initial corrected image with a barrel shape; and fill the arc area in the second initial corrected image with black pixels to obtain the second distortion corrected image; wherein the second distortion corrected image includes four arc black-edge areas formed after the distortion correction processing is performed on the second cropped image.

[0213] In some embodiments, the image cropping module is further configured to: use a specific cropping ratio to crop the first original image into a full-size effective image to obtain the first cropped image.

[0214] In some embodiments, the image cropping module is further configured to: use a specific cropping ratio to crop the second original image into a full-size effective image to obtain the second cropped image.

[0215] In some embodiments, the distortion correction module is further configured to: obtain a first device internal parameter of the first image acquisition device; obtain a first distortion coefficient of the first image acquisition device from the first device internal parameter; and use the first distortion coefficient to perform distortion correction processing on the first cropped image to obtain the first distortion corrected image.

[0216] In some embodiments, the distortion correction module is further configured to: obtain a second device internal parameter of the second image acquisition device; obtain a second distortion coefficient of the second image acquisition device from the second device internal parameter; and use the second distortion coefficient to perform distortion correction processing on the second cropped image to obtain the second distortion corrected image.

[0217] In some embodiments, the distortion correction module is further configured to: obtain a preset camera calibration standard value; use a preset calibration fixture to independently calibrate the first image acquisition device based on the camera calibration standard value, and obtain the first device internal parameter of the first image acquisition device.

[0218] In some embodiments, the distortion correction module is further configured to: obtain a preset camera calibration standard value; use a preset calibration fixture to independently calibrate the second image acquisition device based on the camera calibration standard value to obtain a second device internal parameter of the second image acquisition device.

[0219] In some embodiments, the first type of image acquisition device is an infrared camera, and the second type of image acquisition device is a color camera; the first original image is an infrared image, and the second original image is a color image.

[0220] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0221] The present invention provides a computer program product comprising executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the image processing method described in the present invention.

[0222] An embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided in the embodiment of the present application, for example, the method shown in Figure 4. In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM); or various devices including one or any combination of the above memories.

[0223] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0224] By way of example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). By way of example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0225] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, the method being performed by an electronic device, the method comprising: Acquire an original image pair, the original image pair comprising a first original image acquired by a first type of image acquisition device and a second original image acquired by a second type of image acquisition device; both the first original image and the second original image contain a target object; Performing image cropping on the first original image to obtain a first cropped image; performing image cropping on the second original image to obtain a second cropped image; performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image; performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image; Distortion space alignment is performed on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

2. The method according to claim 1, wherein The first original image is an image with barrel distortion; The performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image includes: performing distortion correction processing on the first cropped image having the barrel distortion to obtain a first initial corrected image having a pincushion shape; The arc region in the first initial corrected image is filled with black pixels to obtain the first distortion-corrected image; wherein the first distortion-corrected image includes two arc black-edge regions formed after the first cropped image is subjected to distortion correction processing.

3. The method according to claim 1 or 2, wherein: The second original image is an image with barrel distortion; The performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image includes: performing distortion correction processing on the second cropped image having the barrel distortion to obtain a second initial corrected image having a pincushion shape; The arc region in the second initial corrected image is filled with black pixels to obtain the second distortion-corrected image; wherein the second distortion-corrected image includes two arc black-edge regions formed after the second cropped image is subjected to distortion correction processing.

4. The method according to any one of claims 1 to 3, wherein: The distortion spatial alignment includes spatial alignment and image cropping; performing the distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object includes: Using a center-alignment method, spatially aligning the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned overlapping image; The image is cropped on the spatially aligned overlapping images to obtain a spatially aligned image containing the target object.

5. The method according to any one of claims 1 to 4, wherein: The method of spatially aligning the first distortion-corrected image and the second distortion-corrected image by aligning toward the center to obtain a spatially aligned overlapping image includes: determining a first object centerline of the target object in the first distortion-corrected image, and determining a second object centerline of the target object in the second distortion-corrected image; determining a relative positional relationship between the first distortion-corrected image and the second distortion-corrected image; Determining, based on the relative positional relationship, a first distance between a first image edge of the first distortion-corrected image and a centerline of the first object, and a second distance between a second image edge of the second distortion-corrected image and the centerline of the second object; wherein the first image edge is an edge determined based on the relative positional relationship and away from the second distortion-corrected image, and the second image edge is an edge determined based on the relative positional relationship and away from the first distortion-corrected image; Based on the first distance and the second distance, the first distortion-corrected image and the second distortion-corrected image are spatially aligned to obtain a spatially aligned overlapping image.

6. The method according to any one of claims 1 to 5, wherein: The step of spatially aligning the first distortion-corrected image and the second distortion-corrected image based on the first distance and the second distance to obtain a spatially aligned overlapping image includes: determining a first alignment direction of the first distortion-corrected image and a second alignment direction of the second distortion-corrected image according to the relative positional relationship; wherein the first alignment direction is opposite to the second alignment direction; determining a first movement distance of the first distortion-corrected image based on the first distance and the second distance; determining a second movement distance of the second distortion-corrected image based on the first distance and the second distance; The first distortion-corrected image is moved along the first alignment direction according to the first movement distance, and the second distortion-corrected image is moved along the second alignment direction according to the second movement distance to obtain the spatially aligned overlapping image.

7. The method according to any one of claims 1 to 6, wherein: The method of spatially aligning the first distortion-corrected image having the arc black edge region and the second distortion-corrected image having the arc black edge region by aligning toward the middle to obtain the overlapped image after spatial alignment includes: determining a first center line of the first distortion-corrected image and a second center line of the second distortion-corrected image; Based on the first center line and the second center line, the first distortion-corrected image with the arc black edge region and the second distortion-corrected image with the arc black edge region are spatially aligned to obtain a spatially aligned overlapping image.

8. The method according to any one of claims 1 to 7, wherein: The first original image is an image with pincushion distortion; The performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image includes: performing distortion correction processing on the first cropped image having the pincushion distortion to obtain a first initial corrected image having a barrel shape; The arc region in the first initial corrected image is filled with black pixels to obtain the first distortion-corrected image; wherein the first distortion-corrected image includes four arc black-edge regions formed after the first cropped image is subjected to distortion correction processing.

9. The method according to any one of claims 1 to 8, wherein: The second original image is an image with pincushion distortion; The performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image includes: performing distortion correction processing on the second cropped image having the pincushion distortion to obtain a second initial corrected image having a barrel shape; The arc area in the second initial corrected image is filled with black pixels to obtain the second distortion-corrected image; wherein the second distortion-corrected image includes four arc black-edge areas formed after the second cropped image is subjected to distortion correction processing.

10. The method according to any one of claims 1 to 9, wherein: The step of cropping the first original image to obtain a first cropped image includes: The first original image is cropped to a full-size effective image using a specific cropping ratio to obtain the first cropped image.

11. The method according to any one of claims 1 to 10, wherein: The step of cropping the second original image to obtain a second cropped image includes: The second original image is cropped to a full-size effective image using a specific cropping ratio to obtain the second cropped image.

12. The method according to any one of claims 1 to 11, wherein: The performing distortion correction processing on the first cropped image to obtain a first distortion-corrected image includes: Acquire a first device internal parameter of the first image acquisition device; Obtaining a first distortion coefficient of the first image acquisition device from an internal parameter of the first device; The first cropped image is subjected to distortion correction processing using the first distortion coefficient to obtain the first distortion-corrected image.

13. The method according to any one of claims 1 to 12, wherein: The performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image includes: Acquire a second device internal parameter of the second image acquisition device; Obtaining a second distortion coefficient of the second image acquisition device from an internal parameter of the second device; The second cropped image is subjected to distortion correction processing using the second distortion coefficient to obtain the second distortion-corrected image.

14. The method according to any one of claims 1 to 13, wherein: The obtaining of a first device internal parameter of the first image acquisition device includes: Get the preset camera calibration standard value; Using a preset calibration fixture, the first image acquisition device is independently calibrated based on the camera calibration standard value to obtain a first device internal parameter of the first image acquisition device.

15. The method according to any one of claims 1 to 14, wherein: The obtaining of a second device internal parameter of the second image acquisition device includes: Get the preset camera calibration standard value; Using a preset calibration fixture, the second image acquisition device is independently calibrated based on the camera calibration standard value to obtain a second device internal parameter of the second image acquisition device.

16. The method according to any one of claims 1 to 15, wherein: The first type of image acquisition device is an infrared camera, and the second type of image acquisition device is a color camera; The first original image is an infrared image, and the second original image is a color image.

17. An image processing device, comprising: an acquisition module configured to acquire an original image pair, the original image pair comprising a first original image acquired by a first type of image acquisition device and a second original image acquired by a second type of image acquisition device; both the first original image and the second original image contain a target object; an image cropping module, configured to crop the first original image to obtain a first cropped image; performing image cropping on the second original image to obtain a second cropped image; a distortion correction module configured to perform distortion correction processing on the first cropped image to obtain a first distortion-corrected image; performing distortion correction processing on the second cropped image to obtain a second distortion-corrected image; The spatial alignment module is configured to perform distortion spatial alignment on the first distortion-corrected image and the second distortion-corrected image to obtain a spatially aligned image containing the target object.

18. An electronic device comprising: a memory for storing executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 16 when executing the executable instructions stored in the memory.

19. A computer-readable storage medium storing executable instructions, for causing a processor to execute the executable instructions to implement the image processing method according to any one of claims 1 to 16.

20. A computer program product comprising executable instructions stored in a computer-readable storage medium; When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the image processing method according to any one of claims 1 to 16 is implemented.

Images