Object recognition method and related device

Abstract

The present application discloses an object recognition method and related equipment, and the related embodiments can be applied to various scenarios such as cloud technology, artificial intelligence, and smart transportation; the reference pixel can be determined from each pixel of the infrared image, and the depth information of the reference pixel can be obtained; the depth information of each pixel in the infrared image can be obtained according to the position information and depth information of the reference pixel; the position mapping processing of each pixel in the infrared image is performed based on the depth information corresponding to each pixel, and the corresponding mapping point position information in the visible light image is obtained; the pixel points in the infrared image and the visible light image are aligned according to the mapping point position information; the object to be identified is identified based on the aligned infrared image and the visible light image to obtain the object recognition result. The present application can realize pixel alignment of the infrared image and the visible light image in the scene without the depth map, so as to perform object recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object recognition method and related equipment. Background Art

[0002] With the development of computer technology, image processing technology has been applied to more and more fields. For example, biometric recognition technology is widely used in many fields such as access control and attendance, information security and electronic certificates. Specifically, biometric recognition technology is a technology that automatically extracts biometric features from the image to be identified, and then authenticates the identity based on these features. In the process of biometric recognition, infrared images and visible light images of the object to be identified are usually collected. Among them, infrared images can be used for liveness detection of the object to be identified. Before liveness detection, it is generally necessary to perform pixel alignment processing on the collected infrared and visible light images.

[0003] In the current related technologies, the pixel alignment processing of infrared light images and visible light images is mainly achieved by relying on depth maps, but this method is not suitable for recognition scenarios that lack depth maps. Summary of the invention

[0004] An embodiment of the present application provides an object recognition method and related equipment. The related equipment may include an object recognition device, an electronic device, a computer-readable storage medium and a computer program product. In a scene where a depth map is missing, pixel alignment of infrared light images and visible light images can be achieved to perform object recognition.

[0005] The present application provides an object recognition method, including:

[0006] Collect infrared light images and visible light images of the object to be identified;

[0007] Determine a reference pixel from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified;

[0008] Acquire depth information of each pixel in the infrared image relative to the object to be identified according to the position information of the reference pixel in the infrared image and the depth information;

[0009] Based on the depth information corresponding to each pixel in the infrared light image, position mapping processing is performed on each pixel in the infrared light image to obtain mapping point position information corresponding to each pixel in the infrared light image in the visible light image;

[0010] Performing alignment processing on pixel points in the infrared light image and the visible light image according to the mapping point position information;

[0011] Based on the aligned infrared light image and the visible light image, object recognition is performed on the object to be recognized to obtain an object recognition result of the object to be recognized.

[0012] Accordingly, an embodiment of the present application provides an object recognition device, including:

[0013] A collection unit, used for collecting infrared light images and visible light images of the object to be identified;

[0014] A determination unit, configured to determine a reference pixel from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified;

[0015] an acquiring unit, configured to acquire depth information of each pixel in the infrared image relative to the object to be identified according to position information of the reference pixel in the infrared image and the depth information;

[0016] A mapping unit, configured to perform position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, so as to obtain mapping point position information corresponding to each pixel in the infrared light image in the visible light image;

[0017] an alignment unit, configured to align pixel points in the infrared light image and the visible light image according to the mapping point position information;

[0018] The recognition unit is used to perform object recognition on the object to be recognized based on the infrared light image and the visible light image after the alignment processing, and obtain the object recognition result of the object to be recognized.

[0019] Optionally, in some embodiments of the present application, the mapping unit may include a mapping subunit and a position transformation subunit, as follows:

[0020] The mapping subunit is used to map each pixel point in the infrared light image to the imaging space of the visible light image based on the depth information corresponding to each pixel point in the infrared light image, so as to obtain the spatial position information of the target mapping point corresponding to each pixel point in the infrared light image in the imaging space;

[0021] The position transformation subunit is used to perform position transformation processing on the spatial position information of the target mapping point according to the position transformation relationship between the imaging plane and the imaging space of the visible light image, so as to obtain the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image.

[0022] Optionally, in some embodiments of the present application, the mapping subunit can be specifically used to obtain the mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image based on the depth information corresponding to each pixel point in the infrared light image; perform position transformation processing on the mapping point spatial position information according to the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, and obtain the target mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the visible light image.

[0023] Optionally, in some embodiments of the present application, the step of “obtaining, based on the depth information corresponding to each pixel in the infrared image, spatial position information of a mapping point corresponding to each pixel in the infrared image in the imaging space of the infrared image” may include:

[0024] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of each pixel point in the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0025] According to the depth information corresponding to each pixel point in the infrared light image and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image is determined.

[0026] Optionally, in some embodiments of the present application, the acquisition unit may include a determination subunit, a construction subunit and an acquisition subunit, as follows:

[0027] The determining subunit is used to determine the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image according to the position information of the reference pixel point in the infrared light image and the depth information;

[0028] A construction subunit, used to construct a plane calibration equation corresponding to the object to be identified based on the spatial position information of the mapping point;

[0029] The acquisition subunit is used to acquire the depth information of each pixel in the infrared light image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared light image and the reference pixel.

[0030] Optionally, in some embodiments of the present application, the acquisition subunit can be specifically used to perform interpolation operations on the position information of each pixel point in the infrared light image and the position information of the reference pixel point to obtain the position relationship parameters between each pixel point in the infrared light image and the reference pixel point; based on the position relationship parameters and the plane calibration equation, determine the depth information of each pixel point in the infrared light image relative to the object to be identified.

[0031] Optionally, in some embodiments of the present application, the step of “determining, according to the position relationship parameter and the plane calibration equation, the depth information of each pixel in the infrared light image relative to the object to be identified” may include:

[0032] According to the position relationship parameters, interpolation operation is performed on the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0033] Determine the depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the initial position information of the mapping point.

[0034] Optionally, in some embodiments of the present application, the reference pixel points include at least three pixel points that are not on the same straight line.

[0035] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the object recognition method provided in the embodiment of the present application.

[0036] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the object recognition method provided in the embodiment of the present application.

[0037] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the object recognition method provided in the embodiment of the present application.

[0038] The embodiment of the present application provides an object recognition method and related equipment, which can collect infrared light images and visible light images of an object to be recognized; determine reference pixels from each pixel of the infrared light image, and obtain depth information of the reference pixels relative to the object to be recognized; obtain depth information of each pixel in the infrared light image relative to the object to be recognized based on the position information of the reference pixels in the infrared light image and the depth information; perform position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, and obtain mapping point position information corresponding to each pixel in the infrared light image in the visible light image; perform alignment processing on the pixels in the infrared light image and the visible light image based on the mapping point position information; perform object recognition on the object to be recognized based on the aligned infrared light image and visible light image, and obtain the object recognition result of the object to be recognized. The present application can realize pixel alignment of infrared light images and visible light images in a scene without a depth map, so as to perform object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1a is a schematic diagram of a scene of an object recognition method provided in an embodiment of the present application;

[0041] Figure 1b is a flow chart of an object recognition method provided by an embodiment of the present application;

[0042] Figure 1c is an illustration of an object recognition method provided in an embodiment of the present application;

[0043] Figure 1d is another illustration of the object recognition method provided by an embodiment of the present application;

[0044] Figure 1e is another illustration of the object recognition method provided by an embodiment of the present application;

[0045] Figure 1f is another illustration of the object recognition method provided by an embodiment of the present application;

[0046] Figure 1g is another illustration of the object recognition method provided by an embodiment of the present application;

[0047] Figure 1his another flow chart of the object recognition method provided by an embodiment of the present application;

[0048] Figure 2 is another flow chart of the object recognition method provided by an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the structure of an object recognition device provided in an embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0052] The present application provides an object recognition method and related equipment, which may include an object recognition device, an electronic device, a computer-readable storage medium, and a computer program product. The object recognition device may be integrated into an electronic device, which may be a terminal or a server.

[0053] It is understandable that the object recognition method of this embodiment can be executed on a terminal, can be executed on a server, or can be executed by both a terminal and a server. The above examples should not be construed as limiting the present application.

[0054] like Figure 1a As shown, the object recognition method is performed by a terminal and a server together as an example. The object recognition system provided in the embodiment of the present application includes a terminal 10 and a server 11, etc. The terminal 10 and the server 11 are connected via a network, such as a wired or wireless network connection, etc., wherein the object recognition device can be integrated in the terminal.

[0055] The terminal 10 can be used to: collect infrared images and visible light images of the object to be identified; determine reference pixels from each pixel of the infrared image, and obtain depth information of the reference pixels relative to the object to be identified; obtain depth information of each pixel in the infrared image relative to the object to be identified according to the position information of the reference pixels in the infrared image and the depth information; perform position mapping processing on each pixel in the infrared image based on the depth information corresponding to each pixel in the infrared image, and obtain mapping point position information corresponding to each pixel in the infrared image in the visible light image; perform alignment processing on the pixels in the infrared image and the visible light image according to the mapping point position information; send the aligned infrared image and visible light image to the server 11, so as to perform object recognition on the object to be identified through the server 11, and obtain the object recognition result of the object to be identified. The terminal 10 can include a mobile phone, a smart TV, a tablet computer, a laptop, or a personal computer (PC, Personal Computer), etc. A client can also be set on the terminal 10, and the client can be an application client or a browser client, etc.

[0056] The server 11 can be used to: receive the aligned infrared image and visible light image sent by the terminal 10, and perform object recognition on the object to be recognized based on the aligned infrared image and visible light image to obtain the object recognition result of the object to be recognized; and send the object recognition result to the terminal 10. The server 11 can be a single server, or a server cluster or cloud server composed of multiple servers. In the object recognition method or device disclosed in the present application, multiple servers can be composed of a blockchain, and the server is a node on the blockchain.

[0057] The step of object recognition performed by the server 11 may also be executed by the terminal 10 .

[0058] The object recognition method provided in the embodiment of the present application relates to computer vision technology in the field of artificial intelligence.

[0059] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Among them, artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0060] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0062] This embodiment will be described from the perspective of an object recognition device, which may be integrated into an electronic device, such as a server or a terminal.

[0063] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0064] The object recognition method of the embodiment of the present application can be applied to various scenarios where object recognition is required, such as palm payment, access control and attendance, etc. The present embodiment can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0065] like Figure 1b As shown, the specific process of the object recognition method can be as follows:

[0066] 101. Collect infrared light images and visible light images of the object to be identified.

[0067] The object to be identified may be an object that needs to be biometrically identified, which may be palm print information corresponding to a human body, or a face, etc. This embodiment does not impose any limitation on this.

[0068] The infrared image may be an infrared image of infrared light imaging acquired by an infrared sensor (Sensor); it may be used for liveness detection of the object to be identified. The visible light image may be a color image of natural light imaging acquired by a color sensor (Sensor); it may be used for identity recognition of the object to be identified. It should be noted that, in this embodiment, the infrared image and the visible light image of the object to be identified may be acquired at the same time, so that the infrared image and the visible light image may correspond to each other.

[0069] Specifically, in the process of biometric recognition, it is generally necessary to obtain a depth image of the object to be identified; the depth image can be specifically a depth map obtained by collecting speckle structure infrared light by an infrared sensor and then parsing the speckle by a depth unit. In three-dimensional (3D) computer graphics and computer vision, a depth map is an image or image channel that contains information about the distance from the surface of the collected scene object to the viewpoint. Each pixel of the depth map represents the vertical distance between the plane of the depth camera and the plane of the photographed object, usually represented by 16 bits, in millimeters. In object recognition scenarios, depth images can generally be used for: liveness detection and auxiliary identity recognition; using depth images to assist in identity recognition can greatly improve the accuracy and robustness of identity recognition.

[0070] Among them, liveness detection is a detection method to determine whether the object to be identified is a real person, a photo, or a head model. Generally, the depth image can be used to determine whether the object to be identified is a photo, and the brightness of the infrared light image can be used to determine whether the object to be identified is a silicone head model.

[0071] Among them, speckle structured light is a dotted light arranged in a certain structural pattern projected by an infrared speckle projector. The speckle structured light imaging system consists of an infrared laser projector and an infrared sensor, such as Figure 1cAs shown, these dot-matrix lights are projected onto the surface of an object. After imaging, the infrared sensor can restore the 3D coordinate information of the object's surface based on the principle of triangulation, thereby obtaining a depth image.

[0072] Compared with depth images, point cloud images record the 3D coordinate information of objects in the real world, which can be calculated from depth images and camera parameters. Although depth images also contain the depth information of objects, the horizontal and vertical coordinates of a point in the depth image are pixel coordinates in the imaging plane, while the point cloud image records the physical coordinates of the point in the real world. Point cloud images can be displayed in a 3D (three-dimensional) rendering engine to reflect the 3D position relationship of each point.

[0073] In the current related technologies, RGB-D (RedGreenBlue-Depth) data is usually used for identity recognition. RGB is a color model, which is a color standard in the industry. By adjusting the three color channels of red, green, and blue and superimposing the three colors, a variety of colors can be obtained. RGB is the color representing the three channels of red, green, and blue. Among them, D is the abbreviation of Depth Map. In three-dimensional computer graphics, Depth Map is an image or image channel containing information about the distance of the surface of the scene object from the viewpoint. Depth Map is similar to a grayscale image, except that the pixel value of each pixel is the actual distance from the sensor to the object. Usually, RGB images (i.e., visible light images) and Depth images are registered, so there is a one-to-one correspondence between the pixels of the RGB image and the pixels of the Depth image.

[0074] However, in some identity recognition scenarios, only visible light images and infrared images can be collected, and depth images are missing. In this way, it is impossible to rely on depth images to achieve pixel-level alignment of the three images, where pixel-level alignment of the three images refers to a one-to-one correspondence between pixels of the visible light image, infrared image, and depth image. The present application can provide an object recognition method that can achieve pixel-level alignment of visible light images and infrared images without relying on depth images.

[0075] Specifically, the object recognition method provided by the present application can be applied to the field of palmprint recognition, which is a biometric recognition technology. Through palmprint recognition, the palmprint information of the object to be recognized can be extracted. The palmprint information can specifically refer to the palm image information from the end of the finger to the wrist. The palmprint features of different objects are different, so identity authentication can be performed based on the palmprint information.

[0076] Among them, biometric recognition technology is a technology that uses the inherent physiological characteristics of the human body (such as fingerprints, faces, irises, etc.) and behavioral characteristics (such as handwriting, voice, gait, etc.) to identify individuals through a close combination of computers with high-tech means such as optics, acoustics, biosensors and biostatistics principles.

[0077] 102. Determine a reference pixel from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified.

[0078] In some embodiments, three pixel points that are not on the same straight line can be arbitrarily selected from the pixel points of the infrared light image as reference pixel points.

[0079] Optionally, in this embodiment, the reference pixel points include at least three pixel points that are not on the same straight line.

[0080] In a specific scenario, the depth information of the reference pixel relative to the object to be identified can be obtained by using the component Psensor (Proximity Sensor) to obtain the depth information corresponding to each reference pixel. Specifically, the number of reference pixels can be determined first, and the number of Psensors can be determined based on the number of reference pixels. Each Psensor can only obtain the depth information of one reference pixel. The principle of Psensor measuring depth information can refer to dtof. dtof (direct Time-of-Flight) directly measures the distance based on the time difference between pulse emission and reception, and Psensor can calculate the distance between the imaging plane and the object surface of the object to be identified based on the round-trip time of the emitted light between the imaging plane and the object surface and the propagation speed of light. This distance is the depth information.

[0081] For example, when there is no depth camera, if the object to be identified is planar (such as a palm), a cheap psensor can be used as an alternative to derive the plane depth value of the object to be identified. Three psensors can be installed at positions that are not in the same straight line in the imaging plane of the infrared light image, so that the Psensor can be used to obtain the depth information corresponding to three or more reference pixels.

[0082] 103. Obtain depth information of each pixel in the infrared image relative to the object to be identified based on the position information of the reference pixel in the infrared image and the depth information.

[0083] Specifically, this embodiment can obtain the imaging plane pixel depth of the infrared image according to the position information and depth information of the reference pixel point, that is, the depth information of each pixel point in the infrared image relative to the object to be identified.

[0084] Optionally, in this embodiment, the step of “obtaining depth information of each pixel in the infrared image relative to the object to be identified according to the position information of the reference pixel in the infrared image and the depth information” may include:

[0085] Determine, according to the position information of the reference pixel point in the infrared light image and the depth information, the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image;

[0086] Based on the spatial position information of the mapping points, constructing a plane calibration equation corresponding to the object to be identified;

[0087] According to the plane calibration equation and the positional relationship between each pixel in the infrared light image and the reference pixel, the depth information of each pixel in the infrared light image relative to the object to be identified is obtained.

[0088] Specifically, the step of “determining the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image according to the position information of the reference pixel point in the infrared light image and the depth information” may include:

[0089] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of the reference pixel point in the infrared light image to obtain the initial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image;

[0090] According to the depth information corresponding to the reference pixel point and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image is determined.

[0091] Among them, the position transformation relationship between the imaging plane of the infrared light image and the imaging space can be specifically characterized by the camera intrinsic parameters of the camera sensor corresponding to the infrared light image. In this embodiment, a pixel coordinate system corresponding to the infrared image can be constructed on the imaging plane of the infrared light image (also called pixel plane), and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0092] The camera intrinsic parameters are used to describe the conversion relationship between the three-dimensional spatial coordinates of real-world objects when they are imaged on the camera sensor and the two-dimensional pixel coordinates in the physical imaging plane after imaging. This conversion relationship can be expressed as follows: Figure 1d The simplified camera model shown in the figure indicates that the camera model is specifically a pinhole imaging model. The depth image and the point cloud image in the above embodiment can also be converted to each other through the camera intrinsic parameters.

[0093] The camera coordinate system refers to a coordinate system with the camera optical center O as the origin, the optical axis as the z-axis, and the x and y axes parallel to the x and y axes of the imaging plane. The 3D coordinates of the point cloud obtained by transforming the depth image and the camera intrinsic parameters are the coordinates in the camera coordinate system.

[0094] Specifically, according to the position transformation relationship between the imaging plane and the imaging space of the infrared image, the camera intrinsic parameter of the camera sensor corresponding to the infrared image can be obtained, and then according to the camera intrinsic parameter, the position information of the reference pixel point in the infrared image can be transformed. The processing process can be shown as formula (1):

[0095]

[0096] Among them, f x 、f y , u0 and v0 are the camera internal parameters. Specifically, f x represents the focal length in the x-axis direction, f y represents the focal length in the y-axis direction, u0 and v0 are the main point positions. The position information of the reference pixel in the infrared image can be expressed as (u, v), and the depth information corresponding to the reference pixel is z c , through formula (1), we can obtain the initial position information x of the mapping point corresponding to the reference pixel in the imaging space of the infrared image c ,y c , where x c ,y c That is, the x-axis coordinate and the y-axis coordinate of the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image.

[0097] After getting the initial position information x of the mapping point c ,y c After that, the depth information z corresponding to the reference pixel can be c , construct the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared image (x c ,y c , z c ).

[0098] The step of “constructing a plane calibration equation corresponding to the object to be identified based on the spatial position information of the mapping points” may include:

[0099] Setting an initial plane calibration equation corresponding to the object to be identified, wherein the initial plane calibration equation includes at least one calibration parameter;

[0100] Based on the spatial position information of the mapping point, the at least one calibration parameter is parsed to obtain a parsed calibration parameter;

[0101] The initial plane calibration equation is updated according to the analyzed calibration parameters to obtain the plane calibration equation corresponding to the object to be identified.

[0102] The initial plane calibration equation can be set to AX+BY+CZ+D=0, where A, B, C and D are calibration parameters. In this embodiment, the spatial position information of the mapping point can be substituted into the initial plane calibration equation, so as to analyze the calibration parameters and obtain the specific values ​​of A, B, C and D, that is, the analyzed calibration parameters, thereby obtaining the plane calibration equation of the object to be identified.

[0103] In a specific embodiment, the position information of the three reference pixels m, n, and p in the infrared image can be recorded as m=(x1, y1)n=(x2, y2)p=(x3, y3). According to formula (1) in the above embodiment, the spatial position information of the mapping points corresponding to the three reference pixels in the imaging space of the infrared image can be calculated and recorded as M=(X1, Y1, Z1)N=(X2, Y2, Z2)P=(X3, Y3, Z3). Then, M=(X1, Y1, Z1)N=(X2, Y2, Z2)P=(X3, Y3, Z3) is substituted into formula (2), as follows:

[0104] AX+BY+CZ+D=0 (2)

[0105] By solving the coefficients ABCD in the above formula, the plane calibration equation of the object to be identified can be obtained.

[0106] Optionally, in this embodiment, the step of “obtaining depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared image and the reference pixel” may include:

[0107] Performing interpolation processing on the position information of each pixel in the infrared light image and the position information of the reference pixel to obtain position relationship parameters between each pixel in the infrared light image and the reference pixel;

[0108] Determine the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation.

[0109] In a specific embodiment, the position information of three reference pixels m, n, and p in the infrared image can be recorded as m=(x1, y1)n=(x2, y2)p=(x3, y3). When it is necessary to obtain the depth information of another pixel k in the infrared image, the position relationship parameters between each pixel in the infrared image and the reference pixel can be calculated by interpolation of the three points m, n, and p. The position information of the pixel k in the infrared image is (x4, y4). The interpolation process can be shown as formula (3) and formula (4):

[0110] x4=x1+α*(x2-x1)+β*(x3-x1) (3)

[0111] y4=y1+α*(y2-y1)+β*(y3-y1) (4)

[0112] The coordinates of the m, n, p, and k pixel points can be substituted into equations (3) and (4) to obtain the positional relationship parameters α and β between each pixel point in the infrared image and the reference pixel point.

[0113] Optionally, in this embodiment, the step of “determining the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation” may include:

[0114] According to the position relationship parameters, interpolation operation is performed on the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0115] Determine the depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the initial position information of the mapping point.

[0116] Specifically, in one embodiment, the mapping point spatial position information corresponding to the three reference pixel points in the imaging space of the infrared light image is M=(X1, Y1, Z1)N=(X2, Y2, Z2)P=(X3, Y3, Z3), respectively. The x-axis coordinates and y-axis coordinates in the three mapping point spatial position information can be substituted into the above equations (3) and (4) to obtain the initial mapping point position information X4 and Y4 corresponding to the pixel point k in the imaging space of the infrared light image, wherein X4 and Y4 are specifically the x-axis coordinates and y-axis coordinates of the mapping point spatial position information corresponding to the pixel point k in the imaging space of the infrared light image.

[0117] After obtaining the initial position information X4 and Y4 of the mapping point, it can be substituted into the formula (2) in the above embodiment to obtain the spatial position information (X4, Y4, Z4) of the mapping point corresponding to the pixel point k in the imaging space of the infrared image, that is, the depth information Z4 of the pixel point k relative to the object to be identified. By analogy, the depth information corresponding to each pixel point in the infrared image can be obtained.

[0118] 104. Perform position mapping processing on each pixel in the infrared image based on the depth information corresponding to each pixel in the infrared image to obtain mapping point position information corresponding to each pixel in the infrared image in the visible light image.

[0119] Optionally, in this embodiment, the step of “performing position mapping processing on each pixel in the infrared image based on the depth information corresponding to each pixel in the infrared image to obtain mapping point position information corresponding to each pixel in the infrared image in the visible light image” may include:

[0120] Based on the depth information corresponding to each pixel in the infrared image, each pixel in the infrared image is mapped to the imaging space of the visible light image to obtain the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space;

[0121] According to the position transformation relationship between the imaging plane and the imaging space of the visible light image, the spatial position information of the target mapping point is subjected to position transformation processing to obtain the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image.

[0122] Among them, the position transformation relationship between the imaging plane of the visible light image and the imaging space can be specifically characterized by the camera intrinsic parameters of the camera sensor corresponding to the visible light image. In this embodiment, a pixel coordinate system corresponding to the color image can be constructed on the imaging plane of the visible light image (also called pixel plane), and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the color image can be constructed in the imaging space of the visible light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0123] Among them, the camera intrinsic parameters are parameters used to describe the conversion relationship between the three-dimensional spatial coordinates of a real-world object when it is imaged on a camera sensor and the two-dimensional pixel coordinates in the physical imaging plane after imaging.

[0124] In a specific embodiment, the camera intrinsic parameters can be used to transform the target mapping point in the camera coordinate system corresponding to the color image into the pixel coordinate system. The transformation between the camera coordinate system and the pixel coordinate system can be referred to Figure 1e , where point P (X C , YC , Z C ) is a point in the camera coordinate system, and point p (x, y) is a point in the pixel coordinate system. The coordinate transformation relationship between the two can be converted through the camera intrinsic parameters.

[0125] Specifically, the camera intrinsic parameters of the camera sensor corresponding to the visible light image can be obtained according to the position transformation relationship between the imaging plane of the visible light image and the imaging space, and then the spatial position information of the target mapping point can be transformed according to the camera intrinsic parameters. The processing process can be shown as formula (5):

[0126]

[0127] Among them, f x 、f y , u0 and v0 are the camera internal parameters. Specifically, f x represents the focal length in the x-axis direction, f y represents the focal length in the y-axis direction, u0 and v0 are the main point positions. The spatial position information of the target mapping point can be expressed as (x c ,y c , z c ), through formula (5), we can obtain the mapping point position information (u, v) corresponding to each pixel point in the infrared image in the visible light image, that is, the mapping point position information in the pixel coordinate system corresponding to the color image.

[0128] Optionally, in this embodiment, the step of “mapping each pixel in the infrared image to the imaging space of the visible light image based on the depth information corresponding to each pixel in the infrared image to obtain the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space” may include:

[0129] Based on the depth information corresponding to each pixel point in the infrared light image, obtaining the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0130] According to the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, the mapping point spatial position information is subjected to position transformation processing to obtain the target mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the visible light image.

[0131] Optionally, in some embodiments, the step of "obtaining the spatial position information of the mapping points corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image" can be specifically based on the depth information corresponding to each pixel point in the infrared light image obtained in step 103, and the initial position information of the mapping points corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image, to construct the spatial position information of the mapping points corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image.

[0132] Among them, the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image can be specifically characterized by the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image. In this embodiment, a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and a camera coordinate system corresponding to the color image can be constructed in the imaging space of the visible light image. Both camera coordinate systems are three-dimensional coordinate systems.

[0133] Among them, the camera extrinsic parameters can be used to describe the parameters of the conversion relationship between other three-dimensional coordinate systems and the camera coordinate system. When there are multiple cameras, the coordinates of the object point in the camera coordinate system of one camera can be converted to the coordinate system of another camera through the rotation matrix R and the translation matrix t. The rotation matrix R and the translation matrix (offset vector) t are the extrinsic parameters between the two cameras. This camera extrinsic parameter describes the conversion relationship between the two camera coordinate systems, such as Figure 1f As shown in the figure, there is an offset in angle and distance between the two cameras, which is similar to the relationship between the world coordinate system and the camera coordinate system. The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation: that is, the object will not be deformed, so only rotation and translation are required.

[0134] In a specific embodiment, the camera extrinsics between the camera sensor corresponding to the infrared image and the camera sensor corresponding to the visible light image can be used to transform the spatial position information of the mapping point in the camera coordinate system corresponding to the infrared image into the camera coordinate system corresponding to the color image. The transformation between the camera coordinate system corresponding to the infrared image and the camera coordinate system corresponding to the color image can be referred to in Figure 1g , where point P (X w , Y w , Z w ) is the point in the camera coordinate system corresponding to the infrared image, which can be converted into the point in the camera coordinate system corresponding to the color image using the camera extrinsic parameters.

[0135] Specifically, according to the position transformation relationship between the imaging space of the infrared image and the imaging space of the visible light image, the camera extrinsic parameters between the camera sensor corresponding to the infrared image and the camera sensor corresponding to the visible light image can be obtained, and then the spatial position information of the mapping point can be transformed according to the camera extrinsic parameters. The processing process is shown in formula (6):

[0136]

[0137] Among them, in formula (6), r 00 to r 22 The matrix composed is the rotation matrix R, t x to z The matrix composed of the offset vector T, the rotation matrix R and the offset vector T are the camera external parameters, and the spatial position information of the mapping point can be expressed as (x w ,y w , z w ), through formula (6), we can obtain the spatial position information (x) of the target mapping point corresponding to each pixel in the infrared image in the imaging space of the visible light image. c ,y c , z c ), that is, the spatial position information of the target mapping point in the camera coordinate system corresponding to the color image.

[0138] Optionally, in this embodiment, the step of “obtaining spatial position information of a mapping point corresponding to each pixel in the infrared image in the imaging space of the infrared image based on the depth information corresponding to each pixel in the infrared image” may include:

[0139] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of each pixel point in the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0140] According to the depth information corresponding to each pixel point in the infrared light image and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image is determined.

[0141] Among them, the position transformation relationship between the imaging plane of the infrared light image and the imaging space can be specifically characterized by the camera intrinsic parameters of the camera sensor corresponding to the infrared light image. In this embodiment, a pixel coordinate system corresponding to the infrared image can be constructed on the imaging plane of the infrared light image (also called pixel plane), and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0142] In a specific embodiment, the camera intrinsic parameters can be used to convert the position information of the pixel points in the pixel coordinate system corresponding to the infrared image into the camera coordinate system. For example, point P is a point in the camera coordinate system, and point p is a point in the pixel coordinate system. The coordinate transformation relationship between the two can be converted using the camera intrinsic parameters.

[0143] Specifically, the camera intrinsic parameters of the camera sensor corresponding to the infrared image can be obtained according to the position transformation relationship between the imaging plane and the imaging space of the infrared image, and then the position information of each pixel in the infrared image can be transformed according to the camera intrinsic parameters. The processing process can be shown as formula (7):

[0144]

[0145] Among them, f x 、f y , u0 and v0 are the camera internal parameters. Specifically, f x represents the focal length in the x-axis direction, f y represents the focal length in the y-axis direction, u0 and v0 are the main point positions. The position information of each pixel in the infrared image can be expressed as (u, v), and the depth information corresponding to each pixel in the infrared image is z c , through formula (1), we can obtain the initial position information x of the mapping point corresponding to each pixel in the infrared image in the imaging space of the infrared image c ,y c , where x c ,y c That is, the x-axis coordinate and the y-axis coordinate of the spatial position information of the mapping point corresponding to each pixel point in the infrared image in the imaging space of the infrared image.

[0146] After getting the initial position information x of the mapping point c ,y c Then, based on the depth information z corresponding to each pixel in the infrared image, c , construct the spatial position information (x) of the mapping points corresponding to each pixel in the infrared image in the imaging space of the infrared image c ,y c , z c ).

[0147] 105. Perform alignment processing on pixel points in the infrared light image and the visible light image according to the mapping point position information.

[0148] Specifically, the visible light image can be used as a reference, and based on the mapping point position information, each pixel in the infrared light image can be aligned with the corresponding pixel in the visible light image, that is, the pixel points in the infrared light image are aligned with the pixel points representing the same target object in the visible light image, so that after alignment, the infrared light image corresponds to the position of the pixel points where the same target object is located in the visible light image.

[0149] In the specific scenario of palmprint recognition, since the palm (specifically, the object to be identified) can be regarded as a plane, this embodiment can use PSensor to calculate the slope distance corresponding to the palm (specifically, it can be the depth information of each pixel point in the infrared light image relative to the object to be identified in the above embodiment), and then based on the slope distance, the infrared light image and the visible light image are pixel-aligned, thereby realizing pixel-level alignment of the two images at a low cost.

[0150] It should be noted that the object recognition method provided by the present application can also determine the reference pixel from each pixel of the visible light image, and obtain the depth information of the reference pixel relative to the object to be recognized; then, according to the position information and depth information of the reference pixel in the visible light image, obtain the depth information of each pixel in the visible light image relative to the object to be recognized; then, based on the depth information corresponding to each pixel in the visible light image, perform position mapping processing on each pixel in the visible light image to obtain the mapping point position information corresponding to each pixel in the visible light image in the infrared light image, and finally, based on the mapping point position information, align the pixels in the infrared light image and the visible light image, and then perform object recognition. It can be understood that the acquisition of the depth information corresponding to the pixel and the process of position mapping processing can be referred to the above embodiment, and will not be repeated here.

[0151] In a specific embodiment, if Figure 1h As shown, it is a flow chart for pixel alignment of infrared light images and visible light images; specifically, based on the depth information of the reference pixel in the infrared light image, the depth information corresponding to each pixel in the infrared light image can be obtained; then, the camera intrinsic parameters of the camera sensor corresponding to the infrared light image are used to convert the position information of the pixel in the pixel coordinate system corresponding to the infrared image into the camera coordinate system of the infrared image; then, the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image are used to convert the spatial position information of the mapping point in the camera coordinate system corresponding to the infrared image into the camera coordinate system corresponding to the color image; finally, the camera intrinsic parameters of the camera sensor corresponding to the visible light image are used to convert the target mapping point in the camera coordinate system corresponding to the color image into the pixel coordinate system corresponding to the color image, so as to obtain the mapping result, thereby performing pixel alignment of the visible light image and the infrared light image based on the mapping result.

[0152] 106. Perform object recognition on the object to be recognized based on the aligned infrared light image and the visible light image to obtain an object recognition result of the object to be recognized.

[0153] After the alignment process, the aligned infrared image can be subjected to liveness detection. After the liveness detection is passed, object recognition can be performed based on the aligned visible light image. Object recognition can specifically be a technology that exchanges biometric multimedia information for object identity information. In some embodiments, relevant payment operations can also be performed based on the object recognition result.

[0154] In a specific scenario, such as a palmprint recognition scenario, object recognition is performed on an object to be recognized, and specifically, palmprint information of the object to be recognized may be extracted. After the palmprint information is obtained, it may be matched with palmprint feature information stored in a preset palmprint information database, thereby determining the object identity of the object to be recognized. The preset palmprint information database may store a target mapping relationship set, which includes a mapping relationship between preset palmprint feature information and preset object identity; for example, the target mapping relationship set may be a relationship table between preset palmprint feature information and preset object identity.

[0155] In some embodiments, the matching degree between the palmprint information and each preset palmprint feature information can be calculated, and the target palmprint feature information can be determined from the preset palmprint feature information based on the matching degree. For example, the preset palmprint feature information with the highest matching degree with the palmprint information can be determined as the target palmprint feature information; and then, based on the target mapping relationship set, the object identity corresponding to the target palmprint feature information is determined as the object identity of the object to be identified.

[0156] As can be seen from the above, this embodiment can collect infrared light images and visible light images for the object to be identified; determine reference pixels from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified; obtain depth information of each pixel in the infrared light image relative to the object to be identified based on the position information of the reference pixel in the infrared light image and the depth information; perform position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, and obtain mapping point position information corresponding to each pixel in the infrared light image in the visible light image; perform alignment processing on the pixels in the infrared light image and the visible light image based on the mapping point position information; perform object recognition on the object to be identified based on the aligned infrared light image and visible light image, and obtain the object recognition result of the object to be identified. This application can realize pixel alignment of infrared light images and visible light images in a scene where a depth map is missing, so as to perform object recognition.

[0157] According to the method described in the previous embodiment, the object recognition device will be further described in detail below by taking the example of being specifically integrated in a terminal.

[0158] The present application embodiment provides an object recognition method, such as Figure 2 As shown, the specific process of the object recognition method can be as follows:

[0159] 201. The terminal collects infrared light images and visible light images of the object to be identified.

[0160] The object to be identified may be an object that needs to be biometrically identified, which may be palm print information corresponding to a human body, or a face, etc. This embodiment does not impose any limitation on this.

[0161] The infrared image may be an infrared image of infrared light imaging acquired by an infrared sensor (Sensor); it may be used for liveness detection of the object to be identified. The visible light image may be a color image of natural light imaging acquired by a color sensor (Sensor); it may be used for identity recognition of the object to be identified. It should be noted that, in this embodiment, the infrared image and the visible light image of the object to be identified may be acquired at the same time, so that the infrared image and the visible light image may correspond to each other.

[0162] 202. The terminal determines a reference pixel from each pixel of the infrared light image, and obtains depth information of the reference pixel relative to the object to be identified.

[0163] In some embodiments, three pixel points that are not on the same straight line can be arbitrarily selected from the pixel points of the infrared light image as reference pixel points.

[0164] Optionally, in this embodiment, the reference pixel points include at least three pixel points that are not on the same straight line.

[0165] In a specific scenario, the depth information of the reference pixel relative to the object to be identified can be obtained by using the component Psensor (Proximity Sensor) to obtain the depth information corresponding to each reference pixel. Specifically, the number of reference pixels can be determined first, and the number of Psensors can be determined based on the number of reference pixels. Each Psensor can only obtain the depth information of one reference pixel. The principle of Psensor measuring depth information can refer to dtof. dtof (direct Time-of-Flight) directly measures the distance based on the time difference between pulse emission and reception, and Psensor can calculate the distance between the imaging plane and the object surface of the object to be identified based on the round-trip time of the emitted light between the imaging plane and the object surface and the propagation speed of light. This distance is the depth information.

[0166] For example, when there is no depth camera, if the object to be identified is planar (such as a palm), a cheap psensor can be used as an alternative to derive the plane depth value of the object to be identified. Three psensors can be installed at positions that are not in the same straight line in the imaging plane of the infrared light image, so that the Psensor can be used to obtain the depth information corresponding to three or more reference pixels.

[0167] 203. The terminal obtains depth information of each pixel in the infrared image relative to the object to be identified based on the position information of the reference pixel in the infrared image and the depth information.

[0168] Optionally, in this embodiment, the step of “obtaining depth information of each pixel in the infrared image relative to the object to be identified according to the position information of the reference pixel in the infrared image and the depth information” may include:

[0169] Determine, according to the position information of the reference pixel point in the infrared light image and the depth information, the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image;

[0170] Based on the spatial position information of the mapping points, constructing a plane calibration equation corresponding to the object to be identified;

[0171] According to the plane calibration equation and the positional relationship between each pixel in the infrared light image and the reference pixel, the depth information of each pixel in the infrared light image relative to the object to be identified is obtained.

[0172] Specifically, the step of “determining the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image according to the position information of the reference pixel point in the infrared light image and the depth information” may include:

[0173] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of the reference pixel point in the infrared light image to obtain the initial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image;

[0174] According to the depth information corresponding to the reference pixel point and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image is determined.

[0175] Among them, the position transformation relationship between the imaging plane of the infrared light image and the imaging space can be specifically characterized by the camera intrinsic parameters of the camera sensor corresponding to the infrared light image. In this embodiment, a pixel coordinate system corresponding to the infrared image can be constructed on the imaging plane of the infrared light image (also called pixel plane), and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0176] The step of “constructing a plane calibration equation corresponding to the object to be identified based on the spatial position information of the mapping points” may include:

[0177] Setting an initial plane calibration equation corresponding to the object to be identified, wherein the initial plane calibration equation includes at least one calibration parameter;

[0178] Based on the spatial position information of the mapping point, the at least one calibration parameter is parsed to obtain a parsed calibration parameter;

[0179] The initial plane calibration equation is updated according to the analyzed calibration parameters to obtain the plane calibration equation corresponding to the object to be identified.

[0180] The initial plane calibration equation can be set to AX+BY+CZ+D=0, where A, B, C and D are calibration parameters. In this embodiment, the spatial position information of the mapping point can be substituted into the initial plane calibration equation, so as to analyze the calibration parameters and obtain the specific values ​​of A, B, C and D, that is, the analyzed calibration parameters, thereby obtaining the plane calibration equation of the object to be identified.

[0181] Optionally, in this embodiment, the step of “obtaining depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared image and the reference pixel” may include:

[0182] Performing interpolation processing on the position information of each pixel in the infrared light image and the position information of the reference pixel to obtain position relationship parameters between each pixel in the infrared light image and the reference pixel;

[0183] Determine the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation.

[0184] Optionally, in this embodiment, the step of “determining the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation” may include:

[0185] According to the position relationship parameters, interpolation operation is performed on the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0186] Determine the depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the initial position information of the mapping point.

[0187] 204. The terminal maps each pixel in the infrared image to the imaging space of the visible light image based on the depth information corresponding to each pixel in the infrared image, and obtains the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space.

[0188] Optionally, in this embodiment, the step of “mapping each pixel in the infrared image to the imaging space of the visible light image based on the depth information corresponding to each pixel in the infrared image to obtain the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space” may include:

[0189] Based on the depth information corresponding to each pixel point in the infrared light image, obtaining the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0190] According to the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, the mapping point spatial position information is subjected to position transformation processing to obtain the target mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the visible light image.

[0191] Among them, the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image can be specifically characterized by the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image. In this embodiment, a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and a camera coordinate system corresponding to the color image can be constructed in the imaging space of the visible light image. Both camera coordinate systems are three-dimensional coordinate systems.

[0192] Optionally, in this embodiment, the step of “obtaining spatial position information of a mapping point corresponding to each pixel in the infrared image in the imaging space of the infrared image based on the depth information corresponding to each pixel in the infrared image” may include:

[0193] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of each pixel point in the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0194] According to the depth information corresponding to each pixel point in the infrared light image and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image is determined.

[0195] Among them, the position transformation relationship between the imaging plane of the infrared light image and the imaging space can be specifically characterized by the camera intrinsic parameters of the camera sensor corresponding to the infrared light image. In this embodiment, a pixel coordinate system corresponding to the infrared image can be constructed on the imaging plane of the infrared light image (also called pixel plane), and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared image can be constructed in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0196] 205. The terminal performs position transformation processing on the spatial position information of the target mapping point according to the position transformation relationship between the imaging plane and the imaging space of the visible light image, and obtains the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image.

[0197] 206. The terminal aligns the pixels in the infrared light image and the visible light image according to the mapping point position information.

[0198] Specifically, the visible light image can be used as a reference, and based on the mapping point position information, each pixel in the infrared light image can be aligned with the corresponding pixel in the visible light image, that is, the pixel points in the infrared light image are aligned with the pixel points representing the same target object in the visible light image, so that after alignment, the infrared light image corresponds to the position of the pixel points where the same target object is located in the visible light image.

[0199] 207. The terminal performs object recognition on the object to be recognized based on the aligned infrared light image and the visible light image to obtain an object recognition result of the object to be recognized.

[0200] After the alignment process, the aligned infrared image can be subjected to liveness detection. After the liveness detection is passed, object recognition can be performed based on the aligned visible light image. Object recognition can specifically be a technology that exchanges biometric multimedia information for object identity information. In some embodiments, relevant payment operations can also be performed based on the object recognition result.

[0201] As can be seen from the above, in this embodiment, the terminal can collect infrared images and visible light images of the object to be identified; determine reference pixels from each pixel of the infrared image, and obtain depth information of the reference pixels relative to the object to be identified; obtain depth information of each pixel in the infrared image relative to the object to be identified based on the position information of the reference pixel in the infrared image and the depth information; based on the depth information corresponding to each pixel in the infrared image, map each pixel in the infrared image to the imaging space of the visible light image, and obtain the target mapping point spatial position information corresponding to each pixel in the infrared image in the imaging space; according to the position transformation relationship between the imaging plane of the visible light image and the imaging space, perform position transformation processing on the target mapping point spatial position information, and obtain the mapping point position information corresponding to each pixel in the infrared image in the visible light image; according to the mapping point position information, align the pixels in the infrared image and the visible light image; based on the aligned infrared image and the visible light image, perform object recognition on the object to be identified, and obtain the object recognition result of the object to be identified. The present application can realize pixel alignment of infrared light images and visible light images in scenarios where a depth map is lacking, thereby performing object recognition.

[0202] In order to better implement the above method, the present application embodiment also provides an object recognition device, such as Figure 3 As shown, the object recognition device may include a collection unit 301, a determination unit 302, an acquisition unit 303, a mapping unit 304, an alignment unit 305, and a recognition unit 306, as follows:

[0203] (1) Collection unit 301;

[0204] The acquisition unit is used to acquire infrared light images and visible light images of the object to be identified.

[0205] (2) determining unit 302;

[0206] The determination unit is used to determine a reference pixel point from each pixel point of the infrared light image, and obtain depth information of the reference pixel point relative to the object to be identified.

[0207] Optionally, in some embodiments of the present application, the reference pixel points include at least three pixel points that are not on the same straight line.

[0208] (3) Acquisition unit 303;

[0209] An acquisition unit is used to acquire the depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information of the reference pixel point in the infrared light image and the depth information.

[0210] Optionally, in some embodiments of the present application, the acquisition unit may include a determination subunit, a construction subunit and an acquisition subunit, as follows:

[0211] The determining subunit is used to determine the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image according to the position information of the reference pixel point in the infrared light image and the depth information;

[0212] A construction subunit, used to construct a plane calibration equation corresponding to the object to be identified based on the spatial position information of the mapping point;

[0213] The acquisition subunit is used to acquire the depth information of each pixel in the infrared light image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared light image and the reference pixel.

[0214] Optionally, in some embodiments of the present application, the acquisition subunit can be specifically used to perform interpolation operations on the position information of each pixel point in the infrared light image and the position information of the reference pixel point to obtain the position relationship parameters between each pixel point in the infrared light image and the reference pixel point; based on the position relationship parameters and the plane calibration equation, determine the depth information of each pixel point in the infrared light image relative to the object to be identified.

[0215] Optionally, in some embodiments of the present application, the step of “determining, according to the position relationship parameter and the plane calibration equation, the depth information of each pixel in the infrared light image relative to the object to be identified” may include:

[0216] According to the position relationship parameters, interpolation operation is performed on the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0217] Determine the depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the initial position information of the mapping point.

[0218] (4) mapping unit 304;

[0219] A mapping unit is used to perform position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, so as to obtain the mapping point position information corresponding to each pixel in the infrared light image in the visible light image.

[0220] Optionally, in some embodiments of the present application, the mapping unit may include a mapping subunit and a position transformation subunit, as follows:

[0221] The mapping subunit is used to map each pixel point in the infrared light image to the imaging space of the visible light image based on the depth information corresponding to each pixel point in the infrared light image, so as to obtain the spatial position information of the target mapping point corresponding to each pixel point in the infrared light image in the imaging space;

[0222] The position transformation subunit is used to perform position transformation processing on the spatial position information of the target mapping point according to the position transformation relationship between the imaging plane and the imaging space of the visible light image, so as to obtain the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image.

[0223] Optionally, in some embodiments of the present application, the mapping subunit can be specifically used to obtain the mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image based on the depth information corresponding to each pixel point in the infrared light image; perform position transformation processing on the mapping point spatial position information according to the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, and obtain the target mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the visible light image.

[0224] Optionally, in some embodiments of the present application, the step of “obtaining, based on the depth information corresponding to each pixel in the infrared image, spatial position information of a mapping point corresponding to each pixel in the infrared image in the imaging space of the infrared image” may include:

[0225] According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of each pixel point in the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image;

[0226] According to the depth information corresponding to each pixel point in the infrared light image and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image is determined.

[0227] (5) Alignment unit 305;

[0228] An alignment unit is used to align the pixel points in the infrared light image and the visible light image according to the mapping point position information.

[0229] (6) identification unit 306;

[0230] The recognition unit is used to perform object recognition on the object to be recognized based on the infrared light image and the visible light image after the alignment processing, and obtain the object recognition result of the object to be recognized.

[0231] As can be seen from the above, in this embodiment, the acquisition unit 301 can acquire an infrared light image and a visible light image of the object to be identified; the determination unit 302 determines a reference pixel from each pixel of the infrared light image, and obtains the depth information of the reference pixel relative to the object to be identified; the acquisition unit 303 obtains the depth information of each pixel in the infrared light image relative to the object to be identified based on the position information of the reference pixel in the infrared light image and the depth information; the mapping unit 304 performs position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, and obtains the mapping point position information corresponding to each pixel in the infrared light image in the visible light image; the alignment unit 305 performs alignment processing on the pixels in the infrared light image and the visible light image based on the mapping point position information; the recognition unit 306 performs object recognition on the object to be identified based on the aligned infrared light image and the visible light image, and obtains the object recognition result of the object to be identified. The present application can realize pixel alignment of infrared light images and visible light images in scenarios where a depth map is lacking, thereby performing object recognition.

[0232] The present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0233] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0234] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0235] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0236] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0237] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0238] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0239] Collect infrared images and visible light images of the object to be identified; determine reference pixels from each pixel of the infrared image, and obtain depth information of the reference pixels relative to the object to be identified; obtain depth information of each pixel in the infrared image relative to the object to be identified based on the position information of the reference pixel in the infrared image and the depth information; perform position mapping processing on each pixel in the infrared image based on the depth information corresponding to each pixel in the infrared image, and obtain mapping point position information corresponding to each pixel in the infrared image in the visible light image; perform alignment processing on the pixels in the infrared image and the visible light image based on the mapping point position information; perform object recognition on the object to be identified based on the aligned infrared image and visible light image, and obtain an object recognition result of the object to be identified.

[0240] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0241] As can be seen from the above, this embodiment can collect infrared light images and visible light images for the object to be identified; determine reference pixels from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified; obtain depth information of each pixel in the infrared light image relative to the object to be identified based on the position information of the reference pixel in the infrared light image and the depth information; perform position mapping processing on each pixel in the infrared light image based on the depth information corresponding to each pixel in the infrared light image, and obtain mapping point position information corresponding to each pixel in the infrared light image in the visible light image; perform alignment processing on the pixels in the infrared light image and the visible light image based on the mapping point position information; perform object recognition on the object to be identified based on the aligned infrared light image and visible light image, and obtain the object recognition result of the object to be identified. This application can realize pixel alignment of infrared light images and visible light images in a scene where a depth map is missing, so as to perform object recognition.

[0242] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0243] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any object recognition method provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0244] Collect infrared images and visible light images of the object to be identified; determine reference pixels from each pixel of the infrared image, and obtain depth information of the reference pixels relative to the object to be identified; obtain depth information of each pixel in the infrared image relative to the object to be identified based on the position information of the reference pixel in the infrared image and the depth information; perform position mapping processing on each pixel in the infrared image based on the depth information corresponding to each pixel in the infrared image, and obtain mapping point position information corresponding to each pixel in the infrared image in the visible light image; perform alignment processing on the pixels in the infrared image and the visible light image based on the mapping point position information; perform object recognition on the object to be identified based on the aligned infrared image and visible light image, and obtain an object recognition result of the object to be identified.

[0245] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0246] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0247] Since the instructions stored in the computer-readable storage medium can execute the steps in any object recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any object recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0248] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned object recognition aspect.

[0249] The object recognition method and related equipment provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the present application are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An object recognition method, characterized in that: include: Collect infrared light images and visible light images of the object to be identified; Determine a reference pixel from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified; Determine, according to the position information of the reference pixel point in the infrared light image and the depth information, the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image; Based on the spatial position information of the mapping points, constructing a plane calibration equation corresponding to the object to be identified; Acquire depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared image and the reference pixel; Based on the depth information corresponding to each pixel in the infrared image, each pixel in the infrared image is mapped to the imaging space of the visible light image to obtain the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space; According to the position transformation relationship between the imaging plane and the imaging space of the visible light image, position transformation processing is performed on the spatial position information of the target mapping point to obtain the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image; Performing alignment processing on pixel points in the infrared light image and the visible light image according to the mapping point position information; Based on the aligned infrared light image and the visible light image, object recognition is performed on the object to be recognized to obtain an object recognition result of the object to be recognized.

2. The method according to claim 1, characterized in that The method of mapping each pixel in the infrared image to the imaging space of the visible light image based on the depth information corresponding to each pixel in the infrared image to obtain the spatial position information of the target mapping point corresponding to each pixel in the infrared image in the imaging space includes: Based on the depth information corresponding to each pixel point in the infrared light image, obtaining the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image; According to the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, the mapping point spatial position information is subjected to position transformation processing to obtain the target mapping point spatial position information corresponding to each pixel point in the infrared light image in the imaging space of the visible light image.

3. The method according to claim 2, characterized in that The acquiring, based on the depth information corresponding to each pixel in the infrared image, the mapping point spatial position information corresponding to each pixel in the infrared image in the imaging space of the infrared image comprises: According to the position transformation relationship between the imaging plane and the imaging space of the infrared light image, position transformation processing is performed on the position information of each pixel point in the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image; According to the depth information corresponding to each pixel point in the infrared light image and the initial position information of the mapping point, the spatial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image is determined.

4. The method according to claim 1, characterized in that: The obtaining, according to the plane calibration equation and the positional relationship between each pixel in the infrared image and the reference pixel, the depth information of each pixel in the infrared image relative to the object to be identified includes: Performing interpolation processing on the position information of each pixel in the infrared light image and the position information of the reference pixel to obtain position relationship parameters between each pixel in the infrared light image and the reference pixel; Determine the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation.

5. The method according to claim 4, characterized in that Determining the depth information of each pixel point in the infrared light image relative to the object to be identified according to the position relationship parameter and the plane calibration equation includes: According to the position relationship parameters, interpolation operation is performed on the spatial position information of the mapping point corresponding to the reference pixel point in the imaging space of the infrared light image to obtain initial position information of the mapping point corresponding to each pixel point in the infrared light image in the imaging space of the infrared light image; Determine the depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the initial position information of the mapping point.

6. The method according to claim 1, characterized in that The reference pixel points include at least three pixel points that are not on the same straight line.

7. An object recognition device, characterized in that: include: A collection unit, used for collecting infrared light images and visible light images of the object to be identified; A determination unit, configured to determine a reference pixel from each pixel of the infrared light image, and obtain depth information of the reference pixel relative to the object to be identified; an acquisition unit, configured to determine spatial position information of a mapping point corresponding to the reference pixel point in an imaging space of the infrared light image according to the position information of the reference pixel point in the infrared light image and the depth information; Based on the spatial position information of the mapping points, constructing a plane calibration equation corresponding to the object to be identified; Acquire depth information of each pixel in the infrared image relative to the object to be identified according to the plane calibration equation and the positional relationship between each pixel in the infrared image and the reference pixel; A mapping unit, configured to map each pixel point in the infrared light image to the imaging space of the visible light image based on the depth information corresponding to each pixel point in the infrared light image, and obtain the spatial position information of the target mapping point corresponding to each pixel point in the infrared light image in the imaging space; and perform position transformation processing on the spatial position information of the target mapping point according to the position transformation relationship between the imaging plane of the visible light image and the imaging space, and obtain the mapping point position information corresponding to each pixel point in the infrared light image in the visible light image; an alignment unit, configured to align pixel points in the infrared light image and the visible light image according to the mapping point position information; The recognition unit is used to perform object recognition on the object to be recognized based on the infrared light image and the visible light image after the alignment processing, and obtain the object recognition result of the object to be recognized.

8. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to perform the operations in the object recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the object recognition method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the object recognition method according to any one of claims 1 to 6 are implemented.

Images