Object identification method, object identification device, electronic device and computer program

Abstract

The present application discloses an object identification method, apparatus, electronic device, computer-readable storage medium, and computer program product, which are applicable to various scenarios such as cloud technology, artificial intelligence, and intelligent transportation, and which determine a reference pixel point from each pixel point of an infrared light image, obtain depth information of the reference pixel point, obtain depth information of each pixel point in the infrared light image based on the position information of the reference pixel point and the depth information, perform a position mapping process for each pixel point in the infrared light image based on the depth information corresponding to each pixel point, obtain corresponding mapping point position information in the visible light image, perform an alignment process for the pixel points in the infrared light image and the visible light image based on the mapping point position information, and perform object identification for the object to be identified based on the infrared light image and the visible light image after the alignment process, so as to obtain an object identification result.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed based on and claims priority from a Chinese patent application having application number 202210295192.8 and filing date March 23, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer technology, and in particular to an object identification method, apparatus, electronic device, computer-readable storage medium and computer program product. [Background technology]

[0003] With the development of computer technology, image processing technology is being applied to an increasing number of fields. For example, biometric identification technology is widely applied in many fields such as access control, information security, and electronic certificates. Specifically, biometric identification technology is a technology that automatically extracts biometric features from an image to be identified and then performs identity authentication based on these features. In the process of biometric identification, infrared and visible light images of the object to be identified are usually collected, where the infrared image may be used to perform liveness detection on the object to be identified. Before liveness detection, pixel alignment processing must be performed on the collected infrared and visible light images.

[0004] Related techniques mainly rely on depth maps to achieve pixel alignment processing for infrared and visible light images, but such methods are not applicable to identification scenarios where depth maps are lacking. Summary of the Invention

[0005] Embodiments of the present application provide an object identification method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can achieve pixel alignment for infrared light images and visible light images in scenarios where a depth map is lacking, thereby performing object identification.

[0006] An embodiment of the present application provides an object identification method executed by an electronic device, the object identification method comprising: collecting infrared and visible light images of an object to be identified; determining a reference pixel point from each pixel point of the infrared light image and obtaining depth information of the reference pixel point relative to the object to be identified; acquiring depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information and the depth information of the reference pixel point in the infrared light image; performing position mapping processing for each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding mapping point position information in the visible light image for each pixel point in the infrared light image; performing a registration process on pixel points in the infrared light image and the visible light image based on the mapping point position information; and performing object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtaining an object identification result for the object to be identified.

[0007] An embodiment of the present application provides an object identification device, the object identification device comprising: a collection unit configured to collect infrared and visible light images of the object to be identified; a determining unit configured to determine a reference pixel point from each pixel point of the infrared light image and obtain depth information of the reference pixel point relative to the object to be identified; an acquisition unit configured to acquire depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information of the reference pixel point in the infrared light image and the depth information; a mapping unit configured to perform a position mapping process on each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image; a registration unit configured to perform registration processing on pixel points in the infrared light image and the visible light image based on the mapping point position information; and an identification unit configured to perform object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtain an object identification result for the object to be identified.

[0008] An embodiment of the present application provides an electronic device, the electronic device including a processor and a memory, wherein a plurality of instructions are stored in the memory, and the processor loads the instructions to perform steps in an object identification method provided by the embodiment of the present application.

[0009] An embodiment of the present application further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements steps in the object identification method provided by the embodiment of the present application.

[0010] An embodiment of the present application further provides a computer program product including a computer program or instructions that, when executed by a processor, implements steps in the object identification method provided by an embodiment of the present application.

[0011] The embodiments of the present application obtain depth information of pixel points from an infrared light image, and assign depth information to pixel points in a visible light image through mapping between the infrared light image and the visible light image. The depth information can be restored from images collected by inexpensive sensors based on infrared light and visible light, and object identification can be supported without relying on expensive depth sensors, and have a wide range of application scenarios. [Brief explanation of the drawings]

[0012] [Figure 1a] 1 is a scenario schematic diagram of an object identification method according to an embodiment of the present application; [Figure 1b] 1 is a flowchart of an object identification method according to an embodiment of the present application; [Figure 1c] 1 is a structural schematic diagram of a speckle structured light imaging system according to an embodiment of the present application; [Figure 1d] FIG. 1 is a schematic diagram of a pinhole imaging model according to an embodiment of the present application. [Figure 1e] FIG. 2 is a schematic diagram of a transformation relationship between a camera coordinate system and a pixel coordinate system according to an embodiment of the present application; [Figure 1f] FIG. 2 is a schematic diagram of the relationship between camera coordinate systems according to an embodiment of the present application; [Figure 1g] 1 is a schematic diagram of the relationship between a camera coordinate system corresponding to an infrared view and a camera coordinate system corresponding to a color view according to an embodiment of the present application; [Figure 1h] 4 is another flowchart of an object identification method according to an embodiment of the present application; [Figure 2] 4 is another flowchart of an object identification method according to an embodiment of the present application; [Figure 3] 1 is a structural schematic diagram of an object identification device according to an embodiment of the present application; [Figure 4] 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0013] In order to more clearly explain the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments are briefly introduced above. Obviously, the drawings described above are only some embodiments of the present application, and those skilled in the art can also obtain other drawings based on these drawings without any creative efforts.

[0014] The technical solutions in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments of the present application without any creative efforts belong to the protection scope of the present application.

[0015] The embodiments of the present application provide an object identification method and related devices. The related devices can include an object identification device, an electronic device, a computer-readable storage medium, and a computer program product. The object identification device can be specifically integrated into an electronic device, and the electronic device can be a device such as a terminal or a server.

[0016] It can be understood that the object identification method of the embodiments of the present application may be executed by a terminal, a server, or jointly executed by a terminal and a server. The above examples should not be understood as limitations on the present application.

[0017] As shown in Fig. 1a, an object identification method is jointly performed by a terminal and a server. The object identification system provided by the embodiments of the present application includes a terminal 10 and a server 11. The terminal 10 and the server 11 are connected via a network, for example, via a wired or wireless network, and the object identification device may be integrated into the terminal.

[0018] Here, the terminal 10 may be configured to collect an infrared image and a visible light image of an object to be identified, determine a reference pixel point from each pixel point in the infrared image, obtain depth information of the reference pixel point for the object to be identified, obtain depth information of each pixel point in the infrared image for the object to be identified based on the position information of the reference pixel point in the infrared image and the depth information, perform a position mapping process for each pixel point in the infrared image based on the depth information corresponding to each pixel point in the infrared image, obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared image, perform a registration process for the pixel points in the infrared image and the visible light image based on the mapping point position information, transmit the registered infrared image and the visible light image to the server 11, perform object identification for the object to be identified via the server 11, and obtain an object identification result for the object to be identified. Here, the terminal 10 may include a mobile phone, a smart TV, a tablet computer, a notebook computer, a personal computer (PC), or the like. A client may be set in the terminal 10, and the client may be an application program client or a browser client, etc.

[0019] Here, the server 11 may be configured to receive the infrared light image and the visible light image after the alignment process transmitted by the terminal 10, perform object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, obtain an object identification result for the object to be identified, and transmit the object identification result to the terminal 10. Here, the server 11 may be a single server, or may be a server cluster or cloud server consisting of multiple servers. In the object identification method or device disclosed in the present application, a blockchain network can be formed by multiple servers, and the servers are nodes on the blockchain network.

[0020] The above steps of object identification by the server 11 may also be performed by the terminal 10.

[0021] The object identification method provided in the embodiments of this application relates to computer vision technology in the field of artificial intelligence. Artificial intelligence (AI) refers to theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology of computer science that aims to understand the essence of intelligence and create new intelligent machines that respond in a manner similar to human intelligence. AI researches the design principles and implementation methods of various intelligent machines, endowing them with the capabilities of sensing, reasoning, and decision-making. AI technology is a comprehensive discipline and covers a wide range of fields, including both hardware and software. AI software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and several themes such as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0022] Each of the embodiments will be described in detail below. It should be noted that the order of description of the following embodiments does not limit the preferred order of the embodiments.

[0023] Although the embodiments of the present application are described in terms of an object identification device, the object identification device may be specifically integrated into an electronic device, which may be a device such as a server or a terminal.

[0024] It is understandable that, when the above examples of the present application are applied to specific products or technologies, the relevant data, such as user information, in accordance with specific embodiments of the present application must obtain the user's permission or consent, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0025] The object identification method of the embodiments of the present application can be applied to various scenarios that require object identification, such as palm payment, access control, etc. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, assisted driving, etc.

[0026] As shown in FIG. 1b, the specific process of the object identification method is as follows:

[0027] In step 101, infrared and visible light images are collected for the object to be identified.

[0028] Here, the object to be identified may be an object that requires biometric feature identification, palm print information corresponding to a human body, a human face, etc., but is not limited to this in the embodiments of the present application.

[0029] Here, the infrared light image is specifically an infrared image obtained by collecting an infrared sensor using pan-infrared light, and may be used to perform liveness detection on the object to be identified. The visible light image is specifically a two-dimensional color image obtained by collecting an image sensor using natural light, and may be used to perform identity identification on the object to be identified. It should be noted that in the embodiment of the present application, the infrared light image and the visible light image of the object to be identified may be collected synchronously, so that the infrared light image and the visible light image can be matched.

[0030] Specifically, in the process of biometric feature identification, it is generally necessary to further acquire a depth image for the object to be identified. Specifically, the depth image may be a depth map obtained by collecting speckle-structured infrared light with an infrared sensor and then analyzing the speckle with a depth unit. In three-dimensional (3D) computer graphics and computer vision, a depth map is an image or image channel containing information about the distance from the surface of the collected scenario object to the viewpoint. Each pixel point in the depth map represents the vertical distance between the depth camera plane and the plane of the photographed object, typically represented by 16 bits and measured in millimeters. In object identification scenarios, depth images may typically be used for liveness detection and auxiliary identification. Using depth images to assist identification can significantly improve the accuracy and robustness of identification.

[0031] Here, biometric detection is a detection method for determining whether the object to be identified is a real person, a photograph, a head mold, etc. Generally, it is possible to determine whether the object to be identified is a photograph based on a depth image, and whether the object to be identified is a silicone head mold based on the brightness of an infrared light image.

[0032] Here, the speckle structured light is a lattice light arranged according to a certain structural rule, projected by an infrared speckle projector. As shown in Figure 1c, the speckle structured light imaging system consists of an infrared laser projector 11 and an infrared sensor 12. These lattice lights are projected onto the surface 13 of the object (i.e., the reference sample). After imaging with the infrared sensor, the 3D coordinate information of the object surface can be restored based on the triangulation principle, thereby obtaining a depth image.

[0033] Compared with a depth image, a point cloud image records the 3D coordinate information of an object in the real world, which is obtained by calculating the depth image and camera parameters. The depth image also contains the depth information of the object, but the abscissa and ordinate of a point in the depth image are pixel coordinates in the imaging plane (hereinafter also referred to as the pixel plane), while the point cloud image records the physical coordinates of the point in the real world. The point cloud image can be displayed in a 3D rendering engine and reflects the three-dimensional positional relationship of each point.

[0034] Related technologies typically use red, green, blue, and depth (RGB-D) data for identification. RGB is a color mode and an industry color standard. Various colors can be obtained by adjusting the three color channels (red, green, and blue) and overlaying them. RGB stands for the three channels (red, green, and blue). Here, D stands for depth map. In 3D computer graphics, a depth map is an image or image channel that contains information about the surface distance of a scene object from the viewpoint. A depth map is similar to a grayscale image, but the pixel value of each pixel represents the actual distance from the sensor to the object. Typically, an RGB image (i.e., a visible light image) is mapped to a depth image, so there is a one-to-one correspondence between the pixel points of the RGB image and the pixel points of the depth image.

[0035] However, in some identity identification scenarios, only visible light images and infrared light images can be collected, and depth images are lacking, making it impossible to rely on depth images to achieve pixel-level alignment of the three images, where pixel point-level alignment of the three images refers to a one-to-one correspondence between pixel points in the visible light image, infrared light image, and depth image. This application provides an object identification method that can achieve pixel-level alignment of visible light images and infrared light images without relying on depth images.

[0036] The object identification method provided in this application can be applied to the field of palmprint identification, which is a biometric feature identification technology. Palmprint identification can extract palmprint information of the object to be identified. Palmprint information specifically refers to palm image information from the fingertips to the wrist. Since the palmprint features of different objects' palms are different, identity authentication can be performed based on the palmprint information.

[0037] Here, biometric identification technology specifically refers to a technology that closely combines high-tech means such as computers, optics, acoustics, biosensors, and biostatistics principles to identify individuals by utilizing physiological characteristics unique to the human body (e.g., fingerprints, face, iris, etc.) and behavioral characteristics (e.g., handwriting, voice, walking style, etc.).

[0038] In step 102, a reference pixel point is determined from each pixel point of the infrared light image, and depth information of the reference pixel point relative to the object to be identified is obtained.

[0039] In some embodiments, three pixel points that are not on the same straight line as a pixel point in the infrared image can be arbitrarily selected as reference pixel points.

[0040] For example, the reference pixel points include at least three pixel points that are not on the same straight line.

[0041] In a specific scenario, for the depth information of the reference pixel points of the object to be identified, a component proximity sensor (Psensor) can be used to obtain the depth information corresponding to each reference pixel point. Specifically, the number of reference pixel points is first determined, and the number of Psensors can be determined based on the number of reference pixel points, with each Psensor being used to obtain the depth information of one reference pixel point. The principle of measuring the depth information of the Psensor can refer to dToF, where direct time-of-flight (dToF) directly measures the distance based on the time difference between the transmission and reception of a pulse. The Psensor can calculate the distance between the imaging plane and the object surface of the object to be identified based on the round-trip time of the transmitted light beam between the imaging plane and the object surface and the propagation speed of light, and the distance is the depth information.

[0042] For example, in the absence of a depth camera, if the object to be identified is planar (e.g., a palm of a hand), Psensor can be used as an alternative solution to derive the planar depth value where the object to be identified is located, and three Psensors can be attached at positions that are not on the same line in the pixel plane of the infrared light image, thereby using Psensor to obtain depth information corresponding to three or more reference pixel points.

[0043] In step 103, based on the position information of the reference pixel point in the infrared light image and the depth information, depth information of each pixel point in the infrared light image relative to the object to be identified is obtained.

[0044] Specifically, the embodiments of the present application can obtain the pixel plane pixel depth of the infrared light image, i.e., the depth information of each pixel point in the infrared light image for the object to be identified, based on the position information and depth information of the reference pixel point.

[0045] For example, in an embodiment of the present application, step 103 may include the steps of: determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the position information and the depth information of the reference pixel point in the infrared light image; constructing a plane orientation equation corresponding to the object to be identified based on the mapping point spatial position information; and obtaining depth information of each pixel point in the infrared light image relative to the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared light image and the reference pixel point.

[0046] In some embodiments, the step of "determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the position information of the reference pixel point in the infrared light image and the depth information" may include the steps of: performing a position transformation process on the position information of the reference pixel point in the infrared light image based on a position transformation relationship between a pixel plane of the infrared light image and an imaging space, to obtain corresponding mapping point initial position information of the reference pixel point in the imaging space of the infrared light image; and determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the depth information corresponding to the reference pixel point and the mapping point initial position information.

[0047] Here, the position transformation relationship between the pixel plane of the infrared light image and the imaging space may be specifically represented by the camera internal parameters of the camera sensor corresponding to the infrared light image. In the embodiment of the present application, a pixel coordinate system corresponding to the infrared map can be established in the pixel plane of the infrared light image, and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared map can be established in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0048] 1d, the camera model is specifically a pinhole imaging model, and the camera internal parameters are parameters for describing the transformation relationship between the three-dimensional spatial coordinates (i.e., coordinates on the pixel plane 14) when a real-world object is imaged on the camera sensor and the two-dimensional pixel coordinates on the physical imaging plane 13 after imaging. The depth image and the point cloud image in the above embodiment can also be transformed into each other by the camera internal parameters.

[0049] Here, the camera coordinate system refers to a coordinate system in which the optical center O17 of the camera 16 is the coordinate origin, the optical axis is the z-axis, and the x-axis and y-axis are parallel to the x-axis and y-axis of the imaging plane. The 3D coordinates of the depth image and the point cloud image obtained by transformation using the camera's internal parameters are coordinates in the camera coordinate system (Oxyz).

[0050] Specifically, based on the position transformation relationship between the pixel plane of the infrared light image and the imaging space, the camera internal parameters of the camera sensor corresponding to the infrared light image can be obtained, and further based on the camera internal parameters, position transformation processing can be performed on the position information of the reference pixel point in the infrared light image, and the processing process may be as shown in equation (1).

[0051] TIFF0007801480000001.tif15170, where fx, fy, u0, and v0 are internal camera parameters. Specifically, fx represents the focal length in the x-axis direction, fy represents the focal length in the y-axis direction, and u0 and v0 represent the position of the principal point. The position information of the reference pixel point in the infrared image can be expressed as (u, v), and the depth information corresponding to the reference pixel point is zc. According to equation (1), the initial position information xc and yc of the corresponding mapping point in the imaging space of the infrared image of the reference pixel point can be obtained. Here, xc and yc are the x-axis coordinate and y-axis coordinate included in the spatial position information of the corresponding mapping point in the imaging space of the infrared image of the reference pixel point.

[0052] After obtaining the initial position information xc and yc of the mapping point, the corresponding mapping point spatial position information (xc, yc, zc) in the imaging space of the infrared light image of the reference pixel point can be constructed based on the depth information zc corresponding to the reference pixel point.

[0053] Here, the step of "constructing a plane orientation equation corresponding to the object to be identified based on the mapping point spatial position information" can include the steps of: setting an initial plane orientation equation corresponding to the object to be identified, the initial plane orientation equation including at least one orientation parameter; performing an analysis process on the at least one orientation parameter based on the mapping point spatial position information to obtain analyzed orientation parameters; and updating the initial plane orientation equation based on the analyzed orientation parameters to obtain a plane orientation equation corresponding to the object to be identified.

[0054] Here, the initial plane orientation equation can be set as AX+BY+CZ+D=0, where A, B, C and D are orientation parameters. The mapping point spatial position information is substituted into the initial plane orientation equation, and the initial plane orientation equation is solved, that is, the orientation parameters are analyzed, and specific values ​​of A, B, C and D, that is, the analyzed orientation parameters, are obtained, thereby obtaining the plane orientation equation of the object to be identified.

[0055] In one specific embodiment, the position information of three reference pixel points m, n, and p in the infrared light image can be written as m=(x1, y1), n=(x2, y2), and p=(x3, y3). Based on Equation (1) in the above embodiment, the corresponding mapping point space position information of the three reference pixel points in the imaging space of the infrared light image can be calculated, and these are written as M=(X1, Y1, Z1), N=(X2, Y2, Z2), and P=(X3, Y3, Z3), respectively. Then, M=(X1, Y1, Z1), N=(X2, Y2, Z2), and P=(X3, Y3, Z3) are substituted into the following Equation (2): AX+BY+CZ+D=0 (2) By solving the coefficients A, B, C and D in the above equation, the plane orientation equation of the object to be identified can be obtained.

[0056] In an embodiment of the present application, the step of "obtaining depth information of each pixel point in the infrared image for the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared image and the reference pixel point" can include the steps of: performing an interpolation operation on positional information of each pixel point in the infrared image and positional information of the reference pixel point to obtain positional relationship parameters between each pixel point in the infrared image and the reference pixel point; and determining depth information of each pixel point in the infrared image for the object to be identified based on the positional relationship parameters and the plane orientation equation.

[0057] In one specific embodiment, the position information of three reference pixel points m, n, and p in the infrared light image can be written as m=(x1, y1), n=(x2, y2), and p=(x3, y3). If it is necessary to obtain depth information of another pixel point k on the infrared light image, the position relationship parameters between each pixel point in the infrared light image and the reference pixel point can be calculated by interpolating the three points m, n, and p, and the position information of pixel point k in the infrared light image is (x4, y4), and the interpolation process can be as shown in Equation (3) and Equation (4).

[0058] x4=x1+α*(x2-x1)+β*(x3-x1) (3) y4=y1+α*(y2-y1)+β*(y3-y1) (4) By substituting the coordinates of the m, n, p, and k pixel points into equations (3) and (4), the positional relationship parameters α and β between each pixel point in the infrared image and the reference pixel point can be obtained.

[0059] In an embodiment of the present application, the step of "determining depth information of each pixel point in the infrared light image for the object to be identified based on the positional relationship parameters and the plane orientation equation" can include the steps of: performing an interpolation operation on corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the positional relationship parameters, to obtain corresponding mapping point initial position information of each pixel point in the infrared light image in the imaging space of the infrared light image; and determining depth information of each pixel point in the infrared light image for the object to be identified based on the plane orientation equation and the mapping point initial position information.

[0060] In one embodiment, the corresponding mapping points of three reference pixel points m, n, and p in the imaging space of the infrared light image are M, N, and P, and the spatial position information of the mapping points is called mapping point spatial position information, which is M=(X1, Y1, Z1), N=(X2, Y2, Z2), and P=(X3, Y3, Z3), respectively. The x-axis and y-axis coordinates of these three mapping point spatial position information are substituted into the above equations (3) and (4), thereby obtaining the corresponding mapping point initial position information X4 and Y4 of pixel point k in the imaging space of the infrared light image, where X4 and Y4 are specifically the x-axis and y-axis coordinates of the corresponding mapping point spatial position information of pixel point k in the imaging space of the infrared light image.

[0061] After obtaining the initial position information X4, Y4 of the mapping points, they can be substituted into equation (2) in the above embodiment to obtain the corresponding spatial position information (X4, Y4, Z4) of the mapping point in the imaging space of the infrared image of pixel point k, i.e., the depth information Z4 of pixel point k relative to the object to be identified can be obtained. By analogy, the depth information corresponding to each pixel point in the infrared image can be obtained.

[0062] In step 104, based on the depth information corresponding to each pixel point in the infrared light image, a position mapping process is performed for each pixel point in the infrared light image to obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image.

[0063] In an embodiment of the present application, the step of "performing a position mapping process on each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding mapping point position information in the visible light image for each pixel point in the infrared light image" can include the steps of: mapping each pixel point in the infrared light image to an imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding target mapping point spatial position information in the imaging space for each pixel point in the infrared light image; and performing a position transformation process on the target mapping point spatial position information based on a position transformation relationship between the pixel plane of the visible light image and the imaging space, and obtaining corresponding mapping point position information in the visible light image for each pixel point in the infrared light image.

[0064] Here, the position transformation relationship between the pixel plane of the visible light image and the imaging space may be specifically represented by the camera internal parameters of the camera sensor corresponding to the visible light image. In the embodiments of the present application, a pixel coordinate system corresponding to the color map can be established in the pixel plane of the visible light image, which may be a two-dimensional coordinate system, or a camera coordinate system corresponding to the color map can be established in the imaging space of the visible light image, which may be a three-dimensional coordinate system.

[0065] Here, the camera internal parameters are parameters that describe the transformation relationship between the three-dimensional spatial coordinates when a real-world object is imaged on the camera sensor and the two-dimensional pixel coordinates on the physical imaging plane after imaging.

[0066] In one specific embodiment, the camera internal parameters can be used to convert the target mapping point in the camera coordinate system corresponding to the color map into the pixel coordinate system. The conversion between the camera coordinate system and the pixel coordinate system can be seen in Figure 1e, where P(XC, YC, ZC) is a point in the camera coordinate system, and p(x, y) is a point in the pixel coordinate system, and the coordinate conversion relationship between the two can be converted by the camera internal parameters.

[0067] Specifically, based on the position transformation relationship between the pixel plane of the visible light image and the imaging space, the camera internal parameters of the camera sensor corresponding to the visible light image can be obtained, and then based on the camera internal parameters, position transformation processing can be performed on the target mapping point space position information, and the processing process may be as shown in the following equation (5).

[0068] TIFF0007801480000002.tif15170, where fx, fy, u0, and v0 are internal camera parameters. Specifically, fx represents the focal length in the x-axis direction, fy represents the focal length in the y-axis direction, and u0 and v0 are the positions of the principal point. The spatial position information of the target mapping point can be expressed as (xc, yc, zc). Equation (5) can be used to calculate the corresponding mapping point position information (u, v) in the visible light image for each pixel point in the infrared image, i.e., the mapping point position information in the pixel coordinate system corresponding to the color map.

[0069] In an embodiment of the present application, the step of "mapping each pixel point in the infrared light image into the imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding target mapping point spatial position information in the imaging space of each pixel point in the infrared light image" can include the steps of: obtaining corresponding mapping point spatial position information in the imaging space of the infrared light image for each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image; and performing a position transformation process on the mapping point spatial position information based on a position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, and obtaining corresponding target mapping point spatial position information in the imaging space of the visible light image for each pixel point in the infrared light image.

[0070] In some embodiments, the step of "obtaining corresponding mapping point spatial position information in the imaging space of the infrared light image for each pixel point in the infrared light image" may specifically construct corresponding mapping point spatial position information in the imaging space of the infrared light image for each pixel point in the infrared light image based on the depth information corresponding to each pixel point in the infrared light image obtained in step 103 and the corresponding mapping point initial position information for each pixel point in the infrared light image in the imaging space of the infrared light image.

[0071] For example, for each pixel point, the depth information corresponding to the pixel point in the infrared light image and the corresponding mapping point initial position information in the imaging space of the infrared light image are each one-dimensional and combined as the corresponding mapping point spatial position information of the pixel point in the imaging space of the infrared light image.

[0072] Here, the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image may be specifically expressed by the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image. In the embodiment of the present application, a camera coordinate system corresponding to the infrared map can be established in the imaging space of the infrared light image, and a camera coordinate system corresponding to the color map can be established in the imaging space of the visible light image, and both camera coordinate systems are three-dimensional coordinate systems.

[0073] As shown in Figure 1f, camera extrinsic parameters can be parameters for describing the transformation relationship between another 3D coordinate system and the camera coordinate system. When there are multiple cameras, the coordinates of an object point in one camera's camera coordinate system can be transformed into another camera coordinate system via a rotation matrix R and a translation matrix T. The rotation matrix R and the translation matrix (also called the offset vector) T are extrinsic parameters between these two cameras, and these camera extrinsic parameters describe the transformation relationship between the two camera coordinate systems. c0 and c1 are the cameras in the two camera coordinate systems, respectively. There are angle and distance offsets between the two cameras. Similar to the relationship between the world coordinate system and the camera coordinate system, the transformation from the world coordinate system to the camera coordinate system is a rigid transformation, i.e., the object does not deform, so only rotation and translation are required.

[0074] In one specific embodiment, the spatial position information of the mapping point in the camera coordinate system corresponding to the infrared image can be converted into the camera coordinate system corresponding to the color image by using the external camera parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image. The conversion between the camera coordinate system corresponding to the infrared image and the camera coordinate system corresponding to the color image can be seen in FIG. 1g, where P(Xw, Yw, Zw) is a point in the camera coordinate system corresponding to the infrared image, and P(Xw, Yw, Zw) can be converted into a point in the camera coordinate system corresponding to the color image by using the external camera parameters (i.e., the above-mentioned rotation matrix R and translation matrix T).

[0075] Specifically, based on the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image can be obtained, and then based on the camera extrinsic parameters, a position transformation process can be performed on the mapping point space position information, and the processing process is as shown in Equation (6).

[0076] TIFF0007801480000003.tif15170Here, in equation (6), the matrix consisting of r00 to r22 is the rotation matrix R, and the matrix consisting of tx to tz is the offset vector T. The rotation matrix R and the offset vector T are camera extrinsic parameters, and the mapping point spatial position information can be expressed as (xw, yw, zw).Equation (6) gives the corresponding target mapping point spatial position information (xc, yc, zc) in the imaging space of the visible light image for each pixel point in the infrared light image. , that is, the spatial position information of the target mapping points in the camera coordinate system corresponding to the color map can be obtained.

[0077] In an embodiment of the present application, the step of "obtaining corresponding mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image" may include the steps of: performing a position transformation process on the position information of each pixel point in the infrared light image based on the position transformation relationship between the pixel plane of the infrared light image and the imaging space, to obtain corresponding mapping point initial position information of each pixel point in the infrared light image in the imaging space of the infrared light image; and determining corresponding mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the infrared light image based on the depth information corresponding to each pixel point in the infrared light image and the mapping point initial position information.

[0078] Here, the position transformation relationship between the pixel plane of the infrared light image and the imaging space may be specifically represented by the camera internal parameters of the camera sensor corresponding to the infrared light image. In the embodiment of the present application, a pixel coordinate system corresponding to the infrared map can be established in the pixel plane of the infrared light image, and the pixel coordinate system can be a two-dimensional coordinate system, and a camera coordinate system corresponding to the infrared map can be established in the imaging space of the infrared light image, and the camera coordinate system can be a three-dimensional coordinate system.

[0079] In one specific embodiment, the position information of a pixel point in the pixel coordinate system corresponding to the infrared image can be converted into the camera coordinate system using the camera's internal parameters. For example, point P is a point in the camera coordinate system, and point p is a point in the pixel coordinate system, and the coordinate conversion relationship between the two can be converted using the camera's internal parameters.

[0080] Specifically, based on the position transformation relationship between the pixel plane of the infrared light image and the imaging space, the camera internal parameters of the camera sensor corresponding to the infrared light image are obtained, and then based on the camera internal parameters, a position transformation process is performed on the position information of each pixel point in the infrared light image, so as to obtain the initial position information of the corresponding mapping point in the imaging space of the infrared light image for each pixel point in the infrared light image, and the processing process may be as shown in the following equation (7).

[0081] TIFF0007801480000004.tif15170, where fx, fy, u0, and v0 are internal camera parameters. Specifically, fx represents the focal length in the x-axis direction, fy represents the focal length in the y-axis direction, and u0 and v0 represent the position of the principal point. The position information of each pixel point in the infrared image can be expressed as (u, v), and the depth information corresponding to each pixel point in the infrared image is zc. The initial position information xc and yc of the corresponding mapping point in the imaging space of the infrared image for each pixel point in the infrared image can be calculated using equation (1), where xc and yc are the x-axis and y-axis coordinates of the corresponding mapping point spatial position information in the imaging space of the infrared image for each pixel point in the infrared image.

[0082] After obtaining the initial mapping point position information xc and yc, the corresponding mapping point spatial position information (xc, yc, zc) in the imaging space of the infrared light image for each pixel point in the infrared light image can be constructed based on the depth information zc corresponding to each pixel point in the infrared light image.

[0083] In step 105, a registration process is performed on the pixel points in the infrared light image and the visible light image based on the mapping point position information.

[0084] Specifically, by using the visible light image as a reference, a registration process can be performed on each pixel point in the infrared light image and the corresponding pixel point in the visible light image based on the mapping point position information, that is, the pixel points in the infrared light image are registered with the pixel points in the visible light image that represent the same target object, and the registered infrared light image is made to correspond to the positions of the pixel points in the visible light image where the same target object is located.

[0085] For example, the position in the pixel coordinate system of the entire visible light image is kept unchanged, i.e., the coordinates in the coordinate system of each pixel in the visible light image are kept unchanged, and the position in the pixel coordinate system of the infrared light image is moved as a whole by at least one of rotation and translation, and the coordinates in the pixel coordinate system of each pixel point in the infrared light image are transformed and moved, so that the coordinates of pixel points representing the same target object in the infrared light image and the visible light image are the same.

[0086] In a specific scenario of palmprint identification, the palm (specifically, the object to be identified) can be regarded as a plane, so the embodiment of the present application uses Psensor to calculate the slope distance corresponding to the palm (specifically, it may be the depth information of each pixel point in the infrared image for the object to be identified in the above embodiment), and then performs pixel registration between the infrared image and the visible light image based on the slope distance, thereby realizing pixel-level registration of the two images at low cost.

[0087] It should be noted that the object identification method provided by the present application also determines a reference pixel point from each pixel point in the visible light image, obtains depth information of the reference pixel point for the object to be identified, then obtains depth information of each pixel point in the visible light image for the object to be identified based on the position information and depth information of the reference pixel point in the visible light image, further performs a position mapping process for each pixel point in the visible light image based on the depth information corresponding to each pixel point in the visible light image, obtains corresponding mapping point position information in the infrared image for each pixel point in the visible light image, and finally performs a registration process for the pixel points in the infrared image and the visible light image based on the mapping point position information, so as to further identify the object. It should be understood that the processes of obtaining depth information corresponding to pixel points and the position mapping process can be correspondingly referred to the above embodiments and will not be described in detail herein.

[0088] In one specific embodiment, as shown in FIG. 1h, a flowchart for pixel registration between an infrared image and a visible light image is shown. Specifically, in step 201, depth information corresponding to each pixel point in the infrared image is obtained based on depth information of a reference pixel point in the infrared image. In step 202, position information of the pixel point in the pixel coordinate system corresponding to the infrared image is converted into the camera coordinate system of the infrared image using internal camera parameters of the camera sensor corresponding to the infrared image. In step 203, spatial position information of the mapping point in the camera coordinate system corresponding to the infrared image is converted into the camera coordinate system corresponding to the color image using external camera parameters between the camera sensors corresponding to the infrared image and the visible light image. In step 204, the camera internal parameters of the camera sensor corresponding to the visible light image are used to convert the target mapping point in the camera coordinate system corresponding to the color image into the pixel coordinate system corresponding to the color image to obtain a mapping result, thereby performing pixel registration between the visible light image and the infrared image based on the mapping result.

[0089] In step 106, object identification is performed on the object to be identified based on the infrared light image and the visible light image after the alignment process, and an object identification result for the object to be identified is obtained.

[0090] Here, after the registration process, biometric detection can be performed on the infrared light image after the registration process, and after passing the biometric detection, object identification can be performed based on the visible light image after the registration process. Object identification can specifically be a technology for exchanging object identity information with biometric multimedia information. In some embodiments, a related payment operation can be further performed based on the object identification result.

[0091] In a specific scenario, for example, in a palmprint identification scenario, performing object identification on an object to be identified may specifically involve extracting palmprint information of the object to be identified. After obtaining the palmprint information, the palmprint information can be matched with palmprint feature information stored in a preset palmprint information database, thereby determining the object identity of the object to be identified. The preset palmprint information database may store a target mapping relationship set, which includes mapping relationships between the preset palmprint feature information and the preset object identities. For example, the target mapping relationship set may be a relationship table between the preset palmprint feature information and the preset object identities.

[0092] In some embodiments, the matching degree between the palm print information and each of the predetermined palm print feature information is calculated, and target palm print feature information can be determined from the predetermined palm print feature information based on the matching degree. For example, the predetermined palm print feature information having the highest matching degree with the palm print information can be determined as the target palm print feature information. Furthermore, the object identity corresponding to the target palm print feature information can be determined as the object identity of the object to be identified based on the target mapping relationship set.

[0093] For example, the Euclidean distance between the vectorized representation of the palm print line features (e.g., length, number of lines) and texture features of different palm print information is calculated, and the reciprocal of the Euclidean distance is taken as the matching degree.

[0094] As can be seen from the above, the embodiments of the present application obtain depth information of pixel points from an infrared light image, and assign depth information to pixel points in the visible light image through mapping between the infrared light image and the visible light image, so that 3D data including depth information can be accurately restored using images collected by inexpensive sensors based on infrared light and visible light, thereby enabling subsequent object identification. Since the cost of object identification at the hardware implementation level is reduced compared to related art solutions that rely on depth sensors for object identification, the embodiments of the present application can be widely used in various application scenarios that require large-scale object identification, such as various intelligent transportation.

[0095] Based on the method described in the previous embodiment, the following will be described in more detail by taking the example that the object identification device is specifically integrated into a terminal.

[0096] The embodiment of the present application provides an object identification method, and as shown in FIG. 2, the specific process of the object identification method may be as follows:

[0097] In step 301, the terminal collects infrared and visible light images of the object to be identified.

[0098] Here, the object to be identified may be an object that requires biometric feature identification, palm print information corresponding to a human body, a human face, etc., but is not limited to this in the embodiments of the present application.

[0099] Here, the infrared light image may be an infrared image obtained by collecting an infrared sensor using pan-infrared light and may be used to detect the presence of an object to be identified. The visible light image may be a color image obtained by collecting an image sensor using natural light and may be used to identify the object to be identified. It should be noted that in the embodiment of the present application, the infrared light image and the visible light image of the object to be identified may be collected at the same time, thereby allowing the infrared light image and the visible light image to correspond to each other.

[0100] In step 302, the terminal determines a reference pixel point from each pixel point of the infrared light image, and obtains depth information of the reference pixel point relative to the object to be identified.

[0101] In some embodiments, three pixel points that are not on the same straight line as a pixel point in the infrared image can be arbitrarily selected as reference pixel points.

[0102] In an embodiment of the present application, the reference pixel points include at least three pixel points that are not on the same line.

[0103] In a specific scenario, for the depth information of the reference pixel points of the object to be identified, a Psensor can be used to obtain the depth information corresponding to each reference pixel point. Specifically, the number of reference pixel points is first determined, and the number of Psensors can be determined based on the number of reference pixel points, and each Psensor can only obtain the depth information of one reference pixel point. The principle of measuring the depth information of the Psensor can refer to dToF, which directly measures the distance based on the time difference between the transmission and reception of a pulse. The Psensor can calculate the distance between the imaging plane and the object surface of the object to be identified based on the round-trip time of the transmitted light between the imaging plane and the object surface and the propagation speed of light, and this distance is the depth information.

[0104] For example, in the absence of a depth camera, if the object to be identified is planar (e.g., a palm of a hand), an inexpensive Psensor can be used as an alternative solution to derive the planar depth value where the object to be identified is located, and three Psensors can be attached at positions that are not on the same line in the pixel plane of the infrared light image, thereby using the Psensors to obtain depth information corresponding to three or more reference pixel points.

[0105] In step 303, the terminal obtains depth information of each pixel point in the infrared image for the object to be identified based on the position information of the reference pixel point in the infrared image and the depth information.

[0106] In an embodiment of the present application, the step of "obtaining depth information of each pixel point in the infrared light image for the object to be identified based on the position information and the depth information of the reference pixel point in the infrared light image" can include the steps of: determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the position information and the depth information of the reference pixel point in the infrared light image; constructing a plane orientation equation corresponding to the object to be identified based on the mapping point spatial position information; and obtaining depth information of each pixel point in the infrared light image for the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared light image and the reference pixel point.

[0107] For example, the step of "determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the position information and the depth information of the reference pixel point in the infrared light image" may include the steps of: performing a position transformation process on the position information of the reference pixel point in the infrared light image based on a position transformation relationship between a pixel plane of the infrared light image and an imaging space; and obtaining corresponding mapping point initial position information of the reference pixel point in the imaging space of the infrared light image; The method may include a step of determining corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on depth information corresponding to the reference pixel point and the mapping point initial position information.

[0108] Here, the position transformation relationship between the pixel plane of the infrared light image and the imaging space may be specifically represented by the camera internal parameters of the camera sensor corresponding to the infrared light image. The embodiments of the present application may establish a pixel coordinate system corresponding to the infrared image in the pixel plane (also referred to as pixel plane) of the infrared light image, which may be a two-dimensional coordinate system, or establish a camera coordinate system corresponding to the infrared image in the imaging space of the infrared light image, which may be a three-dimensional coordinate system.

[0109] Here, the step of "constructing a plane orientation equation corresponding to the object to be identified based on the mapping point spatial position information" can include the steps of: setting an initial plane orientation equation corresponding to the object to be identified, the initial plane orientation equation including at least one orientation parameter; performing an analysis process on the at least one orientation parameter based on the mapping point spatial position information to obtain analyzed orientation parameters; and updating the initial plane orientation equation based on the analyzed orientation parameters to obtain a plane orientation equation corresponding to the object to be identified.

[0110] Here, the initial plane orientation equation can be set as AX+BY+CZ+D=0, where A, B, C and D are orientation parameters. In the embodiment of the present application, the mapping point spatial position information is substituted into the initial plane orientation equation, thereby analyzing the orientation parameters, and the specific values ​​of A, B, C and D, that is, the analyzed orientation parameters, can be obtained, thereby obtaining the plane orientation equation of the object to be identified.

[0111] In an embodiment of the present application, the step of "obtaining depth information of each pixel point in the infrared image for the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared image and the reference pixel point" can include the steps of: performing an interpolation operation on positional information of each pixel point in the infrared image and positional information of the reference pixel point to obtain positional relationship parameters between each pixel point in the infrared image and the reference pixel point; and determining depth information of each pixel point in the infrared image for the object to be identified based on the positional relationship parameters and the plane orientation equation.

[0112] In an embodiment of the present application, the step of "determining depth information of each pixel point in the infrared light image for the object to be identified based on the positional relationship parameters and the plane orientation equation" can include the steps of: performing an interpolation operation on corresponding mapping point spatial position information of the reference pixel point in the imaging space of the infrared light image based on the positional relationship parameters, to obtain corresponding mapping point initial position information of each pixel point in the infrared light image in the imaging space of the infrared light image; and determining depth information of each pixel point in the infrared light image for the object to be identified based on the plane orientation equation and the mapping point initial position information.

[0113] In step 304, the terminal maps each pixel point in the infrared light image to the imaging space of the visible light image based on the depth information corresponding to each pixel point in the infrared light image, and obtains corresponding target mapping point spatial position information in the imaging space of each pixel point in the infrared light image.

[0114] In an embodiment of the present application, the step of "mapping each pixel point in the infrared light image into the imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding target mapping point spatial position information in the imaging space of each pixel point in the infrared light image" can include the steps of: obtaining corresponding mapping point spatial position information in the imaging space of the infrared light image for each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image; and performing a position transformation process on the mapping point spatial position information based on a position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, and obtaining corresponding target mapping point spatial position information in the imaging space of the visible light image for each pixel point in the infrared light image.

[0115] Here, the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image may be specifically expressed by the camera extrinsic parameters between the camera sensor corresponding to the infrared light image and the camera sensor corresponding to the visible light image. In the embodiment of the present application, a camera coordinate system corresponding to the infrared map can be established in the imaging space of the infrared light image, and a camera coordinate system corresponding to the color map can be established in the imaging space of the visible light image, and both camera coordinate systems are three-dimensional coordinate systems.

[0116] In an embodiment of the present application, the step of "obtaining corresponding mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image" may include the steps of: performing a position transformation process on the position information of each pixel point in the infrared light image based on the position transformation relationship between the pixel plane of the infrared light image and the imaging space, to obtain corresponding mapping point initial position information of each pixel point in the infrared light image in the imaging space of the infrared light image; and determining corresponding mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the infrared light image based on the depth information corresponding to each pixel point in the infrared light image and the mapping point initial position information.

[0117] Here, the position transformation relationship between the pixel plane of the infrared light image and the imaging space may be specifically represented by the camera internal parameters of the camera sensor corresponding to the infrared light image. The embodiments of the present application may establish a pixel coordinate system corresponding to the infrared image in the pixel plane (also referred to as pixel plane) of the infrared light image, which may be a two-dimensional coordinate system, or establish a camera coordinate system corresponding to the infrared image in the imaging space of the infrared light image, which may be a three-dimensional coordinate system.

[0118] In step 305, the terminal performs a position transformation process on the target mapping point spatial position information based on the position transformation relationship between the pixel plane of the visible light image and the imaging space, to obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image.

[0119] In step 306, the terminal performs registration processing on the pixel points in the infrared light image and the visible light image based on the mapping point position information.

[0120] Specifically, by using the visible light image as a reference, a registration process can be performed on each pixel point in the infrared light image and the corresponding pixel point in the visible light image based on the mapping point position information, that is, the pixel points in the infrared light image are registered with the pixel points in the visible light image that represent the same target object, and the registered infrared light image is made to correspond to the positions of the pixel points in the visible light image where the same target object is located.

[0121] In step 307, the terminal performs object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtains an object identification result for the object to be identified.

[0122] Here, after the registration process, biometric detection can be performed on the infrared light image after the registration process, and after passing the biometric detection, object identification can be performed based on the visible light image after the registration process. Object identification can specifically be a technology for exchanging object identity information with biometric multimedia information. In some embodiments, a related payment operation can be further performed based on the object identification result.

[0123] As can be seen from the above, the embodiments of the present application can achieve pixel registration for infrared and visible light images in scenarios where depth maps are missing, thereby enabling object identification.

[0124] An embodiment of the present application further provides an object identification apparatus, which may include a collecting unit 301, a determining unit 302, an acquiring unit 303, a mapping unit 304, a registration unit 305 and an identification unit 306, as shown in FIG.

[0125] The collection unit 301 is configured to collect infrared and visible light images of the object to be identified.

[0126] The determining unit 302 is configured to determine a reference pixel point from each pixel point of the infrared light image, and obtain depth information of the reference pixel point relative to the object to be identified.

[0127] In some embodiments of the present application, the reference pixel points include at least three pixel points that are not on the same line.

[0128] The obtaining unit 303 is configured to obtain depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information of the reference pixel point in the infrared light image and the depth information.

[0129] In some embodiments of the present application, the acquisition unit may include a determination subunit, a construction subunit, and an acquisition subunit, wherein the determination subunit is configured to determine corresponding mapping point space position information of the reference pixel point in an imaging space of the infrared light image based on position information of the reference pixel point in the infrared light image and the depth information, and the construction subunit is configured to construct a plane orientation equation corresponding to the object to be identified based on the mapping point space position information; The acquisition subunit is configured to acquire depth information of each pixel point in the infrared light image relative to the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared light image and the reference pixel point.

[0130] In some embodiments of the present application, the acquisition subunit may be specifically configured to perform an interpolation operation on the position information of each pixel point in the infrared light image and the position information of the reference pixel point to obtain positional relationship parameters between each pixel point in the infrared light image and the reference pixel point, and determine depth information of each pixel point in the infrared light image relative to the object to be identified based on the positional relationship parameters and the plane orientation equation.

[0131] The mapping unit 304 is configured to perform a position mapping process for each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image.

[0132] In some embodiments of the present application, the mapping unit may include a mapping subunit and a position conversion subunit, wherein the mapping subunit is configured to map each pixel point in the infrared light image to an imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtain corresponding target mapping point spatial position information in the imaging space for each pixel point in the infrared light image, and the position conversion subunit is configured to perform a position conversion process on the target mapping point spatial position information based on a position conversion relationship between a pixel plane of the visible light image and the imaging space, and obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image.

[0133] In some embodiments of the present application, the mapping subunit may be specifically configured to obtain corresponding mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and perform a position transformation process on the mapping point spatial position information based on the position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, to obtain corresponding target mapping point spatial position information of each pixel point in the infrared light image in the imaging space of the visible light image.

[0134] The alignment unit 305 is configured to perform alignment processing on pixel points in the infrared light image and the visible light image based on the mapping point position information.

[0135] The identification unit 306 is configured to perform object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtain an object identification result for the object to be identified.

[0136] An embodiment of the present application further provides an electronic device, and as shown in FIG. 4 , a structural schematic diagram of an electronic device according to an embodiment of the present application is shown. The electronic device may be a terminal, a server, etc. The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 4 does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than those shown, or may combine some components, or may have different components arranged.

[0137] The processor 401 is the control center of the electronic device, connecting each part of the entire electronic device using various interfaces and lines, running or executing software programs and / or modules stored in the memory 402, and accessing data stored in the memory 402 to perform various functions of the electronic device and process data. For example, the processor 401 may include one or more processing cores, and for example, the processor 401 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It may be understood that the modem processor does not have to be integrated into the processor 401.

[0138] The memory 402 may be used to store software programs and modules, and the processor 401 performs various functional applications and data processing by executing the software programs and modules stored in the memory 402. The memory 402 may mainly include a storage program area and a storage data area, where the storage program area may store an operating system, an application program required for at least one function (e.g., audio playback function, image playback function, etc.), etc., and the storage data area may store data created based on the use of the electronic device, etc.

[0139] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically, in the embodiment of the present application, the processor 401 in the electronic device loads executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and executes the application programs stored in the memory 402 by the processor 401, thereby realizing the above-mentioned object identification method in the embodiment of the present application. For specific implementations of the above operations, please refer to the previous embodiment, and will not be described in detail herein.

[0140] As can be understood by those skilled in the art, all or part of the steps in the various methods of the above embodiments may be completed by instructions or by controlling associated hardware through instructions, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0141] An embodiment of the present application provides a computer-readable storage medium storing a plurality of instructions, which can be loaded by a processor to perform any of the steps in the object identification method provided by the embodiment of the present application. Specific implementations of the above operations can refer to the previous embodiments and will not be described in detail here. The computer-readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0142] An embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer-readable storage medium, a processor of a computing device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computing device to perform the methods provided in various selectable implementation forms of the object identification aspect described above.

[0143] Although the object identification method and related devices provided by the embodiments of the present application have been introduced in detail above, this specification applies specific examples to explain the principles and embodiments of the present application, and the description of the above examples is only intended to assist in understanding the method and its core idea of ​​the present application. At the same time, those skilled in the art will recognize that there may be changes in the specific embodiments and application scope based on the idea of ​​the present application, and as stated above, the contents of this specification should not be construed as limitations on the present application.

Claims

1. 1. A method for object identification performed by an electronic device, comprising: collecting infrared and visible light images of an object to be identified; determining a reference pixel point from each pixel point of the infrared light image and obtaining depth information of the reference pixel point relative to the object to be identified; acquiring depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information and the depth information of the reference pixel point in the infrared light image; performing position mapping processing for each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding mapping point position information in the visible light image for each pixel point in the infrared light image; performing a registration process on pixel points in the infrared light image and the visible light image based on the mapping point position information; performing object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtaining an object identification result for the object to be identified.

2. The step of performing a position mapping process on each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding mapping point position information in the visible light image for each pixel point in the infrared light image, includes: mapping each pixel point in the infrared light image to an imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding target mapping point spatial position information in the imaging space of each pixel point in the infrared light image; and performing a position transformation process on the target mapping point space position information based on a position transformation relationship between a pixel plane of the visible light image and an imaging space, to obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image. The object identification method of claim 1 .

3. The step of mapping each pixel point in the infrared light image to an imaging space of the visible light image based on depth information corresponding to each pixel point in the infrared light image, and obtaining corresponding target mapping point spatial position information in the imaging space of each pixel point in the infrared light image, includes: obtaining corresponding mapping point space position information of each pixel point in the infrared light image in an imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image; performing a position transformation process on the mapping point space position information based on a position transformation relationship between the imaging space of the infrared light image and the imaging space of the visible light image, to obtain corresponding target mapping point space position information in the imaging space of the visible light image for each pixel point in the infrared light image; The object identification method of claim 2 .

4. The step of acquiring corresponding mapping point space position information of each pixel point in the infrared light image in an imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image includes: performing a position transformation process on position information of each pixel point in the infrared light image based on a position transformation relationship between a pixel plane of the infrared light image and an imaging space, to obtain initial position information of a corresponding mapping point in the imaging space of the infrared light image for each pixel point in the infrared light image; determining corresponding mapping point spatial position information of each pixel point in the infrared light image in an imaging space of the infrared light image based on depth information corresponding to each pixel point in the infrared light image and the mapping point initial position information; The object identification method according to claim 3 .

5. The step of acquiring depth information of each pixel point in the infrared light image for the object to be identified based on the position information and the depth information of the reference pixel point in the infrared light image includes: determining corresponding mapping point space position information of the reference pixel point in an imaging space of the infrared light image based on the position information of the reference pixel point in the infrared light image and the depth information; constructing a plane orientation equation corresponding to the object to be identified based on the mapping point spatial position information; and acquiring depth information of each pixel point in the infrared light image relative to the object to be identified based on the plane orientation equation and a positional relationship between each pixel point in the infrared light image and the reference pixel point. The object identification method of claim 1 .

6. The step of acquiring depth information of each pixel point in the infrared light image relative to the object to be identified based on the plane orientation equation and the positional relationship between each pixel point in the infrared light image and the reference pixel point includes: performing an interpolation operation on position information of each pixel point in the infrared image and position information of the reference pixel point to obtain positional relationship parameters between each pixel point in the infrared image and the reference pixel point; determining depth information of each pixel point in the infrared image relative to the object to be identified based on the positional relationship parameters and the plane orientation equation; The object identification method according to claim 5 .

7. The step of determining depth information of each pixel point in the infrared light image relative to the object to be identified based on the positional relationship parameters and the plane orientation equation includes: performing an interpolation operation on corresponding mapping point spatial position information of the reference pixel points in the imaging space of the infrared light image based on the positional relationship parameters, to obtain corresponding mapping point initial position information of each pixel point in the infrared light image in the imaging space of the infrared light image; determining depth information of each pixel point in the infrared light image relative to the object to be identified based on the plane orientation equation and the mapping point initial position information; The object identification method of claim 6.

8. The reference pixel points include at least three pixel points that are not on the same straight line. The object identification method of claim 1 .

9. The step of performing a registration process on pixel points in the infrared light image and the visible light image based on the mapping point position information includes: a step of keeping the coordinates of each pixel point in the visible light image in a pixel coordinate system unchanged, and moving the position of the infrared light image in the pixel coordinate system as a whole by at least one of rotation and translation, so that the coordinates of pixel points representing the same target object in the infrared light image and the visible light image after the movement are the same; The object identification method of claim 1 .

10. 1. An object identification device, comprising: a collection unit configured to collect infrared and visible light images of the object to be identified; a determining unit configured to determine a reference pixel point from each pixel point of the infrared light image and obtain depth information of the reference pixel point relative to the object to be identified; an acquisition unit configured to acquire depth information of each pixel point in the infrared light image relative to the object to be identified based on the position information of the reference pixel point in the infrared light image and the depth information; a mapping unit configured to perform a position mapping process on each pixel point in the infrared light image based on depth information corresponding to each pixel point in the infrared light image, and obtain corresponding mapping point position information in the visible light image for each pixel point in the infrared light image; a registration unit configured to perform registration processing on pixel points in the infrared light image and the visible light image based on the mapping point position information; an identification unit configured to perform object identification on the object to be identified based on the infrared light image and the visible light image after the alignment process, and obtain an object identification result for the object to be identified.

11. a memory for storing an application program; a processor that executes the application program in the memory to perform the object identification method according to any one of claims 1 to 9.

12. A computer program that causes a computer to execute the object identification method described in any one of claims 1 to 9.

Images