A method, device, smart glasses and storage medium for multi-camera half-face recognition in VR

Through multi-camera technology, half-face and eye images under VR glasses are obtained and spliced, and combined with iris recognition, the problems of insufficient security of face recognition and image transmission under VR glasses are solved, achieving higher recognition accuracy and video calls real-time image display.

CN117423147BActive Publication Date: 2024-11-05SHENZHEN HUAHOM TETHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311433900.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-31
Publication Date
2024-11-05
Estimated Expiration
2043-10-31

AI Technical Summary

Technical Problem

When the user's eyes are blocked, the facial recognition security is insufficient or cannot be recognized, and the user's image cannot be transmitted to others in real time, affecting the interactive experience of video calls.

Method used

Multi-camera technology is adopted to obtain images of the human eye and lower half of the face through infrared and visible light cameras, perform stitching and feature extraction, and combine iris recognition to realize stitching and recognition of half-face images, and output virtual display.

Benefits of technology

It improves the accuracy and security of identity recognition, allows the user's real-time image to be recorded and displayed in video calls, and enhances the interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117423147B_ABST
    Figure CN117423147B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of smart glasses, and specifically discloses a multi-camera half-face recognition method, device, smart glasses and storage medium in VR. The method includes: using multiple infrared cameras to shoot human eye images illuminated by infrared light, splicing the images corresponding to the left and right eyes, and obtaining a second eye front view image. Use multiple visible light cameras to shoot a first lower half face image and splice it into a second lower half face image. Extract the skin color of the second lower half face image, color the skin of the second eye front view image, use a pre-trained network to perform color mapping on the eyes, and then splice the colored eye image and the lower half face image into a half face image, judge the identity through facial recognition or combined with iris recognition, and output the half face image for virtual display. The present invention can identify based on the eye image and the lower half face image of the user wearing VR glasses, and can also be combined with VR iris recognition, thereby improving the recognition rate of user identity recognition and improving the security of recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of smart glasses, and in particular to a multi-camera half-face recognition method and device in VR, smart glasses and a storage medium. Background Art

[0002] There are many kinds of smart glasses, including mainstream VR glasses and eye trackers, which are used for virtual display and eye tracking respectively. In the prior art, after wearing VR glasses, the user's eyes will be blocked, and only the lower half of the face will be exposed. In this case, if face recognition is required, the lower half of the face does not have enough facial features, which will lead to insufficient security or ineffective recognition.

[0003] Also, when the user is making a remote online video call with others, the VR glasses can display the image of others on the built-in display for the user to watch, but the user's image cannot be provided to them. Of course, in some cases, the user can use a third-party camera to shoot in front of the face and transmit the image to others, but even so, the user's eyes are blocked by the VR glasses. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a multi-camera half-face recognition method, device, smart glasses and storage medium in VR, so as to provide a technical solution for tracking and displaying human eyes.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present invention, a method for multi-camera half-face recognition in VR is disclosed, the method comprising:

[0007] Acquire a plurality of human eye images illuminated by infrared light and taken by a plurality of infrared cameras arranged at preset positions, wherein the plurality of human eye images correspond to a left eye and a right eye respectively, and the human eye images include eyes and skin around the eyes;

[0008] Splicing the plurality of human eye images corresponding to the left eye and the right eye respectively to obtain first eye frontal view images corresponding to the left eye and the right eye respectively, and splicing the two first eye frontal view images to obtain a second eye frontal view image;

[0009] Acquire a plurality of first lower half face images captured by a plurality of visible light cameras arranged in preset positions;

[0010] splicing the plurality of the first lower half face images into a complete second lower half face image;

[0011] Extracting the skin color in the second lower half face image, coloring the skin in the second eye front view image, performing color mapping on the eyes in the second eye front view image based on a pre-trained color transformation network between a human eye grayscale image and real face skin and real iris, and splicing the colored second eye front view image and the second lower half face image into a half face image;

[0012] Performing facial recognition on the half-face image alone, or / and combining with the iris in the second eye frontal image for recognition, to determine the identity of the person corresponding to the half-face image, and outputting the half-face image for virtual display.

[0013] Furthermore, the plurality of human eye images corresponding to the left eye and the right eye are respectively spliced ​​to obtain first eye frontal images corresponding to the left eye and the right eye, respectively, including:

[0014] Collecting features of the plurality of human eye images obtained of the same human eye, wherein the features include one or more of pupil contour, reflected light spots generated by infrared light irradiation, and texture shape and direction of the iris;

[0015] Based on the collected features, register the multiple human eye images to obtain the spatial coordinate transformation parameters of each of the human eye images;

[0016] Based on the spatial coordinate transformation parameters, the plurality of human eye images are transformed and spliced ​​to obtain the first eye frontal view image.

[0017] Furthermore, the plurality of human eye images corresponding to the left eye and the right eye are respectively spliced ​​to obtain first eye frontal images corresponding to the left eye and the right eye, respectively, including:

[0018] According to the physical positions and camera parameters of the different infrared cameras, for the human eye image taken by any of the infrared cameras, mark the image mask of the overlapping area with the images taken by other infrared cameras;

[0019] For the human eye image taken by each infrared camera, human eye segmentation is performed based on the marked mask area, and the iris, pupil, and sclera areas therein are marked;

[0020] Estimating the projection transformation matrix required for stitching human eye images taken by different infrared cameras;

[0021] An iterative method is adopted to sequentially splice two adjacent human eye images according to the projection transformation matrix, so as to finally obtain a complete frontal view image of the first eye.

[0022] Furthermore, stitching the plurality of first lower half face images into a complete second lower half face image comprises:

[0023] Constructing two or more nonlinear scale spaces of the first lower half face images according to an algorithm based on scale space theory and nonlinear scale space extreme point detection;

[0024] Calculating extreme points in the nonlinear scale space to obtain feature points;

[0025] Calculate the main direction within a preset area of ​​the feature point;

[0026] In a preset area of ​​each feature point, a feature vector is calculated for describing information of surrounding pixels of the feature point;

[0027] Feature matching is performed based on a distance metric between any two of the feature vectors, so that two or more of the first lower half face images are spliced.

[0028] Furthermore, after feature matching, the following steps are also included:

[0029] Randomly select a group of samples from the matching data set of the feature vector, and calculate and evaluate model parameters based on the selected samples;

[0030] Calculate the projection error between all other samples in the matching data set and the evaluation model. If the error is less than a threshold, record them in the qualified set, and take the qualified set with the largest number of elements as the optimal set;

[0031] Repeatedly select samples for calculation, and if the number of elements in the current qualified set is greater than the previous optimal set, update the current qualified set to the new optimal set, and perform a new round of sample selection calculation until the set number of cycles is reached;

[0032] The feature vector in the final optimal set is used as a result and applied to a spliced ​​image, so that two or more of the first lower half face images are spliced.

[0033] Furthermore, after two or more of the first lower half face images are spliced ​​together, color and brightness compensation is performed on the spliced ​​image.

[0034] Furthermore, when color and brightness compensation is performed, it includes:

[0035] respectively calculating the brightness of the common area of ​​each of the first lower half face images;

[0036] respectively calculating the colors of the common areas of the first lower half face images;

[0037] The image with the highest brightness is selected as the global brightness and color, and the color and brightness compensation is performed on the stitched image.

[0038] According to a second aspect of the present disclosure, a multi-camera half-face recognition device in VR is provided, comprising: a first image acquisition module, used to acquire a plurality of human eye images illuminated by infrared light and taken by a plurality of infrared cameras arranged in preset positions, the plurality of human eye images corresponding to a left eye and a right eye respectively, and the human eye images including eyes and skin around the eyes; a first splicing module, used to splice the plurality of human eye images corresponding to the left eye and the right eye respectively, to obtain first frontal eye images corresponding to the left eye and the right eye respectively, and to splice the two first frontal eye images to obtain a second frontal eye image; a second image acquisition module, used to acquire a plurality of first lower half face images taken by a plurality of visible light cameras arranged in preset positions; a second splicing module, Used to stitch together multiple first lower half face images into a complete second lower half face image; a half-face fusion module, used to extract the skin color in the second lower half face image, colorize the skin in the second eye front view image, color map the eyes in the second eye front view image based on a pre-trained color transformation network between the human eye grayscale image and the real skin and real iris of the face, and stitch the colored second eye front view image and the second lower half face image into a half-face image; a recognition module, used to perform facial recognition on the half-face image alone, or / and recognize the iris in the second eye front view image in combination to determine the identity of the person corresponding to the half-face image, and output the half-face image for virtual display.

[0039] According to a third aspect of the present disclosure, there is provided a pair of smart glasses, comprising: a plurality of infrared cameras, which are respectively directed to the left and right human eyes and arranged in preset positions; an infrared floodlight emitter, which is arranged facing the human eyes; a plurality of visible light cameras, which are respectively directed to the lower half of the face and arranged in preset positions; a display screen; one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the multi-camera half-face recognition method in VR as described above.

[0040] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the multi-camera half-face recognition method in VR is implemented as described above.

[0041] The technical solution disclosed in this disclosure has the following beneficial effects:

[0042] The present invention can identify users wearing VR glasses based on the eye images and lower face images, and can also be combined with VR iris recognition, thereby improving the recognition rate of user identity recognition and also improving the security of recognition; after the present invention splices the lower face and eye images, it can also be used in video calls between the user and others, so that the user's real-time image can be recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a multi-camera half-face recognition method in VR in an embodiment of this specification;

[0044] Figure 2 This is a structural block diagram of a multi-camera half-face recognition device in VR in an embodiment of this specification;

[0045] Figure 3 It is a terminal device for executing a multi-camera half-face recognition method in VR in an embodiment of this specification;

[0046] Figure 4 This is a structural block diagram of a smart glasses in an embodiment of this specification;

[0047] Figure 5 It is a computer-readable storage medium that stores a multi-camera half-face recognition method in VR in an embodiment of this specification. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0049] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0050] like Figure 1 As shown, the embodiment of this specification provides a multi-camera half-face recognition method in VR, and the execution subject of the method can be a computer, a server, a wearable smart device, etc. The method can specifically include the following steps S101 to S104:

[0051] In step S101, a plurality of human eye images illuminated by infrared light, captured by a plurality of infrared cameras arranged at preset positions, are obtained, wherein the plurality of human eye images correspond to the left eye and the right eye respectively, and the human eye images include the eyes and the skin around the eyes.

[0052] The number of infrared cameras can be any number greater than 2, such as 3, 4, 5, etc. The positions of each infrared camera relative to each other are known, and the positions relative to each other can be equally spaced or symmetrical. In addition, the use of infrared cameras plus infrared light can avoid the limitations of visible light cameras. For example, visible light cameras need light sources to obtain clear photos. Since the object of the photo is the human eye, the visible light source will cause discomfort to the human eye, thereby affecting the collection effect.

[0053] In step S102, the plurality of human eye images corresponding to the left eye and the right eye are spliced ​​respectively to obtain first eye frontal view images corresponding to the left eye and the right eye respectively, and the two first eye frontal view images are spliced ​​to obtain a second eye frontal view image.

[0054] Among them, since the positions of each infrared camera relative to each other are known, the coordinate system of each infrared camera can be converted to each other, and the coordinates of each pixel of the human eye image obtained by each infrared camera can be converted to each other. Therefore, the two images can be spliced ​​into one, which is operable and accurate. For example, the two human eye images can be preprocessed first, such as cropping, correction, etc., and then the overlapping part (feature part) of the two images is calculated, and then the images are aligned, such as by rotation, translation, scaling, etc., so that the overlapping parts are aligned, and the two aligned images can be fused, such as linear mixing, multi-band fusion, etc., to fuse the two images to obtain the spliced ​​image. These operations can be encapsulated into an algorithm so that they can be automatically executed when the image is input to obtain the first spliced ​​image. Based on the same principle, after obtaining two first eye front view images respectively, they can be spliced ​​and fused to obtain the second eye front view image.

[0055] In step S103, a plurality of first lower half face images captured by a plurality of visible light cameras arranged at preset positions are obtained.

[0056] Among them, the same principle as step S101, the number of visible light cameras can be any number greater than 2, such as 3, 4, 5, etc., which can be arranged under the VR glasses, each shooting a fixed area of ​​the face. The visible light camera can also be set at a position away from the VR glasses and in front of the human body, but in this case, the position between the VR glasses and the visible light camera needs to be calibrated before shooting.

[0057] In step S104, a plurality of the first lower half face images are spliced ​​into a complete second lower half face image.

[0058] The stitching process of the first lower half face image may refer to S102.

[0059] In step S105, the skin color in the second lower half face image is extracted, the skin in the second eye front view image is colored, the eyes in the second eye front view image are color mapped based on a pre-trained color transformation network between the human eye grayscale image and the real skin and real iris of the face, and the colored second eye front view image and the second lower half face image are spliced ​​into a half-face image.

[0060] Among them, since the infrared camera takes a grayscale image and does not show the skin color, the lower half of the face image taken by the visible light camera is colored and colored to the non-eye position in the second eye frontal image. Specifically, the average color of the second lower half of the face image can be taken.

[0061] Of course, in another possible embodiment, the coloring of the eyes and surrounding skin in the second frontal eye image can be performed by the same color transformation network. Specifically, the pre-trained color transformation network can be a neural network based on deep learning. When training the color transformation network, grayscale images of human eyes and corresponding color face images can be collected as training data. The color face image can contain complete features such as human eye iris color, eyebrow color, and facial skin color. The color transformation network is obtained by training using a deep learning framework. The input of the network is a grayscale image, and the output is the corresponding color skin around the eye and color iris image. The network can also be a gray eye image and a color lower face image as input, and the output is the corresponding color skin around the eye and color iris image. Specifically, the real skin of a face can be white, yellow, black, or brown, the iris can be blue, green, or black, the eyebrows can be black, white, yellow, or red, etc., and the above colors can also coexist in multiple colors, such as the iris of one eye is blue and the iris of the other eye is black, or half of an eyebrow is white and the other half is black.

[0062] In step S106, facial recognition is performed on the half-face image alone, or / and combined with iris recognition in the second eye frontal image to determine the identity of the person corresponding to the half-face image, and the half-face image is output for virtual display.

[0063] Specifically, a half-face feature extraction network can be trained offline through a deep learning method, and then the network can be used to extract half-face image features. Whether it is the same person can be determined based on the similarity between the features. At the same time, the iris identified in the second eye frontal image can be combined to refine the iris recognition.

[0064] In another use, the half-face image can be directly output to the video chat as a real-time image of the user talking to the outside world. It can also be output after further processing. Specifically, since the spliced ​​half-face image contains a complete eye image and a face image outside the VR eye coverage, the blind spots of the infrared camera and the visible light camera are missing. These blind spots can also be filled by neural network training, or simply covered by virtual decorations.

[0065] Furthermore, after obtaining the second eye frontal image with color, it can be displayed on the display screen in front of the VR smart glasses. In this way, when the user wears the glasses, his eye image can be seen by others, which improves the interactive experience when talking with others face to face, or can be used in conjunction with other devices that need to recognize eye movements.

[0066] It is worth noting that the display screen can be opaque and only used to display the second spliced ​​image at the front. Correspondingly, the smart glasses are also provided with other display screens on the side facing the user's eyes to display virtual content. The display screen can also be transparent, and the human eye can directly observe the computer screen, physical space, etc. in front through the display screen.

[0067] It is worth noting that after the transformation and stitching of the eye images, the coordinate system of the second eye front view image may be based on the coordinate system of a certain infrared camera, which may be different from the orthographic coordinate system. Therefore, after obtaining the second eye front view image, it is also necessary to calculate the transformation matrix between the second eye front view image and the orthographic coordinate system according to the different shooting angles and camera parameters of the infrared camera. The orthographic coordinate system is generally the coordinate system of the front-end display screen, and based on the transformation matrix, the second spliced ​​image is converted to the coordinate system of the front-end display screen so that the second eye front view image is orthographic.

[0068] In one implementation, the above step S102 may specifically include the following steps S201-203:

[0069] In step S201, feature collection is performed on the multiple human eye images obtained of the same human eye, wherein the feature includes one or more of pupil contour, reflected light spot generated by infrared light irradiation, texture shape and direction of the iris.

[0070] Among them, the collected features can be the outline of the pupil of the human eye, the reflected light spot caused by infrared light irradiation, the texture shape and direction of the iris, etc. In addition, the collected iris features will be used separately in the subsequent iris recognition.

[0071] In step S202, the plurality of human eye images are registered based on the collected features to obtain the spatial coordinate transformation parameters of each of the human eye images.

[0072] Among them, registration is the process of aligning multiple images into a common coordinate system. Through registration, the spatial coordinate transformation parameters of each eye image, such as translation, rotation and scaling, can be obtained.

[0073] In step S203, based on the spatial coordinate transformation parameters, the plurality of human eye images are transformed and spliced ​​to obtain the first frontal eye image.

[0074] Among them, transformation and stitching are two processes. During the transformation, based on the features and spatial coordinate transformation parameters, the stitching process of the two eye images can be confirmed. Subsequently, the two first eye frontal images can be fused and stitched together. Similarly, multiple eye images can be fused and stitched into one.

[0075] In one implementation, step S101 may further include the following steps S301-S304:

[0076] In step S301, according to the physical positions and camera parameters of the different infrared cameras, for the human eye image taken by any of the infrared cameras, an image mask is marked that has overlapping areas with images taken by other infrared cameras.

[0077] In step S302, the human eye image taken by each infrared camera is segmented based on the marked mask area, and the iris, pupil, and sclera areas therein are marked.

[0078] In step S303, the projection transformation matrix required for stitching the human eye images taken by different infrared cameras is estimated;

[0079] The projection transformation matrix required for stitching human eye images taken by different infrared cameras is estimated, and its cost function is constructed based on the texture coefficients of the iris, pupil, and sclera regions marked in step S301, wherein the iris region has the highest overall weight, followed by the pupil region, and the sclera region has the lowest weight. The projection transformation matrix is ​​calculated using an iterative method so that the texture similarity of the mapped image is the highest. The process of estimating the projection transformation matrix can be roughly calculated based on the physical positions and parameters of different infrared cameras and the common features (iris, pupil, sclera, etc.) in two adjacent human eye images.

[0080] In step S304, an iterative method is used to sequentially splice two adjacent human eye images according to the projection transformation matrix, and finally a complete frontal view image of the first eye is obtained.

[0081] In one implementation, step S103 may include the following steps S401-S405:

[0082] In step S401, two or more nonlinear scale spaces of the first lower half face images are constructed according to an algorithm based on scale space theory and nonlinear scale space extreme point detection.

[0083] Specifically, the algorithm is the KAZE algorithm, and the parameter θ in the scale space can be described as follows:

[0084]

[0085] Among them, θ0 is the base scale, o is the index of the group, and S represents the index of the layer within the group. The scale parameter of the key point is calculated based on the number of groups and layers within the group where the key point is located and the combination of equations.

[0086] The scale parameters of each layer in the group that constructs the Gaussian pyramid are calculated according to the following formula:

[0087]

[0088]

[0089] The scale parameters of the same layer in different groups are the same, so the scale calculation formula of a layer image in the group is as follows:

[0090]

[0091] To ensure the continuity of scale space, a set of images is obtained by sampling the second to last layer of the previous set. We assume that the initial scale of the first set is θ. Then the scale parameters of each layer of the first set are θ, rθ, r 2 θ and so on. The scale of the penultimate layer can be defined as follows:

[0092] r s θ=2θ

[0093] Since the nonlinear diffusion filter model is time-based, it is necessary to convert the scale parameter into evolution time. Assume that the standard deviation used in the Gaussian scale space is σ. Then, the convolution of the Gaussian is equivalent to the convolution of the duration t = σ 2 / 2, all images in the nonlinear scale space can be obtained through the AOS algorithm, which can be expressed as:

[0094]

[0095] In step S402, extreme points in the nonlinear scale space are calculated to obtain feature points.

[0096] Among them, the maximum value in the nonlinear scale space can be used as a feature point.

[0097] In step S403, the main direction within the preset area of ​​the feature point is calculated.

[0098] Specifically, assuming that the scale parameter of the feature point is θ1, the search radius is set to 6θ1, six 60-degree fan-shaped areas are divided within the radius, and the sum of the Haar wavelet features in each fan-shaped area is calculated. The direction with the largest sum of wavelet features is determined as the main direction.

[0099] In step S404, a feature vector for describing information of surrounding pixels of each feature point is calculated in a preset area of ​​each feature point.

[0100] Among them, for the scale parameter θ i feature point, we take a gradient image with a size of 24θ on the gradient image centered on the feature point. i x24θ iThen, the window is divided into 4×4 pieces of size 9θ i ×9θ i There is a width of 2θ between adjacent sub-regions. i In the process of calculating the sub-region description vector, for each sub-region, a vector with weight θ1 = 2.5θ is used. i To describe the vector for each subregion:

[0101] d v =(∑L x ,∑L y ,∑|L x |,∑|L y |)

[0102] The weight of each sub-region of the vector dv is then weighted by another 4×4 Gaussian window, where θ2 = 1.5θ i Finally, a 64-dimensional description vector, i.e., feature vector, is obtained through normalization.

[0103] In step S405, feature matching is performed based on the distance measurement between any two of the feature vectors, so that two or more of the first lower half face images are spliced.

[0104] In one embodiment, after feature matching, individual mismatches and over-concentration may occur. To solve this problem, the RANSAC algorithm is applied to eliminate mismatched points.

[0105] Randomly selecting a group of samples from the matching data set of the feature vector, and calculating and evaluating model parameters based on the selected samples;

[0106] Calculate the projection error between all other samples in the matching data set and the evaluation model. If the error is less than a threshold, record them in the qualified set, and take the qualified set with the largest number of elements as the optimal set;

[0107] Repeatedly select samples for calculation, and if the number of elements in the current qualified set is greater than the previous optimal set, update the current qualified set to the new optimal set, and perform a new round of sample selection calculation until the set number of cycles is reached;

[0108] The feature vector in the final optimal set is used as a result and applied to a spliced ​​image, so that two or more of the first lower half face images are spliced.

[0109] The feature points are accurately matched through the above iterative process, which effectively eliminates mismatched point pairs, obtains the optimal feature points, and eliminates distortion and ghosting problems in the stitching process.

[0110] In one implementation, after two or more of the first lower half face images are spliced ​​together, color and brightness compensation is performed on the spliced ​​image.

[0111] Among them, when performing color and brightness compensation, it includes: calculating the brightness of the common area of ​​each of the first lower half face images respectively; calculating the color of the common area of ​​each of the first lower half face images respectively; selecting the image with the highest brightness as the global brightness and color, and performing color and brightness compensation on the spliced ​​image.

[0112] Specifically, the formula is as follows:

[0113]

[0114]

[0115]

[0116] In the above formula, L avg Represents the average brightness difference, L i1 is the brightness of the overlapping area of ​​the first image, L i2 is the brightness of the overlapping area of ​​the second image. Similarly, A avg and B avg is the average color difference, A i1 and B i1 is the color of the overlapping area of ​​the first image, A i2 and B i2 is the color of the overlapping area of ​​the second image. In terms of compensation, an image with higher brightness can be selected as a standard to compensate the global brightness and color of the other image. Finally, the compensated images are fused to obtain the final stitched image.

[0117] Based on the same idea, Figure 2As shown, an exemplary embodiment of the present disclosure further provides a multi-camera half-face recognition device in VR, comprising: a first image acquisition module 201, used to acquire a plurality of human eye images illuminated by infrared light, taken by a plurality of infrared cameras arranged in preset positions, the plurality of human eye images corresponding to the left eye and the right eye respectively, and the human eye images including the eyes and the skin around the eyes; a first splicing module 202, used to splice the plurality of human eye images corresponding to the left eye and the right eye respectively, to obtain first eye frontal images corresponding to the left eye and the right eye respectively, and to splice the two first eye frontal images to obtain a second eye frontal image; a second image acquisition module 203, used to acquire a plurality of first lower half face images taken by a plurality of visible light cameras arranged in preset positions; a second splicing module 204, used to splice the plurality of human eye images corresponding to the left eye and the right eye respectively, to obtain first eye frontal images corresponding to the left eye and the right eye respectively, and to obtain a second eye frontal image by splicing the two first eye frontal images; Block 204 is used to stitch together multiple first lower half face images into a complete second lower half face image; a half-face fusion module 205 is used to extract the skin color in the second lower half face image, colorize the skin in the second eye front view image, perform color mapping on the eyes in the second eye front view image based on a pre-trained color transformation network between a human eye grayscale image and real human face skin and real iris, and stitch the colored second eye front view image and the second lower half face image into a half-face image; a recognition module 206 is used to perform facial recognition on the half-face image alone, or / and recognize the iris in the second eye front view image in combination to determine the identity of the person corresponding to the half-face image, and output the half-face image for virtual display.

[0118] Based on the multi-camera half-face recognition device in VR, it is possible to identify the user wearing VR glasses based on the eye image and the lower face image, and can also be combined with VR iris recognition, thereby improving the recognition rate of user identity recognition and also improving the security of recognition; after the lower face and eye images are spliced ​​together, the present invention can also be used in video calls between the user and others, so that the user's real-time image can be recorded.

[0119] Based on the same idea, the embodiment of this specification also provides a multi-camera half-face recognition device in VR, such as Figure 3 shown.

[0120] The multi-camera half-face recognition device in VR can be the server provided in the above embodiment.

[0121] The multi-camera half-face recognition device in VR may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and the memory 302 may store one or more storage applications or data. Among them, the memory 302 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only storage unit. The application stored in the memory 302 may include one or more program modules (not shown in the figure), such program modules include but are not limited to: an operating system, one or more applications, other program modules and program data, each of these examples or some combination may include the implementation of a network environment. Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the multi-camera half-face recognition device in VR. The multi-camera half-face recognition device in VR may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more I / O interfaces (input and output interfaces) 305, one or more external devices 306 (such as keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable users to interact with the device, and / or communicate with any device that enables the device to communicate with one or more other computing devices (such as routers, modems, etc.). Such communication may be performed through the I / O interface 305. In addition, the device may also communicate with one or more networks (such as a local area network (LAN)) through the wired or wireless interface 304.

[0122] Specifically in this embodiment, the multi-camera half-face recognition device in VR includes a memory 302, and one or more programs, wherein the one or more programs are stored in the memory 302, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the multi-camera half-face recognition device in VR, and is configured to be executed by one or more processors 301. The one or more programs include the following computer executable instructions:

[0123] Acquire multiple human eye images illuminated by infrared light and taken by multiple infrared cameras arranged in preset positions, wherein the multiple human eye images correspond to the left eye and the right eye respectively, and the human eye images include the eyes and the skin around the eyes; splice the multiple human eye images corresponding to the left eye and the right eye respectively to obtain first eye frontal images corresponding to the left eye and the right eye respectively, and splice the two first eye frontal images to obtain a second eye frontal image; acquire multiple first lower face images taken by multiple visible light cameras arranged in preset positions; splice the multiple first lower face images into a complete second lower face image image; extract the skin color in the second lower half face image, colorize the skin in the second eye front view image, perform color mapping on the eyes in the second eye front view image based on a pre-trained color transformation network between the human eye grayscale image and the real skin and real iris of the face, and splice the colored second eye front view image and the second lower half face image into a half face image; perform facial recognition on the half face image alone, or / and recognize the iris in the second eye front view image in combination to determine the identity of the person corresponding to the half face image, and output the half face image for virtual display.

[0124] Based on the same idea, the embodiment of this specification also provides a smart glasses, such as Figure 4 As shown, the smart glasses include: a plurality of infrared cameras 401, which are respectively directed to the left and right human eyes and arranged according to preset positions; an infrared floodlight emitter 402, which is arranged facing the human eyes; a plurality of visible light cameras 406, which are respectively directed to the lower half of the face and arranged according to preset positions; a display screen 403, which is used for displaying at the front end; one or more processors 404; a storage device 405, which is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors 404 implement the following method:

[0125] Acquire multiple human eye images illuminated by infrared light and taken by multiple infrared cameras arranged in preset positions, wherein the multiple human eye images correspond to the left eye and the right eye respectively, and the human eye images include the eyes and the skin around the eyes; splice the multiple human eye images corresponding to the left eye and the right eye respectively to obtain first eye frontal images corresponding to the left eye and the right eye respectively, and splice the two first eye frontal images to obtain a second eye frontal image; acquire multiple first lower face images taken by multiple visible light cameras arranged in preset positions; splice the multiple first lower face images into a complete second lower face image image; extract the skin color in the second lower half face image, colorize the skin in the second eye front view image, perform color mapping on the eyes in the second eye front view image based on a pre-trained color transformation network between the human eye grayscale image and the real skin and real iris of the face, and splice the colored second eye front view image and the second lower half face image into a half face image; perform facial recognition on the half face image alone, or / and recognize the iris in the second eye front view image in combination to determine the identity of the person corresponding to the half face image, and output the half face image for virtual display.

[0126] Based on the same idea, the exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0127] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0128] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0131] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0132] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0133] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0134] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0135] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0136] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0137] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A multi-camera half-face recognition method in VR, characterized in that: The method comprises: Acquire a plurality of human eye images illuminated by infrared light and taken by a plurality of infrared cameras arranged at preset positions, wherein the plurality of human eye images correspond to a left eye and a right eye respectively, and the human eye images include eyes and skin around the eyes; Splicing the plurality of human eye images corresponding to the left eye and the right eye respectively to obtain first eye frontal view images corresponding to the left eye and the right eye respectively, and splicing the two first eye frontal view images to obtain a second eye frontal view image; Acquire a plurality of first lower half face images captured by a plurality of visible light cameras arranged in preset positions; splicing the plurality of the first lower half face images into a complete second lower half face image; Extract the skin color in the second lower half face image, colorize the skin in the second eye front view image, perform color mapping on the eyes in the second eye front view image based on a pre-trained color transformation network between a human eye grayscale image and real face skin, and between a human eye grayscale image and a real iris, and splice the mapped and colored second eye front view image and the second lower half face image into a half face image, wherein the color transformation network is a neural network based on deep learning, and when training the color transformation network, collect human eye grayscale images and corresponding color face images as training data, wherein the color face images contain complete human eye iris color, eyebrow color, and face skin color, and the color transformation network is obtained by training using a deep learning framework, wherein the input of the color transformation network is a grayscale image, and the output is a corresponding color eye surrounding skin and color iris image; Performing facial recognition on the half-face image alone, or / and combining with the iris in the second eye frontal image for recognition, to determine the identity of the person corresponding to the half-face image, and outputting the half-face image for virtual display.

2. The multi-camera half-face recognition method in VR according to claim 1, characterized in that: The method of splicing the plurality of human eye images corresponding to the left eye and the right eye to obtain first eye frontal images corresponding to the left eye and the right eye respectively includes: Collecting features of the plurality of human eye images obtained of the same human eye, wherein the features include one or more of pupil contour, reflected light spots generated by infrared light irradiation, and texture shape and direction of the iris; Based on the collected features, register the multiple human eye images to obtain the spatial coordinate transformation parameters of each of the human eye images; Based on the spatial coordinate transformation parameters, the plurality of human eye images are transformed and spliced ​​to obtain the first eye frontal view image.

3. The multi-camera half-face recognition method in VR according to claim 1, characterized in that: The method of splicing the plurality of human eye images corresponding to the left eye and the right eye to obtain first eye frontal images corresponding to the left eye and the right eye respectively includes: According to different physical positions and camera parameters of the infrared cameras, for the human eye image taken by any of the infrared cameras, mark the image mask of the overlapping area with the images taken by other infrared cameras; For the human eye image taken by each infrared camera, human eye segmentation is performed based on the marked mask area, and the iris, pupil, and sclera areas therein are marked; Based on the physical positions and parameters of different infrared cameras, and based on the iris, pupil, and sclera regions marked in two adjacent human eye images, estimating the projection transformation matrix required for stitching human eye images taken by different infrared cameras; An iterative method is adopted to sequentially splice two adjacent human eye images according to the projection transformation matrix, so as to finally obtain a complete frontal view image of the first eye.

4. A multi-camera half-face recognition device in VR, characterized in that: include: A first image acquisition module, used to acquire a plurality of human eye images illuminated by infrared light, taken by a plurality of infrared cameras arranged at preset positions, wherein the plurality of human eye images correspond to a left eye and a right eye respectively, and the human eye images include eyes and skin around the eyes; A first stitching module is used to stitch the plurality of human eye images corresponding to the left eye and the right eye respectively to obtain first eye frontal view images corresponding to the left eye and the right eye respectively, and stitch the two first eye frontal view images to obtain a second eye frontal view image; A second image acquisition module, used to acquire a plurality of first lower half face images captured by a plurality of visible light cameras arranged at preset positions; A second stitching module, used for stitching the plurality of the first lower half face images into a complete second lower half face image; a half-face fusion module, for extracting the skin color in the second lower half face image, coloring the skin in the second eye front view image, color mapping the eyes in the second eye front view image based on a pre-trained color transformation network between a human eye grayscale image and real face skin, and between a human eye grayscale image and a real iris, and splicing the mapped and colored second eye front view image and the second lower half face image into a half-face image, wherein the color transformation network is a neural network based on deep learning, and when training the color transformation network, human eye grayscale images and corresponding color face images are collected as training data, wherein the color face images contain complete human eye iris color, eyebrow color, and face skin color, and the color transformation network is obtained by training using a deep learning framework, wherein the input of the color transformation network is a grayscale image, and the output is a corresponding color eye surrounding skin and color iris image; A recognition module is used to perform facial recognition on the half-face image alone, or / and recognize the iris in the second eye frontal image to determine the identity of the person corresponding to the half-face image, and output the half-face image for virtual display.

5. A smart glasses, characterized in that: include: Multiple infrared cameras are set up to point at the left and right eyes respectively and arranged in preset positions; Infrared flood emitter, set towards the human eye; Multiple visible light cameras are respectively set to point at the lower half of the face and arranged in preset positions; Display screen; one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the multi-camera half-face recognition method in VR as described in claim 1.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-camera half-face recognition method in VR is implemented as described in any one of claims 1 to 3.