Image processing method and apparatus, storage medium, and electronic device
Through matrix conversion and image fusion processing of depth cameras and color cameras, the problem of large computing volume and low frame rate in high-resolution virtual viewpoint image generation in holographic remote video communication is solved, and the efficiency and user experience of virtual viewpoint image generation algorithm are improved. .
Patent Information
- Application Number
- PCT/CN2024/088397
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
In holographic remote video communication scenarios, the high-resolution virtual viewpoint image generation algorithm for dynamic scenes is too large, resulting in the system being unable to realize real-time display, resulting in display stuttering.
By acquiring the images collected by the depth camera and the color camera, the three-dimensional point cloud is projected to the color image based on the matrix transformation relationship between the two, the user's sight is determined and the image fusion process is performed, high-definition gaze area images and low-definition images are generated, and finally they are fused. The calculation amount is reduced for the virtual viewpoint image to be displayed.
It effectively reduces the computing amount of virtual viewpoint image generation, improves frame rate, improves user experience, and realizes high-definition display of user gaze area and low-resolution generation of non-gaze area.
Smart Images

Figure CN2024088397_05122024_PF_FP_ABST
Abstract
Description
Image processing method and device, storage medium, and electronic device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] The present disclosure claims priority from application number 202310622670.6, application date May 29, 2023, and invention title: Image processing method and device, storage medium, and electronic device. The entire contents of the Chinese patent application are incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a storage medium, and an electronic device. Background Art
[0004] In application scenarios such as remote video communication and VR games, holographic images can be created for users on the call, providing users with a complete and realistic visual experience. Taking the scenario of holographic remote video communication as an example, multiple cameras (color cameras or depth cameras) are generally used to capture high-definition images of users and scenes. After encoding, the images are sent to the user device at the other end via the network. The original viewpoint images are decoded and restored on the device, and one or more specified virtual viewpoint images are generated according to user needs. The images are displayed on the naked-eye 3D display device to present a holographic communication effect to the user. However, the current high-resolution virtual viewpoint image generation algorithm for dynamic scenes is usually completed using a deep learning algorithm. This results in a long time-consuming process when generating high-resolution (1k*1k or above) virtual viewpoint images due to the large amount of computation, making it impossible to achieve real-time display of the system, resulting in display freezes.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.
[0006] Summary of the Invention
[0007] According to a first aspect of the present disclosure, there is provided an image processing method, the method comprising:
[0008] Acquire a target depth image captured by a depth camera and a target color image captured by a color camera; wherein the target color image includes multiple color images captured by multiple color cameras respectively;
[0009] Based on the matrix transformation relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected onto the target color image, and the input viewpoint image is determined according to the matching projection points;
[0010] Determining the user's line of sight, and determining a second gaze area corresponding to the input viewpoint image based on the user's line of sight, and performing image fusion processing based on the gaze area image corresponding to the second gaze area to obtain a high-definition gaze area image; and
[0011] Scaling and fusing the input viewpoint image to obtain a low-definition image;
[0012] The high-definition gaze area image and the low-definition image are subjected to image fusion processing to obtain a virtual viewpoint image to be displayed.
[0013] In some exemplary embodiments, determining the user's line of sight and determining the corresponding second gaze area in the input viewpoint image based on the user's line of sight includes:
[0014] A first gaze area of a user's line of sight in a current virtual viewpoint image on a display screen is determined, and the first gaze area is projected onto the target color image to determine a second gaze area in the input viewpoint image.
[0015] In some exemplary embodiments, obtaining a first gaze area of a current virtual viewpoint image of a user's line of sight on a display screen includes:
[0016] Performing key point detection on the facial image captured by the first camera to determine the key point coordinates of the key points in the facial image coordinate system;
[0017] Performing circular fitting processing according to the key point coordinates of the eye key points to determine the pupil coordinates of the eye pupil in the first camera coordinate system;
[0018] performing plane fitting processing based on the pixel coordinates of the key points of the eye key points, and determining the eye center coordinates of the eye center point in the face coordinate system based on the fitting plane; determining the eye center coordinates of the eye center point in the first camera coordinate system based on the coordinate conversion relationship between the first camera and the face coordinate system;
[0019] According to the coordinate conversion relationship between the first camera coordinate system and the screen coordinate system, the line of sight direction in the screen coordinate system is determined according to the pupil coordinates and the coordinates of the center of the human eye;
[0020] Delineate gaze areas based on gaze direction.
[0021] In some exemplary embodiments, performing key point detection on the facial image captured by the first camera to determine the key point coordinates of the key points in the facial image coordinate system includes:
[0022] Performing face detection on the face image to determine pixel coordinates of preset key points in the image coordinate system;
[0023] Determine the key point coordinates of the key point in the first camera coordinate system based on the coordinate transformation relationship between the image coordinate system and the first camera coordinate system;
[0024] A transformation matrix between the face coordinate system and the first camera coordinate system is obtained, so as to use the transformation matrix to obtain the key point coordinates of the key point in the face coordinate system.
[0025] In some exemplary embodiments, determining the pupil coordinates of the pupil of the eye in the first camera coordinate system by performing a circular fitting process based on the key point coordinates of the eye key points includes:
[0026] Obtain an eye region image according to a set of eye key points;
[0027] Performing a circular fitting process on the eye region image to obtain pixel coordinates of the pupil in the image coordinate system;
[0028] Calculate an average depth value based on the coordinate information of each key point in the eye key point set in the first camera coordinate system, and configure the depth value as the depth value of the eye pupil;
[0029] The pupil coordinates of the eye pupil in the first camera coordinate system are determined by combining the pixel coordinates of the eye pupil in the image coordinate system and the depth value of the eye pupil.
[0030] In some exemplary embodiments, determining the eye center coordinates of the eye center in the face coordinate system based on the fitting plane includes:
[0031] Determining the geometric center of the fitting plane;
[0032] Constructing a perpendicular line of the fitting plane through the geometric center in the first direction;
[0033] The center point of the human eye is configured at a preset length of the vertical line to obtain the center coordinates of the human eye in the face coordinate system.
[0034] In some exemplary embodiments, projecting the original three-dimensional point cloud corresponding to the target depth image onto the target color image based on the matrix transformation relationship between the depth camera and the color camera to determine the input viewpoint image according to the matching projection points includes:
[0035] Based on the conversion relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected onto each of the target color images to obtain the depth information of each matching projection point in each target color image;
[0036] The target color image is projected onto a preset virtual viewpoint image according to matching projection points in the target color image to obtain a plurality of input viewpoint images.
[0037] In some exemplary embodiments, the method further includes: performing image recognition on the target color image, and clearing a background area according to a result of the image recognition, so as to calculate an input viewpoint image according to the target color image after clearing the background area.
[0038] According to a second aspect of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:
[0039] An image data acquisition module is used to acquire a target depth image acquired by a depth camera and a target color image acquired by a color camera; wherein the target color image includes multiple color images acquired by several color cameras respectively;
[0040] A projection transformation module is used to project the original three-dimensional point cloud corresponding to the target depth image to the target color image based on the matrix transformation relationship between the depth camera and the color camera, and determine the input viewpoint image according to the matching projection points;
[0041] a high-definition image acquisition module, configured to determine a user's line of sight, determine a second gaze area corresponding to the input viewpoint image based on the user's line of sight, and perform image fusion processing based on a gaze area image corresponding to the second gaze area to acquire a high-definition gaze area image; and
[0042] A low-definition image acquisition module, configured to perform scaling and fusion processing on the input viewpoint image to acquire a low-definition image;
[0043] The virtual viewpoint image generation module is used to perform image fusion processing on the high-definition gaze area image and the low-definition image to obtain a virtual viewpoint image to be displayed.
[0044] According to a third aspect of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method described above is implemented.
[0045] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:
[0046] processor; and
[0047] a memory for storing executable instructions of the processor;
[0048] Wherein, the processor is configured to implement the above-mentioned image processing method by executing the executable instructions.
[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:
[0051] FIG1 schematically shows a diagram of an image processing method in an exemplary embodiment of the present disclosure.
[0052] FIG2 schematically shows a schematic diagram of a system architecture according to an exemplary embodiment of the present disclosure.
[0053] FIG3 schematically shows a schematic diagram of camera position distribution according to an exemplary embodiment of the present disclosure.
[0054] FIG4 schematically shows a schematic diagram of a facial key point position distribution in an exemplary embodiment of the present disclosure.
[0055] FIG5 schematically shows a schematic diagram of a gaze area in an exemplary embodiment of the present disclosure.
[0056] FIG6 schematically shows a composition diagram of an image processing device in an exemplary embodiment of the present disclosure.
[0057] FIG7 schematically shows a composition diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0059] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0060] In the field of holographic remote video communication, in order to provide customers with a complete and realistic visual experience, multiple cameras are typically used to capture high-definition images of the user and the scene. These images are then encoded and transmitted over the network to the user device at the other end. The original viewpoint images are then decoded and restored on the device. Based on user needs, one or more specified virtual viewpoint images are generated and displayed on a naked-eye 3D display device to present a holographic communication effect to the user. However, current algorithms for generating high-resolution virtual viewpoint images for dynamic scenes typically use deep learning algorithms. This results in a long computational overhead when generating high-resolution virtual viewpoint images, making it impossible to achieve real-time system performance and causing display lag.
[0061] In view of the shortcomings and deficiencies of the prior art, this exemplary embodiment provides an image processing method that can be applied to the generation of holographic images. Referring to FIG1 , the provided image processing method may include:
[0062] Step S11, acquiring a target depth image captured by a depth camera and a target color image captured by a color camera; wherein the target color image includes a plurality of color images captured by a plurality of color cameras respectively;
[0063] Step S12: projecting the original three-dimensional point cloud corresponding to the target depth image onto the target color image based on the matrix conversion relationship between the depth camera and the color camera, and determining the input viewpoint image according to the matching projection points;
[0064] Step S13, determining the user's line of sight, and determining a second gaze area corresponding to the input viewpoint image according to the user's line of sight, performing image fusion processing based on the gaze area image corresponding to the second gaze area to obtain a high-definition gaze area image; and
[0065] Step S14, scaling and fusing the input viewpoint image to obtain a low-definition image;
[0066] Step S15 , performing image fusion processing on the high-definition gaze area image and the low-definition image to obtain a virtual viewpoint image to be displayed.
[0067] The image processing method provided in this example embodiment projects the original three-dimensional point cloud corresponding to the target depth image onto each target color image to determine the corresponding multiple input viewpoint images based on the projection points, and then performs image fusion based on the image of the second gaze area of the user's line of sight on each input viewpoint image, so as to obtain a high-definition gaze area image of the user's gaze area; then, each input viewpoint image is scaled and fused to obtain a low-definition image; and then, the high-definition gaze area image and the low-definition image are fused to obtain a virtual viewpoint image, so that the user's gaze area can be generated at the original size while the non-gaze area is reduced to a specified size, thereby effectively reducing the amount of data involved in the calculation and reducing the amount of calculation of the model.
[0068] Below, each step of the image processing method in this exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.
[0069] In step S11, a target depth image captured by a depth camera and a target color image captured by a color camera are acquired; wherein the target color image includes a plurality of color images respectively captured by a plurality of color cameras.
[0070] In this example embodiment, referring to the system architecture shown in FIG2 , in a remote video communication scenario based on holographic imaging, a transmitter 201 and a receiver 202 may be provided. Data can be transmitted between the transmitter 201 and the receiver 202 via a network such as a wired communication link or a wireless communication link. Referring to FIG3 , the terminal devices of the transmitter 201 and the receiver 202 may include a display screen. A color camera C0 is provided at the center of the upper portion of the display screen as a face capture camera. A depth camera D0 is provided below camera C0 to obtain depth information of the user's gaze area. Color cameras C1-C4 are symmetrically arranged at each corner of the display screen to capture original viewpoint images. For each camera, the user's experience area is approximately 1 meter away from the screen. The angle of each camera can be adjusted so that the optical axis of each camera intersects at the center of the human body, and the viewing angle of each camera can fully cover the area where the human body is located. The display screen may be a naked-eye 3D display device.
[0071] On each terminal device side, the transformation matrix between each camera (color camera, depth camera) can be calibrated in advance using a calibration plate and a plane mirror. j T i ,i≠j,i,j=C0,C1,C2,C3,C4,D0; and the internal parameters K of each camera i , i=C0,C1,C2,C3,C4,D0; and the transformation matrix from the color camera C0 coordinate system to the screen coordinate system s T C0 The virtual viewpoint position can be set according to user needs. C0T n OK, the internal reference is K n .
[0072] The terminal device can control the color camera and depth camera arranged on the display screen to synchronously capture color and depth images and transmit them to the terminal device at the other end in real time. At the same time, the intrinsic parameters of each camera and the transformation matrix between each camera coordinate system can be transmitted to the terminal device at the other end.
[0073] Alternatively, in some exemplary embodiments, the method may further include: performing image recognition on the target color image, and clearing a background area according to an image recognition result, so as to calculate an input viewpoint image according to the target color image after clearing the background area.
[0074] Specifically, for the terminal device, after using the color cameras C1-C4 and the depth camera D0 to capture the color image and the depth image, it can first perform image recognition processing, extract the human body area in the image, and delete the background area; then send the target color image containing only the human body area to the opposite device. This can reduce the amount of data transmitted for the image and improve data transmission efficiency. Alternatively, after receiving the target color image, the terminal device can first perform background recognition and delete the background area in the color image, retaining only the human body area as the target color image. This can improve the efficiency of subsequent projection point calculations. For example, classic background removal algorithms such as Back Ground Matting can be used to remove the background area and retain only the user's body area in the color image.
[0075] In step S12, based on the matrix conversion relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected onto the target color image, and the input viewpoint image is determined according to the matching projection points.
[0076] In this example implementation, the above step S12 may include:
[0077] Step S121: Based on the conversion relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected onto each of the target color images to obtain the depth information of each matching projection point in each target color image;
[0078] Step S122 : projecting the matching projection points in the target color image onto a preset virtual viewpoint image to obtain a plurality of input viewpoint images.
[0079] Specifically, the color images captured by color cameras C1-C4, that is, the target color images, can be used as original viewpoint images. The target depth images captured by the depth cameras can first be converted into corresponding original three-dimensional point cloud data. For the acquired original three-dimensional point cloud data, the transformation matrix between the pre-calibrated depth camera coordinates and the color camera coordinate system and the camera intrinsic parameters can be used to project the user's body area in the original three-dimensional point cloud data onto each target color image, thereby obtaining the depth information corresponding to the pixel points in the human body area on each color image. Specifically, when projecting the original three-dimensional point cloud and the target color image, if the projection point is not within the human body area, the pixel point can be discarded from the target color image. After obtaining the depth information of each pixel point in the human body area on each target color image, it can be projected to the desired virtual viewpoint image, and its pixel value is the color on the target color image. At this point, the input viewpoint image corresponding to each target color image can be obtained.
[0080] Alternatively, when the target color image for the projection operation is a human body area image, when the depth image is projected onto the target color image, if any pixel point in the human body area on the color image lacks a corresponding projection point, the depth information corresponding to the projection point closest to the pixel point can be selected as the depth information of the pixel point.
[0081] In step S13, the user's sight line is determined, and a second gaze area corresponding to the input viewpoint image is determined according to the user's sight line. Image fusion processing is performed based on the gaze area image corresponding to the second gaze area to obtain a high-definition gaze area image.
[0082] In this example embodiment, the terminal device can use the images captured by camera C0 in real time to calculate the user's current gaze area on the display screen. Therefore, step S13 can be performed simultaneously with step S12; alternatively, the terminal device can pre-calculate the user's corresponding gaze area.
[0083] In this example embodiment, determining the user's line of sight and determining the corresponding second gaze area in the input viewpoint image based on the user's line of sight includes: determining a first gaze area of the user's line of sight in the current virtual viewpoint image on the display screen, projecting the first gaze area onto the target color image to determine the second gaze area in the input viewpoint image.
[0084] Specifically, obtaining a first gaze area of a current virtual viewpoint image of a user's sight on the display screen includes:
[0085] Step S21, performing key point detection on the face image captured by the first camera to determine the key point coordinates of the key points in the face image coordinate system;
[0086] Step S22, performing a circular fitting process based on the key point coordinates of the eye key points to determine the pupil coordinates of the eye pupil in the first camera coordinate system;
[0087] Step S23, performing plane fitting processing based on the pixel coordinates of the key points of the eye key points, and determining the eye center coordinates of the eye center point in the facial coordinate system based on the fitted plane; and determining the eye center coordinates of the eye center point in the first camera coordinate system based on the coordinate conversion relationship between the first camera and the facial coordinate system;
[0088] Step S24, determining the sight direction in the screen coordinate system according to the pupil coordinates and the eye center coordinates based on the coordinate conversion relationship between the first camera coordinate system and the screen coordinate system;
[0089] Step S25: defining a gaze area according to the sight direction.
[0090] In this example implementation, in the above step S21, key point detection is performed on the facial image captured by the first camera to determine the key point coordinates of the key points in the facial image coordinate system, including:
[0091] Performing face detection on the face image to determine pixel coordinates of preset key points in the image coordinate system;
[0092] Determine the key point coordinates of the key point in the first camera coordinate system based on the coordinate transformation relationship between the image coordinate system and the first camera coordinate system;
[0093] A transformation matrix between the face coordinate system and the first camera coordinate system is obtained, so as to use the transformation matrix to obtain the key point coordinates of the key point in the face coordinate system.
[0094] Specifically, the first camera may be the color camera C0 mentioned above. Referring to FIG4 , the corresponding key point distribution position may be pre-configured for the face. When the color camera C0 is used to capture the face image, distortion correction may be performed first. For the face image after distortion correction, the coordinates of each pixel in the face image coordinate system may be obtained. Based on the key point distribution shown in FIG4 , the pixel coordinates (u0, v0) to (u 67 ,v 67). The coordinates of each key point in the C0 camera coordinate system can be determined by using the internal parameters of camera C0; the pixel coordinates of each key point in the face image are matched one-to-one with the face template in the face coordinate system, and the face template in the face coordinate system can be matched one-to-one with each key point in the face image by using the pose estimation algorithm, so as to obtain the pose of the face in the camera C0 coordinate system; the transformation matrix from the face coordinate system to the camera C0 coordinate system can be estimated based on the coordinate information of the face template at each key point in the face coordinate system and the pose information of the face in the camera C0 coordinate system. C0 T face Based on the transformation matrix, the three-dimensional coordinates (X0, Y0, Z0)~(X 67 ,Y 67 ,Z 67 ).
[0095] In this example implementation, in the above step S22, performing a circular fitting process based on the key point coordinates of the eye key points to determine the pupil coordinates of the eye pupil in the first camera coordinate system includes:
[0096] Obtain an eye region image according to a set of eye key points;
[0097] Performing a circular fitting process on the eye region image to obtain pixel coordinates of the pupil in the image coordinate system;
[0098] Calculate an average depth value based on the coordinate information of each key point in the eye key point set in the first camera coordinate system, and configure the depth value as the depth value of the eye pupil;
[0099] The pupil coordinates of the eye pupil in the first camera coordinate system are determined by combining the pixel coordinates of the eye pupil in the image coordinate system and the depth value of the eye pupil.
[0100] Specifically, referring to FIG4 , the key points 37-42 corresponding to the left eye region can be obtained by using traditional image algorithms such as binarization, erosion, and expansion to obtain the pupil region, and then perform circular fitting to obtain the pixel coordinates (u $ ,v $ ), take the average depth of the three-dimensional key points 37 to 42 as the depth of the current left eye pupil in the camera C0 coordinate system, and according to the internal parameter K C0 Calculate its three-dimensional coordinates P $ for (X $ ,Y $ ,Z $ Correspondingly, the key points 43-48 in the right eye area can use the same method to obtain the three-dimensional coordinates of the right eye through hole in the camera C0 coordinate system.
[0101] In this example implementation, in the above step S23, determining the eye center coordinates of the eye center in the face coordinate system based on the fitting plane may specifically include:
[0102] Determining the geometric center of the fitting plane;
[0103] Constructing a perpendicular line of the fitting plane through the geometric center in the first direction;
[0104] The center point of the human eye is configured at a preset length of the vertical line to obtain the center coordinates of the human eye in the face coordinate system.
[0105] Specifically, in the face coordinate system, for the left eye area, the plane P can be fitted using key points 37 to 42, and the geometric center Center can be calculated. A perpendicular line L is drawn from the plane to the inside of the head through Center. Given that the diameter of the human eye is approximately 21 to 25 mm, the perpendicular line with a length of 23 mm is taken as the center of the human eye C. e , then the coordinates of the left eye center point are Similarly, using key points 43-48, the coordinates of the right eye center point in the face coordinate system can be determined according to the above method. Then, based on the transformation matrix between the camera C0 coordinate system and the face coordinate system obtained above, C0 T face , the eye center coordinates of the left eye center and the right eye center in the camera C0 coordinate system can be calculated.
[0106] In this exemplary embodiment, in the above step S24, in the camera C0 coordinate system, according to the left eye pupil coordinate P $ The left eye's sight line can be determined by the coordinates of the left eye's center point and the coordinates of the left eye's through hole. Similarly, the right eye's sight line can be determined by the coordinates of the right eye's through hole and the coordinates of the right eye's center point. Based on the pre-calibrated transformation matrix between the camera's C0 coordinate system and the screen's coordinate system s T C0 , transforming the left eye sight and the right eye sight into the screen coordinate system respectively, so that the gaze points of the left eye sight and the right eye sight on the screen can be determined, that is, the user's current gaze point on the virtual viewpoint image.
[0107] In this example implementation, in step S25, a gaze line is established based on the left and right eye gaze points in the screen coordinate system. The midpoint of the gaze line is used as the center of the gaze area. Referring to FIG5 , the r*r rectangular area is defined as the gaze area of the user's eye on the screen at that moment. Here, r can be set based on user needs. After obtaining the coordinates of the gaze area in the screen coordinate system, the gaze area Rect on the virtual viewpoint image can be determined based on the screen resolution and physical size of the display device. This is the first gaze area of the current virtual viewpoint image displayed on the display screen by the user's line of sight. For example, the size of the first gaze area can be set to 256*256.
[0108] In this example implementation, in step S13, for the acquired target depth image, the corresponding three-dimensional point cloud data can be transformed by the transformation matrix n T D Convert to the virtual viewpoint coordinate system; and convert to the virtual viewpoint image according to the virtual viewpoint internal parameters. Get the 3D point cloud in the gaze area Rect on the virtual viewpoint image, and then transform the point cloud through the transformation matrix i T n ,i=C1,C2,C3,C4 and internal reference K i , i=C1,C2,C3,C4 are projected onto the input viewpoint image, and the projection point is surrounded by the minimum rectangular frame. To ensure the consistency of size, the center of the minimum rectangular frame is used as the center to select the same size of Rect as the gaze area Rect on each input viewpoint image i , i=C1,C2,C3,C4. Thus, the gaze area image of the second gaze area corresponding to each input viewpoint image is obtained. Among them, the transformation matrix n T D It can be the transformation matrix between the depth camera coordinate system and the virtual viewpoint coordinate system. The virtual viewpoint coordinate system can be the coordinate system corresponding to the virtual camera, and the virtual viewpoint image is the desired virtual viewpoint image, that is, the image desired to be displayed on the naked eye 3D device.
[0109] A virtual viewpoint image generation model based on the UNet network can be pre-trained. The gaze area images corresponding to the four input viewpoint images can be used as input to the model. They are spliced according to the channel dimension to form 256*256*12 input data. The model is used to perform image fusion processing and output a high-definition gaze area image. Generally speaking, a UNet network can include a backbone feature extraction network, which uses the backbone part to downsample to obtain multiple feature layers. Each feature extraction layer that performs downsampling can be a stack of convolution and maximum pooling. Using the backbone feature extraction part, we can obtain five preliminary valid feature layers. The UNet network can also include an enhanced feature extraction network, which uses the five obtained preliminary valid feature layers for upsampling and feature fusion to obtain a final valid feature layer that integrates all features. The UNet network can also include a prediction network, which uses the final valid feature layer obtained to classify each feature point, which is equivalent to classifying each pixel point.
[0110] In step S14, the input viewpoint image is scaled and fused to obtain a low-definition image.
[0111] In this example embodiment, each input viewpoint image can be scaled. For example, the original size of 1024*1024 can be reduced to 256*256. The scaled input viewpoint image is then input into the UNet-based image fusion model, which outputs a fused image of 256*256. This image is then stretched to 1024*1024, thereby obtaining a low-resolution image of the non-interest area.
[0112] In step S15, image fusion processing is performed on the high-definition gaze area image and the low-definition image to obtain a virtual viewpoint image to be displayed.
[0113] In this example embodiment, after obtaining a high-definition gaze area image and a low-definition image, the two images can be fused to produce a fused virtual viewpoint image. To smooth the transition between the high-definition and low-definition areas, a mean filter can be applied to the junction of the high-definition and low-definition areas, with a processing width of 16 pixels, or 1 / 16 of the side length of the high-definition area. In the generated virtual viewpoint image, the area of the user's gaze is the high-definition image, and the remaining areas are the low-definition image.
[0114] The image processing method provided by the embodiments of the present disclosure can be applied to the generation of virtual viewpoint images in naked-eye 3D scenes. A facial image can be captured using color camera C0, and the facial image can be used to calculate the user's gaze area on the screen. This can then determine the user's gaze area in the virtual viewpoint image. For the depth and color images captured by depth camera D0 and color cameras C1-C4, background recognition can first be performed on each image, and the background area can be deleted, retaining the human body area image. The depth image can then be projected onto each color image using the transformation matrix and memory between the cameras to determine the matching projection points. This allows the input viewpoint image to be obtained, and the corresponding gaze area on the input viewpoint image is then used as a high-definition image as input to the UNet network model, thereby outputting a high-definition image corresponding to the user's gaze area. For example, the original image resolution can be 1024*1024. A 256*256 gaze area image is captured from each input viewpoint image and then fed into the model for fusion processing, outputting a 256*256 high-definition gaze area image. At the same time, each input viewpoint image can be scaled from 1024*1024 to 256*256. The output image, which was originally 256*256, is then stretched to 1024*1024 in the input model to produce a low-resolution image, achieving hierarchical processing of high-definition and low-definition areas. The high-definition and low-definition images are then fused to produce the desired virtual viewpoint image. This image ensures that the area where the user is looking is high-definition, while the remaining areas are low-definition. This provides a virtual viewpoint image generation scheme based on the gaze area, addressing the computationally expensive and low-frame-rate issues associated with generating high-resolution virtual viewpoint images in holographic remote video communication scenarios. This significantly improves the frame rate of the virtual viewpoint image generation algorithm and enhances the user experience. By analyzing the user's gaze direction, the user's gaze area on the screen (i.e., the virtual viewpoint image) is determined. The virtual viewpoint image is generated at its original size for the area of interest, while the less important areas outside the gaze area are compressed to a lower resolution, effectively reducing the computational complexity of the model.
[0115] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0116] Furthermore, referring to FIG6 , an image processing device 60 is provided in the embodiment of this example, which can be applied to a terminal device; the device includes: an image data acquisition module 601, a projection transformation module 602, a high-definition image acquisition module 603, a low-definition image acquisition module 604, and a virtual viewpoint image generation module 605. Among them:
[0117] The image data acquisition module 601 can be used to acquire a target depth image captured by a depth camera and a target color image captured by a color camera; wherein the target color image includes multiple color images captured by several color cameras respectively.
[0118] The projection transformation module 602 can be used to project the original three-dimensional point cloud corresponding to the target depth image to the target color image based on the matrix transformation relationship between the depth camera and the color camera, and determine the input viewpoint image according to the matching projection points.
[0119] The high-definition image acquisition module 603 can be used to determine the user's line of sight, and determine the corresponding second gaze area in the input viewpoint image based on the user's line of sight, and perform image fusion processing based on the gaze area image corresponding to the second gaze area to obtain a high-definition gaze area image.
[0120] The low-definition image acquisition module 604 may be configured to perform scaling and fusion processing on the input viewpoint image to acquire a low-definition image.
[0121] The virtual viewpoint image generation module 605 may be configured to perform image fusion processing on the high-definition gaze area image and the low-definition image to obtain a virtual viewpoint image to be displayed.
[0122] In some exemplary embodiments, the high-definition image acquisition module 603 can be used to determine a first gaze area of the user's line of sight in the current virtual viewpoint image on the display screen, and project the first gaze area onto the target color image to determine a second gaze area in the input viewpoint image.
[0123] In some exemplary embodiments, the high-definition image acquisition module 603 can be used to perform key point detection on the facial image captured by the first camera to determine the key point coordinates of the key points in the facial image coordinate system; perform circular fitting processing based on the key point coordinates of the eye key points to determine the pupil coordinates of the eye pupil in the first camera coordinate system; perform plane fitting processing based on the key point pixel coordinates of the eye key points, and determine the eye center coordinates of the human eye center point in the facial coordinate system based on the fitting plane; based on the coordinate conversion relationship between the first camera and the facial coordinate system, determine the eye center coordinates of the human eye center point in the first camera coordinate system; based on the coordinate conversion relationship between the first camera coordinate system and the screen coordinate system, determine the line of sight direction in the screen coordinate system based on the pupil coordinates and the eye center coordinates; and define the gaze area based on the line of sight direction.
[0124] In some exemplary embodiments, the high-definition image acquisition module 603 can be used to perform face detection on the facial image, determine the pixel coordinates of the preset key points in the image coordinate system; determine the key point coordinates of the key points in the first camera coordinate system based on the coordinate transformation relationship between the image coordinate system and the first camera coordinate system; obtain the transformation matrix between the facial coordinate system and the first camera coordinate system, and use the transformation matrix to obtain the key point coordinates of the key points in the facial coordinate system.
[0125] In some exemplary embodiments, the high-definition image acquisition module 603 can be used to acquire an eye area image based on a set of eye key points; perform circular fitting processing on the eye area image to obtain the pixel coordinates of the eye pupil in the image coordinate system; calculate the average depth value based on the coordinate information of each key point in the eye key point set in the first camera coordinate system, and configure the depth value as the depth value of the eye pupil; determine the pupil coordinates of the eye pupil in the first camera coordinate system in combination with the pixel coordinates of the eye pupil in the image coordinate system and the depth value of the eye pupil.
[0126] In some exemplary embodiments, the high-definition image acquisition module 603 can be used to determine the geometric center of the fitting plane; construct a perpendicular line to the fitting plane in a first direction through the geometric center; and configure the human eye center point at a preset length of the perpendicular line to obtain the human eye center coordinates of the human eye center point in the face coordinate system.
[0127] In some exemplary embodiments, the projection transformation module 602 can be used to project the original three-dimensional point cloud corresponding to the target depth image to each of the target color images based on the conversion relationship between the depth camera and the color camera to obtain the depth information of each matching projection point in each target color image; and project the matching projection points in the target color image to a preset virtual viewpoint image to obtain multiple input viewpoint images.
[0128] In some exemplary embodiments, the apparatus further includes an image background removal module configured to perform image recognition on the target color image and remove a background region based on the image recognition result, so as to calculate an input viewpoint image based on the target color image after removing the background region.
[0129] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0130] FIG. 7 shows a schematic diagram of an electronic device suitable for implementing an embodiment of the present invention.
[0131] It should be noted that the electronic device 1000 shown in FIG7 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0132] As shown in Figure 7, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. For example, the central processing unit 101 can execute the steps shown in Figure 1 to implement the above-mentioned pedestrian detection method. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.
[0133] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card or a modem. The LCD can be a naked-eye 3D display device. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed in the drive 1010 as needed so that computer programs read therefrom can be installed into the storage section 1008 as needed.
[0134] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a storage medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009 and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the various functions defined in the system of the present application are performed.
[0135] Specifically, the electronic device can be a smart device such as a server, tablet computer, or laptop computer, and can execute the aforementioned method for managing IoT device interactions applied to a proxy server or IoT platform. Alternatively, the electronic device can be an IoT device and can execute the aforementioned method for managing IoT device interactions applied to an IoT device.
[0136] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.
[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0138] The units involved in the embodiments of the present invention may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not limit the units themselves.
[0139] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device or can exist independently without being installed in the electronic device. The above-mentioned storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method applied to a proxy server, an IoT platform, or an IoT device.
[0140] Furthermore, the figures above are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0141] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.
[0142] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, It is characterized in that The method comprises: Acquire a target depth image acquired by a depth camera and a target color image acquired by a color camera; wherein the target color image includes a plurality of color images acquired by a plurality of color cameras respectively; Based on the matrix conversion relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected to the target color image, and the input viewpoint image is determined according to the matching projection points; Determine the user's line of sight, and determine a second gaze area corresponding to the input viewpoint image according to the user's line of sight, and perform image fusion processing based on a gaze area image corresponding to the second gaze area to obtain a high-definition gaze area image; and Scaling and fusing the input viewpoint image to obtain a low-definition image; The high-definition gaze area image and the low-definition image are subjected to image fusion processing to obtain a virtual viewpoint image to be displayed.
2. The image processing method according to claim 1, It is characterized in that The determining of the user's sight line, and determining a corresponding second gaze area in the input viewpoint image according to the user's sight line, includes: A first gaze area of a user's line of sight in a current virtual viewpoint image on a display screen is determined, and the first gaze area is projected onto the target color image to determine a second gaze area in the input viewpoint image.
3. The image processing method according to claim 2, It is characterized in that The step of obtaining a first gaze area of a current virtual viewpoint image on a display screen by a user's line of sight includes: Perform key point detection on the face image captured by the first camera to determine the key point coordinates of the key points in the face image coordinate system; Performing circular fitting processing according to the key point coordinates of the eye key points to determine the pupil coordinates of the eye pupil in the first camera coordinate system; Perform plane fitting processing according to the pixel coordinates of the key points of the eye key points, and determine the eye center coordinates of the eye center point in the face coordinate system according to the fitting plane; determine the eye center coordinates of the eye center point in the first camera coordinate system based on the coordinate conversion relationship between the first camera and the face coordinate system; According to the coordinate conversion relationship between the first camera coordinate system and the screen coordinate system, the sight direction in the screen coordinate system is determined according to the pupil coordinates and the eye center coordinates; Delineate gaze areas based on gaze direction.
4. The image processing method according to claim 3, It is characterized in that The step of performing key point detection on the face image captured by the first camera to determine the key point coordinates of the key points in the face image coordinate system includes: Performing face detection on the face image to determine pixel coordinates of preset key points in the image coordinate system; Determine the key point coordinates of the key point in the first camera coordinate system based on the coordinate transformation relationship between the image coordinate system and the first camera coordinate system; A transformation matrix between the face coordinate system and the first camera coordinate system is obtained, so as to use the transformation matrix to obtain the key point coordinates of the key point in the face coordinate system.
5. The image processing method according to claim 3, It is characterized in that The determining the pupil coordinates of the pupil of the eye in the first camera coordinate system by performing circular fitting processing according to the key point coordinates of the eye key points includes: Obtain an eye region image according to a set of eye key points; Performing a circular fitting process on the eye region image to obtain the pixel coordinates of the eye pupil in the image coordinate system; Calculate an average depth value according to the coordinate information of each key point in the eye key point set in the first camera coordinate system, and configure the depth value as the depth value of the eye pupil; The pupil coordinates of the eye pupil in the first camera coordinate system are determined by combining the pixel coordinates of the eye pupil in the image coordinate system and the depth value of the eye pupil.
6. The image processing method according to claim 3, It is characterized in that The step of determining the eye center coordinates of the eye center in the face coordinate system according to the fitting plane includes: Determining the geometric center of the fitting plane; Constructing a perpendicular line of the fitting plane through the geometric center toward the first direction; The human eye center point is configured at a preset length of the vertical line to obtain the human eye center coordinates of the human eye center point in the face coordinate system.
7. The image processing method according to claim 1, It is characterized in that The method of projecting the original three-dimensional point cloud corresponding to the target depth image to the target color image based on the matrix conversion relationship between the depth camera and the color camera to determine the input viewpoint image according to the matching projection points includes: Based on the conversion relationship between the depth camera and the color camera, the original three-dimensional point cloud corresponding to the target depth image is projected onto each of the target color images to obtain the depth information of each matching projection point in each target color image; The target color image is projected onto a preset virtual viewpoint image according to matching projection points in the target color image to obtain a plurality of input viewpoint images.
8. The image processing method according to claim 1 or 7, It is characterized in that The method further comprises: Image recognition is performed on the target color image, and the background area is cleared according to the image recognition result, so as to calculate the input viewpoint image according to the target color image after the background area is cleared.
9. An image processing device, It is characterized in that The device comprises: An image data acquisition module, used to acquire a target depth image acquired by a depth camera and a target color image acquired by a color camera; wherein the target color image includes a plurality of color images acquired by a plurality of color cameras respectively; A projection transformation module, used to project the original three-dimensional point cloud corresponding to the target depth image to the target color image based on the matrix transformation relationship between the depth camera and the color camera, and determine the input viewpoint image according to the matching projection points; a high-definition image acquisition module, configured to determine the user's line of sight, and determine a second gaze area corresponding to the input viewpoint image according to the user's line of sight, and perform image fusion processing based on a gaze area image corresponding to the second gaze area to acquire a high-definition gaze area image; and A low-definition image acquisition module, used for performing scaling and fusion processing on the input viewpoint image to acquire a low-definition image; A virtual viewpoint image generation module is used to perform image fusion processing on the high-definition gaze area image and the low-definition image. Processing to obtain the virtual viewpoint image to be displayed.
10. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.
11. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the image processing method according to any one of claims 1 to 8 by executing the executable instructions.
Citation Information
Patent Citations
Three-dimensional enhancing realizing method for multi-viewpoint free stereo display
CN102568026A
System and method for virtually-augmented visual simultaneous localization and mapping
CN110582798A
Synchronous switching method and device for multi-view live video, equipment and medium
CN111447461A
Image processing method and device, storage medium and electronic equipment
CN116828165A
A method and apparatus for encoding an rendering a 3D scene with inpainting patches
EP3709651A1
Cited By
Eyeball depth image restoration method, system and equipment
CN120278921A
Dense rendering method and device suitable for 3D light field display for 3D model of radiation field and computer equipment
CN121095408A
Cage trolley coordinate system establishment method and device
CN121169951A
Metacosm scene image projection method and projection device
CN121193902A
