Light field image processing method and apparatus

Abstract

The present disclosure relates to the technical field of image processing, and specifically provides a light field image processing method and device. A light field image processing method comprises: acquiring a plurality of to-be-processed images respectively collected by a plurality of cameras arranged on a collection device; for each to-be-processed image, determining a boundary range of a target object on the to-be-processed image; performing cutting processing on the to-be-processed image according to the boundary range and a preset image scale to obtain a target image corresponding to each to-be-processed image; and performing viewpoint fusion processing on each target image according to target viewpoint information to obtain a target light field image. In the present disclosure, the to-be-processed image is cut based on the boundary range of the target object, which can reduce the risk of placing the target object at the edge of the image or even cutting off the target object, improve the quality of the cut image, provide a better data basis for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, in particular to a light field image processing method and device. BACKGROUND

[0002] Light field can record higher dimensional light data, thereby obtaining higher precision three-dimensional information than traditional two-dimensional imaging and traditional three-dimensional imaging represented by binocular stereo vision. Light field video can accurately perceive dynamic environment, so that users experience immersive viewing experience.

[0003] With the development of machine learning technology, a multi-view stereo matching model (MVS Net) based on deep neural network is used to perform three-dimensional stereo reconstruction on input multi-view images, which has good precision and efficiency.

[0004] However, for real-time light field video scenes, due to the unfixed position of the person, the cutting of the multi-view input image of the MVS Net in the traditional scheme cannot guarantee the effect of the person, resulting in poor network output effect. SUMMARY

[0005] To improve the cutting effect of multi-view images in light field video scenes and further improve the processing effect and efficiency of light field video, the embodiments of the present disclosure provide a light field image processing method, device, electronic equipment, video communication system and storage medium.

[0006] In a first aspect, the embodiments of the present disclosure provide a light field image processing method, comprising:

[0007] obtaining a plurality of to-be-processed images respectively collected by a plurality of cameras arranged on a collection device; the plurality of to-be-processed images are collection images including different perspectives of a target object;

[0008] For each to-be-processed image, determining a boundary range of the target object on the to-be-processed image;

[0009] According to the boundary range and a preset image scale, performing cutting processing on the to-be-processed image to obtain a target image corresponding to each to-be-processed image;

[0010] According to target viewpoint information, performing viewpoint fusion processing on each target image to obtain a target light field image corresponding to the target viewpoint information; the target viewpoint information represents position information of an observer's eye at a display device end.

[0011] In some implementations, determining the boundary range of the target object on the image to be processed for each image includes:

[0012] For each image to be processed, the image is binarized to obtain a binary image of the target object.

[0013] Based on the pixel values ​​on the binary image, the target object is searched row by row and column by column to obtain the horizontal and vertical boundaries of the target object on the binary image.

[0014] The boundary range is determined based on the horizontal boundary and the vertical boundary.

[0015] In some implementations, the step of performing a boundary search on the target object row by row and column by column based on the pixel values ​​on the binary image to obtain the horizontal and vertical boundaries of the target object on the binary image includes at least one of the following:

[0016] Based on the pixel values ​​on the binary image, the number of black pixels in each column is detected sequentially from left to right. In response to the fact that the number of black pixels in the first column and the subsequent preset number of columns are all greater than the first preset threshold, the coordinate information corresponding to the first column is determined as the left boundary of the horizontal boundary.

[0017] The number of black pixels in each column is detected sequentially from right to left. In response to the fact that the number of black pixels in the second column and the subsequent preset number of columns are all greater than the second preset threshold, the coordinate information corresponding to the second column is determined as the right boundary of the horizontal boundary.

[0018] The number of black pixels in each row is detected sequentially from top to bottom. In response to the fact that the number of black pixels in the first row and the subsequent preset number of rows are all greater than the third preset threshold, the coordinate information corresponding to the first row is determined as the upper boundary of the vertical boundary.

[0019] The number of black pixels in each row is detected sequentially from bottom to top. In response to the fact that the number of black pixels in the second row and the subsequent preset number of rows is greater than the fourth preset threshold, the coordinate information corresponding to the second row is determined as the lower boundary of the vertical boundary.

[0020] In some implementations, the step of binarizing each image to obtain a binary image of the target object includes:

[0021] Each image to be processed is subjected to image matting to obtain a foreground image of the target object corresponding to each image to be processed;

[0022] Each foreground image is binarized to obtain a binary image of the target object.

[0023] In some embodiments, before performing a boundary search on the target object row by row and column by column based on the pixel values ​​in the binary image to obtain the horizontal and vertical boundaries of the target object in the binary image, the method further includes:

[0024] Based on the pixel values ​​on the binary image, a sliding window of a preset size is used to search on the binary image at a preset step size;

[0025] In each sliding window, the pixels included in the sliding window are denoised based on the sum of the pixel values ​​of the pixels included in the sliding window.

[0026] In some implementations, cropping the image to be processed according to the boundary range and a preset image scale to obtain a target image corresponding to each image to be processed includes:

[0027] The coordinates of the center point of the target object are determined based on the boundary range;

[0028] The center point coordinates of the target object are determined as the center point coordinates of the target image, and the image to be processed is cropped according to the preset image scale to obtain the target image.

[0029] In some implementations, viewpoint fusion processing is performed on each target image based on target viewpoint information to obtain a target light field image corresponding to the target viewpoint information, including:

[0030] At least two of the multiple images to be processed are input into a pre-trained deep network model to obtain the depth map of the target object output by the deep network model.

[0031] Based on the target viewpoint information, viewpoint fusion processing is performed on the depth map to obtain a target viewpoint depth map under the viewpoint corresponding to the target viewpoint information;

[0032] The target image, the target viewpoint depth map, and the target viewpoint information are input into a pre-trained viewpoint fusion model to obtain the target light field image output by the viewpoint fusion model.

[0033] In some embodiments, the method is applied to the acquisition device; after performing viewpoint fusion processing on each target image based on the target viewpoint information to obtain a target light field image corresponding to the target viewpoint information, the method further includes:

[0034] The target light field image is sent to the display device so that the display device renders and displays the target light field image.

[0035] Secondly, this disclosure provides a light field image processing apparatus, comprising:

[0036] The image acquisition module is configured to acquire multiple images to be processed by multiple cameras installed on the acquisition device; the multiple images to be processed are acquired images including different perspectives of the target object;

[0037] The boundary search module is configured to determine the boundary range of the target object on the image to be processed for each image to be processed;

[0038] The cropping module is configured to crop the image to be processed according to the boundary range and the preset image scale to obtain the target image corresponding to each image to be processed.

[0039] The viewpoint fusion module is configured to perform viewpoint fusion processing on each target image based on the target viewpoint information to obtain a target light field image corresponding to the target viewpoint information; the target viewpoint information represents the position information of the observer's eye on the display device.

[0040] In some implementations, the boundary search module is configured to:

[0041] For each image to be processed, the image is binarized to obtain a binary image of the target object.

[0042] Based on the pixel values ​​on the binary image, the target object is searched row by row and column by column to obtain the horizontal and vertical boundaries of the target object on the binary image.

[0043] The boundary range is determined based on the horizontal boundary and the vertical boundary.

[0044] In some implementations, the boundary search module is configured to:

[0045] Based on the pixel values ​​on the binary image, the number of black pixels in each column is detected sequentially from left to right. In response to the fact that the number of black pixels in the first column and the subsequent preset number of columns are all greater than the first preset threshold, the coordinate information corresponding to the first column is determined as the left boundary of the horizontal boundary.

[0046] The number of black pixels in each column is detected sequentially from right to left. In response to the fact that the number of black pixels in the second column and the subsequent preset number of columns are all greater than the second preset threshold, the coordinate information corresponding to the second column is determined as the right boundary of the horizontal boundary.

[0047] The number of black pixels in each row is detected sequentially from top to bottom. In response to the fact that the number of black pixels in the first row and the subsequent preset number of rows are all greater than the third preset threshold, the coordinate information corresponding to the first row is determined as the upper boundary of the vertical boundary.

[0048] The number of black pixels in each row is detected sequentially from bottom to top. In response to the fact that the number of black pixels in the second row and the subsequent preset number of rows is greater than the fourth preset threshold, the coordinate information corresponding to the second row is determined as the lower boundary of the vertical boundary.

[0049] In some implementations, the boundary search module is configured to:

[0050] Each image to be processed is subjected to image matting to obtain a foreground image of the target object corresponding to each image to be processed;

[0051] Each foreground image is binarized to obtain a binary image of the target object.

[0052] In some implementations, the boundary search module is configured to:

[0053] Based on the pixel values ​​on the binary image, a sliding window of a preset size is used to search on the binary image at a preset step size;

[0054] In each sliding window, the pixels included in the sliding window are denoised based on the sum of the pixel values ​​of the pixels included in the sliding window.

[0055] In some implementations, the trimming module is configured to:

[0056] The coordinates of the center point of the target object are determined based on the boundary range;

[0057] The center point coordinates of the target object are determined as the center point coordinates of the target image, and the image to be processed is cropped according to the preset image scale to obtain the target image.

[0058] In some implementations, the viewpoint fusion module is configured to:

[0059] At least two of the multiple images to be processed are input into a pre-trained deep network model to obtain the depth map of the target object output by the deep network model.

[0060] Based on the target viewpoint information, viewpoint fusion processing is performed on the depth map to obtain a target viewpoint depth map under the viewpoint corresponding to the target viewpoint information;

[0061] The target image, the target viewpoint depth map, and the target viewpoint information are input into a pre-trained viewpoint fusion model to obtain the target light field image output by the viewpoint fusion model.

[0062] In some embodiments, the apparatus described herein is applied to the acquisition device, and further includes a transmission module configured to:

[0063] The target light field image is sent to the display device so that the display device renders and displays the target light field image.

[0064] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0065] processor; and

[0066] A memory storing computer instructions for causing the processor to perform the method according to any embodiment of the first aspect.

[0067] Fourthly, embodiments of this disclosure provide a video communication system, including:

[0068] The display device includes an image acquisition device and a first controller;

[0069] The acquisition device includes a plurality of cameras and a second controller, wherein at least one of the first controller and the second controller is configured to perform the method according to any embodiment of the first aspect.

[0070] Fifthly, embodiments of this disclosure provide a storage medium storing computer instructions for causing a computer to perform the method described according to any embodiment of the first aspect.

[0071] The light field image processing method of this disclosure includes acquiring multiple images to be processed by multiple cameras mounted on an acquisition device; for each image to be processed, determining the boundary range of a target object on the image to be processed; cropping the image to be processed according to the boundary range and a preset image scale to obtain a target image corresponding to each image to be processed; and performing viewpoint fusion processing on each target image according to target viewpoint information to obtain a target light field image. In this disclosure, cropping the image to be processed based on the boundary range of the target object can reduce the risk of placing the target object at the edge of the image or even cropping it out, improve the quality of the cropped image, provide a better data foundation for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0073] Figure 1 This is an architecture diagram of a video communication system according to some embodiments of this disclosure.

[0074] Figure 2 This is a schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure.

[0075] Figure 3 This is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

[0076] Figure 4 This is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

[0077] Figure 5 This is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

[0078] Figure 6 This is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

[0079] Figure 7 This is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

[0080] Figure 8 This is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

[0081] Figure 9 This is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

[0082] Figure 10 This is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

[0083] Figure 11 These are comparison diagrams showing the effects of light field image processing methods according to some embodiments of this disclosure.

[0084] Figure 12 This is a flowchart of a light field image processing method according to some embodiments of the present disclosure.

[0085] Figure 13 This is a schematic diagram of a light field image processing method according to some embodiments of the present disclosure.

[0086] Figure 14 This is a structural block diagram of a light field image processing apparatus according to some embodiments of the present disclosure.

[0087] Figure 15 This is a structural block diagram of an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0088] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0089] The light field is defined as the amount of light passing through each point in every direction. Light field images can record light data with higher dimensions than traditional two-dimensional images, thus presenting more precise three-dimensional information than traditional two-dimensional imaging and traditional three-dimensional imaging represented by binocular stereo vision.

[0090] Light field video can accurately perceive the dynamic environment. Combined with eye-tracking technology, the video can follow the change of the user's viewpoint in real time, thus presenting the user with an immersive naked-eye 3D viewing experience.

[0091] Data acquisition for light field video requires the use of a camera array, which includes several or even dozens of cameras with different viewpoints. Each camera is responsible for acquiring images from one viewpoint. Then, the multi-view stereo matching and reconstruction (MVS) algorithm is used to combine the acquired image data with a new viewpoint to perform viewpoint fusion and obtain the light field image from the new viewpoint.

[0092] With the development of machine learning technology, the use of multi-view stereo matching models based on deep neural networks (MVS Net) to reconstruct three-dimensional stereo images has greatly improved the computational efficiency and presentation effect of real-time light field video.

[0093] The input to MVS Net is multi-view images acquired by a camera array. In related technologies, in order to adapt to the scale of the input images of the model and to remove some irrelevant background information and improve the computational efficiency and effect of the model, the large-resolution images (e.g., 4096 pixels * 3000 pixels) acquired by the camera array often need to be cropped to a small-scale input image (e.g., 2560 pixels * 2560 pixels) that conforms to the model input before being input into MVS Net.

[0094] In related technologies, cropping of input images is generally based on preset fixed positions and scales. However, practical experience has shown that in real-time light field video call scenarios, both parties are primarily focused on each other's human figures. However, the actual position of the person in the captured image may not be located at the center of the cropping area. This results in the person being located at the edge of the input image, or even part of the person's body being cropped out, during the cropping algorithm process. This makes it difficult for model matching and reconstruction or network training, leading to poor quality of the final output light field image.

[0095] Based on the deficiencies of the aforementioned related technologies, this disclosure provides a light field image processing method, apparatus, electronic device, video communication system, and storage medium, aiming to improve the cropping effect of multi-viewpoint images in light field video scenes, thereby improving the processing effect and efficiency of light field video.

[0096] Figure 1 The following diagram illustrates the architecture of a video communication system in some embodiments of this disclosure, in conjunction with... Figure 1 The application scenarios of the embodiments of this disclosure are described.

[0097] like Figure 1 As shown, in some embodiments, the video communication system includes a capture device 100 and a display device 200, which establish a communicable connection via a wired or wireless network.

[0098] In an exemplary one-way video communication scenario, the acquisition device 100 can acquire image data of the scene where user A is located and send the image data to the display device 200. The display device 200 obtains the current viewpoint position of user B by tracking the eye position of user B, combines the viewpoint position with the light field image data sent by the acquisition device 100 to synthesize a viewpoint image, and renders and displays the synthesized light field image on the display device 200.

[0099] Alternatively, the display device 200 can track the eye position of user B to obtain the current viewpoint position of user B, and send the viewpoint position to the acquisition device 100. The acquisition device 100 then combines the viewpoint position with the image data of the scene where user A is located that it has acquired to synthesize the viewpoint image to obtain a light field image, and sends the light field image to the display device 200. The display device 200 then renders and displays the image based on the received light field image.

[0100] It is understood that the above example only uses one-way video communication as an example, but the solution disclosed herein is not limited to one-way video communication scenarios. For two-way video communication scenarios, the display device 200 can also simultaneously acquire light field image data of the scene where user B is located and send the light field image data to the acquisition device 100. The acquisition device 100 can also simultaneously track the eye position of user A to obtain the current viewpoint position of user A, combine the viewpoint position with the light field image data sent by the display device 200 to synthesize a viewpoint image, and render and display the synthesized light field image on the acquisition device 100. Those skilled in the art will understand this, and this disclosure will not elaborate further.

[0101] Taking two-way video communication as an example, Figure 2 The present disclosure shows a schematic diagram of the structure of an electronic device in some embodiments. The electronic device can be either a data acquisition device 100 or a display device 200, and the present disclosure does not limit it in this regard.

[0102] like Figure 2 As shown, the electronic device includes a display screen 110, camera arrays C1 to C4, and an image acquisition device C5.

[0103] The display screen 110 is used to display light field images. The display screen 110 can be any screen component suitable for implementation, such as an LCD (Liquid Crystal Display) screen, an OLED (Organic Light-Emitting Diode) screen, etc., and this disclosure does not limit it.

[0104] A camera array comprises multiple cameras deployed in an array on electronic devices, such as... Figure 2 In the example, the camera array includes four cameras, C1 to C4, deployed at four opposite corners of the electronic device. Because each camera has a different position and viewing angle, the camera array can capture multiple images of the scene from different perspectives. Of course, the number of cameras and their deployment method in a camera array are not limited to this. Figure 2 Examples, and any other suitable implementation methods, are not limited in this disclosure.

[0105] The image acquisition device C5 is a camera used to achieve user eye tracking. It can be, for example, a high-precision RGB camera. That is, the image acquisition device C5 acquires images of the current scene and performs image detection to determine the current user's viewpoint information, which represents the position of the user's eyes. Figure 2 In the example, the image acquisition device C5 is located above the center of the display screen 110 of the electronic device. However, it is understood that this disclosure does not limit the specific location of the image acquisition device C5.

[0106] In the above Figure 1 and Figure 2 Based on the video communication system shown, the light field image processing method of the present disclosure will be described below.

[0107] It is worth noting that, for ease of understanding, the following embodiments of this disclosure will be described using a one-way video communication scenario as an example, that is, the acquisition device 100 acts as the light field data acquisition end and the display device 200 acts as the light field video display end. The principle of the two-way video communication scenario is exactly the same, and this disclosure will not repeat it here.

[0108] Furthermore, in the embodiments of this disclosure, one or more steps of the light field image processing method described below can be performed by the acquisition device 100, by the display device 200, or by the acquisition device 100 and the display device 200 interacting together. The executing entities of the method steps will be specifically described later in this disclosure; therefore, method steps whose executing entities are not explicitly limited below are not considered to be restrictive.

[0109] like Figure 3 As shown, in some embodiments, the light field image processing method of this disclosure includes:

[0110] S310. Acquire multiple images to be processed by multiple cameras installed on the acquisition device.

[0111] In the aforementioned video communication scenario, the camera array on the acquisition device side, comprising multiple cameras, can simultaneously acquire a single image, which is the image to be processed as described in this disclosure. The image to be processed includes a target object, such as a human body. Furthermore, due to the different positions and viewing angles of each camera, the multiple images to be processed acquired by the camera array are images including the target object from different perspectives.

[0112] Combination Figure 1 , Figure 2As shown, when user A is located at the acquisition device 100, cameras C1 to C4 on the acquisition device 100 simultaneously acquire an image including user A, thus obtaining four acquisition images including user A from different perspectives. These four acquisition images are the four images to be processed as described in this disclosure.

[0113] S320. For each image to be processed, determine the boundary range of the target object on the image to be processed.

[0114] As mentioned above, after obtaining the image to be processed, it cannot be directly input into the MVS network model. Instead, the image needs to be cropped.

[0115] There are many reasons for cropping images, including but not limited to the following:

[0116] 1) In video communication scenarios, both parties often focus only on each other's bodies and pay little attention to the background of the image. However, due to the large field of view of the camera, the range of scenes that can be captured is very wide, resulting in a low proportion of people in the image to be processed. Therefore, it is necessary to crop out the area of ​​people to highlight them.

[0117] 2) Excessive background information in the image to be processed can interfere with the computation and training of the MVS network, increasing computational overhead and difficulty, and leading to a deterioration in viewpoint fusion performance. Therefore, it is necessary to crop irrelevant background information to reduce computational overhead and difficulty.

[0118] 3) The MVS network model has requirements for the size of the input image, and it is generally a square image. For example, the input image size of a typical MVS network is 2560 pixels * 2560 pixels. However, images captured by a camera are larger in scale and rarely square. For example, the size of a camera image is 4096 pixels * 3000 pixels. Therefore, the captured images need to be cropped to fit the input image size of the network model.

[0119] Traditional cropping methods typically crop the image to be processed to fit the size of the MVS input image at a fixed position. For example... Figure 4 In the example, such as Figure 4 As shown in (a), the outer solid frame represents the scale of the image to be processed, and the inner dashed frame represents the preset cropping position and range for cropping the image to be processed.

[0120] In scenarios such as light field video communication, the person is not always located in the center of the image being processed, for example... Figure 4As shown in (b), the figure may appear in a non-central position in the image to be processed. If cropping is performed at a fixed position, the figure will shift to the edge of the image, or even partially crop out. This will make the MVS matching reconstruction or network training process difficult, resulting in poor quality of the final output light field image.

[0121] Therefore, in this embodiment of the present disclosure, after obtaining each image to be processed, the image to be processed is not cropped based on a fixed position. Instead, the boundary range of the target object on the image to be processed is first determined, and then the image to be processed is cropped based on the boundary range of the target object.

[0122] The boundary range of the target object refers to the image region enclosed by the various boundaries of the target object on the image to be processed. For example... Figure 4 In the scenario shown, the target object in the image to be processed is the human body. The area occupied by the human body in the image to be processed can be represented by a rectangle. The left side of the rectangle is the left boundary of the target object, the right side is the right boundary of the target object, the top side is the top boundary of the target object, and the bottom side is the bottom boundary of the target object. Thus, the rectangular area enclosed by the four boundaries (top, bottom, left, and right) is the boundary range of the target object.

[0123] In some embodiments of this disclosure, the boundary range of the target object on the image to be processed can be obtained by performing a boundary search on the image. The specific process of the boundary search is described in the following embodiments of this disclosure.

[0124] In some embodiments of this disclosure, before performing boundary search on the image to be processed, foreground and background segmentation, binarization, and other processing can be performed on the image to be processed, thereby improving the accuracy and effect of boundary search for the target object. It is understood that these processing methods are optional and not mandatory, and will be described in detail below.

[0125] S330. Based on the boundary range and preset image scale, the image to be processed is cropped to obtain the target image corresponding to each image to be processed.

[0126] In this embodiment of the disclosure, after processing each image to be processed to obtain the boundary range of the target object on each image to be processed, the image to be processed can be cropped according to the boundary range of the target object.

[0127] The preset image scale refers to the scale of the target image obtained after cropping the image to be processed. Taking 3D stereo matching reconstruction using the MVS network as an example, the preset image scale can be set according to the scale of the input image of the MVS network. For example, in one example, if the input image scale required by the MVS network is 2560 pixels * 2560 pixels, then the preset image scale can be set to 2560 pixels * 2560 pixels.

[0128] Taking an image to be processed as an example, after determining the boundary range of the target object on the image to be processed, the image to be processed needs to be cropped to a preset image size according to the boundary range, so as to obtain the target image after cropping the image to be processed.

[0129] In some embodiments, the center point coordinates of the target object can be determined by the boundary range of the target object on the image to be processed. Then, the image is expanded outward based on the center point coordinates. The expanded image range is the preset image scale size. Then, the image to be processed is cropped based on the expanded image range to obtain the target image corresponding to the image to be processed. This process is described in the embodiments below.

[0130] By cropping each image to be processed through the above process, the target image corresponding to each image to be processed can be obtained. The scale of the target image is the preset image scale.

[0131] S340. Perform viewpoint fusion processing on each target image based on the target viewpoint information to obtain the target light field image corresponding to the target viewpoint information.

[0132] Combination Figure 1 The video communication scenario shown illustrates that the image to be processed refers to the scene image of user A captured by acquisition device 100, while the final target light field image is a 3D image to be rendered and displayed on display device 200. Therefore, the motion tracking effect of the target light field image needs to be determined based on user B's viewpoint information. In other words, the target light field image ultimately presented on display device 200 needs to be fused with user B's viewpoint information; therefore, display device 200 needs to acquire user B's viewpoint information.

[0133] In this embodiment, the target viewpoint information represents the position information of the observer's eyes at the display device 200, reflecting the current viewpoint position viewed by the observer at the display device 200. This target viewpoint information is acquired by the image acquisition device C5 at the display device 200 and obtained using an eye-tracking algorithm. It is understood that the process of determining the target viewpoint information at the display device 200 can be obtained by those skilled in the art by referring to relevant technologies and combining them with traditional eye-tracking algorithms; therefore, this disclosure will not elaborate on this process.

[0134] After determining the target viewpoint information and each cropped target image, the viewpoint information can be used to perform viewpoint fusion processing on each target image, and finally the target light field image corresponding to the target viewpoint information can be obtained.

[0135] In some implementations, the viewpoint fusion processing of the target image can be achieved using an MVS network model based on deep neural networks (DNNs). That is, the MVS network is pre-trained to obtain an MVS network model with good convergence performance. Then, the target viewpoint information and each target image are used as inputs to the MVS network model. The MVS network performs viewpoint fusion processing based on these inputs, predicts, and outputs the target light field image. The structure and principle of the MVS network are described in the following implementations and will not be detailed here.

[0136] After obtaining the target light field image, the display device 200 can render and display the target light field image. It can be understood that since the target light field image is generated based on the current target viewpoint information of the observer at the display device 200, a naked-eye 3D effect that follows the change of the observer's viewpoint can be achieved, so that the observer at the display device 200 has an immersive video experience.

[0137] It is worth noting that, in the entire process described above, only the process of acquiring the image to be processed must be performed by the acquisition device 100, and the process of rendering and displaying the target light field image must be performed by the display device 200. The remaining process can be performed by the acquisition device 100, the display device 200, or a third-party device (such as a server) that establishes a communication connection with both of them. This disclosure does not impose any restrictions on this.

[0138] For example, in a typical scenario, the aforementioned process S310 to S340 is all processed by the acquisition device 100. After obtaining the target light field image, the acquisition device 100 sends the target light field image to the display device 200, which then renders and presents the target light field image.

[0139] For example, in another scenario, only the process of acquiring the image to be processed in S310 is performed by the acquisition device 100, and then the acquisition device 100 sends the image to be processed to the display device 200. The rest of the process is performed by the display device 200, and finally the target light field image is rendered and displayed.

[0140] For example, in another example scenario, only the process of acquiring the image to be processed in S310 is performed by the acquisition device 100, and then the acquisition device 100 sends the image to be processed to a third-party server. The rest of the process is performed by the third-party server, and then the third-party server sends the obtained target light field image to the display device 200, which renders and presents the target light field image.

[0141] Of course, those skilled in the art will understand that the implementation methods for the executing entity are not limited to the above examples, and this disclosure will not elaborate further on this.

[0142] It is understood that in this embodiment of the disclosure, the cropping of the image to be processed is not based on a fixed position, but is based on the boundary range of the target object. This ensures that the target object is always located in the center of the cropped image, reducing the risk of placing the target object at the edge of the image or even cropping it out.

[0143] Furthermore, since the target object occupies a larger area in the target image, the effect and accuracy of subsequent MVS network feature extraction are improved, thus enhancing the quality of the light field image generated during network training and prediction. Moreover, because the target object is located at the center of the target image, it can include as much surrounding scene information as possible, combining more contextual feature information to further improve the accuracy and performance of the MVS network.

[0144] As can be seen from the above, in this embodiment of the present disclosure, cropping the image to be processed based on the boundary range of the target object can reduce the risk of placing the target object at the edge of the image or even cropping it out, improve the quality of the cropped image, provide a better data foundation for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication.

[0145] In the embodiments described below, this disclosure will use... Figure 1 and Figure 2 Taking the video communication system shown as an example, the light field image processing method of this disclosure will be further explained.

[0146] like Figure 5 As shown, in some embodiments, the light field image processing method of this disclosure, the process of determining the boundary range of a target object on each image to be processed, includes:

[0147] S510. For each image to be processed, perform binarization processing on the image to be processed to obtain a binary image of the target image.

[0148] It is understood that in the embodiments of this disclosure, the boundary range of the target object refers to the image area enclosed by the various boundaries of the target object on the image to be processed. In other words, in the video communication scenario of the example of this disclosure, the focus is mainly on the boundary of the target object, so the original image to be processed can be converted into a binary image.

[0149] A binary image is an image in which each pixel has only two possible values, typically represented by pixel values ​​0 and 1. 0 represents a black pixel, and 1 represents a white pixel; that is, a binary image contains only black and white pixels. In this embodiment, a binary image can be used to segment target objects and non-target objects by pixels. For example, in one example, pixels belonging to the target object are black pixels (pixel value 0), and other pixels are white pixels (pixel value 1).

[0150] Furthermore, it is understood that in real video communication scenarios, the image to be processed captured by the acquisition device 100 will include not only the foreground target object but also the background area. In some embodiments of this disclosure, before binarizing the image to be processed, the image can be matted to remove the background area and retain only the foreground target object. The following describes how... Figure 6 Please provide an explanation.

[0151] like Figure 6 As shown, in some embodiments, the light field image processing method of this disclosure includes a process of binarizing the image to be processed, comprising:

[0152] S511. Perform image matting on each image to be processed to obtain the foreground image of each image to be processed, including the target object.

[0153] S512. Perform binarization processing on each foreground image to obtain a binary image of the target object.

[0154] In this embodiment of the present disclosure, after obtaining multiple images to be processed through the acquisition device 100, image segmentation can be performed on each image to achieve the matting of the foreground target object and the background.

[0155] It's understandable that the purpose of image cutout is to separate the target object from the background, for example, for... Figure 1 In the video communication scenario shown, the image matting process involves separating the foreground person (user A) from the background, so that the foreground image only includes the person, and the background area can be filled with a single pixel.

[0156] In related technologies, there are many image matting algorithms, and those skilled in the art can use any matting algorithm. For example, in one example, a DNN-based image segmentation model can be used. Each image to be processed is input into a pre-trained image segmentation model, and the foreground image output by the model can be obtained.

[0157] In other implementations, in conjunction with video communication scenarios, taking video conferencing as an example, the large-screen electronic devices used to implement video conferencing are often in relatively fixed positions. As a result, during video communication, the background part of the scene image captured by the acquisition device 100 hardly changes, and generally only the people or objects in the foreground will move.

[0158] Therefore, in this embodiment of the present disclosure, the acquisition device 100 can pre-acquire and save a background image that does not include the target object. For example, the acquisition device 100 can acquire and save the background image when it is powered on. Then, when performing image segmentation on the image to be processed, the image to be processed is quickly cut out based on the difference between the image to be processed and the pre-saved background image. This example method is simple and has higher efficiency in image cutting out.

[0159] For example, in one instance, after performing image matting on an image to be processed, the resulting foreground image might look like this: Figure 7 As shown in (a). It is understandable that the colors are lost after the grayscale conversion of the accompanying drawings in the specification. Figure 7 (a) presents a grayscale image, which may itself be a color (RGB) image, as will not be described further in this disclosure.

[0160] In this embodiment of the disclosure, after obtaining the foreground image corresponding to each image to be processed, each foreground image can be binarized to obtain a binary image corresponding to each image to be processed.

[0161] by Figure 7 Taking the foreground image shown in (a) as an example, in some implementations, the foreground image can first be converted into a grayscale image. The purpose of converting to grayscale is to remove the colors from the foreground image and reduce the computational load. The process of converting to grayscale is represented as follows:

[0162] gray=0.302*R+0.566*G+0.132*B (1)

[0163] In formula (1), gray represents the grayscale value of each pixel in the grayscale image, and R, G, and B represent the RGB values ​​of each pixel in the foreground image. Formula (1) can be used to convert each pixel in the foreground image to a grayscale value, resulting in, for example... Figure 7 The grayscale image shown in (b) is shown in the image.

[0164] After obtaining the grayscale image corresponding to the foreground image, the grayscale image can be converted into a binary image. For example, in one example, a grayscale threshold can be preset, and then the grayscale value of each pixel in the grayscale image can be compared with the grayscale threshold. If the grayscale value of a pixel is greater than or equal to the grayscale threshold, the pixel value of that pixel can be set to 1, i.e., white; if the grayscale value of a pixel is less than the grayscale threshold, the pixel value of that pixel can be set to 0, i.e., black. The specific value of the grayscale threshold can be selected according to the actual situation, and this disclosure does not impose any restrictions on it.

[0165] Therefore, after traversing all pixels of the entire grayscale image, a binary image containing only 0 and 1 pixels can be obtained, as in the example above. Figure 7 After converting the grayscale image shown in (b) into a binary image, we can obtain the following: Figure 7 As shown in the binary image in (c), the binary image only includes black and white pixels, where black pixels represent target objects and white pixels represent non-target objects.

[0166] The above description only covers the binarization process for one image to be processed. For multiple images to be processed acquired by the acquisition device 100, the above process can be used sequentially to obtain the binary image of each image to be processed.

[0167] In some implementations, after obtaining the binary images of each image to be processed, considering that noise may appear in the images—for example, white areas of the binary image may contain black noise, and black areas may contain white noise—this noise may interfere with subsequent boundary searches. Therefore, in some implementations, after obtaining the binary images, further denoising processing can be performed on the binary images.

[0168] Continue to refer to Figure 6 As shown, in some embodiments, the light field image processing method of this disclosure includes the following process for denoising a binary image:

[0169] S513. Based on the pixel values ​​on the binary image, perform a search on the binary image with a preset step size using a sliding window of a preset scale.

[0170] S514. In each sliding window, the pixels included in the sliding window are denoised based on the sum of the pixel values ​​of the pixels included in the sliding window.

[0171] In this embodiment, a sliding window is used to traverse the binary image to denoise the black and white noise points on the binary image. For example, a sliding window of preset size m*n can traverse the entire binary image with a preset step size d. The specific values ​​of the preset size m*n and the preset step size d can be selected according to the denoising accuracy requirements.

[0172] In an example scenario, the sliding window has a scale of 2x2 and a preset step size d = 1. That is, the sliding window can be... Figure 8 As shown, the preset scale is 2*2, and the step size is 1 pixel when sliding on the binary image each time.

[0173] Taking a single slide as an example, the sliding window can select 4 pixels on the binary image, that is... Figure 8 The pixels A, B, C, and D shown have a pixel value of either 0 or 1.

[0174] Therefore, the sum of pixel values ​​of pixels A to D in the sliding window can be calculated, and the denoising process can be achieved according to the relationship of formula (2) based on the sum of pixel values:

[0175]

[0176] Combination Figure 8 As shown in formula (2), if the sum of the pixel values ​​of pixels A to D is 0, it means that all pixels A to D are black pixels, the sliding window is located inside the target object, and pixels A to D are not noise. If the sum of the pixel values ​​of pixels A to D is 1 or 2, it means that there is 1 or 2 white pixels in pixels A to D, and it can be considered that there is white noise. All pixel values ​​in the sliding window are set to 0, that is, the white noise is converted into black pixels. If the sum of the pixel values ​​of pixels A to D is 3, it means that there is 1 black pixel and 3 white pixels in pixels A to D, and it can be considered that there is black noise. Therefore, the pixel value of the black pixel is set to 1, and it is converted into white. If the sum of the pixel values ​​of pixels A to D is 4, it means that all pixels A to D are white pixels, the sliding window is located in the background area, and pixels A to D are not noise.

[0177] By using the denoising process of the above formula (2), the entire binary image can be traversed through a sliding window to complete the denoising process and obtain the denoised binary image.

[0178] The above description only covers the denoising process for one binary image. For the multiple binary images corresponding to the images to be processed acquired by the acquisition device 100, the above process can be used sequentially to complete the denoising process for each binary image.

[0179] S520. Based on the pixel values ​​on the binary image, perform boundary search on the target object row by row and column by column to obtain the horizontal and vertical boundaries of the target object on the binary image.

[0180] In this embodiment of the disclosure, the boundary range of the target object includes a horizontal boundary and a vertical boundary. That is, the boundary range of the target object is a rectangular frame area, which is enclosed by the two horizontal boundaries and two vertical boundaries of the rectangular frame to form the entire boundary range.

[0181] For example, in one example, the binary image of the image to be processed can be as follows: Figure 7 As shown in (c), the process of performing boundary search on a binary graph can be as follows: Figure 9 As shown below, in conjunction with Figure 9 The boundary search process of this disclosure example is described.

[0182] like Figure 9 As shown, firstly, based on the pixel values ​​in the binary image, the pixel values ​​of each column are detected sequentially from left to right, and the number of black pixels in each column is counted, represented as... In the formula, i represents the i-th pixel, n represents the image height, and value... i This represents the pixel value of the i-th pixel.

[0183] If cout(0) = 0, it means that the number of pixels with a value of 0 in this column is 0, that is, the number of black pixels is 0. Then continue to the next column for detection until the number of black pixels in a certain column is not 0.

[0184] For example, if the number of black pixels in the k-th column is not zero, meaning that the column contains black pixels, then the detection continues to the next column (k+1 columns), and the count p = 1. If the k+1 columns do not contain black pixels, then the black pixels detected in the k-th column are noise, and the count p is reset to zero. Conversely, if the k+1 columns contain black pixels, then the count p is incremented by 1, i.e., p = 2, and the detection continues to the k+2 columns.

[0185] Thus, until the count p reaches the preset value, it means that the k-th column of pixels and the subsequent preset number of columns of pixels all include black pixels. At this point, the coordinate information corresponding to the k-th column of pixels can be determined as the left boundary of the target object on the binary image.

[0186] Similarly, by performing detection column by column from right to left, the right boundary of the target object on the binary image can be obtained. Those skilled in the art can undoubtedly understand and fully implement this by referring to the foregoing, and this disclosure will not elaborate further. After determining the left and right boundaries, the horizontal boundary of the target object's boundary range can be determined.

[0187] For vertical boundaries, such as Figure 9 As shown, based on the pixel values ​​in the binary image, the pixel values ​​of each row are detected sequentially from top to bottom, and the number of black pixels in each row is counted, represented as... In the formula, i represents the i-th pixel, m represents the image width, and value... i This represents the pixel value of the i-th pixel.

[0188] If cout(0) = 0, it means that the number of pixels with a value of 0 in the row is 0, that is, the number of black pixels is 0. Then continue to the next row for detection until the number of black pixels in a certain row is not 0.

[0189] For example, if the number of black pixels in row J is not zero, meaning that row J contains black pixels, then the detection continues to the next row (row j+1), and the count p = 1. If row j+1 does not contain black pixels, then the black pixels detected in row j are noise, and the count p is reset to zero. Conversely, if row j+1 contains black pixels, the count p is incremented by 1, i.e., p = 2, and the detection continues to row j+2.

[0190] Thus, until the count p reaches the preset value, it means that the pixels in the j-th row and the subsequent preset number of rows of pixels all include black pixels. At this point, the coordinate information corresponding to the pixels in the j-th row can be determined as the upper boundary of the target object on the binary image.

[0191] Similarly, by performing detection row by row from bottom to top, the lower boundary of the target object on the binary image can be obtained. Those skilled in the art can undoubtedly understand and fully implement this by referring to the foregoing, and this disclosure will not elaborate further. After determining the upper and lower boundaries, the vertical boundary of the target object's boundary range can be determined.

[0192] It is worth noting that, see Figure 9 As shown, in the video communication scenario of this disclosure example, considering that only the upper body of the user is often captured, when performing boundary search on the human body, the lower boundary of the binary image can be determined as the lower boundary of the target object by default. That is, there is no need to perform boundary search on the lower boundary; it is only necessary to determine the left boundary, the upper boundary, and the right boundary.

[0193] Additionally, see Figure 9 As shown, after determining the left boundary, during the upper boundary search, it is not necessary to detect all pixels in the entire row; only the pixels from the left boundary to the rightmost edge need to be detected. Similarly, after determining the upper boundary, during the right boundary search, it is not necessary to detect all pixels in the entire column; only the pixels from the upper boundary to the bottom edge need to be detected. This reduces redundant detection of a large number of pixels and improves computational efficiency.

[0194] S530. Determine the boundary range based on the horizontal and vertical boundaries.

[0195] After determining the horizontal and vertical boundaries of the target object in the binary image, the boundary range of the target object in the image to be processed can be determined based on the coordinate information of the horizontal and vertical boundaries. For example... Figure 9In the example, the rectangular area selected by the left, top, right, and bottom boundaries represents the boundary range of the target object.

[0196] The above description only applies to the boundary search process for one image to be processed. For multiple images to be processed acquired by the acquisition device 100, the boundary range of the target object on each image to be processed can be obtained by sequentially going through the above process.

[0197] As can be seen from the above, in this embodiment of the present disclosure, the boundary range of the target object can be quickly determined by the binary map boundary search method, and the boundary detection accuracy of the target object is higher, providing an accurate data basis for subsequent image cropping.

[0198] like Figure 10 As shown, in some embodiments, the light field image processing method of this disclosure includes a process of cropping the image to be processed based on a boundary range, comprising:

[0199] S531. Determine the coordinates of the center point of the target object based on the boundary range.

[0200] S532. Determine the center point coordinates of the target object as the center point coordinates of the target image, and crop the image to be processed according to the preset image scale to obtain the target image.

[0201] Still with Figure 9 As shown in the example, after obtaining the boundary range of the target object, the coordinates (left, top) of the upper left corner of the boundary range can be obtained based on the coordinate information of the left boundary (left) and the coordinate information of the upper boundary (top).

[0202] Meanwhile, given that the size of the binary image is m*n, assuming in an example the size of the binary image is 4096*3000, the coordinates (Cx, Cy) of the center point of the target object are represented as:

[0203] Cx = left + (right - left) / 2

[0204] Cy = top + (3000 - top) / 2

[0205] After determining the center point coordinates O(Cx, Cy) of the boundary range of the target object, these center point coordinates O(Cx, Cy) can be used as the center point coordinates of the target image. Then, by expanding outward according to the preset image scale based on these center point coordinates, the image range of the target image can be obtained. Finally, the image to be processed can be cropped based on this image range to obtain the target image.

[0206] For example Figure 9In the example, assuming the target image has a preset image scale of 2560*2560 pixels, the image range after expanding the preset image scale based on the center point coordinates O(Cx, Cy) is shown by the dashed line in the figure, and the scale of the dashed box is 2560*2560 pixels. Then, the image to be processed is cropped based on the image range of this dashed box to obtain the target image.

[0207] See Figure 11 As shown, Figure 11 The four images in the first row are the four images to be processed, which were captured by cameras C1 to C4 of the acquisition device 100. Figure 11 The four images in the second row show the effect of cropping the image to be processed based on a fixed position in the traditional method. Figure 11 The four images in the third row represent the effect of cropping the image to be processed using the method of this embodiment, based on boundary search of the target object.

[0208] pass Figure 11 As can be seen from the comparison, after cropping the image using the traditional method, the person is not located in the center of the image, showing a significant offset, and parts of the person's body may even be cropped out. However, after cropping the image using the embodiment of this disclosure, the person remains in the center of the image, providing an accurate data foundation for subsequent viewpoint fusion.

[0209] As can be seen from the above, in this embodiment of the present disclosure, cropping the image to be processed based on the boundary range of the target object can reduce the risk of placing the target object at the edge of the image or even cropping it out, improve the quality of the cropped image, provide a better data foundation for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication.

[0210] In this embodiment of the disclosure, after determining the target viewpoint information and each cropped target image, the viewpoint fusion processing of each target image can be performed using the target viewpoint information, and finally the target light field image corresponding to the target viewpoint information can be obtained.

[0211] In some implementations, the viewpoint fusion process for the target image can be achieved using an MVS network model based on deep neural networks (DNNs). The following section will discuss this in conjunction with... Figure 12 Please provide an explanation.

[0212] like Figure 12 As shown, in some embodiments, the light field image processing method of this disclosure, which performs viewpoint fusion processing on each target image based on target viewpoint information, includes:

[0213] S1210. Input at least two of the multiple images to be processed into a pre-trained deep network model to obtain the depth map of the target object output by the deep network model.

[0214] S1220. Perform viewpoint fusion processing on the depth map based on the target viewpoint information to obtain the target viewpoint depth map under the viewpoint corresponding to the target viewpoint information.

[0215] S1230. Input the target image, target viewpoint depth map, and target viewpoint information into the pre-trained viewpoint fusion model to obtain the target light field image output by the viewpoint fusion model.

[0216] like Figure 13 As shown in this embodiment, the MVS network model includes a deep network model and a viewpoint fusion model. The deep network model is used to predict the depth features of the target object and obtain a depth map of the target object. The deep network model can be, for example, a depth prediction model based on the Depth Nnet architecture. The viewpoint fusion model is used to pre-determine the viewpoint image of the target object from a new viewpoint. The viewpoint fusion model can be, for example, a neural network model based on the MVS Net architecture.

[0217] In this embodiment of the disclosure, combined with Figure 1 and Figure 2 In the scenario shown, the cameras C1 to C4 on the acquisition device 100 first need to be calibrated to obtain the intrinsic and extrinsic parameter matrices of each camera.

[0218] Specifically, firstly, the optical axes of the four cameras can be adjusted to the midpoint of the plane of sharpness, which is the plane in which the camera image is sharp; the focal lengths of the other cameras are related. Then, using the coordinate system of camera C1 as the world coordinate system, the Zhang calibration method is used to calibrate the other cameras, obtaining the intrinsic and extrinsic parameter matrices for each camera. The intrinsic parameter matrices are represented as K1, K2, K3, and K4, respectively, and the extrinsic parameter matrices are represented as (R1, t1), (R2, t2), (R3, t3), and (R4, t4), respectively, where R represents the relative rotation matrix and t represents the relative translation matrix.

[0219] It is understandable that camera intrinsic and extrinsic parameters are necessary parameters for the camera to achieve coordinate system transformation. These parameters are in matrix form, namely the camera intrinsic parameter matrix and the extrinsic parameter matrix. The camera intrinsic parameter matrix is ​​used to transform the camera coordinate system to the pixel coordinate system, while the camera extrinsic parameter matrix is ​​used to transform the world coordinate system to the camera coordinate system.

[0220] The specific algorithm and process for camera calibration can undoubtedly be understood and fully implemented by those skilled in the art by referring to Zhang's calibration method in related technologies, and this disclosure will not elaborate further on this.

[0221] Continue to refer toFigure 13 As shown, in actual video communication scenarios, the synchronization box can control the acquisition devices 100 cameras C1 to C4 to simultaneously acquire images, resulting in images I1 to I4 to be processed. The synchronization box is a synchronization hardware device whose function is to control the cameras C1 to C4 to acquire images synchronously, so that the images I1 to I4 to be processed are images of the target object from different perspectives acquired at the same time.

[0222] In this embodiment of the disclosure, at least two images from I1 to I4 to be processed are required as input to the deep network model, for example... Figure 13 In the example, the images I1 and T2 to be processed can be used as input to the deep network model. The deep network model predicts the depth information of each point on the target object, and its output is a depth map of the target object. Each pixel in the depth map represents the depth value of a point on the target object.

[0223] Deep network models can be pre-trained based on labeled sample data. The traditional supervised training process can be used for the training of deep network models, and this disclosure will not elaborate further.

[0224] After obtaining the depth map of the target object, it is necessary to perform viewpoint fusion processing on the depth map in conjunction with the target viewpoint information to obtain the target viewpoint depth map under the new viewpoint. As mentioned above, the target viewpoint information refers to the current viewpoint information of user B at the display device 200. That is, the depth map needs to be converted to the viewpoint corresponding to the target viewpoint information.

[0225] It is worth noting that the process of viewpoint fusion processing of depth maps by combining target viewpoint information requires coordinate system transformation based on the camera's intrinsic and extrinsic parameter matrices. Since the image to be processed needs to be cropped, the scale of the cropped target image changes compared to the image to be processed, thus changing the camera's intrinsic parameter matrix, necessitating updates to the intrinsic parameter matrix of each camera.

[0226] For example, taking camera C1 as an example, its intrinsic parameter matrix K1 is represented as:

[0227]

[0228] The image to be processed has a size of 4096*3000 pixels, and the target image after cropping has a size of 2560*2560 pixels. Therefore, the horizontal scaling factor Scale_x = 4096 / 2560, the vertical scaling factor Scale_y = 3000 / 2560, the horizontal offset Trans_x = 4096 / 2 - Cx, and the vertical offset Trans_x = 3000 / 2 - Cy. Thus, the updated intrinsic parameter matrix K1' of camera C1 is expressed as:

[0229]

[0230] The above description only illustrates the intrinsic parameter matrix update process for camera C1. The update process for the intrinsic parameter matrices of cameras C2 to C4 is the same, and will not be repeated here.

[0231] Based on the updated intrinsic and extrinsic parameter matrices, viewpoint fusion processing is performed on the depth map according to the target viewpoint information, transforming the depth map to the viewpoint corresponding to the target viewpoint information, thus obtaining the target viewpoint depth map. The specific process and principles of viewpoint fusion processing can be understood and fully implemented by those skilled in the art by referring to relevant technologies, and will not be elaborated further in this disclosure.

[0232] Continue to refer to Figure 13 As shown, the cropping module, based on the aforementioned method and process of this disclosure, performs cropping processing on the images I1 to I4 acquired by cameras C1 to C4 to obtain the target images corresponding to each image to be processed, namely target images I1' to I4'. Those skilled in the art can refer to the foregoing, and will not repeat it here.

[0233] The input to the viewpoint fusion model includes target viewpoint information, target viewpoint depth map, and target images I1' to I4'. The input to the viewpoint fusion model is the target light field image corresponding to the target viewpoint information. The viewpoint fusion model can be pre-trained based on sample labeled data. The network training process of the viewpoint fusion model can be carried out using the traditional supervised training process, which will not be elaborated further in this disclosure.

[0234] The target light field image can be obtained through the above process. In some embodiments, after obtaining the target light field image, the acquisition device 100 can send the target light field image to the display device 200, and the display device 200 can render and display the target light field image on the display screen. Thus, the user B at the display device 200 can see the light field video that changes with their viewpoint, presenting an immersive naked-eye 3D effect.

[0235] As can be seen from the above, in this embodiment of the present disclosure, cropping the image to be processed based on the boundary range of the target object can reduce the risk of placing the target object at the edge of the image or even cropping it out, improve the quality of the cropped image, provide a better data foundation for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication.

[0236] In some embodiments, this disclosure provides a light field image processing apparatus, see [link to relevant documentation]. Figure 14 As shown, the light field image processing apparatus of this disclosure includes:

[0237] The image acquisition module 10 is configured to acquire multiple images to be processed by multiple cameras installed on the acquisition device; the multiple images to be processed are acquired images from different perspectives of the target object;

[0238] The boundary search module 20 is configured to determine the boundary range of the target object on the image to be processed for each image to be processed;

[0239] The cropping module 30 is configured to crop the image to be processed according to the boundary range and the preset image scale to obtain the target image corresponding to each image to be processed.

[0240] The viewpoint fusion module 40 is configured to perform viewpoint fusion processing on each target image based on the target viewpoint information to obtain a target light field image corresponding to the target viewpoint information; the target viewpoint information represents the position information of the observer's eye on the display device.

[0241] In some implementations, the boundary search module 20 is configured to:

[0242] For each image to be processed, binarization is performed to obtain a binary image of the target object.

[0243] Based on the pixel values ​​in the binary image, the target object is searched row by row and column by column to obtain the horizontal and vertical boundaries of the target object in the binary image.

[0244] The boundary range is determined based on the horizontal and vertical boundaries.

[0245] In some implementations, the boundary search module 20 is configured to:

[0246] Based on the pixel values ​​on the binary image, the number of black pixels in each column is detected sequentially from left to right. In response to the fact that the number of black pixels in the first column and the subsequent preset number of columns are all greater than the first preset threshold, the coordinate information corresponding to the first column is determined as the left boundary of the horizontal boundary.

[0247] The number of black pixels in each column is detected sequentially from right to left. In response to the fact that the number of black pixels in the second column and the subsequent preset number of columns are all greater than the second preset threshold, the coordinate information corresponding to the second column is determined as the right boundary of the horizontal boundary.

[0248] The number of black pixels in each row is detected sequentially from top to bottom. If the number of black pixels in the first row and the subsequent preset number of rows is greater than the third preset threshold, the coordinate information corresponding to the first row is determined as the upper boundary of the vertical boundary.

[0249] The number of black pixels in each row is detected sequentially from bottom to top. If the number of black pixels in the second row and the subsequent preset number of rows is greater than the fourth preset threshold, the coordinate information corresponding to the second row is determined as the lower boundary of the vertical boundary.

[0250] In some implementations, the boundary search module 20 is configured to:

[0251] Each image to be processed is processed by image matting to obtain a foreground image including the target object for each image;

[0252] Binarize each foreground image to obtain a binary image of the target object.

[0253] In some implementations, the boundary search module 20 is configured to:

[0254] Based on the pixel values ​​in the binary image, a sliding window of a preset size is used to search the binary image at a preset step size;

[0255] In each sliding window, the pixels included in the sliding window are denoised based on the sum of the pixel values ​​of the pixels included in the sliding window.

[0256] In some implementations, the trimming module 30 is configured to:

[0257] Determine the coordinates of the center point of the target object based on the boundary range;

[0258] The center point coordinates of the target object are determined as the center point coordinates of the target image. The image to be processed is then cropped according to a preset image scale to obtain the target image.

[0259] In some implementations, the viewpoint fusion module 40 is configured to:

[0260] Input at least two of the multiple images to be processed into a pre-trained deep network model to obtain the depth map of the target object output by the deep network model.

[0261] Viewpoint fusion processing is performed on the depth map based on the target viewpoint information to obtain the target viewpoint depth map under the viewpoint corresponding to the target viewpoint information;

[0262] The target image, target viewpoint depth map, and target viewpoint information are input into a pre-trained viewpoint fusion model to obtain the target light field image output by the viewpoint fusion model.

[0263] In some embodiments, the apparatus of this disclosure is applied to a data acquisition device, which further includes a transmitting module configured to:

[0264] The target light field image is sent to the display device so that the display device can render and display the target light field image.

[0265] As can be seen from the above, in this embodiment of the present disclosure, cropping the image to be processed based on the boundary range of the target object can reduce the risk of placing the target object at the edge of the image or even cropping it out, improve the quality of the cropped image, provide a better data foundation for subsequent viewpoint fusion processing, and thus improve the quality and efficiency of light field video communication.

[0266] In some embodiments, this disclosure provides a video communication system, which can, as... Figure 1 As shown, it includes:

[0267] Display device 200 includes an image acquisition device and a first controller;

[0268] The acquisition device 100 includes multiple cameras and a second controller, at least one of which is used to execute the method according to any of the above embodiments.

[0269] In some embodiments, this disclosure provides a storage medium storing computer instructions for causing a computer to perform the methods of any of the above embodiments.

[0270] In some embodiments, this disclosure provides an electronic device, including:

[0271] processor; and

[0272] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0273] In this embodiment of the disclosure, the electronic device can be either the acquisition device 100 or the display device 200 described above, and this disclosure does not impose any limitations on it. Specifically, Figure 15 A schematic diagram of the structure of an electronic device 600 suitable for implementing the method of this disclosure is shown. Figure 15 The electronic device shown can perform the corresponding functions of the processor, controller and storage medium described above.

[0274] like Figure 15 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded into the memory 602 from a storage section 608. The memory 602 also stores various programs and data required for the operation of the electronic device 600. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0275] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0276] In particular, according to embodiments of this disclosure, the above-described method process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0277] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0278] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this disclosure.

Claims

1. A method of light field image processing, the method comprising: The method comprises: acquiring a plurality of to-be-processed images respectively captured by a plurality of cameras arranged on a collection device; the plurality of to-be-processed images are collection images including different perspectives of a target object; for each to-be-processed image, determining a boundary range of the target object on the to-be-processed image; according to the boundary range and a preset image scale, performing a cutting process on the to-be-processed image to obtain a target image corresponding to each to-be-processed image; performing view point fusion processing on each target image according to target view point information to obtain a target light field image corresponding to the target view point information; the target view point information represents position information of an observer's eye at a display device end; performing view point fusion processing on each target image according to target view point information to obtain a target light field image corresponding to the target view point information, comprising: inputting at least two to-be-processed images in the plurality of to-be-processed images into a pre-trained deep network model to obtain a depth map of the target object output by the deep network model; determining a horizontal scaling coefficient and a vertical scaling coefficient according to an image scale of the to-be-processed image and an image scale of the target image, and determining a horizontal offset and a vertical offset according to a center point coordinate of the to-be-processed image and a center point coordinate of the target image; updating an intrinsic matrix of a camera that captures the to-be-processed image according to the horizontal scaling coefficient, the vertical scaling coefficient, the horizontal offset and the vertical offset; performing view point fusion processing on the depth map based on an extrinsic matrix of the camera, the updated intrinsic matrix and the target view point information to obtain a target view point depth map under a view point corresponding to the target view point information; inputting the target image, the target view point depth map and the target view point information into a pre-trained view point fusion model to obtain the target light field image output by the view point fusion model.

2. The method of claim 1, wherein, The method further comprises: for each to-be-processed image, performing binaryzation processing on the to-be-processed image to obtain a binary image of the target object; based on pixel values on the binary image, sequentially performing boundary search on the target object row by row and column by column to obtain a horizontal boundary and a vertical boundary of the target object on the binary image; based on the horizontal boundary and the vertical boundary, determining the boundary range.

3. The method of claim 2, wherein, The method further comprises: based on the pixel values on the binary image, sequentially detecting the number of black pixels of the column of pixels from left to right, and in response to the number of black pixels of the first column of pixels and the continuous preset number of columns of pixels after the first column of pixels both being greater than a first preset threshold, determining coordinate information corresponding to the first column of pixels as a left boundary of the horizontal boundary. detecting the number of black pixels of the column of pixels from right to left in sequence, and determining the coordinate information corresponding to the second column of pixels as the right boundary of the horizontal boundary in response to the number of black pixels of the second column of pixels and the continuous preset number of column of pixels after the second column of pixels being greater than a second preset threshold value; detecting the number of black pixels of the row of pixels from top to bottom in sequence, and determining the coordinate information corresponding to the first row of pixels as the upper boundary of the vertical boundary in response to the number of black pixels of the first row of pixels and the continuous preset number of rows of pixels after the first row of pixels being greater than a third preset threshold value; detecting the number of black pixels of the row of pixels from bottom to top in sequence, and determining the coordinate information corresponding to the second row of pixels as the lower boundary of the vertical boundary in response to the number of black pixels of the second row of pixels and the continuous preset number of rows of pixels after the second row of pixels being greater than a fourth preset threshold value.

4. The method of claim 2, wherein, The method further comprises: performing binarization processing on each of the to-be-processed images to obtain a binary image of the target object, including: performing cutout processing on each of the to-be-processed images to obtain a foreground image corresponding to each of the to-be-processed images and including the target object; 5. The method of claim 2, wherein, performing binarization processing on each of the foreground images to obtain a binary image of the target object. The method further comprises: searching the target object in sequence row by row and column by column based on pixel values on the binary image to obtain a horizontal boundary and a vertical boundary of the target object on the binary image.

6. The method of claim 1, wherein, performing denoising processing on the pixels included in each sliding window based on the sum of the pixel values of the pixels included in the sliding window. The method further comprises: determining a center point coordinate of the target object according to the boundary range; 7. The method of claim 1, wherein, determining the center point coordinate of the target object as a center point coordinate of the target image, and performing cutting processing on the to-be-processed image to obtain the target image in the preset image scale. The method further comprises:

8. An optical field image processing apparatus, characterized by comprising: sending the target light field image to the display device to enable the display device to render and display the target light field image. The method further comprises: an image acquisition module configured to acquire a plurality of to-be-processed images respectively acquired by a plurality of cameras arranged on a collection device; the plurality of to-be-processed images are collection images of different perspectives including a target object; a boundary search module configured to determine, for each of the to-be-processed images, a boundary range of the target object on the to-be-processed image; a cutting processing module configured to perform cutting processing on the to-be-processed image according to the boundary range and a preset image scale to obtain a target image corresponding to each of the to-be-processed images; and a cutting processing module configured to perform cutting processing on the to-be-processed image according to the boundary range and a preset image scale to obtain a target image corresponding to each of the to-be-processed images. The view point fusion module is configured to perform view point fusion processing on each target image according to target view point information to obtain a target light field image corresponding to the target view point information, wherein the target view point information represents position information of an observer's eye at a display device end. The view point fusion module is configured to: perform view point fusion processing on each target image according to target view point information to obtain a target light field image corresponding to the target view point information, comprising: inputting at least two of the plurality of to-be-processed images into a pre-trained deep network model to obtain a depth map of the target object output by the deep network model; determining a horizontal scaling coefficient and a vertical scaling coefficient according to an image size of the to-be-processed image and an image size of the target image, and determining a horizontal offset and a vertical offset according to a center point coordinate of the to-be-processed image and a center point coordinate of the target image; updating an intrinsic matrix of a camera for collecting the to-be-processed image according to the horizontal scaling coefficient, the vertical scaling coefficient, the horizontal offset, and the vertical offset; performing view point fusion processing on the depth map based on an extrinsic matrix of the camera, the updated intrinsic matrix, and the target view point information to obtain a target view point depth map at a view point corresponding to the target view point information; inputting the target image, the target view point depth map, and the target view point information into a pre-trained view point fusion model to obtain the target light field image output by the view point fusion model.

9. An electronic device, comprising: comprising: a processor; and a memory storing computer instructions for causing the processor to execute the method according to any one of claims 1 to 7.

10. A video communication system, characterized by comprising: a display device comprising an image acquisition apparatus and a first controller; an acquisition device comprising a plurality of cameras and a second controller, at least one of the first controller and the second controller being configured to execute the method according to any one of claims 1 to 7.

11. A storage medium, characterized by storing computer instructions for causing a computer to execute the method according to any one of claims 1 to 7.

Images