Image processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: in a process of displaying an image by a transparent screen, under the condition of determining that each pixel on the transparent screen in a field angle range has the same color according to an RGB value corresponding to a first camera, taking an image collected by the first camera as an initial image; removing the same color from the initial image to obtain a target image; and outputting the target image to another display device. By adopting the method, one user can clearly see the eyes of the other user when the video conference is opened, so that a sense of facing is formed, the communication effect of the two parties during conference is improved, and the quality of the teleconference is improved. According to the method, the image collected by any first camera can be updated according to the image collected by the second camera, so that the situation that the image collected by the first camera is unavailable is avoided.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an image processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] A video conferencing system is a conference system that enables people in two or more locations to have face-to-face conversations through communication equipment and the Internet, and is usually used in remote meetings.

[0003] The video conferencing systems currently in use generally have cameras. In one case, the camera is integrated into the all-in-one conference machine in the upper frame of the display screen. The all-in-one machine is easy to deploy, but the integrated camera configuration is relatively poor. In another case, the camera is separated from the display screen and deployed separately. Generally, users place independent cameras directly above the display screen. There is a common problem in these two cases. The cameras are all above the display screen, and the cameras use a bird's-eye view when shooting people. This bird's-eye view video is transmitted to the other party participating in the video conference, which makes it difficult for the other party to see the eyes of the user on this side, and it is impossible to form a "eye-to-eye" feeling, resulting in less eye contact between the two parties participating in the video conference, affecting the communication effect, and thus reducing the quality of the remote conference.

[0004] Therefore, how to enable one user to see the eyes of the other user clearly during a video conference, form a feeling of "eye contact", improve the communication effect between the two parties during the meeting, and improve the quality of remote meetings. Summary of the invention

[0005] Based on this, it is necessary to provide an image processing method, device, computer equipment, storage medium and program product that can enable one user to see the eyes of the other user clearly during a video conference, form a feeling of "eye contact", improve the communication effect between the two parties during the meeting, and improve the quality of remote conferences.

[0006] In a first aspect, an image processing method is provided, which is applied to a display device, wherein the display device includes a body, a transparent screen, a plurality of first cameras, and a processor; wherein the transparent screen and the plurality of first cameras are both arranged on the body, and the transparent screen covers the lens of the first camera; and the transparent screen and the plurality of first cameras are both connected to the processor signal;

[0007] The method comprises:

[0008] In the process of displaying an image on the transparent screen, obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera;

[0009] When it is determined according to the RGB values ​​corresponding to the first camera that each pixel on the transparent screen within the field of view has the same color, using the image captured by the first camera as the initial image;

[0010] Removing the same color from the initial image to obtain a target image;

[0011] The target image is output to another display device.

[0012] In one embodiment, the display device further includes a second camera, the second camera is disposed on the body, and a lens of the second camera is not covered by the transparent screen;

[0013] The method further comprises:

[0014] When it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels on the transparent screen within the corresponding field of view angle range with different colors, the image captured by the second camera is used as a reference image;

[0015] Taking an image captured by any of the first cameras as an image to be processed, and for a same user captured between the image to be processed and the reference image, determining a first image region where the same user is located in the image to be processed, and a second image region where the same user is located in the reference image;

[0016] The colors of the pixels in the first image area in the image to be processed are replaced with the colors of the pixels in the second image area in the reference image to obtain a target image.

[0017] In one embodiment, the method further comprises:

[0018] Extracting a first image feature of the image to be processed and a second image feature of the reference image; wherein the first image feature at least includes the grayscale of the image to be processed, and the second image feature at least includes the grayscale of the reference image;

[0019] The first image feature and the second image feature are input into an image recognition classifier, and the same user captured between the image to be processed and the reference image is determined according to output information of the image recognition classifier.

[0020] In one of the embodiments, before the transparent screen plays the image, the method further includes:

[0021] For each of the first cameras, obtaining pixel coordinates of pixels on the transparent screen within the field of view of the first camera;

[0022] The step of obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera includes:

[0023] For each of the first cameras, the RGB values ​​of the pixels on the transparent screen within the field of view of the first camera are determined according to the RGB value of each pixel in the image displayed by the transparent screen and the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera.

[0024] In one embodiment, removing the same color from the initial image includes:

[0025] For the first camera corresponding to the initial image, obtaining pixel coordinates of pixels on the transparent screen within the field of view of the first camera;

[0026] Determining, according to pixel coordinates of pixels on the transparent screen within the field of view of the first camera, a sub-region to be processed in the initial image from which the same color is to be removed;

[0027] The same color is removed from the sub-region to be processed.

[0028] In one of the embodiments, before obtaining the pixel coordinates of the corresponding pixels on the transparent screen within the field of view angle range for each of the first cameras, the method further includes:

[0029] The coordinates of the first camera in a two-dimensional coordinate system are calibrated.

[0030] In a second aspect, a display device is provided, including: a body, a transparent screen, a plurality of first cameras and a processor;

[0031] The transparent screen and the first cameras are both arranged on the body, and the transparent screen covers the lenses of the first cameras;

[0032] The transparent screen and the first cameras are all connected to the processor signal;

[0033] The transparent screen is configured to display an image;

[0034] The processor is configured to:

[0035] In the process of displaying an image on the transparent screen, obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera;

[0036] When it is determined according to the RGB values ​​corresponding to the first camera that each pixel on the transparent screen within the field of view has the same color, using the image captured by the first camera as the initial image;

[0037] Removing the same color from the initial image to obtain a target image;

[0038] The target image is output to another display device.

[0039] In one embodiment, the display device further includes a second camera, the second camera is disposed on the body, and a lens of the second camera is not covered by the transparent screen;

[0040] The processor is further configured to:

[0041] When it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels on the transparent screen within the corresponding field of view angle range with different colors, the image captured by the second camera is used as a reference image;

[0042] Using any image captured by the first camera as an image to be processed;

[0043] For the same user captured between the image to be processed and the reference image, determine a first image region where the same user is located in the image to be processed, and a second image region where the same user is located in the reference image;

[0044] The colors of the pixels in the first image area in the image to be processed are replaced with the colors of the pixels in the second image area in the reference image to obtain a target image.

[0045] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0046] According to a fourth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0047] The above-mentioned image processing method, device, computer equipment, storage medium and program product utilize the "transparent" feature of the transparent screen, combined with the first camera embedded under the transparent screen, and solve the problem that the two parties in the video conference cannot "look at each other" that is common in video conferencing systems on the market through flexible camera scheduling and algorithm optimization. The two parties participating in the video conference can freely communicate with each other through eye contact, thereby improving the communication effect between the two parties in the meeting and improving the quality of the video conference. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] FIG1 is a diagram of an application environment of an image processing method according to an embodiment;

[0050] FIG2 is a schematic diagram of a flow chart of an image processing method in one embodiment;

[0051] FIG3 is a schematic diagram of a display device in one embodiment;

[0052] FIG4 is a partial flow diagram of an image processing method in one embodiment;

[0053] FIG5 is a schematic diagram of communication between a display device and another display device in one embodiment;

[0054] FIG6 is a partial flow chart of an image processing method according to an embodiment;

[0055] FIG7 is a schematic flow chart of an image processing method in yet another embodiment;

[0056] FIG8 is a schematic diagram of the structure of a display device in one embodiment;

[0057] FIG9 is a block diagram of an image processing device according to an embodiment;

[0058] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The image processing method provided in the embodiment of the present application can be applied in the application environment as shown in FIG1 . Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, tablet computers, smart TVs, etc. The server 104 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud computing services.

[0061] It should be noted that the image processing method provided in any of the following embodiments is not only applicable to video conferencing scenarios, but also applicable to other two-party video scenarios such as video scenarios through applications.

[0062] In an exemplary embodiment, as shown in FIG. 2 , an image processing method is provided, and the method is described by taking the case where the method is applied to the terminal 102 in FIG. 1 , where the terminal 102 is a display device.

[0063] As shown in FIG3 , the display device includes a body, a transparent screen, a plurality of first cameras and a processor. The transparent screen and the plurality of first cameras are both arranged on the body, and the transparent screen covers the lens of the first camera. The transparent screen and the plurality of first cameras are both connected to the processor signal. The plurality of first cameras shown in FIG3 are camera 1, camera 2, camera 3 and camera 4 respectively. The number and type of the first cameras can be set according to actual needs, and are not limited in this embodiment.

[0064] The image processing method includes the following steps 201 to 204. Among them:

[0065] Step 201 , when the transparent screen displays an image, obtain the RGB value of each pixel on the transparent screen within the field of view of the first camera.

[0066] Among them, the transparent screen is a transparent display screen, which is a new display method. It combines the characteristics of a screen and transparency. It can be used as a screen and can also replace transparent flat glass. Static images, dynamic images (such as video images), etc. can be displayed on the transparent screen. While the user sees the content displayed on the transparent screen, he can also see the objects behind the transparent screen. Similarly, the first camera behind the transparent screen can also capture the user's image and other images (such as the conference room setting).

[0067] The main types of transparent screens include transparent organic light emitting diode (OLED) display screens, transparent light emitting diode (LED) display screens, and transparent liquid crystal display (LCD) display screens, etc. The type of transparent screen used in this embodiment can be selected according to actual needs, and this embodiment does not limit it.

[0068] The first camera may also be referred to as the first camera, which is used to capture images. The lens of the first camera is covered by a transparent screen, so the image captured by the first camera includes two parts, one part is the image displayed on the transparent screen, and the other part is the image captured through the transparent screen. For example, in a conference system, the image captured by the first camera through the transparent screen may include users, conference room scenery, etc. Preferably, the first camera is a color camera (RGB camera, R is Red, G is Green, B is Blue), which can simultaneously capture the three main color channels of red, green and blue of the image, thereby generating a complete color image.

[0069] The field of view (FOV) refers to the angle range in which a camera can receive an image in an imaging scene, and is also often referred to as the field of view. The field of view of different first cameras may be the same or different, and this embodiment does not limit this.

[0070] Since the coordinates of each pixel on the transparent screen and the RGB value of each pixel are known during the process of displaying an image on the transparent screen, after knowing the field of view angle range of each first camera, the coordinates of each pixel on the transparent screen within the field of view angle range of the first camera can be known. For each first camera, based on the coordinates of the pixel on the transparent screen within the field of view angle range of the first camera, the coordinates of each pixel on the transparent screen and the RGB value of each pixel, the RGB value of the pixel on the transparent screen within the field of view angle range of the first camera can be known.

[0071] Among them, RGB values ​​can also be understood as color numbers. Color numbers refer to the numerical representation of color, usually used to describe the specific composition and intensity of color. In the fields of computer and design, color numbers are usually represented by the RGB (red, green, and blue) color model, which defines colors by three values ​​(representing the intensity of the red, green, and blue color channels respectively). For example, the RGB value of red is (255, 0, 0), which means that the intensity of the red channel is the maximum (255), while the intensity of the green and blue channels is 0.

[0072] Step 202: when it is determined according to the RGB value corresponding to the first camera that each pixel on the transparent screen within the field of view has the same color, the image captured by the first camera is used as the initial image.

[0073] The RGB value corresponding to the first camera refers to the RGB value of the pixel on the transparent screen within the field of view of the first camera acquired in step 202 .

[0074] One RGB value represents one color, so the color of each pixel on the transparent screen within the field of view can be determined by the RGB value corresponding to the first camera. The same color is red, for example.

[0075] When there are colored pixels on the transparent screen within the field of view of the lens in front of the first camera, the color of the picture taken by the first camera through the transparent screen pixels is no longer true. In particular, when there are more than two colors of pixels within the field of view of the camera embedded behind the transparent screen, it is difficult to restore the picture to its original color even if the algorithm is used for processing. However, the image processing algorithm can process an image with only one pure color superimposed on the original color image, remove the superimposed pure color, and restore the original color image.

[0076] Therefore, preferably, when it is determined that each pixel on the transparent screen within the field of view angle has the same color, in order to restore the original color image (ie, the target image) more simply and quickly, the image captured by the first camera is used as the initial image.

[0077] It should be noted that the display device is provided with multiple first cameras. If the multiple pixels on the transparent screen corresponding to one of the first cameras have different colors, it can be determined whether each pixel on the transparent screen corresponding to other first cameras has the same color. After it is determined that each pixel on the transparent screen corresponding to the first camera has the same color, the image captured by the first camera is used as the initial image. For example, if the multiple pixels on the transparent screen corresponding to camera 1 and camera 2 have different colors, but each pixel on the transparent screen corresponding to camera 3 has the same color, the image captured by camera 3 is used as the initial image.

[0078] Step 203, removing the same color from the initial image to obtain a target image.

[0079] After removing the same color, the obtained target image can be understood as a real image captured by the first camera when it is not blocked by the transparent screen.

[0080] Optionally, an image processing algorithm may be used to remove the same color in the initial image.

[0081] The image processing algorithm can process an image with only one pure color superimposed on the original color image, remove the superimposed pure color, and restore the original color image (i.e., the target image). Therefore, the image processing algorithm is used to remove the same color from the initial image to obtain the target image.

[0082] Image processing algorithms, such as segmentation algorithms and clustering algorithms based on machine learning, can segment and classify the colors in an image, and then remove the area of ​​the specified color. Please refer to the processing schematic diagram shown in Figure 4. After determining the initial image, the pixel coordinates of each pixel in the initial image are obtained, and the RGB value (color number) of the same color in the initial image is obtained. The image processing algorithm then removes the same color based on the pixel coordinates and RGB values, and restores the original color picture or original color video, that is, the target image is obtained.

[0083] Optionally, before processing the initial image, noise reduction processing, smoothing processing, etc. may be performed on the initial image.

[0084] Step 204: output the target image to another display device.

[0085] The display device is a communication terminal, and the other display device is another communication terminal. The display device and the other display device are two communication terminals in a video conference system. Please refer to FIG5 , the display device and the other display device can communicate with each other via the Internet.

[0086] After obtaining the target image, in order to enable the user on the other display device to see the user on the display device, the display device outputs the target image to the other display device, and the other display device displays the target image. In this way, when the picture or video is displayed through the transparent screen, it can also be achieved that when both parties of the video are watching the picture or video, they can also see the user through the transparent screen, achieving the effect of watching and communicating with the user at the same time.

[0087] In the above-mentioned image processing method, the "transparent" feature of the transparent screen is utilized, combined with the first camera embedded under the transparent screen. Through the system's flexible scheduling of cameras and algorithm optimization, the problem that the two parties in the video conference cannot "look at each other" that is common in the current video conferencing systems on the market is solved. The two parties participating in the video conference can freely communicate with each other through eye contact, which improves the communication effect between the two parties in the meeting and improves the quality of the video conference.

[0088] In an exemplary embodiment, as shown in FIG3 , the display device further includes a second camera. The second camera is disposed on the body, and the lens of the second camera is not covered by the transparent screen. For example, the second camera shown in FIG3 is camera 5.

[0089] 6 , the image processing method further includes steps 601 to 603 . Step 601 may be performed after step 201 .

[0090] Step 601: when it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels with different colors on the transparent screen within the corresponding field of view angle, the image captured by the second camera is used as a reference image.

[0091] There are multiple pixels with different colors on the transparent screen corresponding to the first camera. For example, the corresponding transparent screen has red, green, and blue colors, which proves that the colors of all pixels on the transparent screen within the field of view of the first camera are not the same color.

[0092] Since the lens of the second camera is not blocked by the transparent screen, the pixel color captured by the second camera is the pixel color that is not blocked. Therefore, if it is determined that the colors of all pixels on the corresponding transparent screen within the field of view of each first camera are not the same color, the image captured by the second camera is selected as the reference image.

[0093] Step 602, taking any image captured by the first camera as an image to be processed, and for the same user captured between the image to be processed and the reference image, determining a first image area where the same user is located in the image to be processed, and a second image area where the same user is located in the reference image.

[0094] Each image captured by the first camera can be used as an image to be processed. For example, the image captured by camera 1 can be selected as the image to be processed.

[0095] The content of the image to be processed is matched with the reference image, and the same elements are matched and marked. The same elements include the same user, the same object, the same building, etc.

[0096] In order to achieve the effect of two parties looking at each other in the video, only the same users captured between the image to be processed and the reference image can be marked. The first image area where the same user is located in the image to be processed is determined, and the second image area where the same user is located in the reference image is determined. The number of the same user may be more than one, for example, when there are two users, the first image area includes the first image area corresponding to one user and the first image area corresponding to the other user, and the second image area includes the second image area corresponding to the one user and the second image area corresponding to the other user.

[0097] Step 603: Replace the colors of the pixels in the first image area in the image to be processed with the colors of the pixels in the second image area in the reference image to obtain a target image.

[0098] That is, by replacing the color of the pixels in the first image area with the color of the pixels in the second image area, the real image of the same user can be restored. For example, the color of the pixels in the first image area of ​​camera 1 is replaced with the color of the pixels in the second image area of ​​camera 5.

[0099] After the color replacement of the image to be processed is completed, the target image can be obtained. Since the shooting angle of the image to be processed is through the transparent screen, the target image can be seen by the user on the other side of the video as being able to look at the user at the display device.

[0100] In this embodiment, when it is determined that there are multiple pixels with different colors on the transparent screen within the corresponding field of view according to the RGB values ​​corresponding to each of the first cameras, the color of any image captured by the first camera is replaced according to the image captured by the second camera, thereby solving the problem that the existing image processing algorithm cannot remove multiple colors. Furthermore, the unusability of the image captured by the first camera is avoided, and the problem that the two parties in the video conference cannot "look at each other" that is common in the video conferencing systems on the market is solved through the flexible scheduling of cameras and algorithm optimization by the system, so that the two parties participating in the video conference can freely communicate with each other through eye contact, which improves the communication effect of the two parties in the meeting and improves the quality of the video conference.

[0101] In an exemplary embodiment, before step 602, when determining the same user captured between the image to be processed and the reference image, it includes:

[0102] Step 1: extracting a first image feature of the image to be processed and a second image feature of the reference image.

[0103] The first image feature at least includes the grayscale of the image to be processed, and may also include texture information, RGB values ​​of pixels, etc. of the image to be processed.

[0104] The second image feature includes at least the grayscale of the reference image, and may also include texture information, RGB values ​​of pixels, etc. of the reference image.

[0105] Step 2: input the first image feature and the second image feature into an image recognition classifier, and determine the same user captured between the image to be processed and the reference image according to the output information of the image recognition classifier.

[0106] The image recognition classifier is a machine learning or deep learning model that is used to identify objects in an image and classify them into predefined categories. The image recognition classifier uses the first image feature to identify elements in the image to be processed and the second image feature to identify elements in the reference image. The same user is then matched from the elements in the image to be processed and the elements in the reference image.

[0107] In an exemplary embodiment, before the transparent screen plays the image in step 201, the image processing method further includes:

[0108] Step 1: For each of the first cameras, obtain pixel coordinates of pixels on the transparent screen within the field of view of the first camera.

[0109] For each of the first cameras, the pixel coordinates of all pixels on the transparent screen within the field of view of the first camera are marked. For example, for camera 1 as described above, the pixel coordinates of all pixels on the transparent screen within the field of view of camera 1 are marked. For camera 2, the pixel coordinates of all pixels on the transparent screen within the field of view of camera 2 are marked. The pixel coordinates of all pixels on the transparent screen within the field of view of camera 3 are marked. The pixel coordinates of all pixels on the transparent screen within the field of view of camera 4 are marked.

[0110] Step 201 of obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera includes:

[0111] Step 2: For each of the first cameras, determine the RGB value of the pixel on the transparent screen within the field of view of the first camera according to the RGB value of each pixel in the image displayed by the transparent screen and the pixel coordinates of the pixel on the transparent screen within the field of view of the first camera.

[0112] It is understandable that before displaying an image based on the transparent screen, the RGB value and pixel coordinates of each pixel in the displayed image need to be known first, and then the transparent screen can be controlled to display the image based on the known RGB values. The displayed image is a complete image.

[0113] According to the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera, a sub-image area corresponding to the field of view of the first camera can be determined from the displayed image, and the RGB value of the sub-image area is the RGB value of the pixels on the transparent screen within the field of view of the first camera.

[0114] In an exemplary embodiment, step 203 includes:

[0115] Step 1: for the first camera corresponding to the initial image, the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera are obtained.

[0116] For each of the first cameras, the pixel coordinates of all pixels on the transparent screen within the field of view of the first camera are marked. For example, for camera 1 as described above, the pixel coordinates of all pixels on the transparent screen within the field of view of camera 1 are marked. For camera 2, the pixel coordinates of all pixels on the transparent screen within the field of view of camera 2 are marked. The pixel coordinates of all pixels on the transparent screen within the field of view of camera 3 are marked. The pixel coordinates of all pixels on the transparent screen within the field of view of camera 4 are marked.

[0117] Optionally, before step 1, the coordinates of the first camera in a two-dimensional coordinate system are calibrated to improve the accuracy of pixel coordinates.

[0118] Step 2: Determine the sub-region to be processed in the initial image from which the same color is to be removed according to the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera.

[0119] The coordinates of each pixel on the transparent screen are calibrated in advance. According to the coordinates of each pixel on the transparent screen and the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera, the coordinates of each pixel in the sub-area to be processed can be determined.

[0120] Step three, removing the same color in the sub-region to be processed.

[0121] An image processing algorithm may be used to remove the same color in the initial image.

[0122] The image processing algorithm can process an image with only one pure color superimposed on the original color image, remove the superimposed pure color, and restore the original color image (i.e., the target image). Therefore, the image processing algorithm is used to remove the same color from the initial image to obtain the target image.

[0123] Image processing algorithms, such as segmentation algorithms and clustering algorithms based on machine learning, can segment and classify the colors in an image and then remove areas of specified colors.

[0124] The method provided in this embodiment utilizes the "transparent" feature of the transparent screen, combined with the first camera embedded under the transparent screen, and solves the problem that the two parties in the video conference cannot "look at each other" that is common in video conferencing systems on the market through flexible camera scheduling and algorithm optimization. It allows the two parties participating in the video conference to freely communicate with each other, thereby improving the communication effect between the two parties in the meeting and improving the quality of the video conference.

[0125] For ease of understanding, the image processing method is described in more detail by taking the display device including the plurality of first cameras as camera 1, camera 2, camera 3 and camera 4, and the second camera as camera 5. Meanwhile, please refer to another flow chart of the image processing method shown in FIG. 7. As shown in FIG. 7, the image processing method includes:

[0126] The first step is to initialize the coordinate system. This includes: marking the coordinates of camera 1, camera 2, camera 3, camera 4, and camera 5. Marking the pixel coordinates of all pixels on the transparent screen within the field of view of camera 1. Marking the pixel coordinates of all pixels on the transparent screen within the field of view of camera 2. Marking the pixel coordinates of all pixels on the transparent screen within the field of view of camera 3. Marking the pixel coordinates of all pixels on the transparent screen within the field of view of camera 4.

[0127] In the second step, the transparent screen starts playing images.

[0128] The third step is to obtain the transparent screen pixels within the field of view of each camera of camera 1, camera 2, camera 3, and camera 4 during the transparent screen playback process, so as to facilitate the determination of the field of view of which camera all the pixels on the transparent screen are of the same color and the same color number. And obtain the color number of the pixel display color to prepare for the algorithm to process the picture.

[0129] In the fourth step, the system will first determine whether the color of all pixels on the transparent screen within the field of view of camera 1 is a pure color. If it is the same color, the image captured by camera 1 is used as the initial image, and then the color numbers (RGB values) of all transparent screen pixels within the field of view of camera 1 are extracted. The single color (that is, the same color) superimposed in the initial image is processed by the algorithm to obtain the target image. After obtaining the target image, the system will transmit the target image as the final output image of the system to the other party of the video conference.

[0130] In the fifth step, if it is determined that the colors of all pixels on the transparent screen within the field of view of camera 1 are not the same color, the system will continue to determine whether the colors of all pixels on the transparent screen within the field of view of camera 2 are the same color. If they are the same color, the image captured by camera 2 is used as the initial image, and then the color numbers (RGB values) of all transparent screen pixels within the field of view of camera 2 are extracted. The single color superimposed in the initial image is processed by the algorithm to obtain the target image. After obtaining the target image, the system will transmit the target image as the final output image of the system to the other party of the video conference.

[0131] In the sixth step, if it is determined that the colors of all pixels on the transparent screen within the field of view of camera 2 are not the same color, the system will continue to determine whether the colors of all pixels on the transparent screen within the field of view of camera 3 are the same color. If they are the same color, the image captured by camera 3 is used as the initial image, and then the color numbers (RGB values) of all transparent screen pixels within the field of view of camera 2 are extracted, and the single color superimposed in the initial image is processed by the algorithm to obtain the target image. After obtaining the target image, the system will transmit the target image as the final output image of the system to the other party of the video conference.

[0132] In the seventh step, if it is determined that the colors of all pixels on the transparent screen within the field of view of camera 3 are not the same color, the system will continue to determine whether the colors of all pixels on the transparent screen within the field of view of camera 4 are the same color. If they are the same color, the image captured by camera 4 is used as the initial image, and then the color numbers (RGB values) of all transparent screen pixels within the field of view of camera 2 are extracted, and the single color superimposed in the initial image is processed by the algorithm to obtain the target image. After obtaining the target image, the system will transmit the target image as the final output image of the system to the other party of the video conference.

[0133] In the eighth step, if the system determines that the colors of all pixels on the transparent screen within the field of view of camera 4 are not the same color, the system will select the image captured by camera 1 as the image to be processed, and match the image captured by camera 1 with the image captured by camera 5 (i.e., the second camera) for content. The two images are compared and matched to find the same elements and marked. Then, the color of the element in the image captured by camera 1 (the image to be processed) is replaced with the color of the same element in the image captured by camera 5 (the reference image). The system uses the image after replacing the colors of all elements in the image captured by camera 1 with the colors of the same elements captured by camera 5 as the target image (i.e., the final output image of the system), and transmits it to the other party of the video conference.

[0134] It should be understood that, although the steps in the flowcharts of the embodiments described above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0135] 8 , an embodiment of the present application further provides a display device 800 . The display device 800 includes a body 801 , a transparent screen 802 , a plurality of first cameras 803 , and a processor 804 .

[0136] The transparent screen 802 and the plurality of first cameras 803 are both disposed on the body 801. The transparent screen 802 covers the lens of the first camera 803. The location of the first camera 803 under the transparent screen 802 can be selected according to actual needs, which is not limited in this embodiment. The number of the first cameras 803 can be selected according to actual needs, which is not limited in this embodiment.

[0137] The transparent screen 802 and the plurality of first cameras 803 are all connected to the processor 804 by signal. The processor 804 may be integrated on the transparent screen 802 or may be disposed at other locations, which is not limited in this embodiment.

[0138] The transparent screen 802 is configured to display images.

[0139] The transparent screen 802 is a transparent display screen, which is a new display method. It combines the characteristics of a screen and transparency. It can be used as a screen and can also replace transparent flat glass. Static images, dynamic images (such as video images), etc. can be displayed on the transparent screen 802. While the user sees the content displayed on the transparent screen 802, he can also see the object behind the transparent screen. Similarly, the first camera 803 behind the transparent screen 802 can also capture the user image and other images (such as the conference room scenery).

[0140] The main types of transparent screen 802 include transparent organic light emitting diode (OLED) display screen, transparent light emitting diode (LED) display screen and transparent liquid crystal display (LCD) display screen, etc. The type of transparent screen 802 used in this embodiment can be selected according to actual needs, and this embodiment does not limit it.

[0141] The first camera 803 may also be referred to as a first camera, and is used to capture images. The lens of the first camera 803 is covered by the transparent screen 802, so the image captured by the first camera 803 includes two parts, one part is the image displayed on the transparent screen 802, and the other part is the image captured through the transparent screen 802. For example, in a conference system, the image captured by the first camera 803 through the transparent screen 802 may include users, conference room scenery, etc. Preferably, the first camera 803 is a color camera (RGB camera, R is Red, G is Green, B is Blue), which can simultaneously capture the three main color channels of red, green and blue of the image, thereby generating a complete color image.

[0142] The processor 804 is configured to execute the image processing method provided in any of the above embodiments.

[0143] Specifically, the processor 804 is configured to execute step 201 to step 204. For detailed description of step 201 to step 204, reference may be made to the above related description, which will not be repeated here.

[0144] The above-mentioned display device 800 utilizes the "transparent" feature of the transparent screen 802, combined with the first camera 803 embedded under the transparent screen 802, and solves the problem that the two parties in the video conference cannot "look at each other" that is common in the video conferencing systems on the market through flexible system scheduling of cameras and algorithm optimization. It allows the two parties participating in the video conference to freely communicate with each other through eye contact, thereby improving the communication effect between the two parties in the meeting and improving the quality of the video conference.

[0145] In an exemplary embodiment, referring to FIG8 , the display device further includes a second camera 805, which is disposed on the body 801, and a lens of the second camera 805 is not covered by the transparent screen 802. The location of the second camera 805 on the body 801 can be selected according to actual needs, and preferably, the second camera 805 is disposed on the upper side of the transparent screen 802.

[0146] The processor 804 is further configured to execute steps 601 to 603. For detailed descriptions of steps 601 to 603, reference may be made to the above related descriptions, which will not be repeated here.

[0147] In order to achieve the effect of two parties looking at each other in the video, only the same users captured between the image to be processed and the reference image can be marked. The first image area where the same user is located in the image to be processed is determined, and the second image area where the same user is located in the reference image is determined. The number of the same user may be more than one, for example, when there are two users, the first image area includes the first image area corresponding to one user and the first image area corresponding to the other user, and the second image area includes the second image area corresponding to the one user and the second image area corresponding to the other user.

[0148] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0149] In an exemplary embodiment, as shown in FIG9 , an image processing device 900 is provided, comprising: an acquisition module 901, a processing module 902 and a communication module 903, wherein:

[0150] The acquisition module 901 is used to acquire the RGB value of each pixel on the transparent screen within the field of view of the first camera during the process of displaying an image on the transparent screen.

[0151] The processing module 902 is used to use the image captured by the first camera as the initial image when it is determined according to the RGB value corresponding to the first camera that each pixel on the transparent screen within the field of view angle has the same color.

[0152] Processing module 902 removes the same color from the initial image to obtain a target image.

[0153] The communication module 903 is used to output the target image to another display device.

[0154] Optionally, the display device also includes a second camera, which is arranged on the main body, and the lens of the second camera is not covered by the transparent screen; the processing module 902 is also used to use the image captured by the second camera as a reference image when it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels with different colors on the transparent screen within the corresponding field of view angle; using any image captured by the first camera as the image to be processed, and for the same user captured between the image to be processed and the reference image, determining the first image area where the same user is located in the image to be processed, and the second image area where the same user is located in the reference image; replacing the color of the pixels in the first image area in the image to be processed with the color of the pixels in the second image area in the reference image to obtain the target image.

[0155] Optionally, the processing module 902 is also used to extract a first image feature of the image to be processed and a second image feature of the reference image; wherein the first image feature at least includes the grayscale of the image to be processed, and the second image feature at least includes the grayscale of the reference image; the first image feature and the second image feature are input into an image recognition classifier, and the same user captured between the image to be processed and the reference image is determined based on the output information of the image recognition classifier.

[0156] Optionally, the processing module 902 is further configured to obtain, for each of the first cameras, pixel coordinates of pixels on the transparent screen within the field of view of the first camera.

[0157] For each of the first cameras, the RGB values ​​of the pixels on the transparent screen within the field of view of the first camera are determined according to the RGB value of each pixel in the image displayed by the transparent screen and the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera.

[0158] Optionally, the processing module 902 is also used to obtain the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera corresponding to the initial image; determine the sub-area to be processed in the initial image where the same color is to be removed according to the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera; and remove the same color in the sub-area to be processed.

[0159] Optionally, the processing module 902 is further configured to calibrate the coordinates of the first camera in a two-dimensional coordinate system.

[0160] Each module in the above-mentioned image processing device 900 can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method provided in any of the above embodiments is implemented.

[0162] In one embodiment, a computer program product is provided, including a computer program, which implements the image processing method provided in any of the above embodiments when executed by a processor.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: Applied to a display device, the display device comprises a body, a transparent screen, a plurality of first cameras and a processor; wherein the transparent screen and the plurality of first cameras are both arranged on the body, and the transparent screen covers the lens of the first camera; the transparent screen and the plurality of first cameras are both connected to the processor signal; The method comprises: In the process of displaying an image on the transparent screen, obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera; When it is determined according to the RGB values ​​corresponding to the first camera that each pixel on the transparent screen within the field of view has the same color, using the image captured by the first camera as the initial image; Removing the same color from the initial image to obtain a target image; The target image is output to another display device.

2. The method according to claim 1, characterized in that: The display device further includes a second camera, which is disposed on the body, and a lens of the second camera is not covered by the transparent screen; The method further comprises: When it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels on the transparent screen within the corresponding field of view angle range with different colors, the image captured by the second camera is used as a reference image; Taking an image captured by any of the first cameras as an image to be processed, and for a same user captured between the image to be processed and the reference image, determining a first image region where the same user is located in the image to be processed, and a second image region where the same user is located in the reference image; The colors of the pixels in the first image area in the image to be processed are replaced with the colors of the pixels in the second image area in the reference image to obtain a target image.

3. The method according to claim 2, characterized in that The method further comprises: Extracting a first image feature of the image to be processed and a second image feature of the reference image; wherein the first image feature at least includes the grayscale of the image to be processed, and the second image feature at least includes the grayscale of the reference image; The first image feature and the second image feature are input into an image recognition classifier, and the same user captured between the image to be processed and the reference image is determined according to output information of the image recognition classifier.

4. The method according to claim 2, characterized in that: Before the transparent screen plays the image, the method further includes: For each of the first cameras, obtaining pixel coordinates of pixels on the transparent screen within the field of view of the first camera; The step of obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera includes: For each of the first cameras, the RGB values ​​of the pixels on the transparent screen within the field of view of the first camera are determined according to the RGB value of each pixel in the image displayed by the transparent screen and the pixel coordinates of the pixels on the transparent screen within the field of view of the first camera.

5. The method according to any one of claims 1 to 4, characterized in that: The removing of the same color from the initial image comprises: For the first camera corresponding to the initial image, obtaining pixel coordinates of pixels on the transparent screen within the field of view of the first camera; Determining, according to pixel coordinates of pixels on the transparent screen within the field of view of the first camera, a sub-region to be processed in the initial image from which the same color is to be removed; The same color is removed from the sub-region to be processed.

6. The method according to claim 4, characterized in that Before obtaining the pixel coordinates of the corresponding pixels on the transparent screen within the field of view angle range for each of the first cameras, the method further includes: The coordinates of the first camera in a two-dimensional coordinate system are calibrated.

7. A display device, characterized in that: include: A main body, a transparent screen, a plurality of first cameras and a processor; The transparent screen and the first cameras are both arranged on the body, and the transparent screen covers the lenses of the first cameras; The transparent screen and the first cameras are all connected to the processor signal; The transparent screen is configured to display an image; The processor is configured to: In the process of displaying an image on the transparent screen, obtaining the RGB value of each pixel on the transparent screen within the field of view of the first camera; When it is determined according to the RGB values ​​corresponding to the first camera that each pixel on the transparent screen within the field of view has the same color, using the image captured by the first camera as the initial image; Removing the same color from the initial image to obtain a target image; The target image is output to another display device.

8. The display device according to claim 7, characterized in that The display device further includes a second camera, which is disposed on the body, and a lens of the second camera is not covered by the transparent screen; The processor is further configured to: When it is determined according to the RGB values ​​corresponding to each of the first cameras that there are multiple pixels on the transparent screen within the corresponding field of view angle range with different colors, the image captured by the second camera is used as a reference image; Using any image captured by the first camera as an image to be processed; For the same user captured between the image to be processed and the reference image, determine a first image region where the same user is located in the image to be processed, and a second image region where the same user is located in the reference image; The colors of the pixels in the first image area in the image to be processed are replaced with the colors of the pixels in the second image area in the reference image to obtain a target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images