Image processing method, apparatus and system

Abstract

The present invention relates to the technical field of display, and provides an image processing method, apparatus and system. The image processing method is applied to a first image processing apparatus and comprises: acquiring eye state parameters of a viewer; sending the eye state parameters to a second image processing apparatus; receiving image data returned by the second image processing apparatus, wherein the resolution of the image data is determined by the eye state parameters; and on the basis of the image data, generating a multi-view image required by light field display. The technical solution in the present invention can achieve reduction of the power consumption of the image processing system.

Description

Image processing method, device and system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310785935.4 filed in China on June 29, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of display technology, and in particular to an image processing method, device, and system. Background Art

[0004] 3D displays have long been considered the ultimate dream in display technology development. After years of development, two major 3D display technology systems currently exist on the market: glasses-based and autostereoscopic. Autostereoscopic 3D display, also known as autostereoscopic, is the future trend in 3D display technology.

[0005] The application of holographic video communication technology using naked-eye 3D display technology in video conferencing will become increasingly popular. The holographic video communication system is equipped with multiple cameras and uses neural network modeling technology to generate the multi-viewpoint images required by light field displays. From data acquisition, to data transmission, to algorithm processing, to final display, the system requires the transmission and processing of ultra-large bandwidth data volumes, which places very high requirements on processor performance and network bandwidth, and generates huge power consumption.

[0006] Summary of the Invention

[0007] The technical problem to be solved by the present disclosure is to provide an image processing method, device and system, which can reduce the power consumption of the image processing system.

[0008] To solve the above technical problems, the embodiments of the present disclosure provide the following technical solutions:

[0009] In one aspect, an image processing method is provided, applied to a first image processing device, comprising:

[0010] Obtain the viewer's eye state parameters;

[0011] sending the human eye state parameter to a second image processing device;

[0012] receiving image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter;

[0013] Multi-viewpoint images required for light field display are generated according to the image data.

[0014] In some embodiments, obtaining the viewer's eye state parameters includes:

[0015] An image sensor is used to collect a facial image of the viewer, and eye state parameters of the viewer are determined based on the facial image. The eye state parameters of the viewer include:

[0016] A first parameter indicates that the viewer is in a state of looking at the screen;

[0017] A second parameter indicates that the viewer is in a scanning state;

[0018] The third parameter indicates that the viewer is not looking at the screen.

[0019] In some embodiments, sending the human eye state parameter to the second image processing device includes at least one of the following:

[0020] When it is predicted that the human eye state will change from the scanning state to the screen-gazing state, sending the first parameter to the second image processing device;

[0021] When it is predicted that the human eye state will change from being in a non-watching-the-screen state to being in a watching-the-screen state, sending the first parameter to the second image processing device;

[0022] When it is predicted that the human eye state will change from not looking at the screen state to scanning state, sending the second parameter to the second image processing device;

[0023] After determining that the state of the human eye changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device;

[0024] After determining that the state of the human eye changes from the scanning state to the non-looking-at-the-screen state, sending the third parameter to the second image processing device;

[0025] After determining that the state of the human eye changes from the state of looking at the screen to the state of not looking at the screen, the third parameter is sent to the second image processing device.

[0026] In some embodiments, the method further comprises:

[0027] When it is determined based on the facial image of the viewer that the movement speed of the viewer's eyeballs decreases, predicting that the state of the human eye will change from being in a scanning state to being in a screen-gazing state;

[0028] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen, it is predicted that the state of the human eye will change from being not looking at the screen to being looking at the screen;

[0029] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeballs is increasing, it is predicted that the state of the human eye will change from being in a non-watching screen state to being in a scanning state.

[0030] In some embodiments, generating multi-viewpoint images required for light field display based on the image data includes:

[0031] Establishing a neural network model, the neural network model including multiple input terminals, different input terminals corresponding to image data of different resolutions, the neural network model being used to generate multi-viewpoint images using the image data;

[0032] According to the resolution of the image data returned by the second image processing device, the image data is input into the corresponding input end, and the multi-viewpoint images required for light field display are output.

[0033] An embodiment of the present disclosure provides an image processing method, applied to a second image processing device, comprising:

[0034] receiving a human eye state parameter sent by the first image processing device;

[0035] Determining a target resolution corresponding to the human eye state parameter, and acquiring image data of the target resolution;

[0036] The image data is sent to the first image processing device.

[0037] In some embodiments, acquiring image data of a target resolution includes:

[0038] Acquire image data at the target resolution; or

[0039] Image data of a first resolution is collected, and the image data of the first resolution is compressed to obtain image data of a target resolution, where the first resolution is higher than the target resolution.

[0040] An embodiment of the present disclosure provides a first image processing apparatus, including:

[0041] A human eye state parameter acquisition module, used to acquire the viewer's human eye state parameters;

[0042] A human eye state parameter sending module, configured to send the human eye state parameter to a second image processing device;

[0043] an image data receiving module, configured to receive image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter;

[0044] A generating module is used to generate multi-viewpoint images required for light field display according to the image data.

[0045] In some embodiments, the eye state parameter acquisition module is specifically configured to capture a facial image of the viewer using an image sensor, and determine the eye state parameters of the viewer based on the facial image. The eye state parameters of the viewer include:

[0046] A first parameter indicates that the viewer is in a state of looking at the screen;

[0047] A second parameter indicates that the viewer is in a scanning state;

[0048] The third parameter indicates that the viewer is not looking at the screen.

[0049] In some embodiments, the human eye state parameter sending module is specifically configured to perform at least one of the following:

[0050] When it is predicted that the human eye state will change from the scanning state to the screen-gazing state, sending the first parameter to the second image processing device;

[0051] When it is predicted that the human eye state will change from being in a non-watching-the-screen state to being in a watching-the-screen state, sending the first parameter to the second image processing device;

[0052] When it is predicted that the human eye state will change from not looking at the screen state to scanning state, sending the second parameter to the second image processing device;

[0053] After determining that the human eye state changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device;

[0054] After determining that the state of the human eye changes from the scanning state to the non-looking-at-the-screen state, sending the third parameter to the second image processing device;

[0055] After determining that the state of the human eye changes from the state of looking at the screen to the state of not looking at the screen, the third parameter is sent to the second image processing device.

[0056] In some embodiments, the human eye state parameter acquisition module is specifically configured to predict that the human eye state will transition from a scanning state to a screen-gazing state when it is determined based on the facial image of the viewer that the movement speed of the viewer's eyeballs decreases;

[0057] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen, it is predicted that the state of the human eye will change from being not looking at the screen to being looking at the screen;

[0058] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeballs is increasing, it is predicted that the state of the human eye will change from being in a non-watching screen state to being in a scanning state.

[0059] In some embodiments, the generation module is specifically used to establish a neural network model, which includes multiple input terminals, different input terminals corresponding to image data of different resolutions, and the neural network model is used to generate multi-viewpoint images using image data; according to the resolution of the image data returned by the second image processing device, the image data is input into the corresponding input terminal, and the multi-viewpoint image required for light field display is output.

[0060] An embodiment of the present disclosure provides a second image processing apparatus, including:

[0061] A human eye state parameter receiving module, configured to receive the human eye state parameter sent by the first image processing device;

[0062] A determination module, configured to determine a target resolution corresponding to the human eye state parameter;

[0063] An image data acquisition module, used to acquire image data of target resolution;

[0064] An image data sending module is used to send the image data to the first image processing device.

[0065] In some embodiments, the image data acquisition module is specifically used to acquire image data of a target resolution; or

[0066] Image data of a first resolution is collected, and the image data of the first resolution is compressed to obtain image data of a target resolution, where the first resolution is higher than the target resolution.

[0067] An embodiment of the present disclosure provides an image processing system, comprising:

[0068] The first image processing device is configured to obtain eye state parameters of a viewer; send the eye state parameters to a second image processing device; receive image data returned by the second image processing device, wherein the resolution of the image data is determined by the eye state parameters; and generate multi-viewpoint images required for light field display based on the image data.

[0069] The second image processing device is used to receive the human eye state parameters sent by the first image processing device; determine the target resolution corresponding to the human eye state parameters and obtain image data of the target resolution; and send the image data to the first image processing device.

[0070] The embodiments of the present disclosure have the following beneficial effects:

[0071] In the above scheme, the first image processing device obtains the viewer's human eye state parameters and sends the human eye state parameters to the second image processing device. The second image processing device obtains image data of corresponding resolution based on the human eye state parameters and sends it to the first image processing device. In this way, the resolution of the transmitted and processed image data can be dynamically adjusted according to the viewer's human eye state, reducing the power consumption of processing and transmission while ensuring the viewer's user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] FIG1 is a schematic flow chart of an image processing method on a first image processing device side according to an embodiment of the present disclosure;

[0073] Figure 2 is a schematic diagram of a neural network model;

[0074] FIG3 is a schematic flow chart of an image processing method on the side of a second image processing device according to an embodiment of the present disclosure;

[0075] FIG4 is a schematic structural diagram of a first image processing device according to an embodiment of the present disclosure;

[0076] FIG5 is a schematic structural diagram of a second image processing device according to an embodiment of the present disclosure;

[0077] FIG6 is a schematic diagram of the structure of an image processing system according to an embodiment of the present disclosure;

[0078] 7 and 8 are schematic diagrams of interaction between image processing devices at both ends according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] In order to make the technical problems, technical solutions and advantages to be solved by the embodiments of the present disclosure more clear, they will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0080] Holographic video communication systems are a relatively important application scenario in 3D display. They include image processing devices at both ends. The image processing device at one end collects image data from its own end and sends it to the image processing device at the other end. At the same time, it receives image data from the image processing device at the other end and generates the multi-viewpoint images required for light field display based on the image data from the image processing device at the other end. It can be seen that from data acquisition to data transmission, algorithm processing, and final display, the transmission and processing of ultra-large bandwidth data volumes are required, placing very high demands on processor performance and network bandwidth, resulting in huge power consumption. In actual applications, for example, when the viewer at one end of the image processing device is not looking at the screen or their eyes are scanning, there is no need to collect and transmit large amounts of image data. In this scenario, the amount of image data transmitted and processed can be reduced.

[0081] Embodiments of the present disclosure provide an image processing method, apparatus, and system, which can reduce the power consumption of an image processing system.

[0082] An embodiment of the present disclosure provides an image processing method, which is applied to a first image processing device, as shown in FIG1 , and includes:

[0083] Step 101: Obtaining the viewer's eye state parameters;

[0084] In this embodiment, the first image processing device can capture a facial image of the viewer using a low-resolution, high-frame-rate camera and determine the viewer's eye state parameters based on the facial image. Low-resolution, high-frame-rate cameras have very low power consumption, which helps reduce the power consumption of the entire image processing system. Because changes in eye state are uncertain and rapid, cameras used for eye state detection should have as low a latency as possible.

[0085] In this embodiment, the viewer's eye state parameters may include:

[0086] A first parameter indicates that the viewer is in a state of looking at the screen;

[0087] A second parameter indicates that the viewer is in a scanning state;

[0088] The third parameter indicates that the viewer is not looking at the screen.

[0089] The viewer's eye state parameters can be indicated by 2 bits of data, such as 0 for the first parameter, 1 for the second parameter, and 2 for the third parameter. When the viewer is looking at the screen, that is, the viewer's eyes are stationary and facing the screen, the viewer needs to see a higher-resolution image. When the viewer is scanning, that is, the viewer's eyes are looking at the screen but not focused, such as when the head shakes and the eyes move synchronously, or when the head is relatively stationary but the eyes are scanning. The scanning state is easy to occur in 3D video conferencing scenarios, such as when the viewer is reading or taking notes. In this case, a lower-resolution image can be provided to the viewer. When the viewer is not looking at the screen, that is, the viewer is not looking at the screen, and the viewer may be looking down or looking at other areas outside the screen. In this case, the viewer can be provided with the lowest-resolution image.

[0090] Step 102: Sending the human eye state parameter to a second image processing device;

[0091] After determining the human eye state parameter, the first image processing device sends the human eye state parameter to the second image processing device through the communication connection between the first image processing device and the second image processing device.

[0092] Step 103: receiving image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter;

[0093] After receiving the human eye state parameter sent by the first image processing device, the second image processing device determines the target resolution of the image data sent to the first image processing device based on the human eye state parameter. The second image processing device may pre-store a correspondence between resolutions and human eye state parameters, such as resolution 1 corresponding to the first parameter, resolution 2 corresponding to the second parameter, and resolution 3 corresponding to the third parameter, where resolution 3 is smaller than resolution 2, and resolution 2 is smaller than resolution 1. If the second image processing device receives the first parameter, the target resolution is determined to be resolution 1; if the second image processing device receives the second parameter, the target resolution is determined to be resolution 2; and if the second image processing device receives the third parameter, the target resolution is determined to be resolution 3.

[0094] In some embodiments, the native resolution of the camera that captures image data on the second image processing device is the first resolution. Resolution 1 may be equal to the first resolution, Resolution 2 may be 1 / 2 of the first resolution, and Resolution 3 may be 1 / 4 of the first resolution. Of course, the values ​​of Resolution 1, Resolution 2, and Resolution 3 are not limited to these values. Resolution 2 may also be 1 / 2 of the first resolution, Resolution 3 may be 1 / 8 of the first resolution, and so on.

[0095] In the related art, the second image processing device encodes the image data of the first resolution and sends it to the first image processing device. In this embodiment, after the second image processing device receives the second parameter, the image data of resolution 2 is encoded and sent to the first image processing device. After the second image processing device receives the third parameter, the image data of resolution 3 is encoded and sent to the first image processing device, which can greatly reduce the amount of transmitted data.

[0096] Step 104: Generate multi-viewpoint images required for light field display according to the image data.

[0097] In this embodiment, the first image processing device can establish and train a neural network model, as shown in Figure 2. The neural network model may include multiple input terminals and multiple output terminals, and different input terminals correspond to image data of different resolutions. The neural network model is used to generate multi-view images using image data, and the output of the neural network model is the multi-view image required for light field display. After receiving the image data from the second image processing device, the first image processing device first decodes and detects the resolution of the received image data, and inputs the image data into the corresponding input terminal according to the resolution of the image data returned by the second image processing device, outputs the multi-view image required for light field display, and synthesizes the multi-view images to realize light field image display.

[0098] In a specific example, the neural network model may include three input terminals: input terminal 1, input terminal 2, and input terminal 3, which correspond to image data of resolution 1, image data of resolution 2, and image data of resolution 3, respectively. After receiving the image data of the second image processing device, the resolution of the received image data is first decoded and detected. If image data of resolution 1 is received, the image data of resolution 1 is input into input terminal 1, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display; if image data of resolution 2 is received, the image data of resolution 2 is input into input terminal 2, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display; if image data of resolution 3 is received, the image data of resolution 3 is input into input terminal 3, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display.

[0099] In related art, the first image processing device needs to process image data at a first resolution, resulting in a large amount of processed data. In this embodiment, when the viewer is in a scanning state, only image data at a resolution of 2 needs to be processed, reducing the amount of processed data and thus lowering the power consumption of the image processing system. When the viewer is not looking at the screen, only image data at a resolution of 3 needs to be processed, significantly reducing the amount of processed data and thus lowering the power consumption of the image processing system.

[0100] In this embodiment, the first image processing device obtains the viewer's human eye state parameters and sends the human eye state parameters to the second image processing device, so that the second image processing device obtains image data of corresponding resolution according to the human eye state parameters and sends it to the first image processing device. In this way, the resolution of the transmitted and processed image data can be dynamically adjusted according to the viewer's human eye state, thereby reducing the power consumption of processing and transmission while ensuring the viewer's user experience.

[0101] In this embodiment, it is necessary to prevent the viewer from noticing changes in display resolution or clarity. For example, when the state of the human eye on the first image processing device changes from a scanning state to a screen-looking state, due to the transmission delay, the viewer on the first image processing device still sees the low-resolution image in the scanning state. This image is easily noticeable as having a different clarity from the original resolution image. Therefore, when the state of the human eye changes from a scanning state or a state not looking at the screen to a screen-looking state, the first parameter needs to be sent to the second image processing device in advance; when the state of the human eye changes from a state not looking at the screen to a scanning state, the second parameter needs to be sent to the second image processing device in advance. This can prevent the human eye from seeing an unclear image and ensure the viewer's viewing experience. In addition, since the human eye changes relatively quickly, the transmission delay between the first image processing device and the second image processing device cannot be too high and must be less than 1ms.

[0102] In some embodiments, sending the human eye state parameter to the second image processing device includes at least one of the following:

[0103] When it is predicted that the human eye state will change from the scanning state to the screen-gazing state, sending the first parameter to the second image processing device;

[0104] When it is predicted that the human eye state will change from being in a non-watching-the-screen state to being in a watching-the-screen state, sending the first parameter to the second image processing device;

[0105] When it is predicted that the human eye state will change from not looking at the screen state to scanning state, sending the second parameter to the second image processing device;

[0106] After determining that the human eye state changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device;

[0107] After determining that the state of the human eye changes from the scanning state to the non-looking-at-the-screen state, sending the third parameter to the second image processing device;

[0108] After determining that the state of the human eye changes from the state of looking at the screen to the state of not looking at the screen, the third parameter is sent to the second image processing device.

[0109] In some embodiments, changes in the state of the human eye can be predicted based on the movement trends of the viewer's eyeballs and facial movements, including:

[0110] When it is determined based on the facial image of the viewer that the movement speed of the viewer's eyeballs decreases, predicting that the state of the human eye will change from a scanning state to a screen-gazing state;

[0111] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen, it is predicted that the state of the human eye will change from being not looking at the screen to being looking at the screen;

[0112] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeballs is increasing, it is predicted that the state of the human eye will change from being in a non-watching screen state to being in a scanning state.

[0113] After predicting the change in the human eye state, the predicted human eye state parameters need to be sent to the second image processing device in advance to obtain high-resolution image data to ensure the viewer's viewing experience.

[0114] An embodiment of the present disclosure provides an image processing method, which is applied to a second image processing device, as shown in FIG3 , and includes:

[0115] Step 201: receiving a human eye state parameter sent by a first image processing device;

[0116] After determining the human eye state parameter, the first image processing device sends the human eye state parameter to the second image processing device through the communication connection between the first image processing device and the second image processing device.

[0117] In this embodiment, the viewer's eye state parameters may include:

[0118] A first parameter indicates that the viewer is in a state of looking at the screen;

[0119] A second parameter indicates that the viewer is in a scanning state;

[0120] The third parameter indicates that the viewer is not looking at the screen.

[0121] The viewer's eye state parameters can be indicated by 2 bits of data, such as 0 for the first parameter, 1 for the second parameter, and 2 for the third parameter. When the viewer is looking at the screen, that is, the viewer's eyes are stationary and facing the screen, the viewer needs to see a higher-resolution image. When the viewer is scanning, that is, the viewer's eyes are looking at the screen but not focused, such as when the head shakes and the eyes move synchronously, or when the head is relatively stationary but the eyes are scanning. The scanning state is easy to occur in 3D video conferencing scenarios, such as when the viewer is reading or taking notes. In this case, a lower-resolution image can be provided to the viewer. When the viewer is not looking at the screen, that is, the viewer is not looking at the screen, and the viewer may be looking down or looking at other areas outside the screen. In this case, the viewer can be provided with the lowest-resolution image.

[0122] Step 202: determining a target resolution corresponding to the human eye state parameter, and acquiring image data of the target resolution;

[0123] After receiving the human eye state parameter sent by the first image processing device, the second image processing device determines the target resolution of the image data sent to the first image processing device based on the human eye state parameter. The second image processing device may pre-store a correspondence between resolutions and human eye state parameters, such as resolution 1 corresponding to the first parameter, resolution 2 corresponding to the second parameter, and resolution 3 corresponding to the third parameter, where resolution 3 is smaller than resolution 2, and resolution 2 is smaller than resolution 1. If the second image processing device receives the first parameter, the target resolution is determined to be resolution 1; if the second image processing device receives the second parameter, the target resolution is determined to be resolution 2; and if the second image processing device receives the third parameter, the target resolution is determined to be resolution 3.

[0124] In some embodiments, the native resolution of the camera that captures image data on the second image processing device is the first resolution. Resolution 1 may be equal to the first resolution, Resolution 2 may be 1 / 2 of the first resolution, and Resolution 3 may be 1 / 4 of the first resolution. Of course, the values ​​of Resolution 1, Resolution 2, and Resolution 3 are not limited to these values. Resolution 2 may also be 1 / 2 of the first resolution, Resolution 3 may be 1 / 8 of the first resolution, and so on.

[0125] In this embodiment, the second image processing device can directly acquire image data of the target resolution; or, when the first resolution is higher than the target resolution, the second image processing device acquires image data of the first resolution and compresses the image data of the first resolution to obtain image data of the target resolution.

[0126] Step 203: Send the image data to the first image processing device.

[0127] In the related art, the second image processing device encodes the image data of the first resolution and sends it to the first image processing device. In this embodiment, after the second image processing device receives the second parameter, the image data of resolution 2 is encoded and sent to the first image processing device. After the second image processing device receives the third parameter, the image data of resolution 3 is encoded and sent to the first image processing device, which can greatly reduce the amount of transmitted data.

[0128] In this embodiment, the first image processing device obtains the viewer's human eye state parameters and sends the human eye state parameters to the second image processing device, so that the second image processing device obtains image data of corresponding resolution according to the human eye state parameters and sends it to the first image processing device. In this way, the resolution of the transmitted and processed image data can be dynamically adjusted according to the viewer's human eye state, thereby reducing the power consumption of processing and transmission while ensuring the viewer's user experience.

[0129] An embodiment of the present disclosure provides a first image processing apparatus, as shown in FIG4 , comprising:

[0130] The human eye state parameter acquisition module 31 is used to acquire the human eye state parameters of the viewer;

[0131] A human eye state parameter sending module 32, configured to send the human eye state parameter to a second image processing device;

[0132] An image data receiving module 33 is configured to receive image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter;

[0133] The generating module 34 is configured to generate multi-viewpoint images required for light field display according to the image data.

[0134] In this embodiment, the first image processing device obtains the viewer's human eye state parameters and sends the human eye state parameters to the second image processing device, so that the second image processing device obtains image data of corresponding resolution according to the human eye state parameters and sends it to the first image processing device. In this way, the resolution of the transmitted and processed image data can be dynamically adjusted according to the viewer's human eye state, thereby reducing the power consumption of processing and transmission while ensuring the viewer's user experience.

[0135] In some embodiments, the eye state parameter acquisition module 31 is specifically configured to capture a facial image of the viewer using an image sensor, and determine the eye state parameters of the viewer based on the facial image. The eye state parameters of the viewer include:

[0136] A first parameter indicates that the viewer is in a state of looking at the screen;

[0137] A second parameter indicates that the viewer is in a scanning state;

[0138] The third parameter indicates that the viewer is not looking at the screen.

[0139] In some embodiments, the human eye state parameter sending module 32 is specifically configured to perform at least one of the following:

[0140] When it is predicted that the human eye state will change from the scanning state to the screen-gazing state, sending the first parameter to the second image processing device;

[0141] When it is predicted that the human eye state will change from being in a non-watching-the-screen state to being in a watching-the-screen state, sending the first parameter to the second image processing device;

[0142] When it is predicted that the human eye state will change from not looking at the screen state to scanning state, sending the second parameter to the second image processing device;

[0143] After determining that the state of the human eye changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device;

[0144] After determining that the state of the human eye changes from the scanning state to the non-looking-at-the-screen state, sending the third parameter to the second image processing device;

[0145] After determining that the state of the human eye changes from the state of looking at the screen to the state of not looking at the screen, the third parameter is sent to the second image processing device.

[0146] In some embodiments, the human eye state parameter acquisition module 31 is specifically configured to predict that the human eye state will change from a scanning state to a screen-gazing state when it is determined based on the facial image of the viewer that the movement speed of the viewer's eyeballs decreases;

[0147] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen, it is predicted that the state of the human eye will change from being not looking at the screen to being looking at the screen;

[0148] When it is determined based on the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeballs is increasing, it is predicted that the state of the human eye will change from being in a non-watching screen state to being in a scanning state.

[0149] In some embodiments, the generation module 34 is specifically used to establish a neural network model, which includes multiple input terminals, different input terminals corresponding to image data of different resolutions, and the neural network model is used to generate multi-view images using image data; according to the resolution of the image data returned by the second image processing device, the image data is input into the corresponding input terminal, and the multi-view images required for light field display are output.

[0150] An embodiment of the present disclosure provides a second image processing apparatus, as shown in FIG5 , comprising:

[0151] A human eye state parameter receiving module 41 is configured to receive the human eye state parameter sent by the first image processing device;

[0152] A determination module 42 is configured to determine a target resolution corresponding to the human eye state parameter;

[0153] An image data acquisition module 43 is used to acquire image data of a target resolution;

[0154] The image data sending module 44 is configured to send the image data to the first image processing device.

[0155] In this embodiment, the first image processing device obtains the viewer's human eye state parameters and sends the human eye state parameters to the second image processing device, so that the second image processing device obtains image data of corresponding resolution according to the human eye state parameters and sends it to the first image processing device. In this way, the resolution of the transmitted and processed image data can be dynamically adjusted according to the viewer's human eye state, thereby reducing the power consumption of processing and transmission while ensuring the viewer's user experience.

[0156] In some embodiments, the image data acquisition module 43 is specifically used to acquire image data of target resolution; or

[0157] Image data of a first resolution is collected, and the image data of the first resolution is compressed to obtain image data of a target resolution, where the first resolution is higher than the target resolution.

[0158] An embodiment of the present disclosure provides an image processing system, as shown in FIG6 , including:

[0159] The first image processing device 51 is configured to obtain eye state parameters of the viewer; send the eye state parameters to the second image processing device; receive image data returned by the second image processing device, wherein the resolution of the image data is determined by the eye state parameters; and generate multi-viewpoint images required for light field display based on the image data.

[0160] The second image processing device 52 is configured to receive the human eye state parameters sent by the first image processing device; determine a target resolution corresponding to the human eye state parameters and obtain image data of the target resolution; and send the image data to the first image processing device.

[0161] In this embodiment, the image processing system includes image processing devices at both ends, each of which can be a first image processing device and a second image processing device. When one end collects the viewer's eye state parameters, the local end serves as the first image processing device, and the other end serves as the second image processing device. When one end collects image data of a scene, the local end serves as the second image processing device, and the other end serves as the first image processing device. The first image processing device collects the viewer's eye state parameters at the local end, sends the viewer's eye state parameters to the other end, receives the image data sent by the other end, and generates the multi-viewpoint images required for light field display based on the received image data. The second image processing device receives the eye state parameters at the other end, collects the local end's image data, and sends image data of corresponding resolution based on the eye state parameters to the other end.

[0162] As shown in Figure 6, the first image processing device and the second image processing device both include a human eye state parameter acquisition module, a human eye state parameter sending module, an image data receiving module, a generation module, a human eye state parameter receiving module, a determination module, an image data acquisition module and an image data sending module.

[0163] In this embodiment, the first image processing device can capture a facial image of the viewer using a low-resolution, high-frame-rate camera and determine the viewer's eye state parameters based on the facial image. Low-resolution, high-frame-rate cameras have very low power consumption, which helps reduce the power consumption of the entire image processing system. Because changes in eye state are uncertain and rapid, cameras used for eye state detection should have as low a latency as possible.

[0164] In this embodiment, the viewer's eye state parameters may include:

[0165] A first parameter indicates that the viewer is in a state of looking at the screen;

[0166] A second parameter indicates that the viewer is in a scanning state;

[0167] The third parameter indicates that the viewer is not looking at the screen.

[0168] The viewer's eye state parameters can be indicated by 2 bits of data, such as 0 for the first parameter, 1 for the second parameter, and 2 for the third parameter. When the viewer is looking at the screen, that is, the viewer's eyes are stationary and facing the screen, the viewer needs to see a higher-resolution image. When the viewer is scanning, that is, the viewer's eyes are looking at the screen but not focused, such as when the head shakes and the eyes move synchronously, or when the head is relatively stationary but the eyes are scanning. The scanning state is easy to occur in 3D video conferencing scenarios, such as when the viewer is reading or taking notes. In this case, a lower-resolution image can be provided to the viewer. When the viewer is not looking at the screen, that is, the viewer is not looking at the screen, and the viewer may be looking down or looking at other areas outside the screen. In this case, the viewer can be provided with the lowest-resolution image.

[0169] After determining the human eye state parameter, the first image processing device sends the human eye state parameter to the second image processing device through the communication connection between the first image processing device and the second image processing device. After receiving the human eye state parameter sent by the first image processing device, the second image processing device determines the target resolution of the image data sent to the first image processing device based on the human eye state parameter. The second image processing device may pre-store a correspondence between resolutions and human eye state parameters, such as resolution 1 corresponding to the first parameter, resolution 2 corresponding to the second parameter, and resolution 3 corresponding to the third parameter, wherein resolution 3 is smaller than resolution 2, and resolution 2 is smaller than resolution 1. If the second image processing device receives the first parameter, it determines the target resolution to be resolution 1; if the second image processing device receives the second parameter, it determines the target resolution to be resolution 2; if the second image processing device receives the third parameter, it determines the target resolution to be resolution 3.

[0170] In some embodiments, the native resolution of the camera that captures image data on the second image processing device is the first resolution. Resolution 1 may be equal to the first resolution, Resolution 2 may be 1 / 2 of the first resolution, and Resolution 3 may be 1 / 4 of the first resolution. Of course, the values ​​of Resolution 1, Resolution 2, and Resolution 3 are not limited to these values. Resolution 2 may also be 1 / 2 of the first resolution, Resolution 3 may be 1 / 8 of the first resolution, and so on.

[0171] In the related art, the second image processing device encodes the image data of the first resolution and sends it to the first image processing device. In this embodiment, after the second image processing device receives the second parameter, the image data of resolution 2 is encoded and sent to the first image processing device. After the second image processing device receives the third parameter, the image data of resolution 3 is encoded and sent to the first image processing device, which can greatly reduce the amount of transmitted data.

[0172] In this embodiment, the second image processing device can directly acquire image data of the target resolution; or, when the first resolution is higher than the target resolution, the second image processing device acquires image data of the first resolution and compresses the image data of the first resolution to obtain image data of the target resolution.

[0173] As shown in Figure 7, for the first image processing device at end A, the human eye state of the viewer at end A is obtained, and the human eye state parameter 0, 1 or 2 is transmitted to the second image processing device at end B; for the second image processing device at end B, the human eye state parameter of end A is received, and if the human eye state parameter is 0, image data with the original resolution of the camera is obtained; if the human eye state parameter is 1, image data with 1 / 2 of the original resolution of the camera is obtained; if the human eye state parameter is 2, image data with 1 / 4 of the original resolution of the camera is obtained; after obtaining the image data, the image data is encoded and transmitted to end A.

[0174] In this embodiment, the first image processing device can establish and train a neural network model, as shown in Figure 2. The neural network model may include multiple input terminals and multiple output terminals, and different input terminals correspond to image data of different resolutions. The neural network model is used to generate multi-view images using image data, and the output of the neural network model is the multi-view image required for light field display. After receiving the image data from the second image processing device, the first image processing device first decodes and detects the resolution of the received image data, and inputs the image data into the corresponding input terminal according to the resolution of the image data returned by the second image processing device, outputs the multi-view image required for light field display, and synthesizes the multi-view images to realize light field image display.

[0175] In a specific example, the neural network model may include three input terminals: input terminal 1, input terminal 2, and input terminal 3, which correspond to image data of resolution 1, image data of resolution 2, and image data of resolution 3, respectively. After receiving the image data of the second image processing device, the resolution of the received image data is first decoded and detected. If image data of resolution 1 is received, the image data of resolution 1 is input into input terminal 1, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display; if image data of resolution 2 is received, the image data of resolution 2 is input into input terminal 2, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display; if image data of resolution 3 is received, the image data of resolution 3 is input into input terminal 3, and the multi-view image required for light field display is output, and the multi-view images are synthesized to realize light field image display.

[0176] In related art, the first image processing device needs to process image data at a first resolution, resulting in a large amount of processed data. In this embodiment, when the viewer is in a scanning state, only image data at a resolution of 2 needs to be processed, reducing the amount of processed data and thus lowering the power consumption of the image processing system. When the viewer is not looking at the screen, only image data at a resolution of 3 needs to be processed, significantly reducing the amount of processed data and thus lowering the power consumption of the image processing system.

[0177] In this embodiment, it is necessary to prevent the viewer from noticing changes in display resolution or clarity. For example, when the state of the human eye on the first image processing device changes from a scanning state to a screen-looking state, due to the transmission delay, the viewer on the first image processing device still sees the low-resolution image in the scanning state. This image is easily noticeable as having a different clarity from the original resolution image. Therefore, when the state of the human eye changes from a scanning state or a state not looking at the screen to a screen-looking state, the first parameter needs to be sent to the second image processing device in advance; when the state of the human eye changes from a state not looking at the screen to a scanning state, the second parameter needs to be sent to the second image processing device in advance. This can prevent the human eye from seeing an unclear image and ensure the viewer's viewing experience. In addition, since the human eye changes relatively quickly, the transmission delay between the first image processing device and the second image processing device cannot be too high and must be less than 1ms.

[0178] As shown in Figure 8, when the human eye state parameter changes from 1 to 0 or from 2 to 0 or 1, the change signal must be transmitted to the other end in advance during the change process, that is, the changed human eye state parameter is sent at time t1 in Figure 8. This can prevent the human eye from seeing unclear images and affecting the viewing experience.

[0179] In the various method embodiments of the present disclosure, the serial numbers of the steps cannot be used to limit the order of the steps. For ordinary technicians in this field, without paying any creative work, changes to the order of the steps are also within the scope of protection of the present disclosure.

[0180] It should be noted that the various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, since the embodiments are generally similar to the product embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the product embodiments.

[0181] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applicable to a first image processing device, comprising: Obtain the viewer's eye state parameters; Sending the human eye state parameter to a second image processing device; receiving image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter; Multi-viewpoint images required for light field display are generated according to the image data.

2. The image processing method according to claim 1, characterized in that: The obtaining of the viewer's eye state parameters comprises: The facial image of the viewer is collected by using an image sensor, and the human eye state parameters of the viewer are determined according to the facial image. The human eye state parameters of the viewer include: A first parameter indicates that the viewer is in a state of looking at the screen; A second parameter indicates that the viewer is in a scanning state; The third parameter indicates that the viewer is not looking at the screen.

3. The image processing method according to claim 2, characterized in that: The sending the human eye state parameter to the second image processing device comprises at least one of the following: When it is predicted that the human eye state will change from the scanning state to the screen-fixation state, sending the first parameter to the second image processing device; When it is predicted that the human eye state will change from being in a state of not looking at the screen to being in a state of looking at the screen, sending the first parameter to the second image processing device; When it is predicted that the human eye state will change from a state of not looking at the screen to a state of scanning, sending the second parameter to the second image processing device; After determining that the state of the human eye changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device; After determining that the state of the human eye changes from the scanning state to the non-watching state, sending the third parameter to the second image processing device; After determining that the state of the human eye changes from being in the state of looking at the screen to being in the state of not looking at the screen, The third parameter is sent to the second image processing device.

4. The image processing method according to claim 3, characterized in that: The method further comprises: When it is determined according to the facial image of the viewer that the movement speed of the viewer's eyeballs decreases, it is predicted that the state of the human eye will be converted from a scanning state to a screen-gazing state; When it is determined according to the facial image of the viewer that the face of the viewer moves toward the screen, it is predicted that the state of the human eye will change from being in a state of not looking at the screen to being in a state of looking at the screen; When it is determined according to the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeball increases, it is predicted that the state of the human eye will be converted from being in a non-watching screen state to being in a scanning state.

5. The image processing method according to claim 1, characterized in that: Generating multi-viewpoint images required for light field display according to the image data comprises: Establishing a neural network model, the neural network model comprising a plurality of input terminals, different input terminals corresponding to image data of different resolutions, and the neural network model is used to generate multi-viewpoint images using the image data; According to the resolution of the image data returned by the second image processing device, the image data is input into the corresponding input end, and the multi-viewpoint images required for light field display are output.

6. An image processing method, characterized in that: Applicable to a second image processing device, comprising: Receiving a human eye state parameter sent by the first image processing device; Determine a target resolution corresponding to the human eye state parameter, and obtain image data of the target resolution; The image data is sent to the first image processing device.

7. The image processing method according to claim 6, characterized in that: The acquiring of image data of target resolution comprises: Acquire image data at the target resolution; or Image data of a first resolution is acquired, and the image data of the first resolution is compressed to obtain image data of a target resolution, wherein the first resolution is higher than the target resolution.

8. A first image processing device, characterized in that: include: A human eye state parameter acquisition module, used to acquire the viewer's human eye state parameters; A human eye state parameter sending module, used to send the human eye state parameter to a second image processing device; An image data receiving module, used for receiving image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter; A generating module is used to generate multi-viewpoint images required for light field display according to the image data.

9. The image processing device according to claim 8, characterized in that: The human eye state parameter acquisition module is specifically used to collect the facial image of the viewer using an image sensor, and determine the human eye state parameters of the viewer according to the facial image. The human eye state parameters of the viewer include: A first parameter indicates that the viewer is in a state of looking at the screen; A second parameter indicates that the viewer is in a scanning state; The third parameter indicates that the viewer is not looking at the screen.

10. The image processing device according to claim 9, characterized in that: The human eye state parameter sending module is specifically configured to perform at least one of the following: When it is predicted that the human eye state will change from the scanning state to the screen-fixation state, sending the first parameter to the second image processing device; When it is predicted that the human eye state will change from being in a state of not looking at the screen to being in a state of looking at the screen, sending the first parameter to the second image processing device; When it is predicted that the human eye state will change from a state of not looking at the screen to a state of scanning, sending the second parameter to the second image processing device; After determining that the state of the human eye changes from the state of gazing at the screen to the state of scanning, sending the second parameter to the second image processing device; After determining that the state of the human eye changes from the scanning state to the non-watching state, sending the third parameter to the second image processing device; After determining that the state of the human eye changes from the state of looking at the screen to the state of not looking at the screen, the third parameter is sent to the second image processing device.

11. The image processing device according to claim 10, characterized in that: The human eye state parameter acquisition module is specifically used to determine the When the eye movement speed of the viewer decreases, it is predicted that the state of the human eye will be changed from the scanning state to the screen-fixation state; When it is determined according to the facial image of the viewer that the face of the viewer moves toward the screen, it is predicted that the state of the human eye will change from being in a state of not looking at the screen to being in a state of looking at the screen; When it is determined according to the facial image of the viewer that the face of the viewer is moving toward the screen and the movement speed of the viewer's eyeball increases, it is predicted that the state of the human eye will be converted from being in a non-watching screen state to being in a scanning state.

12. The image processing device according to claim 8, characterized in that: The generation module is specifically used to establish a neural network model, the neural network model includes multiple input terminals, different input terminals correspond to image data of different resolutions, and the neural network model is used to generate multi-viewpoint images using image data; According to the resolution of the image data returned by the second image processing device, the image data is input into the corresponding input end, and the multi-viewpoint images required for light field display are output.

13. A second image processing device, characterized in that: include: A human eye state parameter receiving module, used for receiving the human eye state parameter sent by the first image processing device; A determination module, used to determine a target resolution corresponding to the human eye state parameter; An image data acquisition module, used to acquire image data of target resolution; The image data sending module is used to send the image data to the first image processing device.

14. The image processing device according to claim 13, characterized in that: The image data acquisition module is specifically used to collect image data of target resolution; or Image data of a first resolution is acquired, and the image data of the first resolution is compressed to obtain image data of a target resolution, wherein the first resolution is higher than the target resolution.

15. An image processing system, characterized in that: include: A first image processing device, used to obtain the human eye state parameters of the viewer; Sending the human eye state parameter to a second image processing device; receiving image data returned by the second image processing device, wherein the resolution of the image data is determined by the human eye state parameter; Generate multi-viewpoint images required for light field display according to the image data; The second image processing device is used to receive the human eye state parameter sent by the first image processing device; Determine a target resolution corresponding to the human eye state parameter, and obtain image data of the target resolution; The image data is sent to the first image processing device.

Images