Heterogeneous chip, display device, and display control method

Abstract

The present disclosure relates to the technical field of display, and provides a heterogeneous chip, a display device, and a display control method. The heterogeneous chip of the present disclosure comprises: a first processing core configured to: in response to a first video frame in a target video stream being received, store the first video frame in a memory according to the indication of a first clock, control to capture an image, store the captured image in the memory according to the indication of a second clock, and when the captured image is stored, send an interrupt request to a second processing core; and the second processing core configured to: in response to the interrupt request, call computing resources in the first processing core by means of bus communication to process the captured image so as to obtain human eye position information, and send the human eye position information to the first processing core; wherein in response to the human eye position information being received, the first processing core reads second video frames stored in sequence from the memory, and rearranges pixels of the second video frames on the basis of the human eye position information to obtain updated target video frames.

Description

Heterogeneous chip, display device, and display control method Technical Field

[0001] The present disclosure belongs to the field of display technology, and particularly relates to a heterogeneous chip, a display device, and a display control method. Background Art

[0002] With the rapid development of spatial reality (SR) technology, SR displays are gaining increasing attention. SR displays primarily enable users to enjoy a three-dimensional viewing experience without the naked eye.

[0003] Summary of the Invention

[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provide a heterogeneous chip, a display device and a display control method.

[0005] In a first aspect, a technical solution adopted to solve the technical problem of the present disclosure is a heterogeneous chip, wherein the heterogeneous chip includes a first processing core, a second processing core, and a memory; the first processing core and the second processing core are communicatively connected via a bus;

[0006] The first processing core is configured to, in response to receiving a first video frame in a target video stream, store the first video frame in the memory according to an instruction of a first clock; in response to receiving the first video frame in the target video stream, control capturing an image in an acquisition environment, store the captured image in the memory according to an instruction of a second clock, and send an interrupt request to the second processing core after storing the captured image;

[0007] the second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the first processing core;

[0008] In which, in response to receiving the human eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the human eye position information to obtain an updated target video frame; the second video frame and the first video frame are separated by at least one frame of video frame in the target video stream.

[0009] In some embodiments, the first processing core is configured to, in response to receiving a first video frame in a target video stream, store the first video frame in the memory according to a first clock instruction, specifically comprising:

[0010] In response to receiving a first video frame currently to be played in a target video stream, preprocessing the first video frame to obtain a preprocessed video frame;

[0011] The preprocessed video frame is written into the memory according to the instruction of the first clock via the first bus communication.

[0012] In some embodiments, the first processing core is configured to, in response to receiving a first video frame in a target video stream, control capturing images in a capture environment and store the captured images in the memory, specifically including:

[0013] In response to receiving a first video frame currently to be played in a target video stream, controlling a captured image in a capture environment and preprocessing the captured image to obtain a preprocessed captured image;

[0014] The pre-processed captured image is written into the memory according to the instruction of the second clock through the first bus communication.

[0015] In some embodiments, the second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication to process the captured image to obtain the eye position information, specifically including:

[0016] The second processing core is configured to respond to the interrupt request and send a first resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module obtains the preprocessed captured image in response to the first resource call request, and performs face detection on the preprocessed captured image to obtain a first intermediate result; determines a face detection frame based on the first intermediate result; when the face detection frame is determined, sends a second resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module obtains the face image corresponding to the face detection frame in response to the second resource call request, performs key point feature extraction on the face image, and obtains a second intermediate result; determines key point information based on the second intermediate result; and when the key point information is determined, estimates the eye coordinates based on the key point information to obtain eye position information.

[0017] In some embodiments, the first processing core is configured to obtain the pre-processed captured image in response to the first resource call request, specifically including:

[0018] In response to the first resource call request, the pre-processed captured image is read from the memory through the first bus communication.

[0019] In some embodiments, the first processing core is configured to obtain, in response to the second resource call request, a face image corresponding to the face detection frame, specifically including:

[0020] The second processing module is specifically configured to, in response to receiving the face detection frame, obtain the pre-processed captured image from the memory; obtain a face image corresponding to the face detection frame based on the face detection frame and the pre-processed captured image; and generate the second resource call request based on the face image;

[0021] The first processing core is specifically configured to, in response to the second resource call request, parse the facial image carried in the second resource call request.

[0022] In some embodiments, the first processing core is configured to perform face detection on the pre-processed captured image to obtain a first intermediate result, specifically including:

[0023] The first processing core is integrated with a preset neural network algorithm and is configured to perform face detection on the preprocessed captured image, and write a third intermediate result of the face detection process into the memory through the first bus communication; and when a request is made to call the third intermediate result, the third intermediate result is read from the memory through the first bus communication and continued to be used for face detection until the first intermediate result is obtained.

[0024] In some embodiments, the first processing core is configured to extract key point features from the facial image to obtain a second intermediate result, specifically including:

[0025] The first processing core is integrated with a preset neural network algorithm and is configured to perform key point feature extraction on the facial image and absorb the fourth intermediate result of the key point feature extraction process into the memory through the first bus communication; and when a request is made to call the fourth intermediate result, the fourth intermediate result is read from the memory through the first bus communication and continued to be used for key point feature extraction until the second intermediate result is obtained.

[0026] In some embodiments, the first processing core is configured to, in response to receiving a first video frame in a target video stream, store the first video frame in the memory according to a first clock instruction, specifically comprising:

[0027] In response to receiving a first video frame in a target video stream, performing image segmentation on the first video frame to obtain a left image adapted for a left eye and a right image adapted for a right eye;

[0028] A set of the group image and the right image are written into the memory via the first bus according to the instruction of the first clock.

[0029] In some embodiments, in response to receiving the eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame, specifically including:

[0030] In response to receiving the human eye position information, reading the left image and the right image stored in sequence from the memory through the first bus communication; the second video frame includes a group of left images and right images;

[0031] In response to receiving a set of left and right images, scaling the left image according to a preset screen resolution to obtain first image data adapted for the left eye; scaling the right image according to the preset screen resolution to obtain second image data adapted for the right eye;

[0032] In response to receiving a set of the first image data and the second image data, pixels in the first image data and pixels in the second image data are rearranged according to the eye position information to obtain an updated target video frame.

[0033] In some embodiments, the first processing core is configured to rearrange pixels in the first image data and pixels in the second image data according to the eye position information to obtain an updated target video frame, specifically including:

[0034] In response to receiving a set of the first image data and the second image data, determining the binocular midpoint position information according to the human eye position information;

[0035] The human eye position information and the binocular position information are both positions in a pre-established spatial coordinate system of the display screen;

[0036] For any target row of pixels, determining first size information of the first view image and focus position information of the focus according to the binocular midpoint position information and pre-stored prism parameters;

[0037] Determining second size information of a second view image of the target row of pixels according to first size information of the first view image, focus position information of the focus, and screen parameters;

[0038] For a target pixel in any target row of pixels, based on a position of the target pixel in the target row of pixels, preset pixel size information, and positions of each sub-pixel in the target pixel in the target pixel, respectively determine the center position information of a red sub-pixel, the center position information of a green sub-pixel, and the center position information of a blue sub-pixel;

[0039] determining position information of a left-eye pixel and position information of a right-eye pixel based on the first size information, the second size information, and the center position information of a red sub-pixel, the center position information of a green sub-pixel, and the center position information of a blue sub-pixel in the target pixel;

[0040] For each of the target row pixels, based on the position information of the left eye pixel, the sub-pixel image data of the corresponding position in the first image data is extracted, and based on the position information of the right eye pixel, the sub-pixel image data of the corresponding position in the second image data is extracted to obtain an updated target video frame.

[0041] In a second aspect, an embodiment of the present disclosure further provides a display device comprising a display screen and a heterogeneous chip; the heterogeneous chip comprises a first processing core, a second processing core, and a memory; the first processing core and the second processing core are communicatively connected via a bus;

[0042] The first processing core is configured to, in response to receiving a first video frame in a target video stream, store the first video frame in the memory according to an instruction of a first clock; in response to receiving the first video frame in the target video stream, control capturing an image in an acquisition environment, store the captured image in the memory according to an instruction of a second clock, and send an interrupt request to the second processing core after storing the captured image;

[0043] the second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the first processing core;

[0044] In response to receiving the eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame, and sends the updated target video frame to the display screen;

[0045] The display screen is configured to display according to the received target video frame.

[0046] In some embodiments, the display device further includes an external device;

[0047] The external device is configured to collect the original captured image and send it to the first processing core in response to the control instruction when the first processing core receives the video frame.

[0048] In a third aspect, an embodiment of the present disclosure further provides a display control method, including:

[0049] The first processing core is responsive to receiving a first video frame in a target video stream and storing the frame into the memory according to an instruction of a first clock;

[0050] The first processing core controls, in response to receiving a first video frame in a target video stream, capturing an image in a capture environment, and stores the captured image in a memory according to an instruction of a second clock, and after storing the captured image, sends an interrupt request to the second processing core;

[0051] In response to the interrupt request, the second processing core calls the computing resources in the first processing core through bus communication, processes the captured image, obtains the human eye position information, and sends the human eye position information to the first processing core;

[0052] In response to receiving the eye position information, the first processing core reads a second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame; the second video frame and the first video frame are separated by at least one video frame in the target video stream.

[0053] In a fourth aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the display control method described in the third aspect are performed.

[0054] In the fifth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, wherein a computer program is stored on the computer non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the display control method described in the third aspect are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a schematic diagram of a heterogeneous chip provided by an embodiment of the present disclosure;

[0056] FIG2 is a schematic diagram of data processing performed by a heterogeneous chip according to an embodiment of the present disclosure;

[0057] FIG3 is a schematic diagram of a specific process of performing data processing on a heterogeneous chip according to an embodiment of the present disclosure;

[0058] FIG4 is a schematic diagram of a process for collaborative synchronization of heterogeneous chips provided by an embodiment of the present disclosure;

[0059] FIG5 is a schematic diagram of the pixel rearrangement principle provided by an embodiment of the present disclosure;

[0060] FIG6 is a schematic diagram of a display system including heterogeneous chips provided by an embodiment of the present disclosure;

[0061] FIG7 a is a data processing flow for determining human eye position information according to an embodiment of the present disclosure;

[0062] FIG7 b is a flowchart of determining pixel rearrangement according to an embodiment of the present disclosure;

[0063] FIG8 is a flow chart of a display control method provided by an embodiment of the present disclosure;

[0064] FIG9 is a schematic structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0066] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, "one", "an" or "the" and similar words do not indicate a quantity limitation, but rather indicate the existence of at least one. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0067] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0068] Related technologies typically use a glasses-free 3D display system for data processing. For example, this involves collecting eye position information from viewers while simultaneously receiving video images from a display terminal. This information is then used to rearrange the pixels of the video image, allowing viewers at different viewing positions to see a more three-dimensional display. However, achieving optimal processing results, such as high frame rates and low latency, remains a pressing technical challenge in the glasses-free 3D field.

[0069] In view of this, embodiments of the present disclosure provide a heterogeneous chip that substantially eliminates one or more of the problems caused by limitations and defects of the related art.

[0070] Figure 1 is a schematic diagram of a heterogeneous chip provided by an embodiment of the present disclosure. As shown in Figure 1, the heterogeneous chip 100 includes a first processing core, a second processing core and a memory; the first processing core and the second processing core are connected through a bus communication.

[0071] Exemplarily, the first processing core may be a Field Programmable Gate Array (FPGA) for performing logic operations.

[0072] Exemplarily, the second processing core may be a central processing unit (CPU) for instruction-based computing.

[0073] For example, the memory may be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM). The memory may be shared by the first processing core and the second processing core to save costs. Alternatively, different memories may be provided for different processing cores based on the actual amount of stored data.

[0074] Of course, the first processing core, the second processing core, and the memory in the present disclosure are not limited to the FPGA, CPU, and DDR listed above. For ease of understanding, the present disclosure uses the first processing core as an FPGA, the second processing core as a CPU, and the memory as a DDR as an example for explanation.

[0075] Exemplarily, the heterogeneous chip 100 integrates and packages FPGA, CPU, and DDR, and the FPGA and CPU are connected to each other via a high-speed bus inside the heterogeneous chip 100 .

[0076] Figure 2 is a schematic diagram of data processing performed by a heterogeneous chip according to an embodiment of the present disclosure. As shown in Figure 2, the FPGA processes two image data streams in real time. One stream is an image captured by a camera, processed by the FPGA, and then stored in DDR. The other stream is a video frame from a target video stream received from a standard video terminal. The FPGA performs real-time pixel rearrangement of the video frames based on the final eye position information calculated by the CPU, and finally transmits the video frames to the 3D display for display. The FPGA uses different clock domains to process the different data streams, meaning that the FPGA processes the two image data streams in real time using two asynchronous clock domains.

[0077] Figure 3 is a schematic diagram of the specific process of data processing by the heterogeneous chip provided by the embodiment of the present disclosure. As shown in Figure 3, specifically, the FPGA is configured to respond to the first video frame received in the target video stream and store it in the DDR according to the instructions of the first clock. The target video stream is a pre-set image to be played, which is video data received through the display terminal. The display terminal can be a standard video terminal commonly used by the external playback source, such as HDMI, DP,..., or RX. The target video stream contains multiple video frames arranged in sequence according to the playback timing, and the multiple video frames constitute the playback image screen. According to the playback order in the target video stream, each video frame is received in sequence, and the FPGA responds to receiving any first video frame and stores it in the DDR according to the instructions of the first clock.

[0078] The FPGA is further configured to, in response to receiving a first video frame in a target video stream, control the capture of images in the environment while storing the first video frame, store the captured images in the DDR according to the second clock, and send an interrupt request to the CPU after storing the captured images. The FPGA can capture images in the environment in real time by controlling an external device. The external device can be a device with a camera function, such as a camera. The external device is independent of the heterogeneous chip 100 and is in communication with the heterogeneous chip 100. The environment can be the environment in which the external device is located.

[0079] For example, when the shooting device is a camera, the heterogeneous chip 100 and the camera can communicate through a standard interface such as the Mobile Industry Processor Interface (MIPI) to receive images captured by the camera.

[0080] The FPGA has an asynchronous first clock domain and a second clock domain. The first clock domain is used for real-time processing of the first video frame in the target video stream. The first clock is the clock in the first clock domain. The second clock domain is used for real-time processing of captured images. The second clock is the clock in the second clock domain.

[0081] The FPGA receives the first video frame in the target video stream via a standard video terminal (HDMI), receives the captured image via MIPI, and performs parallel calculations on these two data streams. To ensure synchronous processing of each of the two data streams (if they are asynchronous, it is easy to lose the coordinates of the face detected in a frame because the clocks are at different frequencies, and it is impossible to ensure that each frame performs the same function at the same time), the FPGA of the present disclosure uses the frame synchronization signal when receiving the first video frame in the first clock domain to trigger the signal of the external device in the second clock domain to collect the captured image in real time. This achieves synchronous signal processing between the asynchronous clock domains, ensuring that the signal of each frame is processed in a timely manner.

[0082] Exemplarily, the FPGA is further configured to, in response to receiving a first video frame in a target video stream, simultaneously store the video frame and generate and send a first signal to the CPU to trigger an external device to capture an image. The CPU is configured to, in response to the first signal, send a second signal to the external device to capture the image. The FPGA is further configured to, in response to receiving an image captured by the external device in response to the second signal, store the image in a DDR according to a second clock signal. After storing the image, the FPGA sends an interrupt request to the CPU.

[0083] The CPU is also configured to respond to interrupt requests, call the computing resources in the FPGA through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the FPGA for pixel rearrangement. It should be noted that due to the deployment of convolution, full connection and other operators in the neural network in the CPU, the processing speed is slower than that of the FPGA. Therefore, the present disclosure deploys a dedicated computing acceleration unit in the FPGA, such as convolution, full connection and other operators in the neural network, and uses the dedicated computing acceleration unit to process the captured image, so that the calculation of the human eye position information is more efficient and low-latency.

[0084] The FPGA is also configured to, in response to receiving the eye position information, read the sequentially stored second video frame from the DDR and, based on the eye position information, rearrange the pixels of the second video frame to obtain an updated target video frame. Because FPGAs can process large-scale data streams in parallel and in real time, pixel rearrangement of data stream 2 is most appropriately handled within the FPGA.

[0085] Among them, there is at least one frame of video frame in the target video stream between the second video frame and the first video frame. That is to say, the second video frame read in sequence from the DDR is not the first response to the received "first video frame". It should be noted that the data processing process of the entire heterogeneous chip is a dynamic process, because the signal for collecting and shooting images is triggered by the FPGA in response to the signal of receiving a frame of the first video frame, and the time length for the subsequent processing of the shot image to obtain the human eye position information fastest must exceed the interval time △T of the first video frame played at the normal refresh frequency between two adjacent frames in the target video stream. Therefore, in response to receiving the first video frame t1 of the i-th frame in the target video stream, the time t2 for collecting and shooting images and obtaining the human eye position information fastest exceeds the time node t3 of playing the first video frame of the i+1-th frame during normal video playback. Therefore, when obtaining the human eye position information fastest, the pixel rearrangement can only be performed for the video frame after the first video frame of the i+1-th frame in the target video stream, that is, the first video frame of the i+m-th frame, in sequence. in, = represents rounding up, where t2-t1>t3-t1=ΔT. At this point, the i+mth first video frame is recorded as the latest second video frame stored in sequence read from the DDR. The so-called "latest" refers to the first video frame last stored in the DDR.

[0086] For example, if i=1 and m=2, the present disclosure first performs pixel rearrangement on the first video frame of the third frame in the target video stream. As for the first video frame of the first frame and the first video frame of the second frame, since the human eye position information has not been obtained before the two are played, the pixel rearrangement of these two frames is ignored, and only the pixel rearrangement from the first video frame of the third frame to the first video frame of the last frame is performed.

[0087] FIG4 is a flow chart of the collaborative synchronization of heterogeneous chips provided by an embodiment of the present disclosure. As shown in FIG4 , the FPGA receives the video frame; the FPGA triggers the first signal of the external device to capture the captured image at a fixed time in each frame; the camera capture is started; the CPU calls the computing resources in the FPGA to perform artificial intelligence (AI) detection on the captured image, obtains the human eye position information and sends it to the FPGA; the CPU sends a trigger flag signal to the FPGA indicating that the processing of the human eye position information is completed; the above processing must be completed within one frame of the i+1th video frame, that is, after the i-th video frame is received and before the i+2th video frame is received. If the AI ​​detection processing is too slow, the FPGA can only use the human eye position information initiated by the previous video frame (that is, the i-th video frame). At this time, if the viewer moves, the edge jitter of the experienced 3D image will also appear; therefore, it is necessary to ensure that the above processing must be completed within one frame of the i+1th video frame. Afterwards, the FPGA starts pixel rearrangement based on the trigger flag signal, uses the eye position information corresponding to the i-th frame captured, rearranges the pixels of the i+2-th frame, and outputs the rearranged target video frame. Here, i is a positive integer greater than or equal to 1.

[0088] The present disclosure utilizes a CPU and an FPGA on the same heterogeneous chip 100 to process different parts of the face recognition algorithm respectively, utilizes the FPGA to pre-process the video frames and captured images; utilizes the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, and can also perform various other operator operations based on the instruction set; utilizes the CPU to perform post-processing after the convolution, such as key point detection and coordinate estimation. Among them, utilizing the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, as well as various other operator operations based on the instruction set, can achieve optimal processing effects, such as high frame rate and low latency processing effects. In addition, the FPGA utilizes the frame synchronization signal when receiving the video frame in the first clock domain to trigger the external device in the second clock domain to collect the captured image signal in real time, thereby realizing synchronous signal processing between asynchronous clock domains, thereby ensuring that the signal of each frame is processed in a timely manner.

[0089] For example, the FPGA receives video frames in the target video stream through the standard video terminal (HDMI), receives captured images through MIPI, and performs parallel calculations on these two data streams, calculating the human eye position information on one channel and feeding it back to the other channel for real-time pixel rearrangement processing. Finally, the processed target video frame outputs the video signal to the naked-eye 3D screen through a standard internal interface such as VBO.

[0090] For example, the heterogeneous chip provided by the present disclosure can be applied to a display device in a naked-eye 3D display scenario.

[0091] In some embodiments, the process of FPGA receiving and storing video frames specifically includes: in response to receiving the first video frame currently to be played in the target video stream, preprocessing the first video frame to obtain a preprocessed video frame; and writing the preprocessed video frame into DDR according to the instruction of the first clock through the first bus communication.

[0092] Specifically, as shown in Figure 1, the FPGA includes a video frame preprocessing module B2, a first data writing module B3, and a first bus communication module C1, wherein the first bus communication module C1 can realize the first bus communication; the CPU includes a memory control module C3. The video frame preprocessing module B2 is configured to preprocess the video frame in response to receiving the first video frame in the target video stream to obtain a preprocessed video frame. The preprocessing here mainly includes converting the information of interfaces such as HDMI and DP into a data stream that is easy to process internally, for example, converting RGB information with H, V, and DE synchronization signals into an AXI Stream data stream; performing image segmentation on the video frame to obtain a left image and a right image. The left image and the right image here are recorded as preprocessed video frames.

[0093] The first data writing module B3 is configured to write the pre-processed video frame into the memory control module C3 through the first bus communication module C1 according to the instruction of the first clock.

[0094] The memory control module C3 is configured to control the received pre-processed video frames to be stored in the DDR.

[0095] In some embodiments, the FPGA controls an external device to capture captured images in real time and stores the captured images in the DDR, specifically including: in response to receiving the first video frame currently to be played in the target video stream, controlling the captured images in the capture environment, and preprocessing the captured images to obtain preprocessed captured images; and writing the preprocessed captured images into the DDR according to the instructions of the second clock through the first bus communication.

[0096] Specifically, as shown in Figure 1, the FPGA also includes a captured image preprocessing module A2 and a second data writing module A3. The video frame preprocessing module B2 is further configured to, in response to receiving the first video frame in the target video stream, send a task command to the captured image preprocessing module A2 to trigger the capture of the captured image. That is, in response to receiving the first video frame in the target video stream and preprocessing the video frame, the video frame preprocessing module B2 simultaneously sends a task command to the captured image preprocessing module A2 to trigger the capture of the captured image.

[0097] The captured image preprocessing module A2 is configured to control an external device to capture captured images in real time in response to task commands and preprocess the captured images to produce preprocessed captured images. This preprocessing primarily involves converting the camera's raw data into RGB data according to a standard. This RGB data is referred to as the preprocessed captured image.

[0098] The second data writing module A3 is configured to write the pre-processed captured image into the memory control module C3 via the first bus communication module C1 according to the instruction of the second clock.

[0099] The memory control module C3 is configured to control the received pre-processed captured images to be stored in the DDR.

[0100] In some embodiments, the FPGA is configured to send an interrupt request to the CPU after storing the captured image, specifically including: the CPU is configured to respond to the interrupt request and send a first resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module responds to the first resource call request, obtains the preprocessed captured image, and performs face detection on the preprocessed captured image to obtain a first intermediate result; determines a face detection frame based on the first intermediate result; when the face detection frame is determined, sends a second resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module responds to the second resource call request, obtains a face image corresponding to the face detection frame, performs key point feature extraction on the face image, and obtains a second intermediate result; determines key point information based on the second intermediate result; and when the key point information is determined, estimates the coordinates of the human eye based on the key point information to obtain the human eye position information.

[0101] Specifically, as shown in Figure 1, the FPGA also includes an interrupt scheduling module A4, a second bus communication module C2, and a logic processing module A8, wherein the second bus communication module C2 is used to implement second bus communication; the CPU also includes a face detection module A5 and a key point detection module A9. The interrupt scheduling module A4 is configured to send an interrupt request to the face detection module A5 when the second data writing module A3 writes the pre-processed captured image into the memory control module C3. Here, after the second data writing module A3 writes the pre-processed captured image into the memory control module C3, an interrupt task is sent to the interrupt scheduling module A4, and the interrupt scheduling module A4 receives the interrupt task and sends an interrupt request to the face detection module A5. Alternatively, the interrupt scheduling module A4 may detect the second data writing module A3 in real time, and after detecting that the second data writing module A3 has written the pre-processed captured image into the memory control module C3, an interrupt request is sent to the face detection module A5.

[0102] The operators used to process captured images are deployed in two parts: one part is deployed in the face detection module A5, key point detection module A9 and eye coordinate estimation module A10 on the CPU side; the other part is deployed in the logic processing module A8 on the FPGA side.

[0103] The CPU is configured to, in response to an interrupt request, invoke computing resources in the FPGA via bus communication to process the captured image and obtain eye position information. Specifically, the CPU includes: Face detection module A5 is configured to, in response to the interrupt request, send a first resource invocation request to invoke computing resources to logic processing module A8 via second bus communication module C2. Logic processing module A8 only performs intermediate computations on the captured image.

[0104] Logic processing module A8 is configured to, in response to the first resource call request, obtain a preprocessed captured image and perform face detection on the preprocessed captured image to obtain a first intermediate result. Logic processing module A8 deploys a dedicated computational acceleration unit, such as convolutional and fully connected operators in a neural network, to perform face detection on the captured image (a computational process belonging to the first intermediate stage). During this computational process, logic processing module A8 continuously accesses intermediate data with the DDR, ultimately obtaining a first intermediate result of the first intermediate stage. Compared to CPU-side computational processing, using the FPGA to process the computational process of the first intermediate stage is more efficient and has lower latency.

[0105] Here, the logic processing module A8 obtains the pre-processed captured image stored in the DDR, and the acquisition path includes: reading the pre-processed captured image from the DDR with the help of the first data reading module A7, the first bus communication module C1 and the memory control module C3.

[0106] Afterwards, the logic processing module A8 can send the first intermediate result to the face detection module A5 through the second bus communication module C2 to notify the face detection module A5 that the first intermediate stage of the current captured image has been completed and the next stage of processing can be carried out.

[0107] Face detection module A5 is further configured to determine a face detection frame based on the first intermediate result. In addition to the face detection operators implemented in logic processing module A8, all other operators in the face detection algorithm are executed by CPU-side face detection module A5. Ultimately, CPU-side face detection module A5 determines a face detection frame.

[0108] The key point detection module A9 is configured to send a second resource calling request for calling computing resources to the logic processing module A8 through the second bus communication module C2 in response to receiving the face detection frame.

[0109] Furthermore, the logic processing module A8 is further configured to obtain a facial image corresponding to the face detection frame in response to a second resource call request, perform key point feature extraction on the facial image, and obtain a second intermediate result. The second resource call request carries a face detection frame, and the face detection frame can be used to obtain a facial image corresponding to the face detection frame from the preprocessed captured image. The dedicated computing acceleration unit deployed in the logic processing module A8 is used to perform key point feature extraction on the facial image (belonging to the computation process of the second intermediate stage) to obtain a second intermediate result of the second intermediate stage. Compared with CPU-side computational processing, the computational process of the second intermediate stage processed on the FPGA side is more efficient and has lower latency.

[0110] Afterwards, the logic processing module A8 can send the second intermediate result to the key point detection module A9 through the second bus communication module C2 to notify the key point detection module A9 that the second intermediate stage of the current face image has been completed and the next stage of processing can be carried out.

[0111] The key point detection module A9 is further configured to receive the second intermediate result, determine key point information based on the second intermediate result, and send the key point information to the human eye coordinate estimation module A10 for coordinate estimation.

[0112] The eye coordinate estimation module A10 is configured to estimate eye coordinates based on the received key point information to obtain eye position information. The eye position information includes the true coordinates of the observer's left eye and right eye. The true coordinates are defined within a pre-established display screen spatial coordinate system.

[0113] In some embodiments, the first bus communication module C1 and the second bus communication module C2 can be the same bus communication module or different bus communication modules. When the first bus communication module C1 and the second bus communication module C2 are different bus communication modules, the first bus communication module C1 can be a high-speed bus communication module and the second bus communication module C2 can be a low-speed bus communication module. The first bus communication module C1 provided by the present disclosure is used to schedule the interfaces between various modules and DDR control within the chip to transmit high-speed data streams. The second bus communication module C2 provided by the present disclosure is used to schedule various modules within the chip to transmit low-speed signals. The values ​​of some status registers or calculation results can also be transmitted through this channel. The bandwidth and response speed of the high-speed bus communication module are higher than those of the low-speed bus communication module, and the cost is also higher. Providing two types of bus communication modules can avoid tasks that can transmit data through the low-speed bus communication module from occupying the bandwidth of the high-speed bus communication module, thereby reducing the processing efficiency of certain high-speed data transmission tasks.

[0114] In some embodiments, the FPGA reads the pre-processed captured image from the memory via the first bus communication in response to the first resource call request.

[0115] Specifically, as shown in Figure 1, the FPGA also includes a first data reading module A7. The logic processing module A8 is configured to obtain the preprocessed captured image in response to the first resource call request, specifically including: the logic processing module A8 is configured to send a first image read instruction to the first data reading module A7 in response to the first resource call request. The first data reading module A7 is configured to send the first image read instruction to the memory control module C3 via the first bus communication module C1; the memory control module C3 is further configured to read the latest preprocessed captured image stored in sequence from the DDR in response to the first image read instruction, and transmit it to the first data reading module A7 via the first bus communication module C1. The first data reading module A7 is also configured to send the received preprocessed captured image to the logic processing module A8, so that the logic processing module A8 receives the preprocessed captured image transmitted by the first data reading module A7.

[0116] In some embodiments, the FPGA is configured to obtain a facial image corresponding to a face detection frame in response to a second resource call request, specifically including: a second processing module, specifically configured to obtain a preprocessed captured image from a memory in response to receiving a face detection frame; obtain a facial image corresponding to the face detection frame based on the face detection frame and the preprocessed captured image; generate a second resource call request based on the facial image; the FPGA is specifically configured to parse the facial image carried in the second resource call request in response to the second resource call request.

[0117] Specifically, as shown in Figure 1, the logic processing module A8 is configured to obtain the facial image corresponding to the face detection frame in response to the second resource call request, specifically including: the key point detection module A9 is configured to send a second image read instruction to the memory control module C3 in response to receiving the face detection frame. The memory control module C3 is also configured to read the latest pre-processed captured image stored in sequence from the DDR in response to the second image read instruction, and transmit it to the key point detection module A9. The key point detection module A9 is also configured to obtain the facial image corresponding to the face detection frame based on the face detection frame in response to receiving the pre-processed captured image; generate a second resource call request based on the facial image; here, the second resource call request carries the facial image. The logic processing module A8 is specifically configured to parse the facial image carried in the second resource call request in response to the second resource call request.

[0118] In some embodiments, as shown in FIG1 , logic processing module A8 is configured to obtain a facial image corresponding to a face detection frame in response to a second resource call request. Specifically, logic processing module A8 reads a preprocessed captured image from the DDR using first data reading module A7, first bus communication module C1, and memory control module C3. Then, logic processing module A8 uses the face detection frame carried in the second resource call request to obtain a facial image corresponding to the face detection frame from the preprocessed captured image.

[0119] Specifically, the logic processing module A8 is configured to send a fourth image read instruction to the first data reading module A7 in response to the second resource call request. The first data reading module A7 is configured to send the fourth image read instruction to the memory control module C3 via the first bus communication module C1. The memory control module C3 is further configured to read the latest pre-processed captured image stored in sequence from the DDR in response to the fourth image read instruction and transmit it to the first data reading module A7 via the first bus communication module C1. The first data reading module A7 is further configured to send the received pre-processed captured image to the logic processing module A8. The logic processing module A8 is further configured to receive the pre-processed captured image and, using the face detection frame carried in the second resource call request, obtain a facial image corresponding to the face detection frame from the pre-processed captured image.

[0120] In some embodiments, the FPGA is configured to perform face detection on the preprocessed captured image to obtain a first intermediate result, specifically including: a pre-set neural network algorithm is integrated in the FPGA, and is configured to perform face detection on the preprocessed captured image, and write the third intermediate result of the face detection process into the DDR through the first bus communication; and, when a request is made to call the third intermediate result, the third intermediate result is read from the DDR through the first bus communication, and is continued to be used for face detection until the first intermediate result is obtained.

[0121] Specifically, as shown in Figure 1, the FPGA also includes a third data writing module A6. The logic processing module A8 is configured to perform face detection on the pre-processed captured image, specifically including: a pre-set neural network algorithm is integrated in the logic processing module A8, and is configured to perform face detection on the pre-processed captured image, and send the third intermediate result of the face detection process to the third data writing module A6. The third intermediate result here can be understood as the output result of the network layer in the neural network when performing face detection on the captured image. The third data writing module A6 is configured to write the third intermediate result to the memory control module C3 through the first bus communication module C1. The memory control module C3 is also configured to control the received third intermediate result to be stored in the DDR.

[0122] At the same time, the logic processing module A8 is further configured to send a third intermediate result read instruction to the first data reading module A7 when requesting to call the third intermediate result. The first data reading module A7 is further configured to receive the third intermediate result read instruction and send the third intermediate result read instruction to the memory control module C3 via the first bus communication module C1. The memory control module C3 is further configured to respond to the third intermediate result read instruction sent by the first data reading module A7, read the third intermediate result from the DDR, and transmit it to the first data reading module A7 via the first bus communication module C1. The first data reading module A7 is further configured to send the received third intermediate result to the logic processing module A8. The logic processing module A8 is further configured to receive the third intermediate result transmitted by the first data reading module A7 and continue to use it for face detection until the first intermediate result is obtained. The continuous processing here can be understood as the calculation processing between multiple network layers, where the third intermediate result output by the first network layer may be used as the input of the second network layer. Similarly, the third intermediate result output by the second network layer may be used as the input of the third network layer, and so on. The output result of the last network layer in the neural network is also the first intermediate result. The calculation process of each network layer together constitutes the intermediate process of face detection.

[0123] Exemplarily, when all operators of the face detection algorithm are deployed in the logic processing module A8, the first intermediate result is the face detection frame.

[0124] In some embodiments, the FPGA is configured to perform key point feature extraction on a facial image to obtain a second intermediate result, specifically including: a pre-set neural network algorithm is integrated in the FPGA, and is configured to perform key point feature extraction on a facial image, and write a fourth intermediate result of the key point feature extraction process into the DDR through the first bus communication; and, when a request is made to call the fourth intermediate result, the fourth intermediate result is read from the DDR through the first bus communication, and is continued to be used for key point feature extraction until the second intermediate result is obtained.

[0125] Specifically, as shown in FIG1 , the logic processing module A8 is configured to extract key point features from a facial image to obtain key point information, specifically including: a pre-set neural network algorithm is integrated into the logic processing module A8, and the module is configured to extract key point features from a facial image, and send the fourth intermediate result of the key point feature extraction process to the third data writing module A6. The fourth intermediate result here can be understood as the output result of the network layer in the neural network when extracting key point features from a facial image. The third data writing module A6 is configured to write the fourth intermediate result to the memory control module C3 via the first bus communication module C1. The memory control module C3 is also configured to control the storage of the received fourth intermediate result into the DDR.

[0126] At the same time, the logic processing module A8 is further configured to send a fourth intermediate result read instruction to the first data reading module A7 when requesting to call the fourth intermediate result. The first data reading module A7 is further configured to receive the fourth intermediate result read instruction and send the fourth intermediate result read instruction to the memory control module C3 via the first bus communication module C1. The memory control module C3 is further configured to read the fourth intermediate result from the DDR in response to the fourth intermediate result read instruction sent by the first data reading module A7 and transmit it to the first data reading module A7 via the first bus communication module C1. The first data reading module A7 is further configured to send the received fourth intermediate result to the logic processing module A8. The logic processing module A8 is further configured to receive the fourth intermediate result transmitted by the first data reading module A7 and continue processing until the second intermediate result is obtained. The continuous processing here can be understood as the calculation processing between multiple network layers, where the fourth intermediate result output by the first network layer may be used as the input of the second network layer. Similarly, the fourth intermediate result output by the second network layer may be used as the input of the third network layer, and so on. The output result of the last network layer in the neural network is also the second intermediate result. The calculation process of each network layer together constitutes the intermediate process of key point feature extraction.

[0127] Exemplarily, when all operators of the key point feature extraction algorithm are deployed in the logic processing module A8, the second intermediate result is the key point information.

[0128] In some embodiments, the FPGA is configured to respond to receiving the first video frame in the target video stream and store it in the DDR according to the first clock instruction, specifically including: responding to receiving the first video frame in the target video stream, performing image segmentation on the first video frame to obtain a left image adapted for the left eye and a right image adapted for the right eye; writing a group of images and a right image into the DDR through the first bus according to the instruction of the first clock.

[0129] Specifically, as shown in Figure 1, the video frame preprocessing module B2 is configured to, in response to receiving the current video frame to be played in the target video stream, perform image segmentation on the video frame to obtain a left image adapted for the left eye and a right image adapted for the right eye. Subsequently, the first data writing module B3 is configured to write the set of left and right images to the memory control module C3 via the first bus communication module C1 in accordance with the first clock. The memory control module C3 is configured to control the storage of the received set of left and right images into the DDR.

[0130] For example, the image resolution of the video frame, i.e., the preset screen resolution, is 8K, i.e., 7680×4320 pix. Using an image segmentation algorithm, the video frame is segmented into a left image adapted for the left eye and a right image adapted for the right eye. The left image has an image resolution of 4K, i.e., 4096×2160 pix; the right image has an image resolution of 4K, i.e., 4096×2160 pix.

[0131] For example, if the resolution of the original captured image captured by some cameras is different from the preset screen resolution, the original captured image captured by the camera is scaled before the image is segmented. Bilinear interpolation or bicubic interpolation can be used to scale the original captured image to the captured image. Thereafter, the captured image is segmented to obtain the segmented left and right images.

[0132] In some embodiments, a specific process of acquiring a second video frame, performing pixel rearrangement on the second video frame, and obtaining a target video frame includes: in response to receiving human eye position information, reading sequentially stored left and right images from a DDR via a first bus communication; the second video frame includes a set of left and right images; in response to receiving a set of left and right images, scaling the left image according to a preset screen resolution to obtain first image data adapted for the left eye; scaling the right image according to a preset screen resolution to obtain second image data adapted for the right eye; in response to receiving a set of first image data and second image data, rearranging pixels in the first image data and pixels in the second image data according to the human eye position information to obtain an updated target video frame.

[0133] Specifically, as shown in Figure 1, the FPGA also includes a second data writing module A3, an image scaling module B5, and a pixel rearrangement module B6. The second data writing module A3 is configured to, in response to receiving the third image acquisition instruction, read the latest set of left and right images stored in sequence from the DDR and send them to the image scaling module B5. The image scaling module B5 is configured to, in response to receiving the third image acquisition instruction, read the latest set of left and right images stored in sequence from the DDR, and scale the left image according to a preset screen resolution to obtain first image data adapted for the left eye; scale the right image according to the preset screen resolution to obtain second image data adapted for the right eye; the pre-processed video frame includes the first image data and the second image data.

[0134] Exemplarily, according to the preset screen resolution, the left image is stretched into an 8K image to obtain first image data, and the right image is stretched into an 8K image to generate second image data.

[0135] The pixel rearrangement module B6 is configured to, in response to receiving the eye position information, send a third image acquisition instruction to the image scaling module B5; and, in response to receiving the eye position information, read the latest set of first image data and second image data stored in sequence from the DDR, and rearrange the pixels in the first image data and the second image data according to the eye position information to obtain an updated target video frame. Specifically, a pixel rearrangement algorithm can be used to calculate the pixel rearrangement logic in real time according to the eye coordinates indicated by the eye position information, select pixels adapted for the left eye from the first image data, and select pixels adapted for the right eye from the second image data, and rearrange them to form a naked-eye 3D image adapted for the human eye, i.e., the updated target video frame, and output it.

[0136] Regarding the process of the second data writing module A3 reading the latest set of sequentially stored left and right images from the DDR, as shown in Figure 1, the eye coordinate estimation module A10 is further configured to transmit eye position information to the pixel rearrangement module B6 via the second bus communication module C2. The pixel rearrangement module B6 is configured to transmit a third image read instruction to the second data writing module A3 in response to the eye position information. In response to receiving the third image acquisition instruction, the second data writing module A3 transmits the third image read instruction to the memory control module C3 via the first bus communication module C1. The memory control module C3 is further configured to, in response to the third image read instruction, read the latest set of sequentially stored left and right images from the DDR and transmit the information to the second data reading module B4 via the first bus communication module C1. The second data reading module B4 is further configured to, in response to receiving the latest set of left and right images, transmit the received left and right images to the image scaling module B5.

[0137] In some embodiments, for the pixel rearrangement process, specifically, the pixel rearrangement module B6 is specifically configured to determine the binocular midpoint position information according to the human eye position information in response to receiving a set of first image data and second image data; the human eye position information and the binocular position information are both positions in a pre-established spatial coordinate system (XOZ) about the display screen; for any target row pixel, the first size information of the first viewpoint image and the focus position information of the focus are determined according to the binocular midpoint position information and pre-stored prism parameters; the second size information of the second viewpoint image of the target row pixel is determined according to the first size information of the first viewpoint image, the focus position information of the focus, and the screen parameters; for the target pixel in any target row pixel, the first size information of the second viewpoint image of the target row pixel is determined based on the target pixel in the target row pixel. The method comprises the following steps: first, second, and third size information are used to determine the center position information of the red sub-pixel, the center position information of the green sub-pixel, and the center position information of the blue sub-pixel according to the first size information, the second size information, and the center position information of the red sub-pixel, the center position information of the green sub-pixel, and the center position information of the blue sub-pixel in the target pixel; and second, third, fourth size information are used to determine the position information of the left-eye pixel and the right-eye pixel according to the first size information, the second size information, and the center position information of the red sub-pixel, the green sub-pixel, and the blue sub-pixel in the target pixel. For each target row pixel, the image data of the sub-pixel at the corresponding position in the first image data is extracted according to the position information of the left-eye pixel, and the image data of the sub-pixel at the corresponding position in the second image data is extracted according to the position information of the right-eye pixel, to obtain an updated target video frame.

[0138] FIG5 is a schematic diagram of the pixel rearrangement principle provided by an embodiment of the present disclosure. As shown in FIG5 , the eye position information includes the left eye coordinates ML (XL, ZL) and the right eye coordinates MR (XR, ZR). Based on the eye position information, the midpoint position information M1 (X1, Z1) of both eyes is determined, as shown in the following formula 1:

[0139] Among them, XL represents the coordinate position of the viewer's left eye on the first coordinate axis (for example, the horizontal axis) in the spatial coordinate system; XR represents the coordinate position of the viewer's right eye on the first coordinate axis (for example, the horizontal axis) in the spatial coordinate system; ZL represents the coordinate position of the viewer's left eye on the second coordinate axis (for example, the vertical axis) in the spatial coordinate system; ZR represents the coordinate position of the viewer's right eye on the second coordinate axis (for example, the vertical axis) in the spatial coordinate system; X1 represents the coordinate position of the midpoint of both eyes on the first coordinate axis (for example, the horizontal axis) in the spatial coordinate system; Z1 represents the coordinate position of the midpoint of both eyes on the second coordinate axis (for example, the vertical axis) in the spatial coordinate system.

[0140] Afterwards, for any target row of pixels, the first size information of the first view image is determined based on the binocular midpoint position information M1 and pre-stored prism parameters using the principle of similar triangles, as shown in the following formula 2:

[0141] Further launch:

[0142] Where ΔX represents the first dimension of the first view image, which can be understood as the length of the first view image; P represents the prism pitch; and H represents the distance between the prism and the pixel. The pre-stored prism parameters include the prism pitch and the distance between the prism and the pixel. The prism pitch can be understood as the distance between the center points of two adjacent prisms along the row direction. The distance between the prism and the pixel can be understood as the perpendicular distance between the surface of the prism closest to the pixel and the surface of the pixel closest to the prism.

[0143] It should be noted that the viewpoint map corresponds one-to-one to the prism. The viewpoint map is the line connecting the coordinates of the human eye and the boundary position of each prism, and is projected onto the pixel surface to form a series of arranged virtual images. As shown in Figure 5, a single viewpoint map represents the smallest unit left and right images that enter the human eye.

[0144] It should be noted that the prism selected for calculating the first viewpoint image here is a prism close to the origin O of the spatial coordinate system (XOZ), for example, it can be a prism adjacent to the left side of the coordinate axis Z, or a prism adjacent to the right side of the coordinate axis Z can also be selected.

[0145] For any target row pixel, the focus position information M2 is determined based on the binocular midpoint position information M1 and pre-stored prism parameters using the principle of similar triangles, as shown in the following formula 3:

[0146] Further launch:

[0147] The coordinates of M2 are ( -H). The light path with the focus at the midpoint of both eyes passes through the prism adjacent to the left side of the second coordinate axis and is transmitted to the right point on the reference plane where the target row of pixels is located.

[0148] Based on the first size information ΔX of the first view image, the coordinate position X2 of the first coordinate axis indicated by the focus position information, and the screen parameter W, the second size information of the second view image of the target row of pixels is determined, as shown in the following formula 4: ΔB = MOD(X2 + W / 2, ΔX)………………………….Formula 4

[0149] The second view image is a non-integer number of view images at the edge position (e.g., the left edge position) of the target row of pixels. This means that the length of the second view image does not satisfy ΔX. ΔB represents the second size information of the second view image, which can be understood as the length of the second view image; ΔB < ΔX; MOD() represents the remainder function; and W represents the length of the display area on the display screen, or can be understood as the length of the entire row of pixels in the target row, which is a screen parameter.

[0150] For a target pixel in any target row of pixels, based on the position of the target pixel in the target row of pixels, the preset pixel size information, and the positions of each sub-pixel in the target pixel, the center position information of the red sub-pixel, the center position information of the green sub-pixel, and the center position information of the blue sub-pixel are determined respectively, as shown in the following formula 5:

[0151] Where, ΔR J Indicates the center position information of the red sub-pixel, which actually represents the distance between the center of the red sub-pixel and the left edge of the leftmost pixel in the target row, that is, the distance between the center of the red sub-pixel and the left edge of the display. ΔG J Indicates the center position information of the green sub-pixel, which actually represents the distance between the center of the green sub-pixel and the left edge of the leftmost pixel in the target row, that is, the distance between the center of the green sub-pixel and the left edge of the display. ΔB J Indicates the center position of the blue sub-pixel, representing the distance from the center of the blue sub-pixel to the left edge of the leftmost pixel in the target row, or in other words, the distance from the center of the blue sub-pixel to the left edge of the display. J represents the column number of the target pixel in the target row. N represents the width of the target pixel, as indicated by the target pixel size information.

[0152] According to the first size information ΔX, the second size information ΔB, and the center position information ΔB of the red sub-pixel in the target pixel J , the center position information of the green sub-pixel ΔG J and the center position information of the blue sub-pixel ΔB J , determine the position information of the left eye pixel and the position information of the right eye pixel, see the following formula 6:

[0153] It is further speculated that when Determine the red subpixel ΔR in the target pixel J The position of the left eye pixel; when Determine the red subpixel ΔR in the target pixel J belongs to the position of the right eye pixel. Similarly, when Determine the green subpixel ΔG in the target pixel JThe position of the left eye pixel; when Determine the green subpixel ΔG in the target pixel J belongs to the position of the right eye pixel. Similarly, when Determine the blue subpixel ΔB in the target pixel J The position of the left eye pixel; when Determine the blue subpixel ΔB in the target pixel J The position of the right eye pixel. Then, for each target row pixel, the position information ΔR of the left eye pixel can be used. J , ΔG J or ΔB J , extract the image data of the sub-pixel at the corresponding position in the first image data, and, according to the position information ΔR of the right eye pixel J , ΔG J or ΔB J , extract the sub-pixel image data of the corresponding position in the second image data to obtain an updated target video frame.

[0154] Here, when the FPGA processes pixels internally, it processes them in order from left to right. Therefore, it is only necessary to calculate whether the current pixel should take the first image data of the left eye of the video or the second image data of the right eye based on the pixel sequence number actually to be obtained. In this way, data can be taken from the frame buffer of the first image data and the second image data stored in the DDR in sequence.

[0155] In some embodiments, the heterogeneous chip 100 may be a system on chip (SOC).

[0156] To facilitate understanding of the embodiments of the present disclosure, each functional module in the heterogeneous chip 100 is described in detail. FIG6 is a schematic diagram of a display system including a heterogeneous chip provided by an embodiment of the present disclosure. As shown in FIG6 , A1 represents an external device, such as a camera, for collecting the original captured image of the observer in real time. A2 represents a captured image preprocessing module. The original captured image is collected by the camera and enters the captured image preprocessing module inside the heterogeneous chip 100 through a high-speed serial interface such as MIPI and LVDS. The captured image preprocessing module converts the original captured image into an image in a standard format, and performs segmentation and scaling to obtain a captured image suitable for the subsequent detection algorithm. A3 represents a second data writing module; A6 represents a third data writing module; B3 represents a first data writing module; the data writing modules corresponding to A3, A6 and B3 are all used to write information to the DDR to temporarily store the data to be processed, and the CPU or FPGA retrieves and processes it from the shared DDR. A4 represents an interrupt scheduling module. A5 represents the face detection module, which integrates some face detection operators and has face detection capabilities to obtain a face detection frame for subsequent key point feature extraction. A7 represents the first data reading module; B4 represents the second data reading module. The data reading modules corresponding to A7 and B4 are both used to read information from the DDR for subsequent processing. A8 represents the logic processing module, which integrates dedicated computational acceleration units for face detection and key point feature extraction, such as convolution and fully connected operators in neural networks. Using dedicated computational acceleration units will result in more efficient and low-latency calculations. A9 represents the key point detection module, which has key point feature extraction capabilities. This module can detect key point information on the face for subsequent coordinate estimation. A10 represents the eye coordinate estimation module, which is used to estimate eye coordinates. After detecting key point information, the true coordinates of the eye, i.e., the eye position information, are calculated based on the key point information.

[0157] B1 represents a standard video terminal commonly used by external playback sources, used to transmit video frames in the target video stream. B2 represents a video frame preprocessing module, used to preprocess received video frames. B5 represents an image scaling module, used to stretch the left and right images to obtain corresponding first and second image data. B6 represents a pixel rearrangement module, used to calculate pixel rearrangement logic in real time based on eye position information, ultimately forming a standard video signal for output, i.e., the target video frame. B7 represents the FPGA's video output interface, VBO TX. VBO stands for V-BY-ONE standard video port, and is used to transmit high-speed video serial differential signals within the display screen. Designed as an FPGA output within the logic side, it ultimately drives the display's Tcon to display processed 3D content. B8 represents a display screen, such as a 3D display screen driven by Tcon. C1 represents a high-speed bus communication module (also known as the first bus communication module), used to schedule the interface between various modules and the DDR control within the FPGA. C2 represents the low-speed bus communication module (also known as the second bus communication module), which is used to schedule various modules within the FPGA to transmit low-speed signals. Status register values ​​or calculation results can be transmitted through this channel. C3 represents the memory control module, which is used to operate accessed data according to DDR access timing. C4 represents DDR.

[0158] Among them, A1, B1 and B8 are not part of the heterogeneous chip 100. For the sake of completeness, this disclosure adds some peripheral structures. The data processing process of the heterogeneous chip 100 is described in detail below using a complete example.

[0159] FIG7a is a flowchart of data processing for determining eye position information provided by an embodiment of the present disclosure. As shown in FIG7a, S1-1, start camera acquisition; S1-2, camera acquires original captured image (A1); S1-3, pre-processes original captured image (A2); S1-4, writes to DDR (A3-C1-C3-C4); S1-5, sends interrupt request (A4); S1-6, starts face detection (A5); S1-7, dispatches computing resources in logic processing module A8 through second bus communication module C2; ​​S1-8, logic processing module A 8 accesses the third intermediate result (A6-A7-C1-C3-C4); S1-9, extracts the face detection frame (A5); S1-10, starts key point feature extraction (A9); S1-11, schedules the computing resources in the logic processing module A8 through the second bus communication module C2; ​​S1-12, the logic processing module A8 accesses the fourth intermediate result (A6-A7-C1-C3-C4); S1-13, extracts key point information (A9); S1-14, estimates the human eye coordinates (A10); S1-15, sends the human eye coordinate information to FPGA.

[0160] Figure 7b is a flowchart for determining pixel rearrangement provided by an embodiment of the present disclosure. As shown in Figure 7b, S2-1, receiving the target video stream through the standard video terminal (B1); S2-21, preprocessing the first video frame in the target video stream (B2), and at the same time S2-22, sending the task command (B2-A2-A1); S2-3, writing the preprocessed video frame to the DDR (B3-C1-C3-C4); S2-4, following the above step S1-15, taking the left image and the right image out of the DDR (B4, -C1-C3-C4); S2-5, stretching the left image and the right image (B5); S2-6, pixel rearrangement (A6); S2-7, converting the video signal into the timing required by the output interface and sending it (B7); S2-8, sending it to the display screen (B8).

[0161] In addition, a display device corresponding to the heterogeneous chip is also provided in the embodiment of the present disclosure. Since the technical principles of the problems solved by the heterogeneous chip in the display device in the embodiment of the present disclosure are similar to those of the above-mentioned heterogeneous chip in the embodiment of the present disclosure, the implementation of the heterogeneous chip in the display device can refer to the implementation of the above-mentioned heterogeneous chip, and the repeated parts will not be repeated.

[0162] As shown in Figure 6, the display device is applied to the naked-eye 3D display scene; the display device includes a display screen B8 and a heterogeneous chip; the heterogeneous chip includes a first processing core, a second processing core and a memory; the first processing core and the second processing core are connected through bus communication.

[0163] The first processing core is configured to respond to receiving the first video frame in the target video stream and store it in the memory according to the instructions of the first clock; respond to receiving the first video frame in the target video stream, control the captured image in the acquisition environment, and store the acquired captured image in the memory according to the instructions of the second clock, and after storing the captured image, send an interrupt request to the second processing core.

[0164] The first processing core is, for example, an FPGA, and the memory is, for example, a DDR.

[0165] The second processing core is configured to respond to the interrupt request, call the computing resources in the first processing core through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the first processing core.

[0166] The second processing core is, for example, a CPU.

[0167] Among them, in response to receiving the human eye position information, the first processing core reads the latest second video frame stored in sequence from the memory, and rearranges the pixels of the second video frame according to the human eye position information to obtain an updated target video frame, and sends it to the display screen B8.

[0168] Display screen B8 is configured to display according to the received target video frame.

[0169] The display device provided by the embodiment of the present disclosure utilizes a CPU and an FPGA on the same heterogeneous chip to process different parts of the face recognition algorithm respectively, utilizes the FPGA to pre-process the video frames and captured images; utilizes the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, and can also perform various other operator operations based on the instruction set; utilizes the CPU to perform post-processing after the convolution, such as key point detection and coordinate estimation. Among them, utilizing the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, as well as various other operator operations based on the instruction set, can achieve the best processing effect, such as high frame rate and low latency processing effect. In addition, the FPGA utilizes the frame synchronization signal when receiving the video frame in the first clock domain to trigger the external device in the second clock domain to collect the captured image signal in real time, thereby realizing synchronous signal processing between asynchronous clock domains, thereby ensuring that the signal of each frame is processed in a timely manner.

[0170] In some embodiments, as shown in FIG6 , the display apparatus further includes an external device A1 ; the external device A1 is configured to capture an original captured image and send the image to the FPGA in response to a control instruction when the FPGA receives a video frame.

[0171] In some embodiments, as shown in FIG6 , the display device further includes a standard video terminal B1 commonly used by an external playback source.

[0172] In addition, the embodiment of the present disclosure also provides a display control method corresponding to the heterogeneous chip. Since the technical principles of the problems solved by the display control method in the embodiment of the present disclosure are similar to those of the above-mentioned heterogeneous chip in the embodiment of the present disclosure, the implementation of the display control method can refer to the implementation of the above-mentioned heterogeneous chip, and the repeated parts will not be repeated.

[0173] This display control method is applied to a heterogeneous chip in a naked-eye 3D display scenario; the heterogeneous chip includes a first processing core, a second processing core, and a memory shared by the first and second processing cores; the first and second processing cores are communicatively connected via a bus. FIG8 is a flowchart of the display control method provided by an embodiment of the present disclosure. As shown in FIG8 , the display control method includes steps S11 to S14.

[0174] S11 . The first processing core receives a first video frame in a target video stream and stores the received frame into a memory according to an instruction of a first clock.

[0175] S12. In response to receiving the first video frame in the target video stream, the first processing core controls the captured image in the acquisition environment, stores the acquired captured image in the memory according to the instruction of the second clock, and sends an interrupt request to the second processing core after storing the captured image.

[0176] S13. The second processing core responds to the interrupt request, calls the computing resources in the first processing core through bus communication, processes the captured image, obtains the human eye position information, and sends the human eye position information to the first processing core.

[0177] S14. In response to receiving the eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame.

[0178] The second video frame and the first video frame are separated by at least one video frame in the target video stream.

[0179] The first processing core is, for example, an FPGA, the second processing core is, for example, a CPU, and the memory is, for example, a DDR.

[0180] The display control method provided by the embodiment of the present disclosure utilizes the CPU and FPGA to process different parts of the face recognition algorithm respectively, utilizes the FPGA to pre-process the video frames and captured images; utilizes the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, and can also perform various other operations based on the instruction set; utilizes the CPU to perform post-processing after the convolution, such as key point detection and coordinate estimation. Among them, utilizing the CPU to schedule the high-speed computing unit in the FPGA to perform convolution operations in the algorithm model, as well as various other operations based on the instruction set, can achieve the best processing effect, such as high frame rate and low latency processing effect. In addition, the FPGA utilizes the frame synchronization signal when receiving the video frame in the first clock domain to trigger the external device in the second clock domain to collect the captured image signal in real time, thereby realizing synchronous signal processing between asynchronous clock domains, thereby ensuring that each signal is processed in a timely manner.

[0181] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0182] Figure 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. As shown in Figure 9, the computer device provided in an embodiment of the present disclosure includes: one or more processors 901, memory 902, and one or more I / O interfaces 903. Memory 902 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the display control methods described in the above embodiments; one or more I / O interfaces 903 are connected between the processor and memory and are configured to enable information exchange between the processor and memory.

[0183] Among them, the processor 901 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 902 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 903 is connected between the processor 901 and the memory 902, and can realize information interaction between the processor 901 and the memory 902, including but not limited to a data bus (Bus), etc.

[0184] In some embodiments, the processor 901 , the memory 902 , and the I / O interface 903 are connected to each other via a bus 904 , and further connected to other components of the computing device.

[0185] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided, wherein the non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the steps of the display control method in any of the above embodiments are implemented.

[0186] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0187] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A heterogeneous chip, wherein: The heterogeneous chip includes a first processing core, a second processing core and a memory; The first processing core and the second processing core are communicatively connected via a bus; The first processing core is configured to respond to receiving a first video frame in a target video stream and store it in the memory according to an instruction of a first clock; In response to receiving a first video frame in a target video stream, controlling a captured image in the acquisition environment, storing the captured image in the memory according to an instruction of a second clock, and sending an interrupt request to the second processing core after storing the captured image; the second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the first processing core; In which, in response to receiving the human eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the human eye position information to obtain an updated target video frame; the second video frame and the first video frame are separated by at least one frame of video frame in the target video stream.

2. The heterogeneous chip according to claim 1, wherein: The first processing core is configured to respond to receiving a first video frame in a target video stream and store it in the memory according to a first clock instruction, specifically comprising: In response to receiving a first video frame currently to be played in a target video stream, preprocessing the first video frame to obtain a preprocessed video frame; The preprocessed video frame is written into the memory according to the instruction of the first clock via the first bus communication.

3. The heterogeneous chip according to claim 2, wherein: The first processing core is configured to, in response to receiving a first video frame in a target video stream, control the captured image in the acquisition environment and store the acquired captured image in the memory, specifically comprising: In response to receiving a first video frame currently to be played in a target video stream, controlling a captured image in a capture environment and preprocessing the captured image to obtain a preprocessed captured image; The pre-processed captured image is written into the memory according to the instruction of the second clock through the first bus communication.

4. The heterogeneous chip according to claim 3, wherein: The second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication to process the captured image to obtain the human eye position information, specifically including: The second processing core is configured to respond to the interrupt request and send a first resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module obtains the preprocessed captured image in response to the first resource call request, and performs face detection on the preprocessed captured image to obtain a first intermediate result; determines a face detection frame based on the first intermediate result; when the face detection frame is determined, sends a second resource call request for calling computing resources to the first processing module through the second bus communication, so that the first processing module obtains the face image corresponding to the face detection frame in response to the second resource call request, performs key point feature extraction on the face image, and obtains a second intermediate result; determines key point information based on the second intermediate result; and when the key point information is determined, estimates the eye coordinates based on the key point information to obtain eye position information.

5. The heterogeneous chip according to claim 4, wherein: The first processing core is configured to obtain the pre-processed captured image in response to the first resource call request, specifically including: In response to the first resource call request, the pre-processed captured image is read from the memory through the first bus communication.

6. The heterogeneous chip according to claim 4, wherein: The first processing core is configured to obtain, in response to the second resource call request, a face image corresponding to the face detection frame, specifically comprising: The second processing module is specifically configured to, in response to receiving the face detection frame, obtain the pre-processed captured image from the memory; obtain a face image corresponding to the face detection frame based on the face detection frame and the pre-processed captured image; and generate the second resource call request based on the face image; The first processing core is specifically configured to, in response to the second resource call request, parse the facial image carried in the second resource call request.

7. The heterogeneous chip according to claim 4, wherein: The first processing core is configured to perform face detection on the pre-processed captured image to obtain a first intermediate result, specifically including: The first processing core is integrated with a preset neural network algorithm and is configured to perform face detection on the preprocessed captured image, and write a third intermediate result of the face detection process into the memory through the first bus communication; and when a request is made to call the third intermediate result, the third intermediate result is read from the memory through the first bus communication and continued to be used for face detection until the first intermediate result is obtained.

8. The heterogeneous chip according to claim 4, wherein: The first processing core is configured to extract key point features from the facial image to obtain a second intermediate result, specifically including: The first processing core is integrated with a preset neural network algorithm and is configured to perform key point feature extraction on the facial image and write a fourth intermediate result of the key point feature extraction process into the memory through the first bus communication; and when a request is made to call the fourth intermediate result, the fourth intermediate result is read from the memory through the first bus communication and continued to be used for key point feature extraction until the second intermediate result is obtained.

9. The heterogeneous chip according to claim 2, wherein: The first processing core is configured to respond to receiving a first video frame in a target video stream and store it in the memory according to a first clock instruction, specifically comprising: In response to receiving a first video frame in a target video stream, performing image segmentation on the first video frame to obtain a left image adapted for a left eye and a right image adapted for a right eye; A set of the group image and the right image are written into the memory via the first bus according to the instruction of the first clock.

10. The heterogeneous chip according to claim 9, wherein: In response to receiving the eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame, specifically including: In response to receiving the human eye position information, reading the left image and the right image stored in sequence from the memory through the first bus communication; the second video frame includes a group of left images and right images; In response to receiving a set of left and right images, scaling the left image according to a preset screen resolution to obtain first image data adapted for the left eye; scaling the right image according to the preset screen resolution to obtain second image data adapted for the right eye; In response to receiving a set of the first image data and the second image data, pixels in the first image data and pixels in the second image data are rearranged according to the eye position information to obtain an updated target video frame.

11. The heterogeneous chip according to claim 10, wherein: The first processing core is configured to rearrange pixels in the first image data and pixels in the second image data according to the human eye position information to obtain an updated target video frame, specifically including: In response to receiving a set of the first image data and the second image data, determining the binocular midpoint position information according to the human eye position information; The human eye position information and the binocular position information are both positions in a pre-established spatial coordinate system of the display screen; For any target row of pixels, determining first size information of the first view image and focus position information of the focus according to the binocular midpoint position information and pre-stored prism parameters; Determining second size information of a second view image of the target row of pixels according to first size information of the first view image, focus position information of the focus, and screen parameters; For a target pixel in any target row of pixels, based on a position of the target pixel in the target row of pixels, preset pixel size information, and positions of each sub-pixel in the target pixel in the target pixel, respectively determine the center position information of a red sub-pixel, the center position information of a green sub-pixel, and the center position information of a blue sub-pixel; determining position information of a left-eye pixel and position information of a right-eye pixel based on the first size information, the second size information, and the center position information of a red sub-pixel, the center position information of a green sub-pixel, and the center position information of a blue sub-pixel in the target pixel; For each target row pixel, the sub-pixel image data of the corresponding position in the first image data is extracted according to the position information of the left eye pixel, and the sub-pixel image data of the corresponding position in the second image data is extracted according to the position information of the right eye pixel, to obtain the updated target Video frame.

12. A display device comprising a display screen and a heterogeneous chip; the heterogeneous chip comprises a first processing core, a second processing core, and a memory; The first processing core and the second processing core are communicatively connected via a bus; The first processing core is configured to respond to receiving a first video frame in a target video stream and store it in the memory according to an instruction of a first clock; In response to receiving a first video frame in a target video stream, controlling a captured image in the acquisition environment, storing the captured image in the memory according to an instruction of a second clock, and sending an interrupt request to the second processing core after storing the captured image; the second processing core is configured to, in response to an interrupt request, call computing resources in the first processing core through bus communication, process the captured image, obtain the human eye position information, and send the human eye position information to the first processing core; In response to receiving the eye position information, the first processing core reads the second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame, and sends the updated target video frame to the display screen; The display screen is configured to display according to the received target video frame.

13. The display device according to claim 12, wherein: The display device further includes an external device; The external device is configured to collect the original captured image and send it to the first processing core in response to the control instruction when the first processing core receives the video frame.

14. A display control method, wherein: include: The first processing core is responsive to receiving a first video frame in a target video stream and storing the frame into the memory according to an instruction of a first clock; The first processing core controls, in response to receiving a first video frame in a target video stream, capturing an image in a capture environment, and stores the captured image in a memory according to an instruction of a second clock, and after storing the captured image, sends an interrupt request to the second processing core; The second processing core responds to the interrupt request and calls the first processing core through bus communication. computing resources to process the captured image to obtain eye position information, and send the eye position information to the first processing core; In response to receiving the eye position information, the first processing core reads a second video frame stored in sequence from the memory, and rearranges pixels of the second video frame according to the eye position information to obtain an updated target video frame; the second video frame and the first video frame are separated by at least one video frame in the target video stream.

15. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the display control method according to claim 14 are performed.

16. A computer non-transitory readable storage medium, wherein: The computer non-transitory readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display control method according to claim 14 are executed.

Images