Delay data calculation method, image fusion method and electronic device

Abstract

Embodiments of the present application provide a delay data calculation method, an image fusion method and an electronic device. The delay data calculation method comprises: obtaining a target acquisition image and continuous multiple frames of return data; the acquisition image is obtained by photographing a screen displaying a calibration marker; the return data comprises motion capture data or focal length data; performing marker detection on the target acquisition image to obtain an actual position of the calibration marker in the target acquisition image; calculating a candidate position corresponding to each frame of return data, the candidate position being a position of the calibration marker in the target acquisition image calculated based on each frame of return data; determining target return data from the multiple frames of return data according to a difference between the actual position and the candidate position, and obtaining delay data based on the target return data; the delay data comprises dynamic delay data or zoom delay data. Embodiments of the present application realize automatic calibration of delay data, and the obtained delay data is also more accurate.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a delay data calculation method, an image fusion method, an electronic device and a computer storage medium. BACKGROUND

[0002] XR (Extended Reality, extended reality) virtual shooting is to display a rendering picture (on-screen image) rendered by a virtual engine on an LED screen, and then an actor performs using the LED screen as a background, an image acquisition device (such as a camera) simultaneously shoots the actor and the LED screen, and then the camera picture (acquisition image) shot is synthesized with the rendering picture, so as to place the real actor in a virtual scene, achieving the effect of shooting a scene or a science fiction background in a studio.

[0003] To ensure that the finally synthesized image does not have tearing and other problems, in the image synthesis process, the rendering picture is adjusted in view angle based on the focal length information of the camera when shooting the acquisition image, the position information (motion capture data) of the camera, and then the image obtained after the adjustment is synthesized with the camera picture. In actual shooting, the camera picture, the motion capture data and the focal length data are usually returned independently, and each has different device delays, so there are three different delays: static delay, dynamic delay, and zoom delay. Among them, the static delay refers to the delay from the rendering picture of the engine to the camera picture returned by the camera when the camera position and focal length are unchanged; the dynamic delay is the delay caused by the asynchronous return of the motion capture data and the camera picture; the zoom delay is the delay caused by the asynchronous return of the camera focal length data and the camera picture.

[0004] In related technologies, when delay data is calculated, an operator usually inputs the delay data by himself, and then determines a frame of return data corresponding to the input delay data as target return data from a plurality of frames of candidate return data (candidate rendering picture, candidate motion capture data and candidate focal length data) returned continuously, and then synthesizes the camera picture based on the target return data, and then observes whether there is a tearing phenomenon in the synthesized image through the human eye, and finally selects the final delay data from the synthesized effect after multiple attempts of inputting. The above-mentioned method mainly relies on manual input and trial selection multiple times, and cannot realize automatic calculation of delay data. In addition, the selection process mainly relies on the human eye to judge the synthesis effect, so the final delay data obtained may not be accurate enough. SUMMARY

[0005] Therefore, embodiments of the present application provide a delay data calculation scheme to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of the present application, a delay data calculation method is provided, comprising:

[0007] obtaining a target acquisition image and continuous multiple frames of return data; the target acquisition image is determined from continuous multiple frames of acquisition images returned synchronously with the return data; the acquisition image is obtained by photographing a screen displaying a calibration marker; the return data includes motion capture data or focal length data;

[0008] detecting the target acquisition image to obtain the actual position of the calibration marker in the target acquisition image;

[0009] calculating a candidate position corresponding to each frame of return data; the candidate position is the position of the calibration marker in the target acquisition image calculated based on each frame of return data;

[0010] determining target return data from the multiple frames of return data according to the difference between the actual position and the candidate position, and obtaining delay data based on the target return data; the delay data includes dynamic delay data or zoom delay data.

[0011] According to a second aspect of the embodiments of the present application, an image fusion method is provided, comprising:

[0012] receiving an acquisition image to be fused;

[0013] receiving continuous multiple frames of returned engine rendering images; based on the receiving time information of the acquisition image to be fused and static delay data, determining a rendering image to be fused from the continuous multiple frames of engine rendering images;

[0014] receiving continuous multiple frames of returned motion capture data; based on the receiving time information of the acquisition image to be fused and dynamic delay data, determining target motion capture data from the continuous multiple frames of motion capture data;

[0015] receiving continuous multiple frames of returned focal length data; based on the receiving time information of the acquisition image to be fused and zoom delay data, determining target focal length data from the continuous multiple frames of motion capture data;

[0016] based on the target motion capture data and the target focal length data, performing image fusion on the rendering image to be fused and the acquisition image to be fused to obtain a fused image;

[0017] wherein the dynamic delay data and the zoom delay data are obtained by the method of the first aspect described above.

[0018] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method according to the first aspect or the second aspect.

[0019] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to the first aspect or the second aspect.

[0020] According to the delay data calculation scheme provided by the embodiments of the present application, after the target acquisition image and the continuous multiple frames of return data are acquired, the actual position of the calibration marker point in the target acquisition image is detected, the candidate positions of the calibration marker point in the target acquisition image are calculated based on each frame of return data, the target return data closest to the synchronous return data corresponding to the target acquisition image is determined by comparing the differences between the actual position and each candidate position, and then the delay data is obtained based on the target return data. In the embodiments of the present application, manual input of delay data is not required, and automatic calibration of delay data is realized. In addition, in the embodiments of the present application, the target return data is determined from multiple frames of return data based on the differences between the actual position of the detected calibration marker point and the calculated multiple candidate positions, and then the delay data is obtained, instead of being obtained by trial selection. Therefore, the obtained delay data is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0022] Figure 1 A flowchart of an image synthesis process in an XR virtual shooting process;

[0023] Figure 2 A step flowchart of a delay data calculation method according to Embodiment One of the present application;

[0024] Figure 3 A combination marker symbol schematic diagram;

[0025] Figure 4 A delay calibration flowchart corresponding to Embodiment One of the present application;

[0026] Figure 5A flowchart of steps of an image fusion method according to Embodiment Two of the present application;

[0027] Figure 6 A structural block diagram of a delay data calculation device according to Embodiment Three of the present application;

[0028] Figure 7 A structural block diagram of an image fusion device according to Embodiment Four of the present application;

[0029] Figure 8 A structural schematic diagram of an electronic device according to Embodiment Five of the present application. DETAILED DESCRIPTION

[0030] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the embodiments of the present application shall belong to the scope of protection of the embodiments of the present application.

[0031] Reference Figure 1 , Figure 1 A flowchart of image synthesis in an XR virtual shooting process. In actual shooting, the camera picture, motion capture data and focal length data backhaul are usually independent of each other, and there are different device delays, so there are three different delays: static delay, dynamic delay, and zoom delay. Among them, the static delay refers to the delay from rendering the picture by the engine to obtaining the camera picture backhaul by the camera. Referring to Figure 1 , assuming that the rendering time is T0, the camera picture backhaul time is T2, and the static delay is T2-T0; the dynamic delay is the time difference caused by the asynchronization of motion capture data and camera picture backhaul. Referring to Figure 1 , assuming that the camera shooting time is T1, the camera position at this time is L1, the camera picture backhaul time of this frame is T2, and the motion capture backhaul data L1 time is T3, then T3-T2 is the dynamic delay; the zoom delay refers to the time difference caused by the asynchronization of camera focal length data and camera picture backhaul. Referring to Figure 1 , assuming that the camera shooting time is T1, the camera focal length at this time is F1, the camera picture backhaul time of this frame is T2, and the focal length data time is T4, then T4-T2 is the zoom delay.

[0032] The related art usually inputs the delay data by an operator, determines a frame of return data corresponding to the target return data from a plurality of frames of candidate return data (candidate rendering pictures, candidate motion capture data and candidate focal length data) based on the input delay data, synthesizes the camera picture based on the target return data, and determines whether there is a tearing phenomenon in the synthesized image by human eyes. The above-mentioned method mainly relies on manual input and trial selection, and cannot realize automatic calculation of the delay data. In addition, the selection process mainly relies on human eyes to determine the synthesis effect, and therefore the final delay data may not be accurate enough.

[0033] The delay data calculation scheme provided by the embodiment of the present application detects the actual position of the calibration marker point in the target acquisition image after obtaining the target acquisition image and the plurality of frames of return data, calculates the candidate positions of the calibration marker point in the target acquisition image based on each frame of return data, determines the target return data closest to the synchronous return data corresponding to the target acquisition image by comparing the differences between the actual position and each candidate position, and then obtains the delay data based on the target return data. The embodiment of the present application realizes automatic calibration of the delay data without manual input of the delay data. In addition, the embodiment of the present application determines the target return data from the plurality of frames of return data based on the differences between the actual position of the detected calibration marker point and the plurality of calculated candidate positions, and then obtains the delay data, rather than obtaining the delay data by trial selection, and therefore the obtained delay data is more accurate.

[0034] The embodiment of the present application will be further described below with reference to the accompanying drawings.

[0035] Embodiment One

[0036] Reference Figure 2 , Figure 2 is a flowchart of the steps of a delay data calculation method according to the embodiment one of the present application. Specifically, the delay data calculation method provided by the embodiment includes the following steps:

[0037] Step 202, obtaining a target acquisition image and a plurality of frames of return data.

[0038] The target acquisition image is an image determined from a plurality of frames of acquisition images returned synchronously with the return data; the acquisition image is obtained by photographing a screen displaying a calibration marker; and the return data includes motion capture data or focal length data.

[0039] Specifically, the target acquisition image in the embodiment of the present application can be any one of the above-mentioned continuous multiple frames of acquisition images. Further, in order to avoid the problem that the matching target return data cannot be found in the subsequent steps due to the return speed of the return data being slower than the return speed of the acquisition image, in the embodiment of the present application, the Nth frame of the above-mentioned multiple frames of acquisition images can be determined as the target acquisition image, wherein the total number of the continuous multiple frames of acquisition images is M, M and N are natural numbers, and N < M. For example: 10 continuous frames of return data are acquired in step 202, and correspondingly, 10 continuous frames of acquisition images are also acquired. In order to avoid the problem that the matching target return data cannot be found in the subsequent steps due to the return speed of the return data being slower than the return speed of the acquisition image, the Nth frame (N can take any natural number in the range of [1, 9]) of the above-mentioned 10 continuous frames of acquisition images can be determined as the target acquisition image.

[0040] The motion capture tracking data is used to represent the position of the motion capture tracker fixed on the acquisition device in the motion capture coordinate system. In the case of spatial calibration, the position of the acquisition device can be obtained after the motion capture data is acquired. The focal length data refers to the focal length used by the acquisition device when shooting the acquisition image. The acquisition image can be obtained by shooting the screen displaying the calibration marker symbol at the preset position.

[0041] Referring to Figure 1 In the process of returning the acquisition image (camera picture), the motion capture data and the focal length data are also returned synchronously. Specifically, the image acquisition card, the acquisition device, the screen and the motion capture device can be connected to the same synchronous signal generator to realize the same frequency transmission of the acquisition image, the motion capture data and the focal length data. Therefore, in the process of acquiring continuous multiple frames of acquisition images, continuous multiple frames of motion capture data or focal length data are also acquired synchronously.

[0042] According to the meanings of the static delay, the dynamic delay and the zoom delay, it can be known that the above-mentioned three kinds of delays are different link delays.

[0043] Therefore, when the delay calibration is performed, each delay can be calibrated independently. Specifically:

[0044] The dynamic delay is the delay caused by the asynchronization of the return of the motion capture data and the camera picture. The screen can be continuously shot to obtain continuous multiple frames of acquisition images in the case that the focal length of the acquisition device is fixed but the position is constantly changed (that is, the motion capture data continuously changes), and then the calibration of the dynamic delay is performed. Therefore, in the embodiment of the present application, when the dynamic delay calibration is performed, the return data can include the motion capture data, that is, in the process of performing the dynamic delay calibration, the continuous multiple frames of return data of step 202 can include the constantly changing motion capture data (the continuous multiple frames of motion capture data are all different), and correspondingly, in the shooting process, the focal length data of the acquisition device is fixed and unchanged.

[0045] Similarly, the zoom delay can be calibrated by continuously shooting the screen to obtain continuous multiple frames of captured images in the case that the position of the capture device is fixed but the focal length is constantly changing. Therefore, in the embodiments of the present application, the return data can include focal length data when the zoom delay is calibrated, that is, the continuous multiple frames of return data in step 202 can include constantly changing focal length data (the continuous multiple frames of focal length data are all different) when the zoom delay is calibrated, and correspondingly, the motion capture data is fixed and unchanged during shooting.

[0046] In the embodiments of the present application, the specific form of the calibration marker is not limited, and any symbol shape can be set as the calibration marker according to actual conditions.

[0047] Further, considering that when the position of the capture device is constantly changing, the clarity of the captured images may be low, therefore, if a relatively complex pattern is used as the calibration marker, it may cause subsequent calibration marker detection failure or low detection accuracy; on the other hand, considering that during the delay calibration process, there may also be some shape interference in the surrounding environment, therefore, if a too simple pattern is used as the calibration marker, it may interfere with the detection of the calibration marker due to the existence of similar simple shapes in the surrounding environment. Based on the above considerations, in some embodiments of the present application, a combined marker can be introduced in the screen. Specifically: the captured image can be obtained by shooting the screen displaying the combined marker in the preset position; the combined marker is composed of the calibration marker and the auxiliary marker according to the preset positional relationship. For example, see Figure 3 , Figure 3 The combined marker is a schematic diagram. The combined marker has four circles combined according to a certain positional relationship, in actual application, one or more circles can be specified as the calibration marker, and the remaining circles are auxiliary markers.

[0048] Further, in some embodiments, step 202 can include:

[0049] In response to detecting the preset operation, outputting a return data change prompt information;

[0050] Synchronously receiving continuous multiple frames of return data and continuous multiple frames of captured images;

[0051] If the received continuous multiple frames of return data change in the same direction and the change speed belongs to the preset range, selecting one frame from the continuous multiple frames of captured images as a target captured image; wherein the target captured image is the Nth frame in the continuous multiple frames of captured images, the total number of the continuous multiple frames of captured images is M, M and N are natural numbers, and N < M.

[0052] The setting rule of the frame sequence number is that the later the receiving time is, the larger the value of the frame sequence number is, and the frame sequence number of the received first frame of the collected image is the first frame.

[0053] Further, the feedback data includes motion capture data, and the feedback data change prompt information includes: a change in the pose of the collection device prompt information; or the feedback data includes focal length data, and the feedback data change prompt information includes: a change in the focal length prompt information.

[0054] Specifically, in the embodiments of the present application, the specific content of the preset operation is not limited, and can be set according to actual conditions. For example, the preset operation can be clicking the right mouse button, clicking the "start" icon in the display interface, etc.

[0055] When performing dynamic delay calibration, the feedback data includes motion capture data, and correspondingly, the output feedback information change prompt information can be: a change in the pose of the collection device prompt information. For example, when it is detected that the "start" icon is clicked, the display interface outputs the prompt information "please move the camera position". When performing zoom delay calibration, the feedback data includes focal length data, and correspondingly, the output feedback information change prompt information can be: a change in the focal length prompt information. For example, when it is detected that the "start" icon is clicked, the display interface outputs the prompt information "please continuously zoom".

[0056] In addition, after receiving the continuous multiple frames of feedback data, it can be first judged whether the feedback data changes in the same direction and whether the change speed is within a preset range. If so, it indicates that the change in the position of the collection device or the focal length is relatively stable during the shooting process, so that the phenomenon of blurred image caused by too large change speed can be avoided, which is beneficial to the detection of the calibration marker in the subsequent step. The preset range can be the approximate range of the change speed of the feedback data in actual application process. In this way, the image collection condition in the delay calibration process can be as close as possible to the actual application, so that the final delay calibration result is more accurate.

[0057] The target collected image in the embodiments of the present application can be one of the above-mentioned continuous multiple frames of collected images. Further, in order to avoid the problem that the feedback speed of the feedback data is slower than the feedback speed of the collected image, resulting in the inability to find matching target feedback data in the subsequent step, in the embodiments of the present application, a cache mechanism can be used to set a cache queue: in the case of caching continuous multiple frames of collected images, instead of determining the latest one of the multiple frames of collected images as the target collected image, a non-latest one of the collected images (such as the second last frame, or the third last frame, etc.) is taken as the target collected image for subsequent feedback data matching operation.

[0058] Step 204, a marker symbol in the target acquisition image is detected to obtain an actual position of the calibration marker symbol in the target acquisition image.

[0059] The actual position in this step is the position of the calibration marker symbol in the target acquisition image detected from the target acquisition image by marker symbol detection. That is, in the embodiments of the present application, the position of the calibration marker symbol in the target acquisition image obtained by marker symbol detection is referred to as the actual position. Generally, the position of the calibration marker symbol in the target acquisition image is the position in the image coordinate system corresponding to the target acquisition image.

[0060] The actual position in this step can be calculated by the following method: the calibration marker symbol is taken as a detection target, and target detection is performed on the target acquisition image to obtain the actual position of the calibration marker symbol in the target acquisition image. In this step, the target acquisition image can be input into a target detection model, and the calibration marker symbol is detected by the target detection model to output the position information of the calibration marker symbol detection in the target acquisition image, that is, the actual position of the calibration marker symbol in the target acquisition image.

[0061] When the acquisition image is an image obtained by photographing a screen displaying a combined marker symbol, this step can include:

[0062] The target acquisition image is subjected to combined marker symbol detection to obtain the position of the combined marker symbol in the target acquisition image.

[0063] Based on the preset position relationship and the position of the combined marker symbol in the target acquisition image, the actual position of the calibration marker symbol in the target acquisition image is determined.

[0064] Specifically, the preset position relationship can be position information of the calibration marker symbol in the combined marker symbol. In the embodiments of the present application, the target acquisition image can be first input into a target detection model to output the position of the combined marker symbol in the target acquisition image; and finally, the position of the calibration marker symbol in the target acquisition image is determined according to the position information of the calibration marker symbol in the combined marker symbol, that is, the actual position of the calibration marker symbol in the target acquisition image.

[0065] Step 206, a candidate position corresponding to each frame of return data is calculated, and the candidate position is the position of the calibration marker symbol in the target acquisition image calculated based on each frame of return data.

[0066] Specifically, for each frame of the return data, the intrinsic parameters of the acquisition device can be determined based on the focal length data of the acquisition device; and then based on the intrinsic parameters of the acquisition device, the motion capture data, and the position of the calibration marker in the world coordinate system, the position of the calibration marker in the target acquisition image is calculated through projection transformation, which is the candidate position corresponding to the return data.

[0067] The position of the calibration marker in the world coordinate system can be obtained in the following manner: when the spatial calibration has been completed, the position of the screen in the world coordinate system is known, and the position of the calibration marker in the screen is also known, therefore, the position of the calibration marker in the world coordinate system can be obtained through coordinate transformation.

[0068] In step 208, the target return data is determined from the plurality of frames of return data according to the difference between the actual position and the candidate positions, and the delay data is obtained based on the target return data.

[0069] Specifically, when the return data in step 202 includes motion capture data, the delay data in this step includes dynamic delay data; or when the return data in step 202 includes focal length data, the delay data in this step includes zoom delay data.

[0070] In the examples of the present application, the difference values between the actual position and the candidate positions can be calculated first, and then the return data corresponding to the difference values that meet the preset difference condition is determined as the target return data; then based on the difference between the frame number of the target return data and the frame number of the target acquisition image, the delay data is obtained; wherein the preset difference condition can be that the difference value is the smallest, the difference value is less than a preset difference threshold, etc.

[0071] Further, after the target return data is determined, if the number of the target return data is 1, the difference between the frame number of the target return data and the frame number corresponding to the target acquisition image can be determined as the delay data. For example, the determined target return data is the 4th frame in the continuous multiple frames of return data, and the target acquisition image is the 5th frame in the continuous multiple frames of acquisition images, then the delay data is 1 frame.

[0072] Further, in some embodiments, step 208 can further include:

[0073] calculating the difference values between the actual position and the candidate positions;

[0074] determining the continuous two frames of return data corresponding to the smallest two difference values as the target return data;

[0075] based on the target return data, obtaining interpolated return data using an interpolation algorithm, the difference value between the candidate position corresponding to the interpolated return data and the actual position being less than a preset difference threshold.

[0076] According to the difference value of the returned data, the delay data is obtained.

[0077] Specifically, if the number of target returned data is two (two consecutive frames of returned data), interpolation can be performed between the above-mentioned multiple returned data to obtain multiple interpolated returned data, and the difference value between the frame sequence number of the interpolated returned data with the smallest actual position difference value and the frame sequence number corresponding to the target acquisition image is determined as the delay data.

[0078] For example, if the target returned data is the 4th frame and the 5th frame, interpolation can be performed between the 4th frame and the 5th frame to obtain multiple interpolated returned data. Assuming that the frame sequence number of the interpolated returned data with the smallest actual position difference value is 4.5, and the target acquisition image is the 5th frame in the continuous multiple frames of acquisition images, 0.5 frames can be determined as the delay data, where 0.5 = 5 - 4.5.

[0079] Specifically, taking the calculation of dynamic delay as an example: assuming that the calibration marker is Figure 3 The circle located at the center position of the screen in the combined marker shown in the figure, the actual position O r1 of the marker point obtained in steps 204 and 206 respectively, and the candidate position O P1 (i) corresponding to each frame of motion capture data L(i) can be calculated. After that, the difference value between each O P1 (i) and O r1 is calculated, the two consecutive frames of returned data L(n) and L(n+1) corresponding to the smallest difference value are determined as the target returned data, interpolation is performed on the target returned data L(n) and L(n+1) to determine the position point m (1≥m≥0) that makes the difference value between the calculated candidate position and O r1 smaller (for example, smaller than a preset difference threshold), the position point m corresponds to the interpolated returned data L = L(n)*m + L(n+1)*(1-m), and the frame sequence number of the interpolated returned data is: n+m, if the frame sequence number of the target acquisition image is: 0, then the dynamic delay can finally be obtained as n+m. In this example, i is the frame sequence number corresponding to each frame of motion capture data in the order of late to early according to the returned time, for example: L(i), i = 0 represents the motion capture data with the latest returned time in the continuous multiple frames of returned data, and the target acquisition image is the acquisition image with the latest returned time (frame sequence number is 0) in the continuous multiple frames of acquisition images.

[0080] Further, in some embodiments, the returned data is focal length data, and the number of calibration markers is multiple;

[0081] The difference value between the actual position and each candidate position is calculated, including:

[0082] a position difference value between the actual position and the candidate position of the same calibration marker is calculated;

[0083] a distance difference value between the actual distance and the candidate distance between different calibration markers is calculated;

[0084] The position difference value and the distance difference value are fused to obtain a difference value between the actual position and the candidate position.

[0085] The position difference value represents a distance between the actual position and the candidate position of the same calibration marker.

[0086] The actual distance is a distance between actual positions of different calibration markers; the candidate distance is a distance between candidate positions of different calibration markers; and the distance difference value is a difference value between the actual distance and the candidate distance between different calibration markers.

[0087] In the fusion of the position difference value and the distance difference value, the position difference value and the distance difference value can be used as independent variables, a preset function is used for function operation, and an operation result is used as the difference value between the actual position and the candidate position. In the case where the distance difference value is unchanged, the greater the position difference value, the greater the operation result. In the case where the position difference value is unchanged, the greater the distance difference value, the greater the operation result. In the embodiment of the application, the specific form of the preset function is not limited, and can be set according to actual conditions. For example, the preset function can be x+y, or ax+by, where x and y respectively represent the position difference value and the distance difference value, a and b are constants greater than 0 and less than 1, and a+b=1.

[0088] Specifically, in the embodiment of the application, when the dynamic delay calibration is performed, the backhaul data obtained in step 202 is continuous multiple frames of motion capture data, and the number of calibration markers can be one or a single. If the number of calibration markers is set to one, the subsequent calculation process is relatively simple. If the number of calibration markers is set to multiple, more accurate delay data can be obtained. In the embodiment of the application, the specific number of calibration markers is not limited, and can be set according to actual conditions.

[0089] When the zoom delay calibration is performed, the backhaul data obtained in step 202 is continuous multiple frames of focal length data, to prevent the case where a single calibration marker is always in the center of the frame during continuous shooting. The number of calibration markers can be set to multiple, for example, 2 or 3, and the like.

[0090] When there are multiple calibration markers, when calculating the difference between the actual position and the candidate position, the positional difference between the actual position and the candidate position of the same calibration marker, as well as the distance difference between the actual distance and the candidate distance between different calibration markers, can be considered simultaneously. This makes the candidate position corresponding to the determined target return data closer to the actual position, thereby obtaining more accurate delay data.

[0091] For example: Suppose there are two calibration markers: Figure 3 In the combined marker shown, the circles located at the center of the screen and the circle located at the top are used to obtain the actual positions O of the marker points in steps 204 and 206 respectively. r1 O r2 And the candidate positions O corresponding to the focal length data F(i) of each frame. P1 (i), O P2 After (i), each O can be calculated. P1 (i) and O r1 O P2 (i) and O r2 The difference between them is used as the positional difference value; calculate O r1 With O r2 The distance between them is used as the actual distance to calculate O. P1 (i) and O P2 The distance between (i) is used as the candidate distance, and the difference between the actual distance and the candidate distance is calculated as the distance difference value. Then, the above position difference value and the above distance difference value are fused to obtain the difference value between the actual position and the candidate position. The two consecutive frames of returned data F(n) and F(n+1) corresponding to the minimum difference value are determined as the target returned data. The target returned data F(n) and F(n+1) are interpolated to determine the position point m (1≥m≥0) that makes the calculated difference value smaller (e.g., less than the preset difference threshold). The position point m corresponds to the difference returned data F=F(n)*m+F(n+1)*(1-m). Finally, the zoom delay can be obtained as n+m. Where i is the sequence number of each frame of motion capture data after sorting according to the return time from late to early. For example, when i=1, it represents the focal length data of the frame with the latest return time.

[0092] In addition, the delay data calculation method provided in this application embodiment may also include a calibration process for the static delay, specifically:

[0093] Obtain the rendering time of the target image;

[0094] Receive the target image and obtain the reception time of the target image;

[0095] Based on the receiving time and the rendering time, static delay data is obtained.

[0096] Further, for the convenience of calibration, the rendering operation of a set of image groups can be repeatedly performed on the screen, the set of image groups containing a plurality of first preset images and a second preset image; image rendering and image acquisition are performed at the same frequency, the rendering time of the second preset image is recorded, and the receiving time of the returned second preset image is recorded; based on the receiving time and the rendering time, static delay data can be obtained.

[0097] In the embodiments of the present application, the specific image content in the first preset image and the second preset image is not limited and can be set according to actual conditions. For the convenience of operation, the second preset image can be set as an image convenient for detection, for example, the first preset image is set as a pure black image and the second preset image is set as a pure white image, so that based on the pixel value of the received image, it can be determined whether the first preset image or the second preset image is received.

[0098] For example, a set of image groups contains a total of 21 frames of images, of which 20 frames are pure black images (first preset images) and 1 frame is a pure white image (second preset image), then the pure black image and the pure white image can be continuously put into the screen, after every 20 frames of pure black image, a frame of pure white image is put in; the rendering time of the pure white image and the receiving time of the returned pure white image are recorded, so as to obtain the static delay data.

[0099] Referring to Figure 4 , Figure 4 is a delay calibration process diagram corresponding to the first embodiment of the present application. Hereinafter, the first embodiment of the present application will be described with reference to the schematic diagram shown in Figure 4 .

[0100] First, the device is linked to Genlock to achieve frequency synchronization. Specifically, the image acquisition card, the camera, the screen and the motion capture device can be connected to the same synchronous signal generator to realize the same frequency transmission of the acquired image, motion capture data and focal length data.

[0101] Secondly, the camera is fixed, and the static delay calibration is completed. Specifically, the camera can be arranged in front of the screen. Further, to ensure the accuracy of the static delay calibration, the screen area can account for half or more of the entire image in the captured image. The Genlock output frequency is used to continuously render black and white images on the screen, wherein 20 frames of black images are rendered, and then one frame of white image is rendered. The screen image is captured according to the Genlock output frequency, and the captured image is returned. After detecting the clicking operation of the operator on the "start" icon in the display interface, the static delay calibration operation can be started. The rendering time T0 of the white image is recorded, and the receiving time T2 of the returned white image is received. The static delay result is T2-T0.

[0102] Thirdly, the camera is rotated, and the dynamic delay calibration is completed. Specifically, the camera is arranged in front of the screen. After detecting the clicking operation of the operator on the "start" icon in the display interface, the dynamic delay calibration operation can be started. The display interface displays the prompt information "please move the camera position" to prompt the operator to change the pose of the camera (for example, move the camera left and right, or rotate the camera in a certain direction). The continuous returned motion capture data is obtained, and a plurality of continuous frames of returned data with uniform position changes are selected from the continuous returned motion capture data. Meanwhile, the target captured image (for example, the last received captured image) is determined from the continuous plurality of frames of captured images corresponding to the returned data. Steps 204-208 are executed to obtain the dynamic delay data.

[0103] Fourthly, the zoom is continuously changed, and the zoom delay calibration is completed. Specifically, similar to the third step, the camera is arranged in front of the screen. After detecting the clicking operation of the operator on the "start" icon in the display interface, the zoom delay calibration operation can be started. The display interface displays the prompt information "please continuously zoom" to prompt the operator to change the focal length data of the camera. The continuous returned focal length data is obtained, and a plurality of continuous frames of returned data with uniform focal length changes are selected from the continuous returned focal length data. Meanwhile, the target captured image (for example, the last received captured image) is determined from the continuous plurality of frames of captured images corresponding to the returned data. Steps 204-208 are executed to obtain the zoom delay data.

[0104] In the embodiments of the present application, the static delay calibration, the dynamic delay calibration, and the zoom delay calibration are three independent calibration processes. After the first step is executed, the three delay calibration operations can be executed simultaneously, or the calibration sequence can be set according to the actual situation. Figure 4 In the embodiments of the present application, only the static delay calibration, the dynamic delay calibration, and the zoom delay calibration are taken as examples for description, and do not constitute a limitation on the calibration scheme of the embodiments of the present application.

[0105] The delay data calculation scheme provided by the embodiments of the present application, after obtaining the target acquisition image and the continuous multiple frames of return data, detects the actual position of the calibration marker point in the target acquisition image, also calculates the candidate positions of the calibration marker point in the target acquisition image based on each frame of return data, determines the target return data closest to the synchronous return data corresponding to the target acquisition image by comparing the differences between the actual position and each candidate position, and then obtains the delay data based on the above target return data. The embodiments of the present application do not need manual input of delay data, and realize the automatic calibration of delay data. In addition, in the embodiments of the present application, the target return data is determined from the multiple frames of return data based on the differences between the detected actual position of the calibration marker point and the calculated multiple candidate positions, and then the delay data is obtained, rather than being obtained by trial selection, so that the obtained delay data is more accurate.

[0106] Embodiment Two

[0107] Referring to Figure 5 , Figure 5 is a step flow chart of an image fusion method according to Embodiment Two of the present application. Specifically, the image fusion method provided by the present embodiment includes the following steps:

[0108] Step 502, receiving an acquisition image to be fused.

[0109] In a virtual shooting scene, the rendering engine can render a background image to obtain an engine rendered image, and after rendering the engine rendered image to a display screen, the display screen can be shot to obtain an acquisition image to be fused.

[0110] Step 504, receiving continuous multiple frames of return engine rendered images; based on the receiving time information of the acquisition image to be fused and the static delay data, determining an acquisition image to be fused from the continuous multiple frames of engine rendered images.

[0111] Generally, the rendering process of the engine is continuous, that is, continuous multiple frames of background images can be rendered at a certain rendering frequency to obtain continuous multiple frames of engine rendered images, and the continuous multiple frames of engine rendered images are returned to the electronic device performing image fusion.

[0112] After receiving the returned continuous multiple frames of engine rendering images, the engine rendering image corresponding to the to-be-fused acquisition image can be determined from the returned continuous multiple frames of engine rendering images based on the receiving time information of the to-be-fused acquisition image and the static delay data in step 502, as the to-be-fused rendering image. For example, if the receiving time of the to-be-fused acquisition image is t = 9s and the data return frequency is 1 frame / s, the frame number of the to-be-fused acquisition image can be calculated as the 10th frame, and if the static delay data is 3 frames, the to-be-fused rendering image can be determined as the 7th frame, wherein the setting rule of the frame number is that the later the receiving time, the greater the value of the frame number.

[0113] In step 506, receive the returned continuous multiple frames of motion capture data; determine the target motion capture data from the continuous multiple frames of motion capture data buffer based on the receiving time information of the to-be-fused acquisition image and the dynamic delay data.

[0114] After receiving the returned continuous multiple frames of motion capture data, the dynamic delay data corresponding to the to-be-fused acquisition image can be determined from the returned continuous multiple frames of dynamic delay data based on the receiving time information of the to-be-fused acquisition image and the dynamic delay data in step 502, as the target motion capture data. For example, if the receiving time of the to-be-fused acquisition image is t = 9s and the data return frequency is 1 frame / s, the frame number of the to-be-fused acquisition image can be calculated as the 10th frame, and if the dynamic delay data is 2 frames, the target motion capture data can be determined as the 8th frame, wherein the setting rule of the frame number is that the later the receiving time, the greater the value of the frame number.

[0115] In step 508, receive the returned continuous multiple frames of focal length data; determine the target focal length data from the continuous multiple frames of motion capture data based on the receiving time information of the to-be-fused acquisition image and the zoom delay data.

[0116] After receiving the returned continuous multiple frames of focal length data, the focal length data corresponding to the to-be-fused acquisition image can be determined from the returned continuous multiple frames of zoom delay data based on the receiving time information of the to-be-fused acquisition image and the zoom delay data in step 502, as the target focal length data. For example, if the receiving time of the to-be-fused acquisition image is t = 9s and the data return frequency is 1 frame / s, the frame number of the to-be-fused acquisition image can be calculated as the 10th frame, and if the zoom delay data is also 1 frame, the target focal length data can be determined as the 9th frame, wherein the setting rule of the frame number is that the later the receiving time, the greater the value of the frame number.

[0117] The static delay data, the dynamic delay data, and the zoom delay data can be obtained by the method in Embodiment One.

[0118] At step 510, based on the target motion capture data and the target focal length data, image fusion is performed on the to-be-fused rendered image and the to-be-fused captured image to obtain a fused image.

[0119] Specifically, the target motion capture data is the position information of the capture device when the to-be-fused captured image is captured, and the target focal length data is the focal length information of the capture device when the to-be-fused captured image is captured.

[0120] After the target motion capture data, the target focal length data, the to-be-fused rendered image, and the to-be-fused captured image are obtained, the to-be-fused rendered image can be rendered in a perspective view according to the target focal length data and the target motion capture data, and then the image obtained after the adjustment is fused with the to-be-fused captured image to obtain a fused image.

[0121] According to the image fusion scheme provided in the embodiments of the present application, the target backhaul data is determined based on the delay data with high accuracy, so that the determined target backhaul data is more accurate, and then the image fusion is performed based on the more accurate backhaul data, and the visual effect of the fused image obtained finally is also better.

[0122] Embodiment Three

[0123] Figure 6 FIG. 3 is a structural block diagram of a delay data calculation device according to Embodiment Three of the present application. The device includes:

[0124] The acquisition module 602 is configured to acquire a target captured image and continuous multiple frames of backhaul data, wherein the target captured image is an image determined from the continuous multiple frames of captured images backhauled synchronously with the backhaul data, the captured image is obtained by capturing a screen with a calibration marker, and the backhaul data includes motion capture data or focal length data.

[0125] The actual position obtaining module 604 is configured to perform marker detection on the target captured image to obtain an actual position of the calibration marker in the target captured image.

[0126] The candidate position calculation module 606 is configured to calculate a candidate position corresponding to each frame of backhaul data, wherein the candidate position is a position of the calibration marker in the target captured image calculated based on each frame of backhaul data.

[0127] The delay data obtaining module 608 is configured to determine target backhaul data from the multiple frames of backhaul data according to the difference between the actual position and the candidate position, and obtain delay data based on the target backhaul data, wherein the delay data includes dynamic delay data or zoom delay data.

[0128] Optionally, in some embodiments, the delay data obtaining module 608 is specifically configured to:

[0129] calculate the difference value between the actual position and each candidate position;

[0130] determine the two continuous frames of return data corresponding to the smallest two difference values as target return data;

[0131] based on the target return data, obtain interpolation return data by using an interpolation algorithm, the difference value between the candidate position corresponding to the interpolation return data and the actual position being less than a preset difference threshold;

[0132] obtain delay data according to the difference value return data.

[0133] Optionally, in some embodiments, the return data is focal length data, and the number of calibration markers is multiple; when the delay data obtaining module 608 performs the step of calculating the difference value between the actual position and each candidate position, it is specifically used for:

[0134] calculating the position difference value between the actual position and the candidate position of the same calibration marker;

[0135] calculating the distance difference value between the actual distance and the candidate distance between different calibration markers; the actual distance is the distance between the actual positions of different calibration markers; the candidate distance is the distance between the candidate positions of different calibration markers;

[0136] fusing the position difference value and the distance difference value to obtain the difference value between the actual position and the candidate position.

[0137] Optionally, in some embodiments, the captured image is obtained by photographing a screen displaying a combined marker; the combined marker is composed of a calibration marker and an auxiliary marker according to a preset positional relationship;

[0138] The actual position obtaining module 604 is specifically used for:

[0139] detecting the combined marker in the target captured image to obtain the position of the combined marker in the target captured image;

[0140] determining the actual position of the calibration marker in the target captured image based on the preset positional relationship and the position of the combined marker in the target captured image.

[0141] Optionally, in some embodiments, the obtaining module 602 is specifically used for

[0142] in response to detecting a preset operation, outputting return data change prompt information;

[0143] synchronously receiving continuous multiple frames of return data and continuous multiple frames of captured images;

[0144] If the received continuous multiple frames of backhaul data change in the same direction, and the change speed belongs to a preset range, a frame is selected from the continuous multiple frames of captured images as a target captured image; wherein the target captured image is the Nth frame in the continuous multiple frames of captured images, the total number of the continuous multiple frames of captured images is M, M and N are natural numbers, and N < M.

[0145] Optionally, in some embodiments, the backhaul data includes motion capture data, and the backhaul data change prompt information includes: change of the capture device pose prompt information.

[0146] Alternatively,

[0147] The backhaul data includes focal length data, and the backhaul data change prompt information includes: change of the focal length prompt information.

[0148] The delay calibration device of the embodiment is used to implement the corresponding delay calibration method in the foregoing delay calibration method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the functions of each module in the delay calibration device of the embodiment can be implemented by referring to the description of the corresponding part in the foregoing method embodiments, which will not be described here.

[0149] Embodiment Four

[0150] Figure 7 This is a structural block diagram of an image fusion device according to Embodiment Four of the present application. The device includes:

[0151] The first receiving module 702 is configured to receive a captured image to be fused.

[0152] The second receiving module 704 is configured to receive continuous multiple frames of backhaul engine rendered images, and determine a rendered image to be fused from the continuous multiple frames of backhaul engine rendered images based on the receiving time information of the captured image to be fused and the static delay data.

[0153] The third receiving module 706 is configured to receive continuous multiple frames of backhaul motion capture data, and determine target motion capture data from the continuous multiple frames of backhaul motion capture data based on the receiving time information of the captured image to be fused and the dynamic delay data.

[0154] The fourth receiving module 708 is configured to receive continuous multiple frames of backhaul focal length data, and determine target focal length data from the continuous multiple frames of backhaul motion capture data based on the receiving time information of the captured image to be fused and the zoom delay data.

[0155] The fusion module 710 is configured to perform image fusion on the rendered image to be fused and the captured image to be fused based on the target motion capture data and the target focal length data, to obtain a fused image.

[0156] The dynamic delay data and the zoom delay data are obtained by using the method of Embodiment 1.

[0157] The image fusion apparatus of this embodiment is used to implement the corresponding image fusion method in the foregoing image fusion method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the functions of each module in the image fusion apparatus of this embodiment can be implemented by referring to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described here again either.

[0158] Embodiment Five

[0159] Referring to Figure 8 , a structural schematic diagram of an electronic device according to Embodiment Five of the present application is shown, and the specific implementation of the electronic device is not limited in the embodiments of the present application.

[0160] As shown in Figure 8 , the control terminal can include a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0161] Among them:

[0162] The processor 802, the communications interface 804, and the memory 806 complete the communication with each other through the communications bus 808.

[0163] The communications interface 804 is configured to communicate with other electronic devices or servers.

[0164] The processor 802 is configured to execute the program 810, and specifically can execute the related steps in the foregoing delay data calculation method or image fusion method embodiments.

[0165] Specifically, the program 810 can include program code, and the program code includes computer operation instructions.

[0166] The processor 802 can be a CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0167] The memory 806 is configured to store the program 810. The memory 806 can include a high-speed RAM memory, and can further include a non-volatile memory such as at least one disk memory.

[0168] The program 810 can include a plurality of computer instructions, and the program 810 can specifically cause the processor 802 to perform operations corresponding to the methods described in the foregoing method embodiments through the plurality of computer instructions.

[0169] The specific implementation of each step in the program 810 can refer to the corresponding description in the corresponding steps and units in the foregoing method embodiments, and has corresponding beneficial effects, which will not be described here. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the device and the module described above can refer to the corresponding process description in the foregoing method embodiments, which will not be described here.

[0170] The embodiments of the present application further provide a computer storage medium, which stores a computer program. The program is executed by a processor to implement the method described in any of the foregoing method embodiments. The computer storage medium includes but is not limited to a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0171] The embodiments of the present application further provide a computer program product, which includes computer instructions. The computer instructions instruct a computing device to perform operations corresponding to any of the foregoing method embodiments.

[0172] In addition, it needs to be explained that the information related to the user (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the country and region, and provide corresponding operation entrances for the user to choose authorization or refusal.

[0173] It needs to be pointed out that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0174] The method according to the embodiments of the present application can be implemented in hardware, firmware, or software, or a combination thereof, and can be stored in a recording medium such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded by a network from a remote recording medium or non-transitory machine-readable medium originally stored in a local recording medium and then stored in a local recording medium, so that the method described herein can be stored in a recording medium in the form of such software processing using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the method described herein. In addition, when a general-purpose computer accesses code for implementing the method shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the method shown herein.

[0175] Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0176] The above embodiments are only used to illustrate but not limit the embodiments of the present application, and a person of ordinary skill in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, therefore all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A method for calculating delay data, comprising: obtaining a target capture image and continuous multiple frames of return data; the target capture image is determined from continuous multiple frames of capture images returned synchronously with the return data; the capture image is obtained by shooting a screen displaying a calibration marker; and the return data includes motion capture data or focal length data; marker detection is performed on the target capture image to obtain the actual position of the calibration marker in the target capture image; candidate positions corresponding to each frame of return data are calculated, the candidate position being the position of the calibration marker in the target capture image calculated based on each frame of return data; target return data is determined from multiple frames of return data according to the difference between the actual position and the candidate position, and delay data is obtained based on the target return data; the delay data includes dynamic delay data or zoom delay data; wherein determining target return data from multiple frames of return data according to the difference between the actual position and the candidate position, and obtaining delay data based on the target return data, comprises: calculating the difference value between the actual position and the candidate position, and determining the return data corresponding to the difference value that meets the preset difference condition as the target return data; obtaining delay data based on the difference value between the frame number of the target return data and the frame number of the target capture image.

2. The method of claim 1, wherein, determining target return data from multiple frames of return data according to the difference between the actual position and the candidate position, and obtaining delay data based on the target return data, comprises: calculating the difference value between the actual position and each candidate position; determining the continuous two frames of return data corresponding to the smallest two difference values as the target return data; based on the target return data, an interpolation algorithm is used to obtain interpolation return data, the difference value between the candidate position corresponding to the interpolation return data and the actual position being less than a preset difference threshold; obtaining delay data according to the interpolation return data.

3. The method of claim 2, wherein, the return data is focal length data, and the number of calibration markers is multiple; calculating the difference value between the actual position and each candidate position, comprises: calculating the position difference value between the actual position and the candidate position of the same calibration marker; calculating the distance difference value between the actual distance and the candidate distance between different calibration markers; the actual distance is the distance between the actual positions of different calibration markers; the candidate distance is the distance between the candidate positions of different calibration markers; fusing the position difference value and the distance difference value to obtain the difference value between the actual position and the candidate position.

4. The method of claim 1, wherein, the capture image is obtained by shooting a screen displaying a combined marker; the combined marker is composed of a calibration marker and an auxiliary marker according to a preset positional relationship; marker detection is performed on the target capture image to obtain the actual position of the calibration marker in the target capture image, comprising: combined marker detection is performed on the target capture image to obtain the position of the combined marker in the target capture image; Determine an actual position of the calibration marker in the target acquisition image based on the preset position relationship and the position of the combination marker in the target acquisition image.

5. The method of claim 1, wherein, The target acquisition image and the continuous multiple frames of return data are acquired, including: In response to detecting a preset operation, output return data change prompt information; Synchronously receive the continuous multiple frames of return data and the continuous multiple frames of acquisition images; If the received continuous multiple frames of return data change in the same direction and the change speed belongs to a preset range, select one frame from the continuous multiple frames of acquisition images as a target acquisition image; wherein the target acquisition image is the Nth frame in the continuous multiple frames of acquisition images, the total number of the continuous multiple frames of acquisition images is M, M and N are natural numbers, and N 6. The method of claim 5, wherein, The return data includes motion capture data, and the return data change prompt information includes change acquisition device pose prompt information; Or, The return data includes focal length data, and the return data change prompt information includes change focal length prompt information.

7. An image fusion method, wherein, Further comprising: Receive a to-be-fused acquisition image; Receive the continuous multiple frames of engine rendered images returned; Determine a to-be-fused rendered image from the continuous multiple frames of engine rendered images based on the reception time information of the to-be-fused acquisition image and static delay data; Receive the continuous multiple frames of motion capture data returned; Determine target motion capture data from the continuous multiple frames of motion capture data based on the reception time information of the to-be-fused acquisition image and dynamic delay data; Receive the continuous multiple frames of focal length data returned; Determine target focal length data from the continuous multiple frames of motion capture data based on the reception time information of the to-be-fused acquisition image and zoom delay data; Perform image fusion on the to-be-fused rendered image and the to-be-fused acquisition image based on the target motion capture data and the target focal length data to obtain a fused image; The dynamic delay data and the zoom delay data are obtained by the method in any one of claims 1-6.

8. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operation corresponding to the method in any one of claims 1-7.

9. A computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the method in any one of claims 1-7.

10. A computer program product comprising computer instructions instructing a computing device to perform the method in any one of claims 1-7.

Images