Detection method, device, equipment and system of free-view system

Abstract

The application discloses a free-view system detection method, device, equipment and system, relates to the free-view field, and solves the problem of how to accurately detect the deviation of a shooting device in a free-view system in real time. The free-view system comprises N shooting devices, and the N shooting devices synchronously shoot a first scene. After a first shooting device deviates, the first shooting device can preliminarily determine the deviation according to IMU information, and determine a deviation amount according to an image obtained after the first shooting device deviates and an image of other shooting devices in the system that do not deviate, and determine the degree of deviation of the first shooting device according to the deviation amount. Thus, during the process in which the N shooting devices in the free-view system synchronously shoot the first scene, the shooting device that deviates can be accurately detected in real time.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to detection methods, apparatus, devices and systems for free-viewpoint systems. Background Technology

[0002] Free-viewpoint technology enables real-time, interactive video modes, allowing users to independently select a 360° viewing angle on their devices. It is primarily used in program special effects shooting and live broadcasts. Free-viewpoint technology can also be called free-viewpoint technology. In a free-viewpoint system, multiple shooting devices trigger shutters based on control signals, simultaneously capturing images of the same scene from multiple angles, resulting in multi-frame images. A computing device processes these multi-frame images based on parameters from the multiple shooting devices to generate a free-viewpoint video. During shooting, if a shooting device in the free-viewpoint system experiences a collision or vibration, causing a shift in its position and angle, the resulting free-viewpoint video will also exhibit vibration. Typically, the shift detection of shooting devices in a free-viewpoint system is determined by the subjective viewing experience of the final generated video. However, the time span between detection and the occurrence of the shift is significant, and subjective judgment can easily overlook minute shifts. Therefore, how to accurately detect the shift of shooting devices in a free-viewpoint system in real time is a pressing issue. Summary of the Invention

[0003] The detection method, apparatus, equipment, and system for the free-view system provided in this application solve the problem of how to detect the offset of the shooting device in a free-view system in real time and accurately.

[0004] Firstly, a detection method for a free-viewpoint system is provided. The free-viewpoint system includes N imaging devices that simultaneously capture images of a first scene. Each of the N imaging devices includes a first imaging device, which comprises a camera and an Inertial Measurement Unit (IMU). The method is executed by the first imaging device, which is any one of the imaging devices in the free-viewpoint system. The method includes: when the IMU information of the first imaging device changes, acquiring a first image at a first moment through the camera; acquiring M images acquired by the M imaging devices at the first moment from the M imaging devices; and determining an offset based on the first image and the M images. The offset indicates the degree to which the changed extrinsic parameters of the first imaging device at the first moment have shifted relative to the initial camera extrinsic parameters when acquiring the first image. Here, the M imaging devices are devices in the free-viewpoint system other than the first imaging device, the IMU information of the M imaging devices remains unchanged, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1. <N。

[0005] Thus, after the first shooting device shifts, it can initially determine the shift based on IMU information, and determine the amount of shift based on the image acquired after the first shooting device shifts and the images from other shooting devices in the system that have not shifted. Based on the amount of shift, the degree of shift by the first shooting device can be determined. Therefore, during the synchronous shooting of the first scene by N shooting devices in the free-view system, the shooting device that has shifted can be detected accurately and in real time.

[0006] If the differences between the M images and the first image are significant, the accuracy of the offset determined based on the first image and the M images will be low. To improve the accuracy of the offset determination by the first imaging device, the first imaging device can acquire images from a designated imaging device that has not experienced an offset.

[0007] For example, the M shooting devices include devices adjacent to the first shooting device.

[0008] For example, M shooting devices include shooting devices within the preset camera position range of the free-view system.

[0009] In one possible implementation, determining the offset based on the first image and M images includes: the first imaging device determining a reprojection error based on the first image and M images, the reprojection error being used to characterize the coordinate error of feature points within the detection area in the first image and M images; and determining the offset based on the reprojection error.

[0010] In another possible implementation, after determining the offset based on the first image and M images, the method further includes: the first capturing device sending a prompt message to the computing device, thereby promptly informing the computing device that the pose of the first capturing device has shifted, and preventing the computing device from post-processing the image acquired by the first capturing device after the shift based on the camera parameters of the first capturing device, which could lead to image jitter.

[0011] Once the first imaging device is determined to have shifted, a method for recalibrating the first imaging device can be determined based on the amount of shift.

[0012] In another possible implementation, after determining the offset based on the first image and the M images, the method further includes: if the offset is less than a preset offset, indicating that the first shooting device has a small offset, the first shooting device determines its recalibrated extrinsic parameters based on the offset and initial camera extrinsic parameters, and then sends the recalibrated extrinsic parameters to the computing device. The recalibrated extrinsic parameters are used for post-processing of the images acquired by the first shooting device to generate the free-view video of the first scene. Alternatively, the first shooting device determines the projection points in the M images acquired by the M shooting devices at the second time step based on the background point cloud of the first scene; it then determines the recalibrated extrinsic parameters of the first shooting device based on the feature points of the second image acquired by the first shooting device at the second time step, as well as the projection points, and sends the recalibrated extrinsic parameters to the computing device.

[0013] In another possible implementation, after determining the offset based on the first image and M images, the method further includes: if the offset is greater than or equal to a preset offset, it indicates that the first shooting device has a large degree of offset, and the first shooting device triggers the gimbal where the first shooting device is located to adjust the attitude of the first shooting device.

[0014] Optionally, after triggering the gimbal where the first shooting device is located to adjust the attitude of the first shooting device, the method further includes: the first shooting device determining M projection points in M ​​images acquired by the shooting device at the second moment based on the background point cloud of the first scene; determining the recalibrated extrinsic parameters of the first shooting device based on the feature points of the second image acquired by the first shooting device at the second moment and the projection points, and sending the recalibrated extrinsic parameters to the computing device.

[0015] Secondly, a method for generating free-viewpoint video is provided. The free-viewpoint system includes N shooting devices that simultaneously capture images of a first scene. Each of the N shooting devices contains a first shooting device. The method is executed by a computing device and includes: the computing device post-processing the images acquired by the N shooting devices based on their camera parameters to obtain a first free-viewpoint video. The camera parameters include initial camera intrinsic parameters and initial camera extrinsic parameters. If the computing device obtains recalibrated extrinsic parameters of the first shooting device, it post-processes the images acquired by the first shooting device based on the recalibrated extrinsic parameters and the initial camera intrinsic parameters to obtain a second free-viewpoint video. The first and second free-viewpoint videos are then merged to obtain a free-viewpoint video of the first scene. The recalibrated extrinsic parameters are the updated extrinsic parameters of the initial camera extrinsic parameters.

[0016] In a possible implementation, before post-processing the image acquired by the first imaging device according to the rescaled extrinsic parameters and the initial camera intrinsic parameters of the first imaging device, the method further includes: the computing device receives a prompt message sent by the first imaging device and obtains the rescaled extrinsic parameters of the first imaging device. The prompt message is used to indicate that the pose of the first imaging device has shifted. The rescaled extrinsic parameters are used to post-process the image acquired by the first imaging device for generating a free-viewpoint video of the first scene.

[0017] In another possible implementation, obtaining the rescaled extrinsic parameters of the first imaging device includes: receiving the rescaled extrinsic parameters sent by the first imaging device.

[0018] In another possible implementation, obtaining the rescaled extrinsic parameters of the first imaging device includes: the computing device determines projection points in the M images acquired by the M imaging devices according to the background point cloud of the first scene, where the M imaging devices are devices other than the first imaging device in the free-viewpoint system, the IMU information of the M imaging devices has not changed, N is an integer greater than or equal to 3, M is an integer greater than or equal to 1, and M < N; and determines the rescaled extrinsic parameters of the first imaging device according to the feature points of the second image acquired by the first imaging device and the projection points.

[0019] In another possible implementation, before obtaining the rescaled extrinsic parameters of the first imaging device, the method further includes: pausing the post-processing of the images sent by the first imaging device.

[0020] In a third aspect, a detection device is provided, and the device includes modules for executing the detection method of the free-viewpoint system in the first aspect or any possible design of the first aspect.

[0021] In a fourth aspect, a device for generating a free-viewpoint video is provided, and the device includes modules for executing the method for generating a free-viewpoint video in the second aspect or any possible design of the second aspect.

[0022] In a fifth aspect, an imaging device is provided, including: at least one processor, a memory, a camera, and an IMU. The camera is used to capture images, the IMU is used to obtain the IMU information of the imaging device, the memory is used to store computer programs and instructions, and the processor is used to call the computer programs and instructions to assist in executing the detection method of the free-viewpoint system in the first aspect or any possible design of the first aspect together with the camera and the IMU.

[0023] In a sixth aspect, a data processing system is provided, including: at least one processor and a memory. The memory is used to store computer programs and instructions, and the processor is used to call the computer programs and instructions to implement the method for generating a free-viewpoint video in the second aspect or any possible design of the second aspect.

[0024] In a seventh aspect, a free-viewpoint system is provided, the free-viewpoint system comprising N shooting devices, the N shooting devices synchronously shooting a first scene, the N shooting devices including a first shooting device, the first shooting device including a camera and an IMU, and when the IMU information of the first shooting device changes, the first shooting device executes the detection method of the free-viewpoint system in the first aspect or any possible design of the first aspect.

[0025] Eighthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a shooting device of a free-view system, causing the shooting device of the free-view system to perform the operation steps of the method as described in the first aspect or any possible implementation thereof.

[0026] Ninthly, a computer program product is provided that, when run on a computer, causes a shooting device in a free-view system to perform the operational steps of the method described in the first aspect or any possible implementation thereof.

[0027] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0028] Figure 1 A schematic diagram of a free-viewpoint system provided in an embodiment of this application;

[0029] Figure 2 An example of an offset diagram provided for this application;

[0030] Figure 3 A flowchart illustrating a detection method for a free-viewpoint system provided in this application embodiment;

[0031] Figure 4 This application provides a schematic diagram of IMU information changes;

[0032] Figure 5 This application provides a schematic diagram of M imaging devices;

[0033] Figure 6 A schematic diagram of feature point matching provided in this application;

[0034] Figure 7 A flowchart illustrating a detection method for a free-viewpoint system provided in this application embodiment;

[0035] Figure 8 A flowchart illustrating a method for generating free-viewpoint video provided in this application embodiment;

[0036] Figure 9 A schematic diagram of a detection device provided in an embodiment of this application;

[0037] Figure 10 A schematic diagram of an apparatus for generating free-viewpoint video provided in an embodiment of this application;

[0038] Figure 11 A schematic diagram of a shooting device provided in an embodiment of this application;

[0039] Figure 12 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0040] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant technologies is given first.

[0041] Before multiple shooting devices in a free-viewpoint system simultaneously capture images of the same scene, they are pre-installed on tracks, trusses, or pan-tilt units and deployed at the shooting location. These shooting devices can also be referred to as camera positions. A truss is a structure composed of members connected at both ends by hinges. This application does not limit the deployment method or the number of shooting devices. The deployment method can be circular, such as in a ring or ellipse. Understandably, the more shooting devices there are, the richer the images from different angles, and the more viewing angles the user has; conversely, the fewer shooting devices there are, the fewer images from different angles, and the fewer viewing angles the user has.

[0042] A free-viewpoint system may also include a controller and a computing device. The controller controls multiple camera devices to simultaneously capture images of the same scene. The computing device performs post-processing on the images acquired by the multiple camera devices using their camera parameters to obtain free-viewpoint video. Camera parameters include intrinsic and extrinsic camera parameters.

[0043] Understandably, camera extrinsic parameters are the pose information of the camera relative to other reference points in the world coordinate system. The world coordinate system refers to a spatial coordinate system established based on the shooting location, with its origin located at a point on the site. Pose includes position and orientation. Camera extrinsic parameters include rotation and translation matrices. These matrices together describe the transformation relationship between the world coordinate system and the camera coordinate system. The rotation matrix describes the orientation of the coordinate axes in the world coordinate system relative to the coordinate axes in the camera coordinate system. The translation matrix describes the position of the origin in the camera coordinate system.

[0044] Camera intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length and pixel size. Camera intrinsic parameters represent the transformation relationship between the three-dimensional coordinates of an object in the camera coordinate system and the two-dimensional coordinates in the image coordinate system of the captured image.

[0045] A controller can be a standalone physical device or a virtual machine (VM) on a physical device. For example, a VM with controller functionality can be installed on one of multiple shooting devices.

[0046] A computing device can be a server, a cloud device, or an edge device (e.g., a box carrying a chip with processing capabilities). Computing devices possess strong computing power, enabling them to post-process images acquired by multiple cameras to obtain free-viewpoint video and other computational results.

[0047] For example, Figure 1 This is a schematic diagram of a free-viewpoint system provided in an embodiment of this application. The free-viewpoint system 100 includes N shooting devices (e.g., shooting devices 1_1 to 1_n), a controller 120, and a cloud device 130. The controller 120 is connected to the N shooting devices via wired or wireless means. The N shooting devices communicate with the cloud device 130 via wired or wireless means. System maintenance personnel can set the number of shooting devices according to deployment requirements. N is an integer greater than or equal to 3. Figure 1 As shown, N shooting devices can be deployed in a circular arrangement at the shooting location.

[0048] The imaging device can be an integrated module. It includes a camera, an inertial measurement unit (IMU), a communication module, and a processor. For example, the imaging device can be a terminal, such as a mobile phone, tablet, laptop, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, and a camera, etc.

[0049] Before N shooting devices simultaneously capture images of the same scene, all N shooting devices are calibrated to improve the accuracy of simultaneous shooting. Calibration includes coarse calibration and fine calibration. Coarse calibration involves adjusting the position and orientation of each shooting device so that the center point of the image from each device points to the same orbital center. For example, if the shooting devices are mounted on a gimbal, the orientation of the shooting devices is adjusted by the gimbal's servo system. Alternatively, the position and orientation of the shooting devices can be manually adjusted by system maintenance personnel.

[0050] Fine calibration calculates camera parameters, such as camera distortion parameters, intrinsic camera parameters, and extrinsic camera parameters, based on the common-view relationship between various shooting devices. One method is calibration using manually designed calibration objects. These objects commonly take the form of calibration boards and calibration towers. The surfaces of the calibration objects are affixed with characteristic patterns. The calibration objects are deployed at the shooting location as required. After the shooting devices capture images of the calibration objects with specific patterns, the processor uses a calibration algorithm to identify feature points in the patterns, associates the same feature points captured by different shooting devices, and calculates the camera parameters. Another method is calibration without manually designed calibration objects. After the shooting devices capture images of the shooting location, a calibration algorithm identifies feature points in these patterns. The processor associates the same feature points captured by different shooting devices and calculates the camera parameters. After each of the N shooting devices is calibrated, each shooting device sends its calibrated camera parameters to the cloud device 130.

[0051] After receiving the control signal from controller 120, N shooting devices synchronously capture images of the first scene using their cameras. The control signal can also be called a synchronization control signal. The control signal can be a periodic pulse signal (related to the shooting frame rate) or a periodic signal triggered by a command based on a communication synchronization protocol. After acquiring the images, the N shooting devices send an image stream containing the images acquired by the N shooting devices to cloud device 130 via the communication module. For example, the shooting devices send the images to cloud device 130 via network 140. Network 140 can refer to the Internet.

[0052] Cloud device 130 uses the camera parameters of N shooting devices to post-process the images acquired by the N shooting devices to obtain free-viewpoint video. Optionally, when a user needs to watch the free-viewpoint video, the terminal can download the free-viewpoint video from cloud device 130 for viewing, allowing the user to independently select the 360° viewing angle of the free-viewpoint video on the terminal. For example, the terminal downloads the free-viewpoint video from cloud device 130 via network 140.

[0053] During the process of the first shooting device (e.g., shooting device 1_1) acquiring an image, if the first shooting device experiences a collision or shaking, which may cause a shift in position and orientation, the cloud device 130 uses the camera parameters of the shifted first shooting device to perform post-processing on the image acquired by the first shooting device. The resulting processed image will show shaking, making the user perceive a discontinuous video experience.

[0054] For example, such as Figure 2As shown, the first row represents images captured after coarse calibration by three consecutive shooting devices in the free-viewpoint system. Due to limitations in the accuracy of the coarse calibration, the position and angle of the center focus axis, represented by the upward arrow, are not accurately aligned with the image center. The second row represents the results of post-processing the first row of images using the camera parameters obtained from the fine calibration of the three consecutive shooting devices in the free-viewpoint system. The position and angle of the center focus axis, represented by the upward arrow, are accurately aligned with the image center, and their size and direction are consistent. Therefore, the obtained free-viewpoint video has a smooth perspective switching effect around a fixed center. The third row represents the image post-processing using the camera parameters of the offset second shooting device after the second shooting device shifted. The resulting processed image is jittery, producing a discontinuous experience.

[0055] In this embodiment, when the IMU information acquired by the first imaging device (e.g., imaging device 1_1) changes, the first imaging device determines the offset based on the first image acquired by the first imaging device through the camera at a first moment, and the image acquired by the imaging device in the system that has not shifted at the first moment. Furthermore, the first imaging device can also perform recalibration through quantization, that is, determine whether to perform coarse calibration based on the offset, and then perform fine calibration. Thus, the function of real-time and accurate detection of imaging device offset in the free-view system is realized, as well as the function of real-time and automatic recalibration of the imaging device when an offset occurs.

[0056] Next, combined Figures 3 to 12 The detection method of the free-viewpoint system provided in the embodiments of this application is described in detail. Figure 3 This is a flowchart illustrating a detection method for a free-viewpoint system provided in an embodiment of this application. The explanation assumes that the first imaging device has shifted. The first imaging device can be any imaging device in the free-viewpoint system. Figure 3 As shown, the method includes the following steps.

[0057] Step 310: The first imaging device determines whether the IMU information has changed.

[0058] In a free-viewpoint system, when N shooting devices simultaneously capture images of the first scene, each shooting device's IMU continuously acquires IMU information (or IMU signals), and each shooting device's camera continuously captures images. The shooting devices store the IMU information in main memory (using an IMU buffer queue for storing IMU information) and the images in main memory (using an image buffer queue for storing images). The method for maintaining clock synchronization between the IMU information and the images can refer to conventional synchronization techniques and will not be elaborated upon.

[0059] The first imaging device can obtain a continuous period of IMU information from the IMU buffer queue storage space. Based on this continuous period of IMU information, it can determine whether the IMU information has changed, that is, to preliminarily determine whether the first imaging device has experienced an offset. The time unit of this continuous period can be seconds or milliseconds. For example, a continuous period can be 1 second.

[0060] In some embodiments, the first imaging device can determine whether there is a change in IMU information over a continuous period of time. If there is a change in IMU information over a continuous period of time, it indicates that the IMU information has changed, and it is preliminarily determined that the first imaging device has shifted. If there is no change in IMU information over a continuous period of time, it indicates that the IMU information has not changed, and it is preliminarily determined that the first imaging device has not shifted. The first imaging device does not need to be recalibrated, and the process ends.

[0061] IMU information includes, but is not limited to, triaxial accelerometer signals and triaxial gyroscope signals. For example... Figure 4 As shown in (a), α, β, and γ represent the angular velocity values ​​on the three orthogonal axes of the gyroscope, and X, Y, and Z represent the acceleration values ​​on the three axes of the accelerometer. When the first imaging device is stationary, there is no angular velocity value on any rotational axis, but due to noise and random walk of the gyroscope itself, the angular velocity value is a small value near 0. Since the IMU is affected by gravitational acceleration in the direction towards the Earth's center, the actual acceleration values ​​detected on the three axes of the accelerometer are the result of the orthogonal decomposition of gravitational acceleration in these three directions, and are affected by noise and random walk of the accelerometer, fluctuating slightly around a fixed value. However, when the first imaging device shakes or shifts, the detected value on at least one of the six axes will fluctuate significantly (e.g., Figure 4 As shown in (b)), if the change in IMU information exceeds the threshold, it can be preliminarily determined that the first imaging device has shifted. The threshold values ​​for the gyroscope and accelerometer are set as the variances of their respective noise levels. The variances are determined by calibrating the IMU. The specific calibration method can refer to conventional techniques and will not be elaborated here.

[0062] Optionally, the first imaging device can first perform denoising and smoothing processing on the IMU information over a continuous period of time, and then determine whether there are any changes in the IMU information over the continuous period of time after denoising and smoothing. This reduces noise interference and improves the accuracy of determining whether the IMU information has changed.

[0063] Before the N shooting devices in the free-viewpoint system synchronously shoot the first scene, the N shooting devices are pre-mounted on a fixed frame such as a track, a truss or a pan-tilt head. If the fixed frame where the N shooting devices are located is offset, the entire N shooting devices are also offset, and the IMU information of the N shooting devices all changes. However, relative to other shooting devices in the free-viewpoint system, the first shooting device actually does not shift. Therefore, when the IMU information of the first shooting device changes, it indicates that it is preliminarily determined that the first shooting device has shifted. To further determine whether the first shooting device has shifted, steps 320 and 330 are executed.

[0064] Step 320: The first shooting device obtains a first image at a first moment through a camera.

[0065] It can be understood that the first image at the first moment obtained by the first shooting device through the camera is the image obtained after the first shooting device has shifted.

[0066] Step 330: The first shooting device obtains M images obtained by M shooting devices at the first moment.

[0067] The M shooting devices are the devices in the free-viewpoint system other than the first shooting device whose IMU information has not changed. M is an integer greater than or equal to 1, and M < N. It can be understood that the M shooting devices are the shooting devices in the free-viewpoint system that have not shifted. The M images are the images obtained by the M shooting devices at the first moment after the first shooting device has shifted.

[0068] In some embodiments, each of the N shooting devices in the free-viewpoint system can be provided with a flag bit. The flag bit being 0 indicates that the IMU information of the shooting device has not changed, that is, the shooting device has not shifted; the flag bit being 1 indicates that the IMU information of the shooting device has changed, that is, the shooting device has shifted. If the IMU information of the shooting device changes, the flag bit is set to 1; if the IMU information of the shooting device has not changed, the flag bit is set to 0.

[0069] After the IMU information of the first shooting device changes, an image request message is sent to the shooting devices in the free-viewpoint system. The M shooting devices with the flag bit being 0 can send the M images obtained at the first moment to the first shooting device.

[0070] If the difference between the M images and the first image is large, the accuracy of the offset determined based on the first image and the M images is low. To improve the accuracy of the first shooting device in determining the offset, the first shooting device can obtain images from the non-shifted shooting devices among the specified shooting devices.

[0071] For example, the M shooting devices include the devices adjacent to the first shooting device. Figure 5As shown in (a), it is assumed that the IMU information of shooting device 1_1 changes, the IMU information of shooting device 1_2 does not change, and the IMU information of shooting device 1_n does not change. The M shooting devices include, for example, shooting device 1_2 and shooting device 1_n.

[0072] For example, M shooting devices include shooting devices within a preset camera position range within the free-view system. The preset camera position range can refer to a preset angle range centered on the shooting device whose IMU information changes, or a range of other shooting devices. For example... Figure 5 As shown in (b), assume that the preset camera position range indicates that the number of shooting devices is in the range of 4-6. Assume that the IMU information of shooting device 1_1 changes, while the IMU information of each of the shooting devices 1_2, 1_3, 1_n-1 and 1_n remains unchanged. The M shooting devices include, for example, shooting devices 1_2, 1_3, 1_n-1 and 1_n.

[0073] It should be understood that if the first capturing device acquires M images from the M capturing devices that have not shifted at the first moment, it indicates that the first capturing device has shifted, and the first capturing device executes step 340. If all N capturing devices shift, the first capturing device will not acquire the M images from the M capturing devices that have not shifted at the first moment.

[0074] Step 340: The first imaging device determines the offset based on the first image and M images.

[0075] The offset is used to indicate the degree to which the changed extrinsic parameters of the first imaging device have shifted relative to the initial camera extrinsic parameters when acquiring the first image at the first moment. In some embodiments, the first imaging device determines the reprojection error based on the first image and M images, and determines the offset based on the reprojection error. The reprojection error is used to characterize the coordinate error of feature points within the detection region in the first image and the M images. The reprojection error satisfies the following formula (1).

[0076]

[0077] in, This represents the depth value of point j in the camera coordinate system of the first imaging device. This represents the inverse of the intrinsic parameter matrix of the first imaging device. This indicates that point j corresponds to the pixel coordinates in the first image obtained by the first imaging device. This represents the pixel coordinates of point j in M ​​images. This represents the initial camera extrinsic parameters of the first imaging device. ΔT represents the reprojection error, which is the change in IMU information during the time it takes for the first imaging device to go from moving to stationary.

[0078] The offset satisfies the following formula (2).

[0079]

[0080] in, This represents the initial camera extrinsic parameters of the first imaging device in the world coordinate system. This represents the initial camera extrinsic parameters of the imaging device in the world coordinate system, indicating that no offset has occurred. This represents the relative external parameter of the first shooting device compared to the shooting device that has not shifted.

[0081] It should be noted that, based on the epipolar constraint theory, each feature point in the first image corresponds to a line in each of the M images. For example... Figure 6 As shown, during the feature matching process of the first image from the first perspective and the M images from the second perspective, for each feature point in the first shooting device, a non-shifted perspective with a common viewing area is determined in the image acquired by the non-shifted shooting device. All detected feature points are found near the corresponding epipolar line in the M images, and feature matching is performed sequentially with the feature points in the first shooting device, thereby significantly reducing the matching time.

[0082] Thus, after the first capturing device shifts, it can initially determine the shift based on IMU information, and determine the amount of shift based on the image acquired after the first capturing device shifts and the images from other capturing devices in the system that have not shifted. Based on the amount of shift, it can be confirmed that the first capturing device has indeed shifted. Therefore, during the synchronous capturing of the first scene by N capturing devices in the free-view system, the capturing device that has shifted can be detected in real time and accurately.

[0083] After the first capturing device determines the offset based on the first image and M images, the first capturing device can also perform step 350. Step 350: The first capturing device sends a prompt message to the cloud device 130. The prompt message indicates that the posture of the first capturing device has shifted, so that the cloud device 130 can promptly pause and process the images from the shifted capturing device.

[0084] If any camera in the free-view system shifts, regardless of the magnitude of the shift, the user is immediately prompted to perform coarse calibration again. This makes the free-view system very inflexible and affects other camera devices in the system. Therefore, if the shift is small, coarse calibration can be skipped, meaning fine calibration can proceed directly without coarse calibration.

[0085] After the first imaging device determines the offset based on the first image and M images, the method further includes the following steps.

[0086] Step 360: The first shooting device determines whether the offset is greater than or equal to the preset offset.

[0087] If the offset is less than the preset offset, it means that the angle of the first shooting device is small and no coarse calibration is required. Then proceed to steps 370 and 380.

[0088] If the offset is greater than or equal to the preset offset, it means that the first shooting device has a large offset angle. For example, if the first shooting device is offset by 180°, the first shooting device needs to be coarsely calibrated and step 390 should be executed.

[0089] In some embodiments, the first imaging device may determine whether to perform coarse calibration based on either criterion one or criterion two.

[0090] Criterion 1: Calculate the pose of the first imaging device using the offset, and calculate the change in the angle between the first imaging device and all other imaging devices that have not shifted. If the average change in the angle exceeds 20%, then a coarse calibration is performed. The change in the angle between the first imaging device and all other imaging devices that have not shifted satisfies formula (3).

[0091]

[0092] Where Δθ represents the average value of the angle change, Δθ cur,i This represents the change in the attitude angle between the first shooting device and the M shooting devices. This value can be calculated from the offset and the initial camera extrinsic parameters.

[0093] Criterion 2: When Criterion 1 does not determine that re-coarse calibration is required, obtain the three-dimensional points corresponding to all other shooting devices that have not shifted, and project all three-dimensional points using the initial and current position and posture of the first shooting device, and calculate the average parallax in the horizontal and vertical directions of the image. If the average parallax in a certain direction exceeds 10% of the resolution in that direction, re-coarse calibration is required. The average parallax in the horizontal and vertical directions can be calculated by the following formula (4).

[0094]

[0095]

[0096] Step 370: The first shooting device determines the recalibrated extrinsic parameters of the first shooting device based on the offset and the initial camera extrinsic parameters.

[0097] The recalibrated extrinsic parameters are used for post-processing of images acquired by the first shooting device that generates the free-view video of the first scene.

[0098] The recalibrated extrinsic parameters satisfy the following formula (5).

[0099]

[0100] in, This indicates the initial camera extrinsic parameters of the first shooting device. ΔT represents the recalibrated extrinsic parameter of the first shooting device, and ΔT represents the offset.

[0101] Step 380: The first shooting device sends the recalibrated external parameters to the cloud device 130.

[0102] Step 390: The gimbal on which the first shooting device is located adjusts the attitude of the first shooting device.

[0103] Optionally, the first imaging device can alert system maintenance personnel that an offset has occurred, or the system maintenance personnel can adjust the first imaging device manually. However, the camera extrinsic parameters of the manually adjusted first imaging device will still deviate from the initial camera extrinsic parameters.

[0104] Step 3100: Cloud device 130 performs fine calibration on the first shooting device.

[0105] After the gimbal on which the first shooting device is located adjusts the attitude of the first shooting device, the first shooting device can send fine calibration prompt information to the cloud device 130. The fine calibration prompt information is used to instruct the first shooting device to perform fine calibration.

[0106] The cloud device 130 stores background point clouds. The background point clouds are converted into three-dimensional coordinates in camera coordinates using the initial camera intrinsic parameters. Then, the three-dimensional coordinates in the camera memory are converted into two-dimensional coordinates. The projection points in the M images acquired by the M shooting devices at the second time are determined. The feature points of the second image acquired by the first shooting device at the second time are matched with the projection points to determine the recalibrated extrinsic parameters of the first shooting device.

[0107] The recalibrated extrinsic parameters satisfy the following formula (6).

[0108]

[0109] in, P represents the recalibrated external parameters of the first shooting device. j Represents the 3D coordinates of the background point cloud. K represents the two-dimensional coordinates of the first imaging device. cur This indicates the initial camera intrinsic parameters of the first shooting device.

[0110] Therefore, upon detecting a shift in the first imaging device, it can be automatically recalibrated. The calibration process does not affect the normal operation of the remaining, unshifted imaging devices. After recalibration, the recovered first imaging device can be rejoined to the system and operate normally alongside the other unshifted devices.

[0111] In another example, if the computing resources of the imaging device can provide sufficient computing power to determine offset and recalibration, the offset detection method and recalibration method provided in this application embodiment can both be executed by the imaging device. Figure 7 As shown above, Figure 3 The difference lies in the fact that if the offset is less than the preset offset, the first capturing device can also perform step 3100 to determine the recalibrated extrinsic parameters. Alternatively, if the offset is greater than or equal to the preset offset, the first capturing device performs coarse calibration, then performs fine calibration, and sends the recalibrated extrinsic parameters to the cloud device 130. The specific method for coarse calibration can be found in the description of step 390 above, and the specific method for fine calibration can be found in the description of step 3100 above.

[0112] Figure 8 This is a flowchart illustrating a method for generating free-viewpoint video according to an embodiment of this application. Here, we assume that the first shooting device has shifted, and the computing device is a cloud device 130, as an example for explanation. The first shooting device can be any shooting device in the free-viewpoint system. For information on the shift detection of the free-viewpoint system and the first shooting device, please refer to the description in the above embodiments. Figure 8 As shown, the method includes the following steps.

[0113] Step 810: The cloud device 130 performs post-processing on the images acquired by the N shooting devices according to the camera parameters of the N shooting devices to obtain the first free-view video.

[0114] Before the first shooting device shifts, the cloud device 130 performs post-processing on the images acquired by the N shooting devices based on the camera parameters of the N shooting devices.

[0115] In some embodiments, the cloud device 130 uses images from multiple cameras captured simultaneously to perform three-dimensional reconstruction based on a three-dimensional reconstruction method, recovers the geometric representation of a scene over a period of time, such as a triangular mesh or a three-dimensional point cloud, edits the position and orientation of virtual lenses (such as surrounding the subject), and then renders the image of each virtual lens to generate a surround video.

[0116] In other embodiments, the cloud device 130 uses a two-dimensional image synthesis method with a fixed layout of multiple shooting devices. It uses images from the synchronously captured multiple shooting devices to generate continuous image frames through two-dimensional image processing methods, such as frame interpolation, and then synthesizes a smooth surround video.

[0117] If the first capturing device shifts, the cloud device 130 suspends post-processing of the image sent by the first capturing device. Other capturing devices in the system that have not shifted can then post-process the acquired images according to their camera parameters.

[0118] For example, assume that N shooting devices do not shift in the first time period, the first shooting device shifts in the second time period, and the N shooting devices do not shift in the third time period. Cloud device 130 performs post-processing on the images acquired by the N shooting devices in the first time period based on their camera parameters, obtaining a free-viewpoint video for the first time period. Cloud device 130 performs post-processing on the images acquired in the second time period based on the camera parameters of the N-1 shooting devices that did not shift, obtaining a free-viewpoint video for the second time period. Cloud device 130 does not perform post-processing on the images acquired by the first shooting device in the second time period. Cloud device 130 performs post-processing on the images acquired in the third time period based on the camera parameters of the N-1 shooting devices that did not shift, obtaining a free-viewpoint video for the N-1 shooting devices in the third time period. Cloud device 130 performs post-processing on the images acquired in the third time period based on the initial camera intrinsic parameters and recalibrated extrinsic parameters of the first shooting device, obtaining a free-viewpoint video for the first shooting device in the third time period. The first free-viewpoint video includes the free-viewpoint video for the first time period, the free-viewpoint video for the second time period, the free-viewpoint video for the N-1 shooting devices in the third time period, and the free-viewpoint video for the first shooting device in the third time period.

[0119] Understandably, the first free-viewpoint video is obtained by post-processing the image based on the initial camera intrinsic and extrinsic parameters. For example, the first free-viewpoint video includes a first-time-period free-viewpoint video, a second-time-period free-viewpoint video, and a third-time-period free-viewpoint video from N-1 shooting devices.

[0120] Optionally, after the first capturing device determines the offset based on the first image and M images, and before the first capturing device performs fine calibration, the first capturing device does not send images to the cloud device 130.

[0121] Step 820: The cloud device 130 performs post-processing on the image acquired by the first shooting device based on the recalibrated extrinsic parameters and the initial camera intrinsic parameters of the first shooting device to obtain the second free-viewpoint video.

[0122] Understandably, the second free-viewpoint video is obtained by post-processing the image based on the recalibrated extrinsic parameters and the initial camera intrinsic parameters. For example, the second free-viewpoint video includes the third time-segment free-viewpoint video of the first shooting device.

[0123] Step 830: The cloud device 130 merges the first free-view video and the second free-view video to obtain the free-view video of the first scene.

[0124] It should be noted that before the cloud device 130 performs post-processing on the image acquired by the first shooting device based on the recalibrated extrinsic parameters and the initial camera intrinsic parameters of the first shooting device, the cloud device 130 can also receive a prompt message sent by the first shooting device, which is used to indicate that the posture of the first shooting device has shifted.

[0125] The cloud device 130 can acquire recalibrated extrinsic parameters from the first shooting device. These recalibrated extrinsic parameters are used for post-processing of the images acquired by the first shooting device to generate the free-viewpoint video of the first scene. In some embodiments, the cloud device 130 determines projection points in M ​​images acquired by M shooting devices based on the background point cloud of the first scene, and determines the recalibrated extrinsic parameters of the first shooting device based on feature points in a second image acquired by the first shooting device and the projection points. In other embodiments, the cloud device 130 receives the recalibrated extrinsic parameters sent by the first shooting device.

[0126] Thus, when the first shooting device shifts, the cloud device 130 can temporarily remove it from the free-viewpoint system, while the remaining shooting devices continue operating normally. Using information from the non-shifted devices, the first shooting device is automatically recalibrated. Once recalibrated, the first shooting device is rejoined to fully restore the free-viewpoint video effect. Therefore, the cloud device 130 can automatically handle shifted camera positions without affecting the normal operation of other shooting devices, making the free-viewpoint system highly flexible for live streaming applications.

[0127] It is understood that, in order to achieve the functions in the above embodiments, the imaging device includes hardware structures and / or software modules corresponding to each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0128] Figure 9 This is a schematic diagram of the possible detection devices provided for embodiments of this application. These detection devices can be used to implement the functions of the imaging device in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In the embodiments of this application, the detection device can be as follows: Figure 1Any one of the shooting devices 1_1 to 1_n in the free-viewpoint system shown may also be a module (such as a chip) applied to the shooting device. The shooting devices 1_1 to 1_n synchronously shoot the first scene. Each shooting device includes a camera and an IMU. The camera is used to shoot images. The IMU is used to obtain IMU information.

[0129] As Figure 9 shown, the detection device 900 includes an image acquisition module 910, a detection module 920, a re-calibration module 930, and a communication module 940. The detection device 900 can be applied to a shooting device as Figure 1 shown.

[0130] When the IMU information of the first shooting device (such as: shooting device 1_1) changes, the camera is used to obtain the first image at the first moment.

[0131] The image acquisition module 910 is used to obtain M images acquired by M shooting devices at the first moment. The M shooting devices are devices other than the first shooting device in the free-viewpoint system. The IMU information of the M shooting devices has not changed. N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1, and M < N. The image acquisition module 910 is used to execute step 330.

[0132] The detection module 920 is used to determine an offset according to the first image and the M images. The offset is used to indicate the degree to which the external parameters of the first shooting device change relative to the initial camera external parameters when obtaining the first image at the first moment. The detection module 920 is used to execute step 340.

[0133] The re-calibration module 930 is used to determine the re-calibrated external parameters of the first shooting device according to the offset and the initial camera external parameters. The re-calibrated external parameters are used to post-process the images acquired by the first shooting device for generating the free-viewpoint video of the first scene. The re-calibration module 930 is used to execute step 370.

[0134] Optionally, the re-calibration module 930 is used to determine the projection points in the M images acquired by the M shooting devices at the second moment according to the background point cloud of the first scene; and, according to the feature points of the second image acquired by the first shooting device at the second moment and the projection points, determine the re-calibrated external parameters of the first shooting device. The re-calibrated external parameters are used to post-process the images acquired by the first shooting device for generating the free-viewpoint video of the first scene. The re-calibration module 930 is used to execute step 390 and step 3100.

[0135] The communication module 940 is used to send the recalibrated extrinsic parameters to the cloud device 130. The communication module 940 is used to execute step 380.

[0136] Storage module 950 is used to store images, as well as the application programs required to perform iterative training.

[0137] For a more detailed description of the image acquisition module 910, detection module 920, recalibration module 930, and communication module 940 mentioned above, please refer to [the relevant documentation / reference]. Figure 3 or Figure 7 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0138] like Figure 10 As shown, the apparatus 1000 for generating free-viewpoint video includes a post-processing module 1001, a generation module 1002, and a communication module 1003.

[0139] Post-processing module 1001 is used to post-process the images acquired by the N shooting devices according to the camera parameters of the N shooting devices to obtain a first free-viewpoint video. The camera parameters include initial camera intrinsic parameters and initial camera extrinsic parameters.

[0140] The post-processing module 1001 is further configured to, if the computing device obtains the recalibrated extrinsic parameters of the first shooting device, perform post-processing on the image obtained by the first shooting device according to the recalibrated extrinsic parameters of the first shooting device and the initial camera intrinsic parameters to obtain a second free-viewpoint video, wherein the recalibrated extrinsic parameters are the updated extrinsic parameters of the initial camera extrinsic parameters. The post-processing module 1001 is configured to execute steps 810 and 820.

[0141] The generation module 1002 is used to merge the first free-viewpoint video and the second free-viewpoint video to obtain a free-viewpoint video of the first scene. The generation module 1002 is used to execute step 830.

[0142] Optionally, the apparatus 1000 for generating free-viewpoint video may further include a recalibration module 1004. The recalibration module 1004 is used to acquire recalibration extrinsic parameters of the first shooting device, which are used for post-processing the images acquired by the first shooting device that generate the free-viewpoint video of the first scene.

[0143] The communication module 1003 is used to receive a prompt message sent by the first shooting device, the prompt message being used to indicate that the posture of the first shooting device has shifted.

[0144] The communication module 1003 is also used to receive the recalibrated extrinsic parameters sent by the first shooting device.

[0145] Storage module 1006 is used to store free-viewpoint videos and the applications required to perform iterative training.

[0146] For a more detailed description of the post-processing module 1001, generation module 1002, and communication module 1003 mentioned above, please refer to [the relevant documentation]. Figure 3 or Figure 7 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0147] Figure 11 This is a schematic diagram of the structure of a shooting device 1100 provided in this embodiment. As shown in the figure, the shooting device 1100 includes a processor 1110, a bus 1120, a memory 1130, a communication interface 1140, and a camera 1150.

[0148] It should be understood that in this embodiment, the processor 1110 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0149] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the program in this application.

[0150] The communication interface 1140 is used to enable communication between the imaging device 1100 and external devices or components. In this embodiment, the imaging device 1100 is used to implement... Figure 3 or Figure 7 When the shooting device shown functions, the camera 1150 is used to acquire images, and the communication interface 1140 is used to send images, recalibrated external parameters, and indication information.

[0151] Bus 1120 may include a pathway for transmitting information between the aforementioned components (such as processor 1110 and memory 1130). In addition to a data bus, bus 1120 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1120 in the figure.

[0152] As an example, the imaging device 1100 may include multiple processors. A processor may be a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0153] It is worth noting that, Figure 11 Taking the shooting device 1100 as an example, which includes a processor 1110 and a memory 1130, the processor 1110 and the memory 1130 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0154] The memory 1130 can correspond to the storage medium used to store computer instructions and images in the above method embodiments, such as a disk, like a mechanical hard disk or a solid-state hard disk.

[0155] The aforementioned shooting device 1100 can be a general-purpose device or a special-purpose device. For example, the shooting device 1100 can be a mobile phone terminal, tablet computer, laptop computer, VR device, AR device, MR device or ER device, vehicle-mounted shooting device, etc., or it can be an edge device (e.g., a box carrying a chip with processing capabilities).

[0156] It should be understood that the imaging device 1100 according to this embodiment can correspond to the detection device 900 in this embodiment, and can correspond to the device executing the action according to... Figure 3 or Figure 7 The corresponding subject in any of the methods, and the above and other operations and / or functions of each module in the detection device 900 are respectively for the purpose of implementing Figure 3 or Figure 7 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0157] Since the various modules in the apparatus 1000 for generating free-viewpoint video provided in this application can be distributed and deployed on multiple computers in the same or different environments, this application also provides a method such as... Figure 12 The data processing system shown includes multiple computers 1200, each computer 1200 including a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204. The memory 1201, processor 1202, and communication interface 1203 are interconnected via the bus 1204.

[0158] The memory 1201 may be a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 1201 may store computer instructions. When the computer instructions stored in the memory 1201 are executed by the processor 1202, the processor 1202 and the communication interface 1203 are used to execute a portion of the data processing methods of the software system. The memory may also store data sets; for example, a portion of the storage resources in the memory 1201 may be divided into an area for storing images and programs that implement the function of generating free-viewpoint video according to the embodiments of this application.

[0159] Processor 1202 may be a general-purpose CPU, an application-specific integrated circuit (ASIC), a GPU, or any combination thereof. Processor 1202 may include one or more chips. Processor 1202 may include an AI accelerator, such as an NPU.

[0160] The communication interface 1203 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the computer 1200 and other devices or communication networks. For example, the communication interface 1203 can acquire images and recalibrated extrinsic parameters, or send recalibrated extrinsic parameters back to the imaging device.

[0161] Bus 1204 may include a pathway for transmitting information between various components of computer 1200 (e.g., memory 1201, processor 1202, communication interface 1203).

[0162] Each of the aforementioned computers 1200 establishes a communication path through a communication network. Each computer 1200 runs any one or more of the post-processing module 1001 and the generation module 1002. Any computer 1200 can be a computer in a cloud data center (e.g., a server), a computer in an edge data center, or a terminal computing device.

[0163] The functionality of cloud device 130 can be deployed on each computer 1200. For example, GPUs are used to implement the functionality of cloud device 130.

[0164] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a terminal device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0165] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A detection method for a free-viewpoint system, characterized in that, The free-viewpoint system includes N shooting devices, which simultaneously capture images of a first scene. Each of the N shooting devices includes a first shooting device, which comprises a camera and an inertial measurement unit (IMU). The method is executed by the first shooting device and includes: When the IMU information of the first imaging device changes, the first image at the first moment is acquired through the camera; Acquire M images from M imaging devices at the first moment. The M imaging devices are all devices in the free-view system other than the first imaging device. The IMU information of the M imaging devices remains unchanged. N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1. <N; An offset is determined based on the first image and the M images. The offset is used to indicate the degree to which the changing extrinsic parameters of the first imaging device have shifted relative to the initial camera extrinsic parameters when acquiring the first image at the first moment.

2. The method according to claim 1, characterized in that, The M shooting devices include those adjacent to the first shooting device.

3. The method according to claim 1, characterized in that, The M shooting devices include shooting devices within the preset camera position range of the free-view system.

4. The method according to any one of claims 1-3, characterized in that, Determining the offset based on the first image and the M images includes: The reprojection error is determined based on the first image and the M images, and the reprojection error is used to characterize the coordinate error of feature points in the detection area in the first image and the M images; The offset is determined based on the reprojection error.

5. The method according to any one of claims 1-3, characterized in that, After determining the offset based on the first image and the M images, the method further includes: A prompt message is sent to the computing device, indicating that the posture of the first imaging device has shifted.

6. The method according to claim 1, characterized in that, After determining the offset based on the first image and the M images, the method further includes: If the offset is less than a preset offset, the recalibrated extrinsic parameters of the first shooting device are determined based on the offset and the initial camera extrinsic parameters. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

7. The method according to claim 1, characterized in that, After determining the offset based on the first image and the M images, the method further includes: If the offset is less than the preset offset, the projection points in the M images acquired by the M shooting devices at the second moment are determined according to the background point cloud of the first scene. Based on the feature points of the second image at the second moment acquired by the first shooting device and the projection points, the recalibrated extrinsic parameters of the first shooting device are determined. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

8. The method according to any one of claims 1-3, characterized in that, After determining the offset based on the first image and the M images, the method further includes: If the offset is greater than or equal to the preset offset, the gimbal on which the first shooting device is located will adjust the attitude of the first shooting device.

9. The method according to claim 8, characterized in that, After the gimbal on which the first shooting device is located adjusts the attitude of the first shooting device, the method further includes: Based on the background point cloud of the first scene, determine the projection points in the M images acquired by the M shooting devices at the second moment; Based on the feature points of the second image at the second moment acquired by the first shooting device and the projection points, the recalibrated extrinsic parameters of the first shooting device are determined. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

10. The method according to any one of claims 6, 7 or 9, characterized in that, The method further includes: The recalibrated extrinsic parameters are sent to the computing device.

11. A method for generating free-viewpoint video, characterized in that, The free-viewpoint system includes N shooting devices, which simultaneously capture images of a first scene. Each of the N shooting devices includes a first shooting device. The method is executed by a computing device and includes: The images acquired by the N shooting devices are post-processed according to the camera parameters of the N shooting devices to obtain a first free-viewpoint video. The camera parameters include initial camera intrinsic parameters and initial camera extrinsic parameters. If the computing device obtains the recalibrated extrinsic parameters of the first shooting device, it performs post-processing on the image obtained by the first shooting device based on the recalibrated extrinsic parameters of the first shooting device and the initial camera intrinsic parameters to obtain a second free-viewpoint video. The recalibrated extrinsic parameters are the extrinsic parameters updated from the initial camera intrinsic parameters. The first free-view video and the second free-view video are merged to obtain the free-view video of the first scene.

12. The method according to claim 11, characterized in that, Before post-processing the image acquired by the first imaging device based on the recalibrated extrinsic parameters and the initial camera intrinsic parameters of the first imaging device, the method further includes: Receive a prompt message sent by the first shooting device, the prompt message being used to indicate that the posture of the first shooting device has shifted; Obtain the recalibrated extrinsic parameters of the first shooting device. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

13. The method according to claim 12, characterized in that, Obtaining the recalibrated extrinsic parameters of the first imaging device includes: Receive the recalibrated extrinsic parameters sent by the first imaging device; Alternatively, projection points in M ​​images acquired by M shooting devices are determined based on the background point cloud of the first scene. The M shooting devices are those other than the first shooting device in the free-view system. The IMU information of the M shooting devices remains unchanged, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1. <N; The recalibrated extrinsic parameters of the first imaging device are determined based on the feature points of the second image acquired by the first imaging device and the projection points.

14. The method according to claim 12 or 13, characterized in that, Before obtaining the recalibrated extrinsic parameters of the first imaging device, the method further includes: Post-processing of the images sent by the first capturing device is paused.

15. A detection device, characterized in that, The detection device is applied to the shooting equipment in a free-viewpoint system. The free-viewpoint system includes N shooting devices that simultaneously capture images of a first scene. Each of the N shooting devices includes a first shooting device, which comprises a camera and an inertial measurement unit (IMU). The detection device includes: When the IMU information of the first imaging device changes, the camera is used to acquire the first image at the first moment; The image acquisition module is used to acquire M images from M imaging devices at the first moment. The M imaging devices are those other than the first imaging device in the free-view system. The IMU information of the M imaging devices remains unchanged. N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1. <N; The detection module is used to determine the offset based on the first image and the M images. The offset is used to indicate the degree to which the changed extrinsic parameters of the first imaging device have shifted relative to the initial camera extrinsic parameters when acquiring the first image at the first moment.

16. The apparatus according to claim 15, characterized in that, The M shooting devices include those adjacent to the first shooting device.

17. The apparatus according to claim 15, characterized in that, The M shooting devices include shooting devices within the preset camera position range of the free-view system.

18. The apparatus according to any one of claims 15-17, characterized in that, When the detection module determines the offset based on the first image and the M images, it is specifically used for: The reprojection error is determined based on the first image and the M images, and the reprojection error is used to characterize the coordinate error of feature points in the detection area in the first image and the M images; The offset is determined based on the reprojection error.

19. The apparatus according to any one of claims 15-17, characterized in that, The device also includes a communication module; The communication module is used to send a prompt message to the computing device, the prompt message being used to indicate that the posture of the first shooting device has shifted.

20. The apparatus according to claim 15, characterized in that, The device also includes a recalibration module; If the offset is less than a preset offset, the recalibration module is used to determine the recalibration extrinsic parameters of the first shooting device based on the offset and the initial camera extrinsic parameters. The recalibration extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

21. The apparatus according to claim 15, characterized in that, The device also includes a recalibration module; If the offset is less than the preset offset, the recalibration module is used to determine the projection points in the M images acquired by the M shooting devices at the second moment based on the background point cloud of the first scene. Based on the feature points of the second image at the second moment acquired by the first shooting device and the projection points, the recalibrated extrinsic parameters of the first shooting device are determined. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

22. The apparatus according to any one of claims 15-17, characterized in that, The device also includes a triggering module; If the offset is greater than or equal to a preset offset, the triggering module is used to adjust the attitude of the first shooting device by the gimbal where the first shooting device is located.

23. The apparatus according to claim 22, characterized in that, The device also includes a recalibration module; The recalibration module is used to determine the projection points in the M images acquired by the M shooting devices at the second moment based on the background point cloud of the first scene. Furthermore, based on the feature points of the second image at the second moment acquired by the first shooting device and the projection points, the recalibrated extrinsic parameters of the first shooting device are determined. The recalibrated extrinsic parameters are used to post-process the images acquired by the first shooting device that generate the free-view video of the first scene.

24. The apparatus according to any one of claims 20, 21 or 23, characterized in that, The device also includes a communication module; The communication module is used to send the recalibrated extrinsic parameters to the computing device.

25. An apparatus for generating free-viewpoint video, characterized in that, The apparatus for generating free-viewpoint video is applied to a computing device in a free-viewpoint system, the free-viewpoint system comprising N shooting devices, the N shooting devices synchronously shooting a first scene, the N shooting devices including a first shooting device, and the apparatus comprising: The post-processing module is used to post-process the images acquired by the N shooting devices according to the camera parameters of the N shooting devices to obtain a first free-viewpoint video. The camera parameters include initial camera intrinsic parameters and initial camera extrinsic parameters. The post-processing module is further configured to, if the computing device obtains the recalibrated extrinsic parameters of the first shooting device, perform post-processing on the image obtained by the first shooting device according to the recalibrated extrinsic parameters of the first shooting device and the initial camera intrinsic parameters to obtain a second free-viewpoint video, wherein the recalibrated extrinsic parameters are the extrinsic parameters updated from the initial camera intrinsic parameters; The generation module is used to merge the first free-viewpoint video and the second free-viewpoint video to obtain the free-viewpoint video of the first scene.

26. The apparatus according to claim 25, characterized in that, The device also includes a communication module and a recalibration module; The communication module is used to receive a prompt message sent by the first shooting device, the prompt message being used to indicate that the posture of the first shooting device has shifted; The recalibration module is used to obtain the recalibration extrinsic parameters of the first shooting device. The recalibration extrinsic parameters are used to post-process the images obtained by the first shooting device that generate the free-view video of the first scene.

27. The apparatus according to claim 26, characterized in that, When the recalibration module obtains the recalibration extrinsic parameters of the first imaging device, it is specifically used for: Receive the recalibrated extrinsic parameters sent by the first imaging device; Alternatively, projection points in M ​​images acquired by M shooting devices are determined based on the background point cloud of the first scene. The M shooting devices are those other than the first shooting device in the free-view system. The IMU information of the M shooting devices remains unchanged, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 1. <N; The recalibrated extrinsic parameters of the first imaging device are determined based on the feature points of the second image acquired by the first imaging device and the projection points.

28. The apparatus according to claim 26 or 27, characterized in that, The post-processing module is specifically used to pause post-processing of the images sent by the first shooting device.

29. A shooting device, characterized in that, include: At least one processor, memory, camera, and inertial measurement unit (IMU), wherein the camera is used to capture images, the IMU is used to acquire IMU information of the capturing device, the memory is used to store computer programs and instructions, and the processor is used to invoke the computer programs and instructions to assist the camera and the IMU in performing the method as described in any one of claims 1-10.

30. A data processing system, characterized in that, include: At least one processor and a memory, the memory being used to store computer programs and instructions, the processor being used to invoke the computer programs and instructions to perform the method as described in any one of claims 11-14.

31. A free-viewpoint system, characterized in that, The free-viewpoint system includes N shooting devices, which simultaneously capture images of a first scene. The N shooting devices include a first shooting device, which includes a camera and an inertial measurement unit (IMU). When the IMU information of the first shooting device changes, the first shooting device executes the method as described in any one of claims 1-10.

Images