Virtual shooting system based on multi-angle imaging

Abstract

The invention relates to the technical field of virtual imaging, in particular to a virtual shooting system based on multi-angle imaging. Comprising the following steps: performing preliminary alignment of frame time difference by calculating and collecting average collection time of equipment, generating a preliminary synchronization data set by using a time step interpolation algorithm according to a preliminary alignment result, and verifying a synchronization error in the preliminary synchronization data set by using a reference synchronization signal; and determining a starting point of an initial splicing path through a spatial consistency score, generating a primary virtual shooting result, carrying out path splicing optimization by using a region switching mechanism, generating a second-generation virtual shooting result, then realizing virtual shooting through a splitting and recombining mechanism, and carrying out iterative optimization on the splitting and recombining result; according to the method, a dual calibration mechanism of'inter-frame interpolation + reference synchronization signal verification 'is introduced, so that sub-frame-level synchronization of cross-device images is realized, and the problem of time sequence dislocation caused by device asynchronization and frame rate jitter in multi-angle acquisition is accurately eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual imaging, and in particular to a virtual shooting system based on multi-angle imaging. Background Art

[0002] Existing virtual shooting systems typically use multiple cameras to capture images of the same scene from different perspectives, and then achieve multi-perspective fusion through post-processing image stitching. However, existing technologies face the following technical challenges:

[0003] Synchronization issues are difficult to accurately control: Due to issues such as startup delay differences, transmission delays, and inconsistent acquisition frame rates among multi-camera devices, it is difficult to achieve sub-frame synchronization of image frames relying solely on timestamps or external signals. This can easily lead to problems such as "wrong frames" and "ghost images" after stitching.

[0004] Fixed and rigid image stitching paths: Existing stitching methods are mostly based on preset paths or image feature matching. It is difficult to dynamically adjust the stitching order based on the spatial consistency between actual viewpoints, resulting in an incoherent final image.

[0005] Lack of effective splicing structure optimization mechanism: During the splicing process, there is a lack of mechanisms for path scoring, optimization, and reorganization, making it impossible to compare the quality of different splicing paths and select the best one. As a result, the final result is easily affected by the quality of the initial path. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a virtual shooting system based on multi-angle imaging, comprising:

[0008] Multi-angle imaging data acquisition synchronization module, specifically:

[0009] Use multiple cameras to collect data from different angles of the object, and use the frame synchronization mechanism to synchronize the time steps between different acquisition devices, including:

[0010] The average acquisition time of the acquisition device is calculated to perform preliminary alignment of the frame time difference. Based on the preliminary alignment result, a time step interpolation algorithm is used to generate a preliminary synchronization data set. At the same time, the synchronization error in the preliminary synchronization data set is verified using a reference synchronization signal.

[0011] And the data splicing and fusion module, specifically:

[0012] The data in the multi-angle imaging synchronization dataset is used as the path node, and the splicing and fusion of the data are realized through the analysis of spatial connection relationships, including:

[0013] The starting point of the initial splicing path is determined by spatial consistency scoring, and the first-generation virtual shooting results are generated. The path splicing is optimized using the regional exchange mechanism to generate the second-generation virtual shooting results. Then, virtual shooting is realized through the splitting and reorganization mechanism, and the splitting and reorganization results are iteratively optimized.

[0014] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the time step synchronization between different acquisition devices using the frame synchronization mechanism is specifically as follows:

[0015] Set the frame time difference preliminary alignment judgment threshold ΔT ε , according to the offset ΔT i (t) Compare the result with the judgment threshold to perform preliminary alignment of the frame time difference, specifically:

[0016] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) ≤ΔT ε , it means that the acquisition time of the current device is the synchronous frame;

[0017] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) >ΔT ε , it means that the acquisition time of the current device is the advanced frame;

[0018] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) <-ΔT ε , it means that the acquisition time of the current device is a delayed frame.

[0019] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the method of generating the preliminary synchronization data set using the time step interpolation algorithm is as follows:

[0020] The advanced frame and delayed frame in the preliminary frame time alignment result are used as substitute calibration frames, and the synchronization frame is reconstructed based on the time step interpolation algorithm, specifically:

[0021] Perform pixel-level difference between the previous and next frames of the calibration frame to generate a fitting frame compared to the current time window, and replace the calibration frame with the generated fitting frame to achieve synchronous frame reconstruction of the calibration frame in the current time window. Then,

[0022] Set the replacement calibration frame to the previous frame and the next frame to replace the calibration frame;

[0023] Set interpolation weights based on the extracted previous and next frames;

[0024] Generate a fitting frame compared to the current window based on the calculated interpolation weights;

[0025] The generated fitting frame is used to replace the frame to be calibrated, so as to reconstruct the synchronous frame of all devices in the current window, and the reconstructed data is constructed as the initial synchronous data set.

[0026] As a preferred solution of the virtual shooting system based on multi-angle imaging of the present invention, the synchronization error in the preliminary synchronization data set is verified by using the reference synchronization signal as follows:

[0027] Set the trigger time of the reference synchronization signal to S (t) , using the trigger time of the reference synchronization signal to verify the data in the preliminary synchronization data set, we have,

[0028] If the timestamp T corresponding to the image frame in the preliminary synchronization dataset i ' (t) The trigger time of the set reference synchronization signal satisfies the formula |T i ' (t) -S (t) |≤5ms, it means that the error check of the reconstructed synchronization frame is successful. Otherwise, it means that the error check of the reconstructed synchronization frame fails. The fitting frame is regenerated, and the synchronization frame is reconstructed twice. The error check of the synchronization frame is continued until the error check formula is satisfied. If the error check fails three times in a row, it means that the current data acquisition device is faulty and a new data acquisition device is replaced to collect multi-angle image data.

[0029] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the spatial consistency score is specifically as follows:

[0030] Project the two image data onto the same reference plane, and the area of the overlapping region is the area overlap of the two image data;

[0031] Calculate the average Euclidean distance of the features in the two image data respectively, and calculate the Euclidean distance between the data points at the same position in the same reference plane of the two image data. If the Euclidean distance between the two data points is less than the average Euclidean distance of the features in the two image data, it means that the two data points are feature matching points. After all the data points in the two images are matched, calculate the number of all feature matching points, and the calculation result is the number of feature matching between the two image data;

[0032] According to the determined overlap of the acquisition area and the number of feature matches between the two image data, the spatial consistency score between the two image data is calculated by the weight coefficient, then,

[0033]

[0034] Among them, w1, w2, w3 represent weight coefficients, G ij Indicates the overlap of the determined acquisition area, M ij represents the number of feature matches between the two image data, represents the error correction term, Represents the calculated spatial consistency score used to construct the initial stitching path starting points.

[0035] As a preferred solution of the virtual shooting system based on multi-angle imaging of the present invention, the method of determining the starting point of the initial stitching path by using the spatial consistency score is as follows:

[0036] Calculate image data I i The spatial consistency score between the data collected by other acquisition devices in the multi-angle imaging synchronization data set is calculated, and each calculated spatial consistency score is arranged in order from small to large. The largest spatial consistency score among all the spatial consistency score calculation results is used as the starting point P of the initial splicing path. (0) , and the data are spliced in the order of spatial consistency scores to form the initial generation of virtual shooting results.

[0037] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the path splicing optimization using the region exchange mechanism is specifically as follows:

[0038] For the initial stitching path starting point in the initial virtual shooting result, select a local image block area in the next adjacent view image and a local image block area in the previous adjacent view image, where the selected local image block area is the overlapping area between the two image areas;

[0039] The two selected local image block areas are exchanged and the stitching path is rebuilt. The spatial consistency score before and after the two image areas are calculated. Based on the two spatial consistency scores, it is judged whether the area exchange mechanism has achieved the optimization of the stitching path. Specifically:

[0040] If the spatial consistency score after region exchange is higher than the spatial consistency score before region exchange, it means that the region exchange mechanism can optimize the stitching path. Based on the calculated spatial consistency score after region exchange, the stitching path is reconstructed to form the second-generation virtual shooting result.

[0041] If the spatial consistency score after region exchange is lower than the spatial consistency score before region exchange, it means that the region exchange mechanism cannot optimize the stitching path and maintains the virtual shooting result formed by the initial stitching path.

[0042] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the virtual shooting is realized by the splitting and reassembling mechanism as follows:

[0043] According to the generated second-generation virtual shooting results, the second-generation virtual shooting results are disassembled according to the path nodes in the splicing path in the second-generation virtual shooting results, and then,

[0044] For the image data of the acquisition device corresponding to the starting point in the stitching path, calculate the spatial similarity between the image data of the acquisition device corresponding to the current starting point and the image data of other acquisition devices, and select the image data with the greatest spatial similarity as the path node after the starting point of the stitching path;

[0045] Based on the determined path nodes after the starting point, recalculate the spatial similarity between the image data of the acquisition device corresponding to the path node after the starting point and the image data of other acquisition devices, and select the image data with the largest spatial similarity as the next path node;

[0046] According to the largest image data in the spatial similarity calculation process, data splicing is performed in sequence until all image data are spliced together, which means that the splitting and reassembly of the second-generation virtual shooting results are completed.

[0047] As a preferred solution of the virtual shooting system based on multi-angle imaging of the present invention, the iterative optimization of the splitting and reassembling results is specifically as follows:

[0048] Randomly select two adjacent image data from the initial virtual shooting result, the second-generation virtual shooting result, and the splitting and reassembling result, respectively, and the serial numbers of the two adjacent images selected are consistent in each result;

[0049] Calculate the spatial consistency scores of two adjacent image data selected from the initial virtual shooting results, the second-generation virtual shooting results, and the splitting and recombining results respectively, compare the spatial consistency scores of the three results, and perform iterative optimization based on the comparison results. Then,

[0050] If the spatial consistency scores of the three results increase layer by layer, it means that the splitting and reassembling result is the best iterative result, and the splitting and reassembling result at this time is the virtual shooting result;

[0051] If the spatial consistency scores of the three results do not increase layer by layer, it means that the splitting and reorganization results are not the best iterative results. The result with the highest spatial consistency score is located and re-iterative optimization is performed.

[0052] As a preferred solution of the virtual shooting system based on multi-angle imaging described in the present invention, the re-iterative optimization is specifically as follows:

[0053] If the result with the highest spatial consistency score is the second-generation virtual shooting result, it means that the splitting and reassembling has failed, and the splitting and reassembling process is repeated until the spatial consistency score corresponding to the splitting and reassembling is higher than the second-generation virtual shooting result;

[0054] If the result with the highest spatial consistency score is the initial virtual shooting result, it means that the stitching path optimization of the second-generation virtual shooting result has failed. The path optimization is performed again until the spatial consistency score corresponding to the second-generation virtual shooting result is higher than the spatial consistency score of the initial virtual shooting result and lower than the spatial consistency score corresponding to the splitting and reassembling. At this time, the splitting and reassembling result is the virtual shooting result.

[0055] Beneficial effects of the present invention:

[0056] This invention introduces a dual calibration mechanism of "inter-frame interpolation + reference synchronization signal verification" to achieve sub-frame synchronization of images across devices, accurately eliminating the timing misalignment problem caused by device asynchrony and frame rate jitter in multi-angle acquisition;

[0057] Compared to the existing coarse synchronization method based solely on timestamp alignment, this solution uses a combination of "time step interpolation" to generate fitting frames and reference signal error checking to achieve higher-precision data synchronization and adapt to complex shooting environments.

[0058] The second-generation stitching results are formed through "region exchange + stitching path scoring optimization", breaking through the traditional static stitching mode. The stitching results after each exchange are automatically evaluated based on the spatial consistency score, and have "self-learning" path selection capabilities. At the same time, the introduction of path "scoring" and "iterative optimization" mechanisms makes the stitching effect tend to the optimal in continuous optimization;

[0059] By constructing a global stitching path evolution mechanism of "splitting and reassembling + multi-path comparison + third-order iteration", the image stitching problem is modeled as an iteratively optimized path search problem. It not only optimizes the current path, but also splits and reconstructs the existing stitching path, forming a dynamic evolution chain from "first generation → second generation → reassembly → final iteration". Each iteration is compared based on the spatial consistency scoring function, which helps to select the optimal structure from multiple stitching schemes and avoid the local optimal trap. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0061] Figure 1 The figure is a schematic diagram of the overall method steps of a virtual shooting system based on multi-angle imaging of the present invention. DETAILED DESCRIPTION

[0062] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] Example 1

[0064] Reference Figure 1 , which is a first embodiment of the present invention, provides a virtual shooting system based on multi-angle imaging, including a multi-angle imaging data acquisition and synchronization module, and a data splicing and fusion module;

[0065] Specifically, the multi-angle imaging data acquisition and synchronization module is used to use the acquisition equipment to acquire data of the object to be photographed from multiple angles, and synchronize the acquired data in time steps to ensure the accuracy of the subsequently photographed data; the data stitching and fusion module is used to stitch and fuse the synchronized multi-angle imaging data to complete virtual shooting.

[0066] Furthermore, the multi-angle imaging data acquisition synchronization module uses multiple cameras to collect data from different angles of the object, and uses the frame synchronization mechanism to synchronize the time steps between different acquisition devices. The specific implementation is as follows:

[0067] Assuming that there are N devices for multi-angle data acquisition, then C={C1,C2,...,C N}, where C1 represents the first image data acquisition device, C N represents the Nth image data acquisition device, N represents the total number of image acquisition devices, and C represents the device set consisting of the image data acquisition devices;

[0068] For any device in the device set, each device collects a frame of image in the tth time window, then,

[0069] F i (t) ,i=1,2,3...,N

[0070] T i (t) ,i=1,2,3...,N

[0071] Among them, F i (t) represents the image frame collected by the i-th device in the t-th time window, T i (t) It represents the timestamp corresponding to the image frame captured by the i-th device in the t-th time window, i represents the serial number of the device in the device set, and N represents the total number of image acquisition devices;

[0072] And calculate the average acquisition time of all devices in the tth time window, then we have,

[0073]

[0074] Where i represents the serial number of the device in the device set, N represents the total number of image acquisition devices, and T i (t) Indicates the timestamp corresponding to the image frame collected by the i-th device in the t-th time window, Indicates the average collection time of all devices in the tth time window.

[0075] Based on the calculated average acquisition time, preliminary alignment of frame time differences is performed as follows:

[0076] For any acquisition device C i , calculate the offset of the device's acquisition time compared to the average acquisition time, then we have,

[0077]

[0078] Among them, T i (t) Indicates the timestamp corresponding to the image frame collected by the i-th device in the t-th time window, Indicates the average acquisition time of all devices in the tth time window, ΔT i (t) It represents the offset of the acquisition time of the i-th device in the t-th time window compared to the average acquisition time, which is used for preliminary alignment of the frame time difference. Then,

[0079] Set the frame time difference preliminary alignment judgment threshold ΔT ε , according to the comparison result of the offset and the judgment threshold, the preliminary alignment of the frame time difference is performed, specifically:

[0080] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) ≤ΔT ε , it means that the acquisition time of the current device is the synchronous frame;

[0081] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) >ΔT ε , it means that the acquisition time of the current device is the advanced frame;

[0082] If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) <-ΔT ε , it means that the acquisition time of the current device is a delayed frame.

[0083] Based on the preliminary alignment results of the frame times, the initial synchronization dataset is constructed using the time step interpolation algorithm, specifically:

[0084] The advanced frame and delayed frame in the preliminary frame time alignment result are used as substitute calibration frames, and the synchronization frame is reconstructed based on the time step interpolation algorithm, specifically:

[0085] Perform pixel-level difference between the previous and next frames of the calibration frame to generate a fitting frame compared to the current time window, and replace the calibration frame with the generated fitting frame to achieve synchronous frame reconstruction of the calibration frame in the current time window. Then,

[0086] Set the calibration frame to F i (t) , replace the previous frame F of the calibration frame i (t-1) and the next frame F i (t+1) ;

[0087] According to the extracted front and back frames, we set the interpolation weights,

[0088]

[0089] According to the calculated interpolation weight, a fitting frame is generated compared to the current window, then,

[0090]

[0091] in, represents the average acquisition time of all devices in the tth time window, T i(t+1) Indicates the timestamp corresponding to the image frame collected by the i-th device in the t+1-th time window, T i (t-1) Indicates the timestamp corresponding to the image frame collected by the i-th device in the t-1-th time window, Represents the calculated interpolation weight, F i (t-1) Indicates the previous frame of the frame to be calibrated, F i (t+1) Indicates the next frame after the frame to be calibrated. represents the generated fitting frame;

[0092] The generated fitting frame is used to replace the frame to be calibrated, so as to reconstruct the synchronous frame of all devices in the current window, and the reconstructed data is constructed as the initial synchronous data set.

[0093] For the constructed preliminary synchronization data set, the synchronization error of each frame is checked using the reference synchronization signal, specifically:

[0094] Set the trigger time of the reference synchronization signal to S (t) , using the trigger time of the reference synchronization signal to verify the data in the preliminary synchronization data set, we have,

[0095] If the timestamp T corresponding to the image frame in the preliminary synchronization dataset i ' (t) The trigger time of the set reference synchronization signal satisfies the formula |T i ' (t) -S (t) |≤5ms, it means that the error check of the reconstructed synchronization frame is successful. Otherwise, it means that the error check of the reconstructed synchronization frame fails. The fitting frame is regenerated, and the synchronization frame is reconstructed twice. The error check of the synchronization frame is continued until the error check formula is satisfied. If the error check fails three times in a row, it means that the current data acquisition device is faulty and a new data acquisition device is replaced to collect multi-angle image data.

[0096] It should be noted that the iteration conditions for data synchronization in the current window are jointly limited by the acquisition time offset in the current window and the synchronization frame error check, specifically:

[0097] For the data in the current window, the formula ΔT is satisfied at the same time i (t) ≤ΔT ε and |T i ' (t) -S (t) When |≤5ms, it means that the time step synchronization of the multi-angle imaging data in the current window is completed, and the synchronized data set in the current window is constructed;

[0098] After the time step synchronization of the multi-angle imaging data is completed for the data under all windows, the synchronized data sets under all windows are constructed as a multi-angle imaging synchronized data set.

[0099] Furthermore, the data stitching and fusion module stitches and fuses the constructed multi-angle imaging synchronous dataset through a path optimization mechanism. The path optimization mechanism uses the data in the multi-angle imaging synchronous dataset as path nodes and realizes the stitching and fusion between the data through the analysis of spatial connection relationships, thereby completing the virtual shooting. The specific implementation is as follows:

[0100] For multi-angle imaging synchronous dataset {I1,I2,...,I N}, where I1 represents the data collected by the first acquisition device in the multi-angle imaging synchronous data set, I N Represents the data collected by the Nth acquisition device in the multi-angle imaging synchronous dataset, where N represents the total number of image acquisition devices;

[0101] Randomly select data collected by two acquisition devices from the multi-angle imaging synchronous data set I i and I j , according to the number of feature matches between the two data, the overlap of the acquisition area, and the depth map matching error, the spatial consistency score is calculated, and the initial splicing path starting point is constructed based on the calculated spatial consistency score, specifically:

[0102] The image data collected by the two acquisition devices I i and I j , constructed as the corresponding image stitching pair (I i ,I j );

[0103] And according to the regional overlap in the two image data, the overlap of the acquisition area G is determined ij , and the number of feature matches M between the two image data ij , specifically:

[0104] Project the two image data onto the same reference plane, and the area of the overlapping region is the area overlap of the two image data;

[0105] Calculate the average Euclidean distance of the features in the two image data respectively, and calculate the Euclidean distance between the data points at the same position in the same reference plane of the two image data. If the Euclidean distance between the two data points is less than the average Euclidean distance of the features in the two image data, it means that the two data points are feature matching points. After all the data points in the two images are matched, calculate the number of all feature matching points, and the calculation result is the number of feature matching between the two image data;

[0106] According to the determined overlap of the acquisition area and the number of feature matches between the two image data, the spatial consistency score between the two image data is calculated by the weight coefficient, then,

[0107]

[0108] Among them, w1, w2, and w3 represent weight coefficients, which are set by the implementer according to the actual application scenario. ij Indicates the overlap of the determined acquisition area, M ij Indicates the number of feature matches between the two image data and the error correction term, which is set by the implementer according to the actual application scenario. Represents the calculated spatial consistency score, which is used to construct the starting point of the initial splicing path, specifically:

[0109] Continue to calculate image data I i The spatial consistency score between the data collected by other acquisition devices in the multi-angle imaging synchronization data set is calculated, and each calculated spatial consistency score is arranged in order from small to large. The largest spatial consistency score among all the spatial consistency score calculation results is used as the starting point P of the initial splicing path. (0) , and the data are spliced in the order of spatial consistency scores to form the initial generation of virtual shooting results.

[0110] Based on the generated initial virtual shooting results, the stitching path is optimized using the region exchange mechanism, specifically:

[0111] For the initial stitching path starting point in the initial virtual shooting result, select a local image block area in the next adjacent view image and a local image block area in the previous adjacent view image, where the selected local image block area is the overlapping area between the two image areas;

[0112] The two selected local image block areas are exchanged and the stitching path is rebuilt. The spatial consistency score before and after the two image areas are calculated. Based on the two spatial consistency scores, it is judged whether the area exchange mechanism has achieved the optimization of the stitching path. Specifically:

[0113] If the spatial consistency score after region exchange is higher than the spatial consistency score before region exchange, it means that the region exchange mechanism can optimize the stitching path. Based on the calculated spatial consistency score after region exchange, the stitching path is reconstructed to form the second-generation virtual shooting result.

[0114] If the spatial consistency score after region exchange is lower than the spatial consistency score before region exchange, it means that the region exchange mechanism cannot optimize the stitching path and maintains the virtual shooting result formed by the initial stitching path.

[0115] For the generated second-generation virtual shooting results, virtual shooting is achieved through a splitting and reorganization mechanism, specifically:

[0116] According to the generated second-generation virtual shooting results, the second-generation virtual shooting results are disassembled according to the path nodes in the splicing path in the second-generation virtual shooting results, and then,

[0117] For the image data of the acquisition device corresponding to the starting point in the stitching path, calculate the spatial similarity between the image data of the acquisition device corresponding to the current starting point and the image data of other acquisition devices, and select the image data with the greatest spatial similarity as the path node after the starting point of the stitching path;

[0118] Based on the determined path nodes after the starting point, recalculate the spatial similarity between the image data of the acquisition device corresponding to the path node after the starting point and the image data of other acquisition devices (excluding the image data of the acquisition device corresponding to the starting point), and select the image data with the largest spatial similarity as the next path node;

[0119] According to the largest image data in the spatial similarity calculation process, data splicing is performed in sequence until all image data are spliced together, which means that the splitting and reassembly of the second-generation virtual shooting results are completed.

[0120] It should be noted that the splitting and reorganization of the second-generation virtual shooting results is iteratively optimized through the initial virtual shooting results and the second-generation virtual shooting results, specifically as follows:

[0121] Randomly select two adjacent image data from the initial virtual shooting result, the second-generation virtual shooting result, and the splitting and reassembling result, respectively. The serial numbers of the two adjacent images selected are consistent in each result (if the two adjacent image data selected are the first and second image data in the initial virtual shooting result, then the serial numbers selected in the second-generation virtual shooting result and the serial numbers in the splitting and reassembling result are also the first and second image data);

[0122] Calculate the spatial consistency scores of two adjacent image data selected from the initial virtual shooting results, the second-generation virtual shooting results, and the splitting and recombining results respectively, compare the spatial consistency scores of the three results, and perform iterative optimization based on the comparison results. Then,

[0123] If the spatial consistency scores of the three results increase layer by layer, it means that the splitting and reassembling result is the best iterative result, and the splitting and reassembling result at this time is the virtual shooting result;

[0124] If the spatial consistency scores of the three results do not increase layer by layer, it means that the split and recombination result is not the best iterative result. Locate the result with the highest spatial consistency score and perform re-iterative optimization. Specifically:

[0125] If the result with the highest spatial consistency score is the second-generation virtual shooting result, it means that the splitting and reassembling has failed, and the splitting and reassembling process is repeated until the spatial consistency score corresponding to the splitting and reassembling is higher than the second-generation virtual shooting result;

[0126] If the result with the highest spatial consistency score is the initial virtual shooting result, it means that the stitching path optimization of the second-generation virtual shooting result has failed. The path optimization is performed again until the spatial consistency score corresponding to the second-generation virtual shooting result is higher than the spatial consistency score of the initial virtual shooting result and lower than the spatial consistency score corresponding to the splitting and reassembling. At this time, the splitting and reassembling result is the virtual shooting result.

[0127] Furthermore, if the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0128] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0129] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A virtual shooting system based on multi-angle imaging, characterized by: include, Multi-angle imaging data acquisition synchronization module, specifically: Use multiple cameras to collect data from different angles of the object, and use the frame synchronization mechanism to synchronize the time steps between different acquisition devices, including: The average acquisition time of the acquisition device is calculated to perform preliminary alignment of the frame time difference. Based on the preliminary alignment result, a time step interpolation algorithm is used to generate a preliminary synchronization data set. At the same time, the synchronization error in the preliminary synchronization data set is verified using a reference synchronization signal. And the data splicing and fusion module, specifically: The data in the multi-angle imaging synchronization dataset is used as the path node, and the splicing and fusion of the data are realized through the analysis of spatial connection relationships, including: The starting point of the initial splicing path is determined by spatial consistency scoring, and the first-generation virtual shooting results are generated. The path splicing is optimized using the regional exchange mechanism to generate the second-generation virtual shooting results. Then, virtual shooting is realized through the splitting and reorganization mechanism, and the splitting and reorganization results are iteratively optimized.

2. The virtual shooting system based on multi-angle imaging according to claim 1, characterized in that: The time step synchronization between different acquisition devices using the frame synchronization mechanism is specifically as follows: Set the frame time difference preliminary alignment judgment threshold ΔT ε , according to the offset ΔT i (t) Compare the result with the judgment threshold to perform preliminary alignment of the frame time difference, specifically: If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) ≤ΔT ε , it means that the acquisition time of the current device is the synchronous frame; If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) >ΔT ε , it means that the acquisition time of the current device is the advanced frame; If the calculated offset compared to the average acquisition time and the set frame time difference are compared to the initial judgment threshold, the formula ΔT is satisfied. i (t) <-ΔT ε , it means that the acquisition time of the current device is a delayed frame.

3. The virtual shooting system based on multi-angle imaging according to claim 2, characterized in that: The method of generating a preliminary synchronization data set using the time step interpolation algorithm is as follows: The advanced frame and delayed frame in the preliminary frame time alignment result are used as substitute calibration frames, and the synchronization frame is reconstructed based on the time step interpolation algorithm, specifically: Perform pixel-level difference between the previous and next frames of the calibration frame to generate a fitting frame compared to the current time window, and replace the calibration frame with the generated fitting frame to achieve synchronous frame reconstruction of the calibration frame in the current time window. Then, Set the replacement calibration frame to the previous frame and the next frame to replace the calibration frame; Set interpolation weights based on the extracted previous and next frames; Generate a fitting frame compared to the current window based on the calculated interpolation weights; The generated fitting frame is used to replace the frame to be calibrated, so as to reconstruct the synchronous frame of all devices in the current window, and the reconstructed data is constructed as the initial synchronous data set.

4. The virtual shooting system based on multi-angle imaging according to claim 3, characterized in that: The synchronization error verification of the preliminary synchronization data set using the reference synchronization signal is specifically performed as follows: Set the trigger time of the reference synchronization signal to S (t) , using the trigger time of the reference synchronization signal to verify the data in the preliminary synchronization data set, we have, If the timestamp T corresponding to the image frame in the preliminary synchronization dataset i ' (t) The trigger time of the set reference synchronization signal satisfies the formula |T i ' (t) -S (t) |≤5ms, it means that the error check of the reconstructed synchronization frame is successful. Otherwise, it means that the error check of the reconstructed synchronization frame fails. The fitting frame is regenerated, and the synchronization frame is reconstructed twice. The error check of the synchronization frame is continued until the error check formula is satisfied. If the error check fails three times in a row, it means that the current data acquisition device is faulty and a new data acquisition device is replaced to collect multi-angle image data.

5. The virtual shooting system based on multi-angle imaging according to claim 4, characterized in that: The spatial consistency scores are as follows: Project the two image data onto the same reference plane, and the area of the overlapping region is the area overlap of the two image data; Calculate the average Euclidean distance of the features in the two image data respectively, and calculate the Euclidean distance between the data points at the same position in the same reference plane of the two image data. If the Euclidean distance between the two data points is less than the average Euclidean distance of the features in the two image data, it means that the two data points are feature matching points. After all the data points in the two images are matched, calculate the number of all feature matching points, and the calculation result is the number of feature matching between the two image data; According to the determined overlap of the acquisition area and the number of feature matches between the two image data, the spatial consistency score between the two image data is calculated by the weight coefficient, then, Among them, w1, w2, w3 represent weight coefficients, G ij Indicates the overlap of the determined acquisition area, M ij represents the number of feature matches between the two image data, represents the error correction term, Represents the calculated spatial consistency score used to construct the initial stitching path starting points.

6. The virtual shooting system based on multi-angle imaging according to claim 5, characterized in that: The determination of the initial splicing path starting point by spatial consistency scoring is as follows: Calculate image data I i The spatial consistency score between the data collected by other acquisition devices in the multi-angle imaging synchronization data set is calculated, and each calculated spatial consistency score is arranged in order from small to large. The largest spatial consistency score among all the spatial consistency score calculation results is used as the starting point P of the initial splicing path. (0) , and the data are spliced in the order of spatial consistency scores to form the initial generation of virtual shooting results.

7. The virtual shooting system based on multi-angle imaging according to claim 6, characterized in that: The path splicing optimization using the regional exchange mechanism is specifically as follows: For the initial stitching path starting point in the initial virtual shooting result, select a local image block area in the next adjacent view image and a local image block area in the previous adjacent view image, where the selected local image block area is the overlapping area between the two image areas; The two selected local image block areas are exchanged and the stitching path is rebuilt. The spatial consistency score before and after the two image areas are calculated. Based on the two spatial consistency scores, it is judged whether the area exchange mechanism has achieved the optimization of the stitching path. Specifically: If the spatial consistency score after region exchange is higher than the spatial consistency score before region exchange, it means that the region exchange mechanism can optimize the stitching path. Based on the calculated spatial consistency score after region exchange, the stitching path is reconstructed to form the second-generation virtual shooting result. If the spatial consistency score after region exchange is lower than the spatial consistency score before region exchange, it means that the region exchange mechanism cannot optimize the stitching path and maintains the virtual shooting result formed by the initial stitching path.

8. The virtual shooting system based on multi-angle imaging according to claim 7, characterized in that: The virtual shooting is realized by the splitting and reassembling mechanism as follows: According to the generated second-generation virtual shooting results, the second-generation virtual shooting results are disassembled according to the path nodes in the splicing path in the second-generation virtual shooting results, and then, For the image data of the acquisition device corresponding to the starting point in the stitching path, calculate the spatial similarity between the image data of the acquisition device corresponding to the current starting point and the image data of other acquisition devices, and select the image data with the greatest spatial similarity as the path node after the starting point of the stitching path; Based on the determined path nodes after the starting point, recalculate the spatial similarity between the image data of the acquisition device corresponding to the path node after the starting point and the image data of other acquisition devices, and select the image data with the largest spatial similarity as the next path node; According to the largest image data in the spatial similarity calculation process, data splicing is performed in sequence until all image data are spliced together, which means that the splitting and reassembly of the second-generation virtual shooting results are completed.

9. The virtual shooting system based on multi-angle imaging according to claim 8, characterized in that: The iterative optimization of the splitting and reorganization results is specifically as follows: Randomly select two adjacent image data from the initial virtual shooting result, the second-generation virtual shooting result, and the splitting and reassembling result, respectively, and the serial numbers of the two adjacent images selected are consistent in each result; Calculate the spatial consistency scores of two adjacent image data selected from the initial virtual shooting results, the second-generation virtual shooting results, and the splitting and recombining results respectively, compare the spatial consistency scores of the three results, and perform iterative optimization based on the comparison results. Then, If the spatial consistency scores of the three results increase layer by layer, it means that the splitting and reassembling result is the best iterative result, and the splitting and reassembling result at this time is the virtual shooting result; If the spatial consistency scores of the three results do not increase layer by layer, it means that the splitting and reorganization results are not the best iterative results. The result with the highest spatial consistency score is located and re-iterative optimization is performed.

10. The virtual shooting system based on multi-angle imaging according to claim 9, characterized in that: The re-iterative optimization is specifically as follows: If the result with the highest spatial consistency score is the second-generation virtual shooting result, it means that the splitting and reassembling has failed, and the splitting and reassembling process is repeated until the spatial consistency score corresponding to the splitting and reassembling is higher than the second-generation virtual shooting result; If the result with the highest spatial consistency score is the initial virtual shooting result, it means that the stitching path optimization of the second-generation virtual shooting result has failed. The path optimization is performed again until the spatial consistency score corresponding to the second-generation virtual shooting result is higher than the spatial consistency score of the initial virtual shooting result and lower than the spatial consistency score corresponding to the splitting and reassembling. At this time, the splitting and reassembling result is the virtual shooting result.

Images