Video superimposition method and device, electronic equipment and readable storage medium

Abstract

The application provides a video superimposition method and device, electronic equipment and readable storage medium, and relates to the field of computer application, which comprises the following steps: screening target frame data from all frame data; determining the update area of the current target frame image in the three-dimensional map according to the current position information and the current attitude information; superimposing the current target frame image on the basis of the original image corresponding to the update area, so as to realize video superimposition according to the video stream of all time periods; the target frame data comprises the current target frame image and the position information and the attitude information associated with the current target frame image; the all frame data is determined by processing the video stream of the current time period; and the video stream is the video data captured by the aerial camera carried by the aircraft in real time. The application can realize real-time image updating, more intuitive viewing of the updating process, efficient form and shortest time realization of image updating of all paths in the whole map range, improvement of work efficiency and reduction of use cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer application, and in particular, to a video superimposition method and device, an electronic device, and a readable storage medium. BACKGROUND

[0002] The prior art method for processing real-time video stream is conventional, and cannot superimpose new images in real time in the existing web-based map, that is, cannot superimpose real-time images on the web-based map when the unmanned aerial vehicle is flying, so that the technical effect of video superimposition cannot be achieved.

[0003] Moreover, the prior art cannot continuously complete long-time three-dimensional map updating and rendering, so that the image updating efficiency of the three-dimensional map is low, and the labor cost is increased. SUMMARY

[0004] The present application provides a video superimposition method and device, an electronic device, and a readable storage medium, to solve the technical defect that the prior art cannot update images in real time and dynamically in a three-dimensional map, and to achieve the use of video stream acquired by an unmanned aerial vehicle in a three-dimensional map for real-time image superimposition rendering during the vehicle's movement, thereby improving the image updating efficiency.

[0005] In a first aspect, the present application provides a video superimposition method, comprising:

[0006] screening target frame data from all frame data;

[0007] determining an updating area of a current target frame image in a three-dimensional map according to current position information and current attitude information;

[0008] superimposing the current target frame image on the basis of an original image corresponding to the updating area, to achieve video superimposition according to video stream of all time periods;

[0009] The target frame data includes a current target frame image, and position information and attitude information associated with the current target frame image.

[0010] The all frame data is determined by processing video stream of a current time period.

[0011] The video stream is video data captured in real time by an aerial camera carried by a flying vehicle.

[0012] According to the video superimposition method provided by the present application, the video stream of the current time period is divided, and target frame data is screened from all frame data, comprising:

[0013] dividing the video stream of the current time period to obtain all frame data corresponding to the video stream;

[0014] Traverse all frame data, according to the data format and / or data length of each frame data, determine target frame data.

[0015] According to the video superimposition method provided by the application, the update region of the current target frame image in the three-dimensional map is determined according to the current position information and the current attitude information, and the method comprises the steps that:

[0016] The first coordinate information is determined according to the longitude, latitude and first height in the current position information;

[0017] The second coordinate information is determined according to the longitude, latitude and second height in the current position information;

[0018] The first view matrix is constructed with the first coordinate information to the second coordinate information direction as the first axis and the north direction as the second axis;

[0019] The first view matrix is rotated according to the current attitude information to determine the second view matrix;

[0020] The update region is determined according to the second view matrix and the terrain of the three-dimensional map;

[0021] The second height is less than the first height;

[0022] The current attitude information comprises the yaw angle information, the pitch angle information and the roll angle information.

[0023] According to the video superimposition method provided by the application, the update region is determined according to the second view matrix and the terrain of the three-dimensional map, and the method comprises the steps that:

[0024] Four space straight lines are determined according to the second view matrix and the view cone matrix of the aerial camera;

[0025] The update region is determined according to the intersection of the four space straight lines and the terrain of the three-dimensional map;

[0026] Each of the space straight lines passes through the vertex of the second view matrix and the vertex of the view cone matrix of the aerial camera.

[0027] According to the video superimposition method provided by the application, the video superimposition is realized according to all time period video streams, and the method further comprises the steps that:

[0028] Each video stream corresponding to the flight path of the aircraft is sequentially acquired according to the flight path of the aircraft;

[0029] Each video stream is sequentially processed to determine the original image and the frame image to be superimposed corresponding to each video stream;

[0030] The sequence superimposes the frame image to be superimposed on the basis of each original image.

[0031] According to the video superimposition method provided by the application, before the target frame data is screened from all frame data, the video stream of the current period is divided.

[0032] The video stream of the current period is acquired according to a low-delay link.

[0033] The video stream at least includes real-time collected video information, position information written at intervals and preset time, and attitude information.

[0034] According to the video superimposition method provided by the application, after the target frame data is screened from all frame data, the method further includes:

[0035] The target frame data is checked.

[0036] When the target frame data is in a normal state, the current target frame image, the position information associated with the current target frame image, and the attitude information are determined according to the target frame data.

[0037] The second aspect further provides a video superimposition device, which includes:

[0038] A processing module is configured to screen target frame data from all frame data.

[0039] A determining module is configured to determine an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information.

[0040] A superimposition module is configured to superimpose the current target frame image on the basis of an original image corresponding to the update area, so as to realize video superimposition according to video streams of all periods.

[0041] The third aspect further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor realizes the video superimposition method when executing the program.

[0042] The fourth aspect further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to realize the video superimposition method.

[0043] The application provides a video superimposition method, device, electronic equipment and readable storage medium, by converting a video stream into frame data, taking a frame image corresponding to the frame data containing position information and attitude information as an update image, determining an update position according to the position information and the attitude information, and finally realizing image update of the current video stream at the update position in a webpage three-dimensional map, and in the process of real-time UAV driving path, since the video stream is continuously acquired, different images at different update positions are updated according to the same method, and along the path direction of the UAV, the picture is continuously superimposed in the webpage three-dimensional map, and a dynamic picture update effect is presented, compared with the non-real-time picture rendering in the prior art, the application can realize real-time image update, more intuitively watch the update process, thereby realizing picture update of all paths in the whole map range in an efficient form and in the shortest time, improving work efficiency and reducing use cost. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0045] Figure 1 is one of the flowcharts of the video superimposition method provided by the application;

[0046] Figure 2 is the flowchart of screening out target frame data provided by the application;

[0047] Figure 3 is the flowchart of determining the update area of the current target frame image in the three-dimensional map provided by the application;

[0048] Figure 4 is the flowchart of determining the update area provided by the application;

[0049] Figure 5 is the second flowchart of the video superimposition method provided by the application;

[0050] Figure 6 is the structural schematic diagram of the video superimposition device provided by the application;

[0051] Figure 7 is the structural schematic diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0053] It should be noted that in the description of the embodiments of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover the non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or equipment comprising the element.

[0054] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally a class, and do not limit the number of objects, for example, the first object can be one or more.

[0055] Figure 1 is one of the flow diagrams of the video superimposition method provided by the present application, and the present application discloses a video superimposition method, comprising:

[0056] Filtering target frame data from all frame data;

[0057] According to the current position information and the current attitude information, determining the update area of the current target frame image in the three-dimensional map;

[0058] Superimposing the current target frame image on the basis of the original image corresponding to the update area, to realize video superimposition according to the video stream of all time periods;

[0059] The target frame data comprises a current target frame image and position information and attitude information associated with the current target frame image;

[0060] The all frame data is determined when processing the video stream of the current time period;

[0061] The video stream is video data captured by an aerial camera carried by the aircraft in real time.

[0062] The image updating in the application can be applied to picture construction, rendering, updating and the like at different coordinates in a three-dimensional map on a webpage. The aircraft and the aerial camera are not limited in type and layout position, and the aircraft can include manned aircraft, unmanned aerial vehicle and the like. The aerial camera can include area array optical instrument, linear array optical instrument, spectral instrument or radar instrument and the like.

[0063] Optionally, the unmanned aerial vehicle is provided with a holder with any one of single lens, double lens, four lenses or no lens, and performs the aerial photography task of observing the earth. Taking the single lens holder as an example, the holder is mainly used to generate images, and the effect of oblique photogrammetry can be achieved by adjusting the angle of the holder. Meanwhile, a computer is arranged on the unmanned aerial vehicle, an interface of the computer is connected with the holder, and the interface is used to transmit the video stream collected by the holder in real time.

[0064] In step 101, the video stream of the current period is segmented, and the target frame data is selected from all frame data. The video stream is the data flow used in unit time after the video file is encoded, also known as video code rate, which is the most important part of video picture quality. The video stream is constructed by continuous frame data, and the frame is the unit of data transmission. The more frames per second, the smoother the displayed motion will be. The video stream of the current period is the video stream acquired in a current period. The video stream of the current period is segmented to obtain a plurality of frame data. The video stream at least includes a target frame data. The target frame data at least includes a current target frame image, position information associated with the current target frame image and attitude information. The position information includes longitude, latitude and altitude. The attitude information includes yaw angle information, pitch angle information and roll angle information. The purpose of the application is to obtain the target frame data including position information and attitude information, and determine the target frame image corresponding to the target frame data.

[0065] In step 102, the update area of the current target frame image in the three-dimensional map is determined according to the current position information and the current attitude information. The application can determine the target frame image acquired by the aircraft at the current position with the current attitude, and determine the update area in the three-dimensional map according to the current position information and the current attitude information.

[0066] In step 103, after the update area corresponding to the current target frame image in the three-dimensional map is acquired, the original image corresponding to the update area in the three-dimensional map is determined, and the current target frame image is superimposed on the basis of the original image corresponding to the update area, so as to realize video superposition according to the video stream of all periods.

[0067] Optionally, before screening the target frame data from all frame data, the method further comprises:

[0068] acquiring the video stream of the current period according to a low-delay link;

[0069] The video stream comprises at least real-time collected video information, position information written at intervals and preset time, and attitude information.

[0070] The application can write the position information and the attitude information into the real-time collected video information, and specifically comprises:

[0071] In the 1st-8th byte, the longitude of the aircraft is written, and the data type is double-precision floating point type.

[0072] In the 9th-16th byte, the latitude of the aircraft is written, and the data type is double-precision floating point type.

[0073] In the 17th-24th byte, the altitude of the aircraft is written, and the data type is double-precision floating point type.

[0074] In the 25th-32th byte, the yaw angle of the camera holder is written, and the data type is double-precision floating point type.

[0075] In the 33th-40th byte, the pitch angle of the camera holder is written, and the data type is double-precision floating point type.

[0076] In the 41th-48th byte, the roll angle of the camera holder is written, and the data type is double-precision floating point type.

[0077] In the 49th-52th byte, the check bit is written, and the data type is integer type.

[0078] After writing the position information and the attitude information into a frame of the video stream, the application sends the video stream to a ground webpage end, so that the ground webpage end acquires the video stream of a period according to a low-delay link.

[0079] Optionally, after screening the target frame data from all frame data, the method further comprises:

[0080] verifying the target frame data;

[0081] When the target frame data is verified to be in a normal state, determining a current target frame image and position information and attitude information associated with the current target frame image according to the target frame data.

[0082] The application reads each frame of the video stream in turn according to the playing order, analyzes the current read frame as a target frame, and checks the analyzed current target frame, and the determination result has two kinds: determination success and determination failure. The main purpose of the check is to determine whether the current target frame is normal, so as to avoid receiving a problematic current target frame. There is no fixed way for the check, and the parity check and the like can be commonly used.

[0083] The determination success is a case where the position information and the attitude information of the current target frame meet the preset condition, which indicates that the position information and the attitude information obtained after the current target frame is decoded are valid information, and the current target frame image corresponding to the current target frame is taken as the to-be-stacked image.

[0084] The determination failure is a case where the position information and the attitude information of the current target frame do not meet the preset condition, which indicates that the position information and the attitude information obtained after the current target frame is decoded are invalid information, and the current frame is discarded, and the next frame of the current frame is read in turn to perform the determination process.

[0085] The application provides a video stacking method and device, electronic equipment and readable storage medium. The video stream is converted into frame data, and the frame image corresponding to the frame data containing position information and attitude information is taken as an update image to determine an update position by using the position information and the attitude information, and finally the image update of the current video stream at the update position in the webpage three-dimensional map is realized. In the process of real-time unmanned aerial vehicle driving path, the video stream is continuously acquired, that is, different images are updated at different update positions by using the same method, and along the path direction of the unmanned aerial vehicle, the picture is continuously stacked in the webpage three-dimensional map, and the picture update effect of dynamic picture presentation is presented. Compared with the non-real-time picture rendering in the prior art, the application can realize real-time image update, more intuitively watch the update process, and realize picture update of all paths in the whole map range in an efficient form and in the shortest time, thereby presenting the technical effect of video stacking, improving the work efficiency, and reducing the use cost.

[0086] Figure 2 The application provides a flowchart for screening target frame data from all frame data, and the screening of the target frame data from all frame data comprises the following steps.

[0087] The video stream of the current time period is divided to obtain all frame data corresponding to the video stream.

[0088] All frame data are traversed, and the target frame data are determined according to the data format and / or data length of each frame data.

[0089] In step 1011, the application provides a way for obtaining all frame data corresponding to the video stream, and in other embodiments, each frame of the current period of the video stream can be read in sequence, and the target frame data is determined during the reading process.

[0090] In step 1012, the application can package the position information and the posture information written into the video stream into a fixed format, and then determine the target frame data according to the data format when the video stream is parsed; and in other embodiments, the position information and the posture information written into the video stream can be set as data with a preset length, and then the target frame data is determined according to the data length when the video stream is parsed.

[0091] Figure 3 is a flowchart of the application for determining an update area of a current target frame image in a three-dimensional map, which determines an update area of a current target frame image in a three-dimensional map according to current position information and current posture information, and includes:

[0092] determining first coordinate information according to the longitude, the latitude and the first elevation in the current position information;

[0093] determining second coordinate information according to the longitude, the latitude and the second elevation in the current position information;

[0094] constructing a first view matrix with the first coordinate information to the second coordinate information direction as a first axis and the north direction as a second axis;

[0095] rotating the first view matrix according to the current posture information to determine a second view matrix;

[0096] determining the update area according to the second view matrix and the terrain of the three-dimensional map;

[0097] the second elevation is less than the first elevation;

[0098] the current posture information includes yaw angle information, pitch angle information and roll angle information.

[0099] In step 1021, the current position information at least includes longitude, latitude and first elevation of the aircraft when shooting at the current position, and optionally, the longitude, latitude and first elevation are converted into first coordinate information in the Earth-Centered Earth-Fixed coordinate system.

[0100] In step 1022, the second elevation is less than the first elevation, taking the first elevation as a reference point, taking another elevation point of a preset distance downward as a second elevation, converting the longitude, latitude and second elevation into second coordinate information in the earth-centered earth-fixed coordinate system, and optionally, in order to reduce calculation error, selecting a distance greater than 100 meters as the preset distance.

[0101] In step 1023, a first view matrix is constructed with the first coordinate information to the second coordinate information direction as a first axis and the north direction as a second axis, and a first view matrix is constructed in the earth-centered earth-fixed coordinate system with the first coordinate information to the second coordinate information direction as a Z axis and the north direction as a Y axis.

[0102] In step 1024, the first view matrix is rotated according to the current attitude information to determine a second view matrix, and the first view matrix is rotated according to the yaw angle information, the pitch angle information and the roll angle information to determine a second view matrix.

[0103] In step 1025, those skilled in the art understand that if the unmanned aerial vehicle is in an absolute shooting angle when shooting, there is no yaw, tilt or roll, then the corresponding four constraint lines can be directly determined according to the four edges on the camera view cone side of the unmanned aerial vehicle and the first view matrix, and then the update area is determined according to the constraint lines and the terrain of the three-dimensional map, in order to reduce calculation error, the yaw angle information, the pitch angle information and the roll angle information are used as factors affecting the determination of the update area, the terrain of the three-dimensional map in the shooting range of the unmanned aerial vehicle is determined, and the update area is determined, the four space straight lines are determined according to the second view matrix after rotation and the four edges on the camera view cone side of the unmanned aerial vehicle, and then the update area is determined according to the four space straight lines and the terrain of the three-dimensional map.

[0104] Figure 4 The flowchart for determining the update area provided by the application, and the update area is determined according to the second view matrix and the terrain of the three-dimensional map, which comprises:

[0105] Four space straight lines are determined according to the second view matrix and the view cone matrix of the aerial camera.

[0106] The update area is determined according to the intersection of the four space straight lines and the terrain of the three-dimensional map.

[0107] Each of the space straight lines passes through the vertex of the second view matrix and the vertex of the view cone matrix of the aerial camera.

[0108] In step 10251, four space straight lines are determined according to the second view matrix and the view frustum matrix of the aerial camera, each of the space straight lines passes through the top points of the second view matrix and the top points of the view frustum matrix of the aerial camera, for example, the second view matrix includes a first top-left corner, a first top-right corner, a first bottom-left corner and a first bottom-right corner, and the view frustum matrix of the aerial camera includes a second top-left corner, a second top-right corner, a second bottom-left corner and a second bottom-right corner.

[0109] In such an embodiment, the four space straight lines include a first space straight line, a second space straight line, a third space straight line and a fourth space straight line. The first space straight line passes through the first top-left corner and the second top-left corner, the second space straight line passes through the first top-right corner and the second top-right corner, the third space straight line passes through the first bottom-left corner and the second bottom-left corner, and the fourth space straight line passes through the first bottom-right corner and the second bottom-right corner.

[0110] In step 10252, the four space straight lines extend to the three-dimensional map direction and form four top corners on the three-dimensional map, the four top corners are connected to determine the update region.

[0111] Figure 5 Figure 2 is a flowchart of a video superimposition method according to an embodiment of the present application, and the video superimposition method comprises the following steps:

[0112] According to the flight path of the aircraft, each video stream corresponding to the flight path is sequentially acquired;

[0113] Each video stream is sequentially processed to determine the original image corresponding to each video stream and the frame image to be superimposed;

[0114] The frame image to be superimposed is sequentially superimposed on each original image.

[0115] In step 201, the present application aims to continuously and real-timely acquire video streams, superimpose the images corresponding to the positions in the three-dimensional map for each video stream, and delete the previous images when a frame image is loaded and superimposed. For the continuous superimposition process, especially when the frame data with a short time interval is extracted for image superimposition, the dynamic effect of video superimposition is presented in the three-dimensional map, so that the working process of the unmanned aerial vehicle is more intuitively, accurately and real-timely acquired, and the real-time control and adjustment of the three-dimensional map rendering are realized. In such an embodiment, the present application converts the frame data in the video stream into target frame images, acquires the position information of the unmanned aerial vehicle and the attitude information of the camera gimbal from the low-latency link every 0.04 seconds, and superimposes the target frame images on the corresponding positions of the three-dimensional map calculated by the 3D rendering engine.

[0116] In step 202, each video stream is sequentially processed to determine the original image corresponding to each video stream and the frame image to be superimposed, and the processing of each video stream can refer to steps 101 to 103. Optionally, each video stream is provided with at least one frame image to be superimposed, and one of the frame images to be superimposed corresponds to one of the original images. In other embodiments, each video stream can also be provided with multiple frame images to be superimposed, that is, each frame image to be superimposed is provided with corresponding position information and attitude information, and each of the frame images to be superimposed corresponds to one of the original images.

[0117] In step 203, the frame image to be superimposed is sequentially superimposed on each original image. Those skilled in the art understand that if the time interval of the obtained target frame data is long, and the time interval is a first interval, two frame images to be superimposed may appear to have an original image that is not superimposed when they are mapped on a three-dimensional map. If the time interval of the obtained target frame data is further shortened, and the time interval is a second interval, two frame images to be superimposed may appear to seamlessly connect and superimpose the original image when they are mapped on a three-dimensional map. If the time interval of the obtained target frame data is further shortened, and the time interval is a third interval, two frame images to be superimposed may appear to have the latter image superimposed on the former image according to the superimposition order when they are mapped on a three-dimensional map. If the time interval of the obtained target frame data is further shortened, and the time interval is a fourth interval, two frame images to be superimposed may appear to have the latter image superimposed on the former image according to the superimposition order, and may also present a dynamic video superimposition effect when they are mapped on a three-dimensional map, wherein the fourth interval is less than the third interval, the third interval is less than the second interval, and the second interval is less than the first interval.

[0118] Figure 6 The figure is a structural schematic diagram of a video superimposition device provided by the application. The application provides a video superimposition device, which comprises a processing module 1 for screening target frame data from all frame data. The working principle of the processing module 1 can refer to the aforementioned step 101, and will not be described here.

[0119] The video superimposition device further comprises a determination module 2 for determining an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information. The working principle of the determination module 2 can refer to the aforementioned step 102, and will not be described here.

[0120] The video superimposition apparatus further comprises a superimposition module 3 for superimposing the current target frame image on the basis of the original image corresponding to the update region, so as to realize video superimposition according to the video stream of all time periods, and the working principle of the superimposition module 3 can refer to the foregoing step 103, which will not be described here.

[0121] The application provides a video superimposition method, device, electronic equipment and readable storage medium, by converting a video stream into frame data, and taking a frame image corresponding to the frame data containing position information and attitude information as an update image, determining an update position by the position information and the attitude information, and finally realizing image update of the current video stream at the update position in the webpage three-dimensional map, and in the process of real-time flight path of the unmanned aerial vehicle, since the video stream is continuously acquired, different images at different update positions are updated according to the same method, and along the path direction of the unmanned aerial vehicle, the picture is continuously superimposed in the webpage three-dimensional map, and the picture effect of dynamic picture update is presented, compared with the non-real-time picture rendering in the prior art, the application can realize real-time image update, more intuitively watch the update process, so that the picture update of all paths in the whole map range is realized in an efficient form and in the shortest time, the work efficiency is improved, and the use cost is reduced.

[0122] Figure 7 is a structural schematic diagram of the electronic equipment provided by the application. As shown in Figure 7 the electronic equipment can include a processor 710, a communications interface 720, a memory 730 and a communications bus 740, wherein the processor 710, the communications interface 720 and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can call logical instructions in the memory 730 to execute a video superimposition method, including: screening target frame data from all frame data; determining an update region of a current target frame image in a three-dimensional map according to current position information and current attitude information; superimposing the current target frame image on the basis of an original image corresponding to the update region, so as to realize video superimposition according to a video stream of all time periods; the target frame data includes the current target frame image and position information and attitude information associated with the current target frame image; the all frame data is determined by processing a video stream of a current time period; and the video stream is video data photographed by an aerial camera carried by a flying vehicle in real time.

[0123] In addition, the logic instructions in the memory 730 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0124] In another aspect, the present application also provides a control software for running programs or instructions on a control terminal, which, when executed by the control terminal, performs the video superimposition method described above, which comprises: screening target frame data from all frame data; determining an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information; superimposing the current target frame image on the basis of the original image corresponding to the update area to realize video superimposition according to video streams of all time periods; the target frame data comprises a current target frame image and position information and attitude information associated with the current target frame image; the all frame data is determined by processing video streams of the current time period; and the video streams are video data captured by an aerial camera carried by an aircraft in real time.

[0125] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform a video superimposition method provided by the above-mentioned methods, which comprises: screening target frame data from all frame data; determining an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information; superimposing the current target frame image on the basis of the original image corresponding to the update area to realize video superimposition according to video streams of all time periods; the target frame data comprises a current target frame image and position information and attitude information associated with the current target frame image; the all frame data is determined by processing video streams of the current time period; and the video streams are video data captured by an aerial camera carried by an aircraft in real time.

[0126] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a video superimposition method provided by any of the above methods, the method comprising: screening target frame data from all frame data; determining an update region of a current target frame image in a three-dimensional map according to current position information and current attitude information; superimposing the current target frame image on a basis of an original image corresponding to the update region to implement video superimposition according to a video stream of all time periods; the target frame data comprising the current target frame image and position information and attitude information associated with the current target frame image; the all frame data being determined when processing a video stream of a current time period; and the video stream being video data captured by an aerial camera carried by an aircraft in real time.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0128] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0129] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of video overlaying, characterized by, The method comprises the following steps: screening target frame data from all frame data; determining an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information, comprising: determining first coordinate information according to longitude, latitude and first elevation in the current position information; determining second coordinate information according to longitude, latitude and second elevation in the current position information; constructing a first view matrix with the direction from the first coordinate information to the second coordinate information as a first axis and the direction of true north as a second axis; rotating the first view matrix according to the current attitude information to determine a second view matrix; determining four space straight lines according to the second view matrix and a view cone matrix of an aerial camera; determining the update area according to the intersection of the four space straight lines and the terrain of the three-dimensional map; each of the space straight lines passes through the vertex of the second view matrix and the vertex of the view cone matrix of the aerial camera; the second elevation is less than the first elevation; the current attitude information comprises yaw angle information, pitch angle information and roll angle information; superimposing the current target frame image on the original image corresponding to the update area to realize video superimposition according to video streams of all time periods; the target frame data comprises a current target frame image and position information and attitude information associated with the current target frame image; the all frame data is determined by processing video streams of a current time period; the video streams are video data captured by an aerial camera carried by a flying vehicle in real time.

2. The video overlay method of claim 1, wherein, The method of screening target frame data from all frame data comprises the following steps: segmenting video streams of a current time period to obtain all frame data corresponding to the video streams; traversing all frame data to determine target frame data according to the data format and / or data length of each frame data.

3. The video overlay method of claim 1, wherein, The method of realizing video superimposition according to video streams of all time periods further comprises the following steps: sequentially obtaining each video stream corresponding to the flight path of a flying vehicle according to the flight path of the flying vehicle; sequentially processing each video stream to determine an original image and a frame image to be superimposed corresponding to each video stream; sequentially superimposing the frame image to be superimposed on each original image.

4. The video overlay method of claim 1, wherein, Before segmenting video streams of a current time period and screening target frame data from all frame data, the method comprises the following steps: obtaining video streams of a current time period according to a low-latency link; the video streams at least comprise real-time collected video information, position information and attitude information written at intervals with a preset time interval.

5. The video overlay method of claim 1, wherein, After screening target frame data from all frame data, the method further comprises the following steps: verifying the target frame data; determining a current target frame image and position information and attitude information associated with the current target frame image according to the target frame data when the target frame data is verified to be in a normal state.

6. A video overlay apparatus characterized by comprising: The device for executing the video superimposition method of claim 1 comprises: a processing module for screening target frame data from all frame data; a determination module for determining an update area of a current target frame image in a three-dimensional map according to current position information and current attitude information. The superimposition module is configured to superimpose the current target frame image on the basis of the original image corresponding to the update region, so as to realize video superimposition according to the video stream of all time periods.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the video superimposition method according to any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is configured to implement the video superimposition method according to any one of claims 1 to 5 when executed by the processor.

Images