Method, device and medium for constructing three-dimensional video based on target monitoring area

Abstract

The embodiments of the present disclosure disclose a method, device and medium for constructing a three-dimensional video based on a target monitoring area. A specific implementation of the method includes: using an initial transformed perspective image to construct a model, classifying and filling the transformed monitoring area view to obtain multiple view filling images, and outputting a first monitoring area perspective transformed view based on the multiple view filling images; determining the initial transformed perspective image construction model as a trained transformed perspective image construction model; in response to detecting a three-dimensional display description instruction for the target monitoring area, obtaining a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display description instruction; according to the transformed perspective image construction model, performing a three-dimensional conversion process on the two-dimensional target monitoring area monitoring video to obtain a three-dimensional target monitoring area monitoring video. This implementation can improve the generation effect of two-dimensional monitoring video images of different perspectives and improve the stability of the generated three-dimensional images.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the computer field, and in particular to a method, device, and medium for constructing a three-dimensional video based on a target monitoring area. Background Art

[0002] At present, the video display of the monitoring area is usually displayed in two-dimensional video to explain the monitoring area. However, if it is only displayed in two dimensions, it is not possible to fully explain it to the technicians. And the conventional video 3D conversion method may cause the converted 3D video to be distorted or the 3D image to be unstable.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a method, device, electronic device and computer-readable medium for constructing a three-dimensional video based on a target monitoring area to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for constructing a three-dimensional video based on a target monitoring area, the method comprising: obtaining monitoring area image samples, wherein the monitoring area image samples comprise: a first monitoring area view and a second monitoring area view, the first monitoring area view and the second monitoring area view corresponding to different angles; determining an image disparity map between the first monitoring area view and the second monitoring area view; performing image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view; using an initial transformed perspective image to construct a model, performing classification filling processing on the transformed monitoring area view to obtain a plurality of view filling images, and performing classification filling processing on the plurality of view filling images according to the plurality of view filling images. image, output a first monitoring area perspective transformation view; determine the difference value between the above-mentioned first monitoring area perspective transformation view and the above-mentioned second monitoring area view; in response to determining that the above-mentioned difference value is less than or equal to the preset difference value, determine the above-mentioned initial transformation perspective image construction model as the trained transformation perspective image construction model; in response to detecting a three-dimensional display description instruction for the target monitoring area, obtain a two-dimensional target monitoring area monitoring video corresponding to the above-mentioned three-dimensional display description instruction; according to the above-mentioned transformation perspective image construction model, perform three-dimensional conversion processing on the above-mentioned two-dimensional target monitoring area monitoring video to obtain a three-dimensional target monitoring area monitoring video, and display the above-mentioned three-dimensional target monitoring area monitoring video on the associated monitoring display terminal.

[0007] In a second aspect, some embodiments of the present disclosure provide a three-dimensional video construction device based on a target monitoring area, the device comprising: a first acquisition unit, configured to acquire monitoring area image samples, wherein the monitoring area image samples comprise: a first monitoring area view and a second monitoring area view, the first monitoring area view and the second monitoring area view corresponding to different angles; a first determination unit, configured to determine an image disparity map between the first monitoring area view and the second monitoring area view; a transformation unit, configured to perform image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view; a filling unit, configured to construct a model using an initial transformed perspective image, perform classification filling processing on the transformed monitoring area view to obtain multiple view filling images, and fill the images according to the multiple views. , outputting a first monitoring area perspective transformation view; a second determining unit, configured to determine a difference value between the first monitoring area perspective transformation view and the second monitoring area view; a third determining unit, configured to, in response to determining that the difference value is less than or equal to a preset difference value, determine the initial transformation perspective image construction model as a trained transformation perspective image construction model; a second acquiring unit, configured to, in response to detecting a three-dimensional display description instruction for a target monitoring area, acquire a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display description instruction; a conversion unit, configured to perform three-dimensional conversion processing on the two-dimensional target monitoring area monitoring video according to the transformation perspective image construction model to obtain a three-dimensional target monitoring area monitoring video, and display the three-dimensional target monitoring area monitoring video on an associated monitoring display terminal.

[0008] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0009] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0010] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the three-dimensional video construction method based on the target monitoring area of ​​some embodiments of the present disclosure, the generation effect of two-dimensional monitoring video images of different perspectives can be improved. The overall filling effect of the two-dimensional monitoring video image can be guaranteed, and the stability of the three-dimensional image is improved. First, a monitoring area image sample is obtained, wherein the above-mentioned monitoring area image sample includes: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles. Secondly, an image disparity map between the above-mentioned first monitoring area view and the second monitoring area view is determined. According to the above-mentioned image disparity map, the above-mentioned first monitoring area view is subjected to image transformation processing to obtain a transformed monitoring area view. Then, the initial transformed perspective image construction model is adopted to classify and fill the above-mentioned transformed monitoring area view to obtain multiple view filling images, and the first monitoring area perspective transformation view is output according to the multiple view filling images. In this way, the stability of the video can be guaranteed and the jitter of the video content can be reduced. Then, the difference value between the above-mentioned first monitoring area perspective transformation view and the above-mentioned second monitoring area view is determined. In response to determining that the above-mentioned difference value is less than or equal to the preset difference value, the above-mentioned initial transformed perspective image construction model is determined as the transformed perspective image construction model that has been trained. Therefore, the filling effect of the transformed perspective image can be improved by constructing a model of transformed perspective images after training, so as to obtain a more robust transformed perspective image and improve the quality of the transformed perspective image. Finally, in response to detecting a three-dimensional display instruction for the target monitoring area, a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display instruction is obtained. According to the above-mentioned transformed perspective image construction model, the above-mentioned two-dimensional target monitoring area monitoring video is subjected to three-dimensional conversion processing to obtain a three-dimensional target monitoring area monitoring video, and the above-mentioned three-dimensional target monitoring area monitoring video is displayed on the associated monitoring display terminal. Therefore, the generation effect of two-dimensional monitoring video images of different perspectives can be improved, the overall filling effect of the two-dimensional monitoring video image is guaranteed, and the stability of the generated three-dimensional image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flow chart of some embodiments of a method for constructing a three-dimensional video based on a target monitoring area according to the present disclosure;

[0013] Figure 2 It is a schematic diagram of the structure of some embodiments of a three-dimensional video construction device based on a target monitoring area according to the present disclosure;

[0014] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0017] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0019] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0020] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] Figure 1 1 is a flowchart of some embodiments of the method for constructing a three-dimensional video based on a target monitoring area according to the present disclosure. 100 of some embodiments of the method for constructing a three-dimensional video based on a target monitoring area according to the present disclosure is shown. The method for constructing a three-dimensional video based on a target monitoring area includes the following steps:

[0022] Step 101, obtaining a monitoring area image sample.

[0023] In some embodiments, an execution subject (e.g., a computing device) of a method for constructing a three-dimensional video based on a target monitoring area may obtain monitoring area image samples. The monitoring area image samples include: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles. The first monitoring area view and the second monitoring area view correspond to different viewing angles. The viewing angle may be set according to actual needs, such as a left viewing angle and a right viewing angle. In this way, a certain viewing angle view may be used to predict and output another viewing angle view. For example, if a first monitoring area view (or a second monitoring area view) is input, a second monitoring area view (or a first monitoring area view) is predicted and output.

[0024] Step 102: determine an image disparity map between the first monitoring area view and the second monitoring area view.

[0025] In some embodiments, the execution subject may determine the image disparity map between the first monitoring area view and the second monitoring area view. The image disparity map between the first monitoring area view and the second monitoring area view may be determined. For example, the execution subject may use the first monitoring area view as a reference and adopt NCC (normalization cross-correlation) or stereo matching algorithm to perform depth information estimation (Depth Estimation) method to determine the image disparity map from the first monitoring area view to the second monitoring area view.

[0026] Step 103 : performing image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view.

[0027] In some embodiments, the execution entity may perform image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view. The transformed monitoring area view is usually the same as the viewing angle of the second monitoring area view. For example, the transformed monitoring area view initially generated based on the depth information is also required to fill the hole area to obtain the second monitoring area view. Therefore, the transformed monitoring area view usually also includes filling mask data. For example, the image transformation processing may be a Depth-Guided warp transformation operation. Depth-Guided warp is usually a warping operation that guides the view through depth information. By remapping the pixel position of the view, image distortion can achieve various deformations of the view, such as elongation, rotation, and compression.

[0028] Step 104, using the initial transformed view image to build a model, classifying and filling the above-mentioned transformed monitoring area view, obtaining multiple view filling images, and outputting the first monitoring area view transformation view according to the multiple view filling images. For example, the initial transformed view image construction model may include: a first filling network, a second filling network, and a third filling network. The first filling network may be a poly-base network. The second filling network may be a DE-base network. The third filling network may be a DL-base network. PolyBase (poly-base) neural network is not a specific technology or product name, but refers to a method or concept of using neural network technology or models for data processing and analysis in a PolyBase environment. PolyBase is a component of Microsoft SQL Server, which is used to query external data sources such as Hadoop, Azure Blob Storage, etc. in SQL Server. Although PolyBase itself does not directly involve neural network technology, neural network models can be integrated and used in the PolyBase environment for data analysis and processing. PolyBase is a SQLServer component that allows users to directly query and analyze big data stored in external data sources (such as Hadoop, Azure Blob Storage, etc.) in SQL Server. It improves the efficiency of big data processing by optimizing query execution plans and reducing data movement. DE-base neural network refers to a neural network model based on deep learning, which is usually used to process large-scale data sets and complex tasks. The core idea of ​​DE-base neural network is to automatically extract features and patterns from a large amount of data through the connection and learning process between multiple layers of neurons, so as to achieve efficient prediction and classification. DL-base neural network refers to a neural network based on deep learning (DL for short). Deep learning is a subset of machine learning, which processes complex data and tasks through multi-layer neural networks.

[0029] In some embodiments, the execution subject may construct a model using the initial transformed view image, perform classification filling processing on the transformed monitoring area view, obtain multiple view filling images, and output the first monitoring area perspective transformed view based on the multiple view filling images. For example, the multiple view filling images may be feature fused to obtain the first monitoring area perspective transformed view.

[0030] In practice, the execution subject may perform classification and filling processing on the transformed monitoring area view through the following steps:

[0031] In the first step, the transformed monitoring area view is input into the first filling network included in the initial transformed view image construction model, and a polygon filling algorithm is used for filling processing to obtain a first view filled image. Polygon filling may refer to: coloring the pixels in the polygon area to form a continuous figure. The polygon filling algorithm may be: a seed filling method, a boundary filling algorithm.

[0032] For example, the rows that transform the view of the surveillance area can be considered as polylines, which are generated and deformed by parallax adjustment before rasterization. First, each edge of the polygon is determined and scanned in the vertical direction. For each row, the intersection of the row with the edge of the polygon is calculated. Then, the line is divided into two parts according to the horizontal position (for example, x The intersection points of each row are sorted by coordinates. Then, the intersection points on each row are connected into line segments to form the projection of the polygon on the row. After the transformed monitoring area view is deformed, the deformed line segments are converted into pixel data, and finally, the pixels are filled between the intersection points of each row.

[0033] In the second step, the transformed monitoring area view is input into the second filling network included in the initial transformed view image construction model, and the filling process is performed through the video editing tool to obtain the second view filling image. The video editing tool can be a Propainte tool. The video editing tool can perform feature propagation of adjacent views and feature double layers through optical flow, fill the current area to be filled through known pixels in other frames, and then fill the remaining unfilled area through the designed filling network.

[0034] In the third step, the transformed monitoring area view is input into the third filling network included in the initial transformed viewing angle image construction model, and the foreground edge in the transformed monitoring area view is extended and filled to obtain a third view filling image.

[0035] The third step may include the following sub-steps:

[0036] The first sub-step is to determine the edge position of the disparity map of the above-mentioned image disparity map, and determine the foreground pixels and background pixels corresponding to the edge position of the above-mentioned disparity map. For example, the edge detection algorithm (Canny) can be used to determine the edge position of the disparity map of the above-mentioned image disparity map, and determine the foreground pixels and background pixels corresponding to the edge position of the above-mentioned disparity map. It can be determined according to the size of the pixel value at the edge position. For example, a pixel point with a pixel value greater than a set value can be determined as a foreground pixel. A pixel point with a pixel value less than a set value can be determined as a background pixel.

[0037] In the second sub-step, a preset number of background pixels located in a preset direction at the edge of the disparity map are replaced with adjacent foreground pixels to obtain a replacement image disparity map. For example, two background pixels at the edge in the horizontal direction may be replaced with pixel values ​​of adjacent foreground pixels.

[0038] The third sub-step is to perform an expansion and filling process on the above-mentioned transformed monitoring area view according to the above-mentioned replacement image disparity map. For example, the transformed monitoring area view can be copied, and an edge detection algorithm can be used to determine the foreground edge of the copied transformed monitoring area view. And a preset number of foreground pixels in a set direction at the foreground edge of the copied transformed monitoring area view are replaced with adjacent background pixels to obtain a third view filling image.

[0039] In the fourth step, the first view filling image, the second view filling image and the third view filling image are determined as a plurality of view filling images.

[0040] In practice, the above execution subject can output the first monitoring area perspective transformation view through the following steps:

[0041] In the first step, the image fusion network included in the initial transformation view image construction model is used to perform image feature fusion processing on multiple view filling images to obtain a filling color image and a mask image corresponding to each view filling image. The image fusion network includes: a coding layer group and a corresponding decoding layer group.

[0042] The first step may include the following sub-steps:

[0043] In the first sub-step, for each coding layer in the coding layer group, the input features of the coding layer are convolved with the relevant image features of the next adjacent coding layer to obtain the output features of the coding layer, wherein the relevant image features are features obtained by upsampling the input features.

[0044] In the second sub-step, the output features of each coding layer in the coding layer group are jump-connected to the corresponding decoding layer to perform image feature fusion.

[0045] In the second step, the filled color image and each mask image are fused to obtain the perspective transformed view of the first monitoring area.

[0046] The second step may include the following sub-steps:

[0047] In a first sub-step, a first fusion coefficient is generated according to two view filling images among the plurality of view filling images and a mask image corresponding to one of the two view filling images.

[0048] The second sub-step is to generate a second fusion coefficient according to the first fusion coefficient, a mask image corresponding to another view filling image of the two view filling images, and other view filling images of the multiple view filling images.

[0049] The third sub-step is to generate a first monitoring area perspective transformation view of the first monitoring area view at another perspective based on the second fusion coefficient, the fill color map, and the corresponding remaining mask map.

[0050] For example, see the following formula:

[0051] .

[0052] in, Represents the first fusion coefficient. Represents the second fusion coefficient. Indicates the perspective change view of the first monitoring area. Indicates that the first view is filled with an image The corresponding mask image. Indicates that the second view is filled with an image The corresponding mask image. Indicates that the third view fills the image The corresponding mask image. Represents a fill color map.

[0053] It should be noted that the above image fusion network may include a coding layer group and a corresponding decoding layer group. One coding layer corresponds to one decoding layer.

[0054] For each coding layer in the image fusion network, in addition to encoding the input features, the FUN unit performs convolution fusion on the input features of the coding layer and the related features of the next adjacent coding layer to obtain the output features of the coding layer. The related features are the features obtained by upsampling the input features. And the output features of the coding layer are jump-connected to the corresponding decoding layer to achieve feature fusion. See the following formula:

[0055] .

[0056] in, Represents the convolutional layer in the FUN unit. Indicates m The input features of the encoding layer. Indicates m +1 encoding layer’s input features are upsampled. Indicates m The feature output of the encoding layer is fused and jump-connected to the corresponding decoding layer.

[0057] Therefore, by extending the edge of the image disparity map, the effect of the subsequent filling network and the accuracy of background content filling can be effectively improved. By effectively updating and fusing multi-resolution features through the image fusion network, the filling results of each filling network can be effectively fused, providing more detailed content optimization.

[0058] Step 105: Determine a difference value between the first monitoring area perspective transformed view and the second monitoring area view.

[0059] In some embodiments, the execution subject may determine the difference between the first monitoring area perspective transformation view and the second monitoring area view. For example, the difference between the first monitoring area perspective transformation view and the second monitoring area view may be determined by a cross entropy loss function or a hinge loss function.

[0060] For another example, the execution subject may determine the difference value between the first monitoring area perspective transformation view and the second monitoring area view by using the following loss function:

[0061] .

[0062] Wherein, L represents the difference value. Indicates the second monitoring area view. Indicates the perspective change view of the first monitoring area.

[0063] Step 106: In response to determining that the difference value is less than or equal to a preset difference value, the initial transformed view image construction model is determined as a trained transformed view image construction model.

[0064] In some embodiments, the execution subject may determine the initial transformed perspective image construction model as a trained transformed perspective image construction model in response to determining that the difference value is less than or equal to a preset difference value.

[0065] Step 107 : in response to detecting a three-dimensional display description instruction for the target monitoring area, obtaining a two-dimensional monitoring video of the target monitoring area corresponding to the three-dimensional display description instruction.

[0066] In some embodiments, the execution subject may, in response to detecting a three-dimensional display instruction for a target monitoring area, obtain a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display instruction. The three-dimensional display instruction may refer to an instruction to display a three-dimensional video display of the corresponding target monitoring area within a preset time period. For example, an instruction to convert a two-dimensional target monitoring area monitoring video shot within a preset time period into a three-dimensional video (3D video). The two-dimensional target monitoring area monitoring video may be a two-dimensional video of the target monitoring area shot by a camera within a preset time period. The target monitoring area may refer to an area that needs to be monitored and defended (e.g., a restricted area).

[0067] Step 108, constructing a model based on the transformed view image, performing three-dimensional conversion processing on the two-dimensional target monitoring area monitoring video to obtain a three-dimensional target monitoring area monitoring video, and displaying the three-dimensional target monitoring area monitoring video on an associated monitoring display terminal.

[0068] In some embodiments, the above-mentioned execution subject can perform three-dimensional conversion processing on the above-mentioned two-dimensional target monitoring area monitoring video according to the above-mentioned transformed perspective image construction model to obtain a three-dimensional target monitoring area monitoring video, and display the above-mentioned three-dimensional target monitoring area monitoring video on the associated monitoring display terminal. For example, the above-mentioned execution subject can perform three-dimensional conversion processing on the above-mentioned two-dimensional target monitoring area monitoring video through Vid3D technology to obtain a three-dimensional target monitoring area monitoring video. For example, each frame image included in the two-dimensional target monitoring area monitoring video can be input into the transformed perspective image construction model to obtain each frame image after the perspective conversion. Then, each frame image after the perspective conversion can be fused with the two-dimensional target monitoring area monitoring video through Vid3D technology to obtain a three-dimensional target monitoring area monitoring video. The monitoring display terminal may refer to an AR or VR display terminal for displaying a three-dimensional target monitoring area monitoring video that is communicatively connected to the above-mentioned execution subject.

[0069] In practice, the execution subject may perform three-dimensional conversion processing on the two-dimensional target monitoring area monitoring video through the following steps:

[0070] In the first step, depth prediction is performed on each video frame image in the above-mentioned two-dimensional target monitoring area monitoring video to generate a video frame disparity map and obtain a video frame disparity map sequence.

[0071] In the second step, for each video frame image, perform the following processing steps:

[0072] First, according to the video frame disparity map corresponding to the video frame image, the video frame image is transformed to obtain an initial transformed video frame image.

[0073] Second, the initial transformed video frame image is input into the transformed view image construction model to obtain a transformed video frame image of the video frame image.

[0074] The third step is to generate a three-dimensional target monitoring area monitoring video based on the above two-dimensional target monitoring area monitoring video and the corresponding transformed video frame images. For example, a 2D video (two-dimensional target monitoring area monitoring video) can be converted into a 3D video (three-dimensional target monitoring area monitoring video) by using AI technology. For another example, a 2D video (two-dimensional target monitoring area monitoring video) can be converted into a 3D video (three-dimensional target monitoring area monitoring video) by using 2D-3D conversion technology.

[0075] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a device for constructing a three-dimensional video based on a target monitoring area. These embodiments of the device for constructing a three-dimensional video based on a target monitoring area are similar to Figure 1 Corresponding to the method embodiments shown, the three-dimensional video construction device based on the target monitoring area can be specifically applied to various electronic devices.

[0076] like Figure 2As shown, in some embodiments, a three-dimensional video construction device 200 based on a target monitoring area includes: a first acquisition unit 201, a first determination unit 202, a transformation unit 203, a filling unit 204, a second determination unit 205, a third determination unit 206, a second acquisition unit 207 and a conversion unit 208. The first acquisition unit 201 is configured to acquire monitoring area image samples, wherein the monitoring area image samples include: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles; the first determination unit 202 is configured to determine an image disparity map between the first monitoring area view and the second monitoring area view; the transformation unit 203 is configured to perform image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view; the filling unit 204 is configured to construct a model using an initial transformed perspective image, perform classified filling processing on the transformed monitoring area view to obtain multiple view filling images, and output a first monitoring area perspective transformed view according to the multiple view filling images; the second determination unit 204 is configured to construct a model using an initial transformed perspective image, perform classified filling processing on the transformed monitoring area view to obtain multiple view filling images, and output a first monitoring area perspective transformed view according to the multiple view filling images; Unit 205 is configured to determine the difference value between the above-mentioned first monitoring area perspective transformation view and the above-mentioned second monitoring area view; the third determination unit 206 is configured to determine the above-mentioned initial transformation perspective image construction model as the trained transformation perspective image construction model in response to determining that the above-mentioned difference value is less than or equal to the preset difference value; the second acquisition unit 207 is configured to acquire the two-dimensional target monitoring area monitoring video corresponding to the above-mentioned three-dimensional display description instruction in response to detecting the three-dimensional display description instruction for the target monitoring area; the conversion unit 208 is configured to perform three-dimensional conversion processing on the above-mentioned two-dimensional target monitoring area monitoring video according to the above-mentioned transformation perspective image construction model to obtain the three-dimensional target monitoring area monitoring video, and display the above-mentioned three-dimensional target monitoring area monitoring video on the associated monitoring display terminal.

[0077] It can be understood that the units recorded in the three-dimensional video construction device 200 based on the target monitoring area are similar to the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the 3D video construction device 200 based on the target monitoring area and the units contained therein, and will not be described in detail here.

[0078] Reference below Figure 3, which shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0080] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0081] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0082] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0083] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0084] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains monitoring area image samples, wherein the monitoring area image samples include: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles; determines an image disparity map between the first monitoring area view and the second monitoring area view; performs image transformation processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view; constructs a model using an initial transformed perspective image, performs classification filling processing on the transformed monitoring area view to obtain multiple view filling images, and performs classification filling processing on the multiple view filling images according to the multiple view images. The invention relates to a method for filling an image with a graph and outputting a perspective transformation view of a first monitoring area; determining a difference value between the perspective transformation view of the first monitoring area and the perspective transformation view of the second monitoring area; in response to determining that the difference value is less than or equal to a preset difference value, determining the initial perspective transformation image construction model as a trained perspective transformation image construction model; in response to detecting a three-dimensional display description instruction for a target monitoring area, obtaining a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display description instruction; performing three-dimensional conversion processing on the two-dimensional target monitoring area monitoring video according to the perspective transformation image construction model to obtain a three-dimensional target monitoring area monitoring video, and displaying the three-dimensional target monitoring area monitoring video on an associated monitoring display terminal.

[0085] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: a first acquisition unit, a first determination unit, a transformation unit, a filling unit, a second determination unit, a third determination unit, a second acquisition unit and a conversion unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain circumstances, for example, the transformation unit may also be described as "a unit that performs image transformation processing on the above-mentioned first monitoring area view according to the above-mentioned image disparity map to obtain a transformed monitoring area view".

[0088] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for constructing a three-dimensional video based on a target monitoring area, comprising: Acquire a monitoring area image sample, wherein the monitoring area image sample includes: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles; Determining an image disparity map between the first monitoring area view and the second monitoring area view; According to the image disparity map, performing image transformation processing on the first monitoring area view to obtain a transformed monitoring area view; Using the initial transformed view angle image to construct a model, performing classification and filling processing on the transformed monitoring area view to obtain a plurality of view filling images, and outputting a first monitoring area view angle transformed view according to the plurality of view filling images; Determine a difference value between the first monitoring area perspective transformed view and the second monitoring area view; In response to determining that the difference value is less than or equal to a preset difference value, determining the initial transformed perspective image construction model as a trained transformed perspective image construction model; In response to detecting a three-dimensional display description instruction for a target monitoring area, acquiring a two-dimensional monitoring video of the target monitoring area corresponding to the three-dimensional display description instruction; The model is constructed according to the transformed view image, the two-dimensional target monitoring area monitoring video is converted into three dimensions to obtain the three-dimensional target monitoring area monitoring video, and the three-dimensional target monitoring area monitoring video is displayed on the associated monitoring display terminal.

2. The method according to claim 1, wherein: The model is constructed by using the initial transformed view image, and the transformed monitoring area view is classified and filled to obtain a plurality of view filling images, including: Inputting the transformed monitoring area view into a first filling network included in the initial transformed view angle image construction model, and performing filling processing using a polygon filling algorithm to obtain a first view filling image; Inputting the transformed monitoring area view into a second filling network included in the initial transformed view angle image construction model, and performing filling processing through a video editing tool to obtain a second view filling image; Inputting the transformed monitoring area view into a third filling network included in the initial transformed view angle image construction model, performing an expansion filling process on the foreground edge in the transformed monitoring area view, and obtaining a third view filling image; The first view filling image, the second view filling image and the third view filling image are determined as a plurality of view filling images.

3. The method according to claim 2, wherein: The step of performing an expansion and filling process on the foreground edge in the transformed monitoring area view includes: Determining a disparity map edge position of the image disparity map, and determining each foreground pixel and background pixel corresponding to the disparity map edge position; Replacing a preset number of background pixels in a preset direction at an edge position of the disparity map with adjacent foreground pixels to obtain a replacement image disparity map; The transformed monitoring area view is extended and filled according to the replacement image disparity map.

4. The method according to claim 1, wherein: The step of constructing a model according to the transformed view angle image and performing a three-dimensional conversion process on the two-dimensional target monitoring area monitoring video to obtain a three-dimensional target monitoring area monitoring video includes: Performing depth prediction on each video frame image in the monitoring video of the two-dimensional target monitoring area to generate a video frame disparity map, thereby obtaining a video frame disparity map sequence; For each video frame image, the following processing steps are performed: According to the video frame disparity map corresponding to the video frame image, performing image transformation on the video frame image to obtain an initial transformed video frame image; Inputting the initial transformed video frame image into the transformed view image construction model to obtain a transformed video frame image of the video frame image; A three-dimensional target monitoring area monitoring video is generated according to the two-dimensional target monitoring area monitoring video and the corresponding transformed video frame images.

5. A three-dimensional video construction device based on a target monitoring area, comprising: A first acquisition unit is configured to acquire a monitoring area image sample, wherein the monitoring area image sample includes: a first monitoring area view and a second monitoring area view, and the first monitoring area view and the second monitoring area view correspond to different angles; A first determining unit is configured to determine an image disparity map between the first monitoring area view and the second monitoring area view; a transforming unit configured to perform image transform processing on the first monitoring area view according to the image disparity map to obtain a transformed monitoring area view; A filling unit is configured to construct a model using an initial transformed view image, perform classified filling processing on the transformed monitoring area view to obtain a plurality of view filling images, and output a first monitoring area view transformed view according to the plurality of view filling images; a second determining unit, configured to determine a difference value between the first monitoring area perspective transformed view and the second monitoring area view; A third determining unit is configured to, in response to determining that the difference value is less than or equal to a preset difference value, determine the initial transformed perspective image construction model as a trained transformed perspective image construction model; A second acquisition unit is configured to, in response to detecting a three-dimensional display description instruction for a target monitoring area, acquire a two-dimensional target monitoring area monitoring video corresponding to the three-dimensional display description instruction; The conversion unit is configured to construct a model based on the transformed perspective image, perform three-dimensional conversion processing on the two-dimensional target monitoring area monitoring video to obtain a three-dimensional target monitoring area monitoring video, and display the three-dimensional target monitoring area monitoring video on an associated monitoring display terminal.

6. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images