Video generation method, video generation device and computer readable storage medium

Abstract

The embodiment of the invention provides a video generation method, a video generation device and a computer readable storage medium. The video generation method according to the embodiment of the invention comprises the following steps: acquiring three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space comprises one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain guide point cloud data of each guide region in the three-dimensional space; acquiring a guide channel of each guide area according to the guide point cloud data of each guide area, and calculating one or more guide points of each guide area; and acquiring a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generating the video by using the three-dimensional point cloud data and the navigation path.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a video generation method, a video generation device and a computer-readable storage medium. Background Art

[0002] With the booming development of virtual reality, panoramic cameras are no longer limited to use in the real world. People can use panoramic cameras in the field of virtual reality, such as in the metaverse, to share various scenes and arrangements of the virtual world to the real world. But even in the metaverse, it is very time-consuming and labor-intensive to travel through various scenes, such as offices, buildings, shopping malls, etc. Therefore, in order to give people a better experience, generating panoramic guide videos is a good choice.

[0003] At present, the production method of panoramic guide video is mainly to generate panoramic video by direct shooting, or to generate point-to-point jump roaming panoramic video after shooting panoramic images at multiple fixed points. These methods have the problems of low production efficiency and lack of intelligence.

[0004] Therefore, there is a need for a method and device that can automatically calculate and generate a panoramic guide video based on three-dimensional image data. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, a video generation method is provided, comprising: obtaining three-dimensional point cloud data of a three-dimensional space, the three-dimensional space including one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; obtaining a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area; obtaining a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generating the video using the three-dimensional point cloud data and the navigation path.

[0006] According to another aspect of the present invention, a video generating device is provided, comprising: an acquiring unit, configured to acquire three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; a segmenting unit, configured to perform area segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; a calculating unit, configured to acquire a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculate one or more navigation points of each navigation area; a generating unit, configured to acquire a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generate the video using the three-dimensional point cloud data and the navigation path.

[0007] According to another aspect of the present invention, a video generating device is provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: obtaining three-dimensional point cloud data of a three-dimensional space, the three-dimensional space including one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; obtaining a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area; obtaining a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generating the video using the three-dimensional point cloud data and the navigation path.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining three-dimensional point cloud data of a three-dimensional space, the three-dimensional space including one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; obtaining a navigation channel of each navigation area based on the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area; obtaining a navigation path of the three-dimensional space based on one or more navigation points of each navigation area in the three-dimensional space, and generating the video using the three-dimensional point cloud data and the navigation path.

[0009] According to the above-mentioned video generation method, video generation device and computer-readable storage medium of the present invention, it is possible to obtain the navigation point cloud data of each navigation area by segmenting the three-dimensional point cloud data of the three-dimensional space, and automatically calculate the navigation points for the navigation area through the information of the navigation channel of the navigation area, and automatically obtain the navigation path accordingly to generate a video for the three-dimensional space. The video generation method, video generation device and computer-readable storage medium of the present invention can provide a method for automatically generating a navigation video for a three-dimensional space, improve the efficiency of video production, and promote the intelligent development in virtual reality scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0011] Figure 1 A flowchart showing a video generation method according to an embodiment of the present invention;

[0012] Figure 2A schematic diagram showing three-dimensional point cloud data of a three-dimensional space according to an example of an embodiment of the present invention;

[0013] Figure 3 A schematic diagram showing navigation point cloud data of a navigation area segmented from three-dimensional point cloud data according to an example of an embodiment of the present invention;

[0014] Figure 4 A schematic diagram showing navigation projection data obtained by projecting navigation point cloud data according to an example of an embodiment of the present invention;

[0015] Figure 5 A schematic diagram showing how the guided projection data is divided into guided channels according to an example of an embodiment of the present invention;

[0016] Figure 6 A schematic diagram of triangulation of a navigation channel according to an example of an embodiment of the present invention is shown;

[0017] Figure 7 A schematic diagram of generating a navigation path according to an example of an embodiment of the present invention is shown;

[0018] Figure 8 A schematic diagram showing video frame generation according to an example of an embodiment of the present invention is shown;

[0019] Fig. 9 A block diagram showing a video generating apparatus according to an embodiment of the present invention;

[0020] Fig.10 A block diagram of a video generating apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] The video generation method, video generation device and computer readable storage medium according to the embodiments of the present invention will be described below with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0022] In the field of virtual reality, the production of guide videos for different scenes often has the problems of low production efficiency and poor user experience. To this end, an embodiment of the present invention provides a video generation method, a video generation device and a computer-readable storage medium.

[0023] Figure 1 FIG. 1 is a flowchart of a video generation method 100 according to an embodiment of the present invention. Figure 1 A video generation method according to an embodiment of the present invention is described.

[0024] In step S101 , three-dimensional point cloud data of a three-dimensional space is acquired, where the three-dimensional space includes one or more navigation areas.

[0025] In this step, at least a portion of the three-dimensional dense point cloud data of the three-dimensional space can be created by at least one of Lidar scanning, Kinect, structured light or multi-view stereo vision (Multi-view Stereo). Optionally, the three-dimensional point cloud data may include coordinate data (x, y, z) of the three-dimensional point cloud in the three-dimensional space obtained by at least one of the above methods. In addition, optionally, the three-dimensional point cloud data may also include various other information such as color data, object texture data, etc. of the three-dimensional point cloud in the three-dimensional space further obtained by at least one of the above methods. The content of the three-dimensional point cloud data obtained for the three-dimensional space varies according to the different requirements of the actual application scenarios, and is not limited here.

[0026] Optionally, the three-dimensional space can be various buildings, streets and other different scenes in the virtual reality scene. The three-dimensional space can include one or more navigation areas for the navigation video. For example, when the three-dimensional space is a building used as an office space, the navigation areas included in the three-dimensional space can be various rooms in the office space (such as offices, conference rooms, lounges, etc.); when the three-dimensional space is a shopping mall, the navigation areas included in the three-dimensional space can be various functional areas on each floor of the shopping mall (such as different brand stores, restaurants, cafes, etc.); when the three-dimensional space is a block, the navigation areas included in the three-dimensional space can be various different houses in the block. The division of each navigation area in the three-dimensional space can be changed differently according to different virtual reality scenes, and is not limited here.

[0027] In step S102, the three-dimensional point cloud data is segmented into regions to obtain navigation point cloud data of each navigation region in the three-dimensional space.

[0028] Optionally, edge detection can be performed on the three-dimensional point cloud data using, for example, an edge detection model in a neural network to obtain corner point data (such as corner point coordinates) corresponding to each navigation area in the three-dimensional space, so as to generate three-dimensional navigation point cloud data for each navigation area. The generated navigation point cloud data can be used in the subsequent acquisition of navigation channels and the generation of videos.

[0029] In step S103, a navigation channel of each navigation area is obtained according to the navigation point cloud data of each navigation area, and one or more navigation points of each navigation area are calculated.

[0030] Optionally, in the process of obtaining the navigation channel, the three-dimensional navigation point cloud data of a certain navigation area may be firstly projected in two dimensions to obtain corresponding navigation projection data. For example, the navigation point cloud data in a building perpendicular to the xy plane may be projected onto the xy plane to generate two-dimensional navigation projection data, wherein the RGB data of the point cloud with the largest z-axis coordinate may be used as the RGB data of the corresponding two-dimensional coordinate point in the navigation projection data.

[0031] After obtaining the navigation projection data, the deep learning model can be used to segment the image in the navigation projection data to divide the navigation channel and the non-navigation channel. Optionally, the navigation channel can be an area in the navigation area that is free of obstacles and can be passed through, while the non-navigation channel can be an area where obstacles are placed. In the process of image segmentation of the navigation projection data, the navigation channel can be divided by outputting the two-dimensional vertex coordinates of the navigation channel. Of course, the above-mentioned method of dividing the navigation channel and the non-navigation channel is only an example. In actual applications, other judgment and division methods can also be used to divide the navigation channels in the navigation area. For example, a specific preset route can be used as a navigation channel in the navigation area, or a traveling area paved with tracks or conveyor belts can be used as a navigation channel in the navigation area. There is no limitation here.

[0032] After the navigation channel corresponding to the navigation area is divided, optionally, the spatial center of gravity of the navigation channel can be calculated, and the calculated spatial center of gravity is used as at least one of the navigation points of the navigation area. Specifically, the spatial center of gravity of the navigation channel can be calculated by triangulating the navigation channel. Among them, the navigation channel can first be divided into multiple triangles using the calculated two-dimensional vertex coordinates of the navigation channel corresponding to the navigation area; then, the weight of each triangle on the center of gravity of the navigation area can be obtained by calculating the area and center of gravity of each triangle to calculate the center of gravity of the navigation channel on the two-dimensional plane; finally, the center of gravity coordinates of the navigation channel in the z direction can be obtained according to the corresponding three-dimensional navigation point cloud data to calculate the spatial center of gravity of the navigation channel. In addition, optionally, any other points in the navigation area that meet the requirements can also be used as the navigation points of the navigation channel, such as the geometric center point of the navigation area and the points near a specific object in the navigation area can be used as the navigation points, and there is no limitation here.

[0033] In another example, one or more points within a certain range around the entrance and / or exit of the navigation area can also be set as navigation points of the navigation area. For example, the navigation point cloud data corresponding to the navigation area can first be image recognized using a deep learning model to identify objects (such as doors) that serve as the entrance and / or exit of the navigation area or areas near the entrance and / or exit. Subsequently, the navigation points of the entrance and / or exit can be calculated based on the results of image recognition. For example, the center of gravity of the door can also be used as the navigation point of the navigation area.

[0034] The above selection and calculation methods for navigation points in the navigation area are only examples. In actual applications, different locations and numbers of navigation points can be set according to the needs of the scene, and there is no limitation here. In addition, the number of navigation points can be adjusted according to different video length requirements or playback speed restrictions to meet the requirements of specific video playback.

[0035] In step S104, a navigation path of the three-dimensional space is obtained according to one or more navigation points of each navigation area in the three-dimensional space, and the video is generated using the three-dimensional point cloud data and the navigation path.

[0036] Optionally, one or more navigation points in each navigation area in the three-dimensional space can be path-planned by various algorithms such as Dijkstra algorithm and A* algorithm to obtain the sorting of all navigation points and generate the navigation path of the three-dimensional space. For example, the navigation points can be sorted in the order of the spatial center of gravity of the first navigation area, the spatial center of gravity of the second navigation area, the spatial center of gravity of the third navigation area, etc., wherein the first navigation area, the second navigation area and the third navigation area can be adjacent to each other in the three-dimensional space. For another example, the navigation points can also be sorted in the order of the spatial center of gravity of the first navigation area, the spatial center of gravity of the door of the first navigation area, the spatial center of gravity of the second navigation area, the door center of gravity of the second navigation area, the spatial center of gravity of the third navigation area, the door center of gravity of the third navigation area, etc. The above-mentioned various path planning methods are only examples. In actual applications, any path planning method can be adopted according to the needs of the scene, and it is not limited here.

[0037] After completing the path planning and generating the navigation path of the three-dimensional space, the video can be generated using the three-dimensional point cloud data and the navigation path. Specifically, a plurality of viewpoints arranged continuously can be determined on the navigation path, such as the first viewpoint, the second viewpoint, the third viewpoint, etc., and then a video frame is formed for each viewpoint according to the three-dimensional point cloud data, and the video frames are combined to generate the video. Among them, the corresponding three-dimensional navigation point cloud data can first be mapped to a unit sphere with the first viewpoint as the center, and the panoramic image obtained by mapping is converted into an equidistant cylindrical projection map as a video frame at the first point of the navigation path. Subsequently, video frames corresponding to the second viewpoint, the third viewpoint, etc. of the navigation path can be generated in sequence, and these video frames can be combined to form the final navigation video. Optionally, when playing the navigation video, the video frame corresponding to a certain viewpoint can be oriented in a certain direction, and the video frame played at a certain viewpoint can also be made to produce a large-angle rotation (such as 180-degree rotation, 270-degree rotation or 360-degree rotation) playback effect.

[0038] In addition, optionally, in the process of generating a video, the method of the embodiment of the present invention may further include: identifying the type of one or more guide areas in the three-dimensional space; generating a commentary corresponding to the guide area according to the identified type of the guide area to be inserted into the video. Specifically, the guide point cloud data corresponding to the guide area may first be image recognized using a deep learning model to identify the type of objects in the guide area; then, the type of the guide area may be determined according to the type of objects in the identified guide area, for example, the object type may be used to identify the probability of corresponding to a certain type of guide area, thereby determining the type of the guide area according to the type with the highest probability. Finally, a corresponding voice commentary may be generated from a pre-stored voice database according to the identified type of the guide area, and the generated voice commentary may be inserted when playing the video of the corresponding guide area.

[0039] The video generation method according to the embodiment of the present invention can obtain the navigation point cloud data of each navigation area by segmenting the three-dimensional point cloud data of the three-dimensional space, and automatically calculate the navigation points for the navigation area through the information of the navigation channel of the navigation area, and automatically obtain the navigation path accordingly to generate a video for the three-dimensional space. The video generation method of the present invention can provide a method for automatically generating a navigation video for the three-dimensional space, improve the efficiency of video production, and promote the intelligent development in the virtual reality scene.

[0040] An example of generating a video according to a video generation method according to an embodiment of the present invention is described below. In this example, the three-dimensional space is an office space in a virtual reality space, which includes multiple rooms, such as an office, a conference room, a lounge, a general manager's office, etc., as multiple navigation areas of the three-dimensional space.

[0041] According to an example of an embodiment of the present invention, three-dimensional point cloud data of a three-dimensional space may be acquired first. Figure 2 A schematic diagram of three-dimensional point cloud data in a three-dimensional space according to an example of an embodiment of the present invention is shown. Figure 2 As shown, the three-dimensional space may include multiple navigation areas in the office space. Optionally, the three-dimensional point cloud data of the three-dimensional space may be created by at least one of Lidar scanning, Kinect, structured light or multi-view stereo vision.

[0042] Subsequently, the three-dimensional point cloud data may be segmented to obtain navigation point cloud data of each navigation area in the three-dimensional space. Optionally, edge detection may be performed on the three-dimensional point cloud data using, for example, an edge detection model in a neural network to obtain corner point data (such as corner point coordinates) corresponding to each navigation area in the three-dimensional space, so as to generate three-dimensional navigation point cloud data of each navigation area. Figure 3 A schematic diagram of navigation point cloud data of a navigation area segmented from three-dimensional point cloud data according to an example of an embodiment of the present invention is shown. Figure 3 In the method, the edge detection model can be used to obtain the corner point coordinate data of a certain navigation area in the three-dimensional point cloud data, and the three-dimensional navigation point cloud data corresponding to the navigation area can be divided according to the corner point coordinate data, such as Figure 3 As shown in the black box in Figure 3 In the above region segmentation operation, the region corresponding to the navigation point cloud data represented by the black frame divided by the corner point coordinates can be recorded as the second navigation region. Figure 3 The remaining navigation point cloud data divided from is also marked as the corresponding navigation area. Figure 3 The navigation area corresponding to the navigation point cloud data divided outside the black box on the right is recorded as the first navigation area, and Figure 3 The navigation area corresponding to the navigation point cloud data divided outside the black frame in the upper left corner is recorded as the third navigation area.

[0043] Afterwards, the navigation channel of each navigation area can be obtained according to the navigation point cloud data of each navigation area, and one or more navigation points of each navigation area can be calculated. In the process of obtaining the navigation channel, the three-dimensional navigation point cloud data of a certain navigation area can be firstly projected in two dimensions to obtain corresponding navigation projection data. For example, the navigation point cloud data in a building perpendicular to the xy plane can be projected onto the xy plane to generate two-dimensional navigation projection data, wherein the RGB data of the point cloud with the largest z-axis coordinate can be used as the RGB data of the corresponding two-dimensional coordinate point in the navigation projection data. Figure 4A schematic diagram showing navigation projection data obtained by projecting navigation point cloud data according to an example of an embodiment of the present invention is shown. Figure 4 As shown, it can be Figure 3 The three-dimensional navigation point cloud data is projected onto the xy plane, and the RGB data of the point cloud with the largest z-axis coordinate is used as the RGB data of the corresponding two-dimensional coordinate point in the navigation projection data to generate two-dimensional navigation projection data.

[0044] After obtaining the guided projection data, the deep learning model can be used to segment the images in the guided projection data to divide the guided channels without obstacles and the non-guided channels with obstacles. Figure 5 FIG. 2 is a schematic diagram showing how the navigation projection data is divided into navigation channels according to an example of an embodiment of the present invention. Figure 5 As shown, deep learning models can be used to Figure 4 The navigation projection data shown is subjected to image segmentation, so that the navigation channels corresponding to the navigation areas are formed using the two-dimensional vertex coordinates of the navigation channels.

[0045] After dividing the navigation channel corresponding to the navigation area, optionally, the spatial center of gravity of the navigation channel can be calculated, and the calculated spatial center of gravity is used as at least one of the navigation points of the navigation area. Figure 6 FIG. 2 shows a schematic diagram of a triangulation of a navigation channel according to an example of an embodiment of the present invention. Figure 6 As shown, the navigation channel can be first divided into i triangles; then, the area A of each triangle can be calculated i and the center of gravity (C ix ,C iy ); Then, the centroid of each triangle can be accumulated and multiplied by the area of ​​each triangle, and then divided by the total area of ​​the navigation channel to calculate the centroid of the navigation channel on the two-dimensional plane (C x ,C y ); Finally, the coordinates of the center of gravity point C in the z direction of the navigation channel can be obtained according to the corresponding three-dimensional navigation point cloud data z =h / 2, where h is the height of the navigation point cloud data in the z direction, that is, the difference between the maximum and minimum coordinates of the navigation point cloud data in the z direction, so as to finally calculate the spatial center of gravity of the navigation channel (C x ,C y ,C z ). The spatial center of gravity of the navigation channel can browse the navigation area more comprehensively. In the example of the embodiment of the present invention, the corresponding spatial center of gravity point can be calculated for each navigation area in the three-dimensional space as the navigation point.

[0046] In addition, a deep learning model may be used to perform image recognition on the navigation point cloud data corresponding to the navigation area to identify the doors at the entrance and / or exit of the navigation area, and the center of gravity of the door may also be used as the navigation point of the navigation area.

[0047] In this example, finally, the navigation path of the three-dimensional space can be obtained according to one or more navigation points of each navigation area in the three-dimensional space, and the video can be generated using the three-dimensional point cloud data and the navigation path.

[0048] Figure 7 A schematic diagram of generating a navigation path according to an example of an embodiment of the present invention is shown. In this example, the navigation points can be sorted in the order of the first navigation area space center point in the lower right corner, the first navigation area door center point, the second navigation area space center point in the middle, the second navigation area door center point, and the third navigation area space center point in the upper left corner to generate a navigation path. Figure 7 As shown, each navigation point in each navigation area can be connected by a straight line indicated by an arrow to show the navigation path and travel direction in the navigation video. Of course, in actual applications, in the navigation area, each navigation point can be connected by various methods such as curves and broken lines according to the needs of specific scenes to plan various different navigation paths and travel directions, avoid obstacles as much as possible, expand the user's viewing angle, and enrich the viewing experience. There is no restriction on this.

[0049] After completing the path planning and generating the navigation path in the three-dimensional space, the video can be generated using the three-dimensional point cloud data and the navigation path. Specifically, a plurality of viewpoints arranged continuously can be determined on the navigation path, such as the first viewpoint, the second viewpoint, the third viewpoint, etc., and then a video frame is formed for each viewpoint according to the three-dimensional point cloud data, and the video frames are combined to generate the video. First, the corresponding three-dimensional navigation point cloud data can be mapped onto a unit sphere with the first viewpoint as the center, and the panoramic image obtained by mapping can be converted into an equirectangular projection image as a video frame at the first point of the navigation path. Figure 8 A schematic diagram of video frame generation according to an example of an embodiment of the present invention is shown. Figure 8 As shown, the coordinates of the center of the first viewpoint can be and the coordinates of a point on the unit sphere Draw a ray to intersect the navigation point cloud data at point (X, Y, Z, R, G, B), and replace the point (R, G, B) to the corresponding coordinate position of the equirectangular projection map to obtain (u 1 ,v 1 ,R,G,B) to form the corresponding video frame.

[0050] Subsequently, video frames corresponding to the second viewpoint, the third viewpoint, etc. of the navigation path may be generated in sequence, and these video frames may be combined to form a final navigation video. Optionally, when playing the navigation video, the video frame corresponding to a certain viewpoint may be oriented in a certain direction, or the video frame played at a certain viewpoint may be played with a 360-degree rotation effect.

[0051] In addition, optionally, in the process of generating the video, the navigation point cloud data corresponding to the navigation area can also be image recognized using a deep learning model to identify the type of objects in the navigation area; then, the type of the navigation area can be determined based on the type of objects in the recognized navigation area, for example, the object type can be used to identify the probability of corresponding to a certain type of navigation area, so as to determine the type of the navigation area based on the type with the highest probability. Finally, a corresponding voice commentary can be generated from a pre-stored voice database based on the recognized type of the navigation area, and the generated voice commentary can be inserted when playing the video of the corresponding navigation area. For example, in this example, the probability calculation can be performed based on the water dispenser and sofa identified in the guide area. For example, the probability of a water dispenser appearing in the office is expressed as 0.8, and the probability of a water dispenser appearing in the general manager's office is expressed as 0.2; the probability of a sofa appearing in the office is expressed as 0.1, and the probability of a sofa appearing in the general manager's office is expressed as 0.9. Then, the probability coding weighting of the identified water dispenser and sofa in [office, general manager's office] can be expressed as [0.9, 1.1]. From the result, the probability of the general manager's office is higher, so the guide area is identified as the general manager's office. After the general manager's office is identified, the corresponding voice commentary can be inserted when the guide video is played to the general manager's office, such as: "This is the general manager's office, with sofas and water dispensers, and the general manager works here." The voice commentary can be pre-stored according to the type of the guide area, or it can be automatically generated based on the results of the object recognition and the type recognition of the guide area.

[0052] Below, refer to Fig. 9 A video generating apparatus according to an embodiment of the present invention will be described. Fig. 9 FIG. 9 is a block diagram of a video generating apparatus 900 according to an embodiment of the present invention. Fig. 9 As shown, the video generating device 900 includes an acquisition unit 910, a segmentation unit 920, a calculation unit 930, and a generation unit 940. In addition to these units, the video generating device 900 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the video generating device 900 according to the embodiment of the present invention are the same as those described above with reference to Figure 1 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0053] Fig. 9 The acquisition unit 910 of the video generation device 900 acquires three-dimensional point cloud data of a three-dimensional space, where the three-dimensional space includes one or more navigation areas.

[0054] The acquisition unit 910 can create at least a portion of the three-dimensional dense point cloud data of the three-dimensional space by at least one of Lidar scanning, Kinect, structured light or multi-view stereo vision (Multi-view Stereo). Optionally, the three-dimensional point cloud data may include coordinate data (x, y, z) of the three-dimensional point cloud in the three-dimensional space acquired by at least one of the above methods. In addition, optionally, the three-dimensional point cloud data may also include other various information such as color data, object texture data, etc. of the three-dimensional point cloud in the three-dimensional space further acquired by at least one of the above methods. The content of the three-dimensional point cloud data acquired for the three-dimensional space varies according to different requirements of the actual application scenarios, and is not limited here.

[0055] Optionally, the three-dimensional space can be various buildings, streets and other different scenes in the virtual reality scene. The three-dimensional space can include one or more navigation areas for the navigation video. For example, when the three-dimensional space is a building used as an office space, the navigation areas included in the three-dimensional space can be various rooms in the office space (such as offices, conference rooms, lounges, etc.); when the three-dimensional space is a shopping mall, the navigation areas included in the three-dimensional space can be various functional areas on each floor of the shopping mall (such as different brand stores, restaurants, cafes, etc.); when the three-dimensional space is a block, the navigation areas included in the three-dimensional space can be various different houses in the block. The division of each navigation area in the three-dimensional space can be changed differently according to different virtual reality scenes, and is not limited here.

[0056] The segmentation unit 920 performs region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space.

[0057] Optionally, the segmentation unit 920 can perform edge detection on the three-dimensional point cloud data using, for example, an edge detection model in a neural network, and obtain corner point data (such as corner point coordinates) corresponding to each navigation area in the three-dimensional space to generate three-dimensional navigation point cloud data for each navigation area. The generated navigation point cloud data can be used for subsequent navigation channel acquisition and video generation processes.

[0058] The calculation unit 930 obtains the navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculates one or more navigation points of each navigation area.

[0059] Optionally, in the process of acquiring the navigation channel, the calculation unit 930 may first perform a two-dimensional projection on the three-dimensional navigation point cloud data of a certain navigation area to obtain corresponding navigation projection data. For example, the navigation point cloud data in a building perpendicular to the xy plane may be projected onto the xy plane to generate two-dimensional navigation projection data, wherein the RGB data of the point cloud with the largest z-axis coordinate may be used as the RGB data of the corresponding two-dimensional coordinate point in the navigation projection data.

[0060] After obtaining the navigation projection data, the computing unit 930 can segment the image in the navigation projection data using a deep learning model to divide the navigation channel and the non-navigation channel. Optionally, the navigation channel can be an area in the navigation area that is free of obstacles and can be passed through, while the non-navigation channel can be an area where obstacles are placed. In the process of image segmentation of the navigation projection data, the navigation channel can be divided by outputting the two-dimensional vertex coordinates of the navigation channel. Of course, the above-mentioned method of dividing the navigation channel and the non-navigation channel is only an example. In actual applications, other judgment and division methods can also be used to divide the navigation channel in the navigation area. For example, a specific preset route can be used as a navigation channel in the navigation area, or a traveling area paved with tracks or conveyor belts can be used as a navigation channel in the navigation area. There is no limitation here.

[0061] After dividing the navigation channel corresponding to the navigation area, optionally, the calculation unit 930 can calculate the spatial center of gravity of the navigation channel, and use the calculated spatial center of gravity as at least one of the navigation points of the navigation area. Specifically, the spatial center of gravity of the navigation channel can be calculated by triangulating the navigation channel. Among them, the navigation channel can first be divided into multiple triangles using the calculated two-dimensional vertex coordinates of the navigation channel corresponding to the navigation area; then, the weight of each triangle on the center of gravity of the navigation area can be obtained by calculating the area and center of gravity of each triangle to calculate the center of gravity of the navigation channel on the two-dimensional plane; finally, the center of gravity coordinates of the navigation channel in the z direction can be obtained according to the corresponding three-dimensional navigation point cloud data to calculate the spatial center of gravity of the navigation channel. In addition, optionally, any other points in the navigation area that meet the requirements can also be used as the navigation points of the navigation channel, such as the geometric center point of the navigation area and the point near a specific object in the navigation area can be used as the navigation point, which is not limited here.

[0062] In another example, one or more points within a certain range around the entrance and / or exit of the navigation area can also be set as navigation points of the navigation area. For example, the navigation point cloud data corresponding to the navigation area can first be image recognized using a deep learning model to identify objects (such as doors) that serve as the entrance and / or exit of the navigation area or areas near the entrance and / or exit. Subsequently, the navigation points of the entrance and / or exit can be calculated based on the results of image recognition. For example, the center of gravity of the door can also be used as the navigation point of the navigation area.

[0063] The above selection and calculation methods for navigation points in the navigation area are only examples. In actual applications, different locations and numbers of navigation points can be set according to the needs of the scene, and there is no limitation here. In addition, the number of navigation points can be adjusted according to different video length requirements or playback speed restrictions to meet the requirements of specific video playback.

[0064] The generating unit 940 obtains a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generates the video using the three-dimensional point cloud data and the navigation path.

[0065] Optionally, the generating unit 940 can perform path planning for one or more navigation points of each navigation area in the three-dimensional space by various algorithms such as Dijkstra algorithm and A* algorithm, obtain the sorting of all navigation points, and generate the navigation path of the three-dimensional space. For example, the navigation points can be sorted in the order of the spatial center of gravity of the first navigation area, the spatial center of gravity of the second navigation area, the spatial center of gravity of the third navigation area, etc., wherein the first navigation area, the second navigation area and the third navigation area can be adjacent to each other in the three-dimensional space. For another example, the navigation points can also be sorted in the order of the spatial center of gravity of the first navigation area, the door center of gravity of the first navigation area, the spatial center of gravity of the second navigation area, the door center of gravity of the second navigation area, the spatial center of gravity of the third navigation area, the door center of gravity of the third navigation area, etc. The above-mentioned various path planning methods are only examples. In actual applications, any path planning method can be adopted according to the needs of the scene, and it is not limited here.

[0066] After completing the path planning and generating the navigation path of the three-dimensional space, the generating unit 940 can generate the video using the three-dimensional point cloud data and the navigation path. Specifically, a plurality of viewpoints arranged continuously on the navigation path, such as the first viewpoint, the second viewpoint, the third viewpoint, etc., can be determined, and then a video frame is formed for each viewpoint according to the three-dimensional point cloud data, and the video frames are combined to generate the video. Among them, the corresponding three-dimensional navigation point cloud data can first be mapped to a unit sphere with the first viewpoint as the center, and the panoramic image obtained by mapping is converted into an equidistant cylindrical projection map as a video frame at the first point of the navigation path. Subsequently, video frames corresponding to the second viewpoint, the third viewpoint, etc. of the navigation path can be generated in sequence, and these video frames can be combined to form the final navigation video. Optionally, when playing the navigation video, the video frame corresponding to a certain viewpoint can be oriented in a certain direction, and the video frame played at a certain viewpoint can also be made to produce a large-angle rotation (such as 180-degree rotation, 270-degree rotation or 360-degree rotation) playback effect.

[0067] In addition, optionally, in the process of generating the video, the generation unit 940 can also identify the type of one or more guide areas in the three-dimensional space; according to the identified type of the guide area, generate a commentary corresponding to the guide area to be inserted into the video. Specifically, the guide point cloud data corresponding to the guide area can first be image recognized using a deep learning model to identify the type of objects in the guide area; then, the type of the guide area can be determined according to the type of objects in the identified guide area. For example, the object type can be used to identify the probability of corresponding to a certain type of guide area, so as to determine the type of the guide area according to the type with the highest probability. Finally, the corresponding voice commentary can be generated from a pre-stored voice database according to the identified type of the guide area, and the generated voice commentary can be inserted when playing the video of the corresponding guide area.

[0068] The video generation device according to the embodiment of the present invention can obtain the navigation point cloud data of each navigation area by segmenting the three-dimensional point cloud data of the three-dimensional space, and automatically calculate the navigation points for the navigation area through the information of the navigation channel of the navigation area, and automatically obtain the navigation path accordingly to generate the video for the three-dimensional space. The video generation method of the present invention can provide a method for automatically generating a navigation video for the three-dimensional space, improve the efficiency of video production, and promote the intelligent development in the virtual reality scene.

[0069] Below, refer to Fig.10 A video generating apparatus according to an embodiment of the present invention will be described. Fig.10 FIG. 1 is a block diagram of a video generating apparatus 1000 according to an embodiment of the present invention. Fig.10As shown, the device 1000 may be a computer or a server.

[0070] like Fig.10 As shown, the video generating device 1000 includes one or more processors 1010 and a memory 1020. Of course, in addition to this, the video generating device 1000 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.10 The components and structure of the video generating device 1000 shown are merely exemplary and non-limiting. The video generating device 1000 may also have other components and structures as required.

[0071] The processor 1010 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1020 to perform desired functions, which may include: acquiring three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; acquiring a navigation channel of each navigation area based on the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area; acquiring a navigation path of the three-dimensional space based on one or more navigation points of each navigation area in the three-dimensional space, and generating the video using the three-dimensional point cloud data and the navigation path.

[0072] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may run the program instructions to implement the functions of the video generation device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the video generation method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0073] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining three-dimensional point cloud data of a three-dimensional space, the three-dimensional space including one or more navigation areas; performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; obtaining a navigation channel for each navigation area based on the navigation point cloud data of each navigation area, and calculating one or more navigation points for each navigation area; obtaining a navigation path for the three-dimensional space based on one or more navigation points for each navigation area in the three-dimensional space, and generating the video using the three-dimensional point cloud data and the navigation path.

[0074] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0075] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0076] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0077] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.

[0078] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0079] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0080] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0081] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0082] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0083] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0084] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0085] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0086] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0087] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0088] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0089] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0090] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A video generation method, comprising: Acquire three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; Performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; Acquire a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculate one or more navigation points of each navigation area; A navigation path in the three-dimensional space is acquired according to one or more navigation points in each navigation area in the three-dimensional space, and the video is generated using the three-dimensional point cloud data and the navigation path.

2. The method of claim 1, wherein: Obtaining 3D point cloud data in 3D space includes: At least a portion of the three-dimensional point cloud data of the three-dimensional space is acquired according to at least one of Lidar, Kinect, structured light, and multi-view stereo vision.

3. The method of claim 1, wherein: Performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space includes: Edge detection is performed on the three-dimensional point cloud data to obtain corner point data corresponding to each navigation area in the three-dimensional space, so as to generate navigation point cloud data for each navigation area.

4. The method of claim 1, wherein: Obtaining the navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area include: The navigation projection data corresponding to the navigation point cloud data is acquired, and image segmentation is performed on the navigation projection data to obtain the navigation channel.

5. The method of claim 4, wherein: Acquiring the navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculating one or more navigation points of each navigation area also includes: The spatial center of gravity of the navigation channel is calculated, and the calculated spatial center of gravity is used as the navigation point of the navigation area.

6. The method of claim 1, wherein: Acquiring a navigation path in the three-dimensional space according to one or more navigation points in each navigation area in the three-dimensional space includes: Path planning is performed on one or more navigation points in each navigation area in the three-dimensional space to obtain the order of all navigation points and generate a navigation path in the three-dimensional space.

7. The method of claim 1, wherein: Generating the video using the three-dimensional point cloud data and the navigation path includes: A plurality of viewpoints are determined on the navigation path, a video frame is formed for each viewpoint according to the three-dimensional point cloud data, and the video frames are combined to generate the video.

8. The method of claim 1, wherein: The method further comprises: Identifying a type of one or more navigation areas in the three-dimensional space; According to the identified type of the navigation area, a commentary corresponding to the navigation area is generated to be inserted into the video.

9. A video generating device, comprising: An acquisition unit configured to acquire three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; a segmentation unit configured to perform region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; A calculation unit, configured to obtain a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculate one or more navigation points of each navigation area; The generating unit is configured to obtain a navigation path of the three-dimensional space according to one or more navigation points of each navigation area in the three-dimensional space, and generate the video using the three-dimensional point cloud data and the navigation path.

10. A video generating device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Acquire three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; Performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; Acquire a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculate one or more navigation points of each navigation area; A navigation path in the three-dimensional space is acquired according to one or more navigation points in each navigation area in the three-dimensional space, and the video is generated using the three-dimensional point cloud data and the navigation path.

11. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Acquire three-dimensional point cloud data of a three-dimensional space, wherein the three-dimensional space includes one or more navigation areas; Performing region segmentation on the three-dimensional point cloud data to obtain navigation point cloud data of each navigation area in the three-dimensional space; Acquire a navigation channel of each navigation area according to the navigation point cloud data of each navigation area, and calculate one or more navigation points of each navigation area; A navigation path in the three-dimensional space is acquired according to one or more navigation points in each navigation area in the three-dimensional space, and the video is generated using the three-dimensional point cloud data and the navigation path.

Images