Method, device, medium and equipment for implementing interactive panoramic video streaming map

Abstract

The present application relates to a method, apparatus, medium, and equipment for implementing an interactive panoramic video stream map, and relates to the technical field of panoramic street view maps, wherein the method comprises: obtaining a map track point selection instruction sent by a terminal of a target user; determining a target road section where a target track point corresponding to the map track point selection instruction is located, and filtering target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section; determining a target playback time node corresponding to the panoramic image of the target track point in the target panoramic video stream data; obtaining a panoramic video playback instruction sent by the terminal, and according to the panoramic video playback instruction, playing and displaying the target panoramic video stream data on the terminal starting from the target playback time node. The present application has the effect of improving the smoothness of roaming in a panoramic street view map.

Description

Technical Field

[0001] The present application relates to the technical field of panoramic street view maps, and in particular to a method, apparatus, medium, and device for implementing an interactive panoramic video stream map. Background Art

[0002] Panoramic Street View is a real-world mapping service that provides users with 360-degree panoramic images of cities, streets, and other environments. This mapping service uses advanced image acquisition and processing technology to capture and stitch together multiple high-definition images to create a continuous, complete 360-degree panoramic image. Panoramic Street View allows users to explore and experience every corner of a real-world environment, providing an immersive map browsing experience.

[0003] At present, the method commonly used to collect panoramic images is: a street view map collection vehicle drives along the road, uses a panoramic camera, and exposes according to the time interval set by the collection personnel or a fixed driving distance, and finally collects a panoramic image. However, due to the influence of the exposure cycle of the panoramic camera during collection, there is a certain spatial interval between two adjacent panoramic images. When users browse the panoramic street view map, there will be a sense of jumpiness in the picture, resulting in poor smoothness of roaming in the panoramic street view map. Summary of the Invention

[0004] In order to improve the smoothness of roaming on a panoramic street view map, the present application provides a method, apparatus, medium, and device for implementing an interactive panoramic video streaming map.

[0005] In a first aspect of the present application, a method for implementing an interactive panoramic video stream map is provided, specifically comprising:

[0006] Obtaining the map track point selection instruction sent by the target user's terminal;

[0007] Determining a target road section where a target track point corresponding to the map track point selection instruction is located, and selecting target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, wherein the panoramic video stream data is a video composed of consecutive panoramic image frames of the corresponding road section;

[0008] Determine the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data;

[0009] Obtain a panoramic video playback instruction sent by the terminal, and according to the panoramic video playback instruction, play and display the target panoramic video stream data on the terminal starting from the target playback time node.

[0010] By adopting the above technical solution, after obtaining the map track point selection instruction, the corresponding target track point is determined, thereby determining the location point on the map where the target user wants to view the panoramic image image, and then determining the target road section where the target track point is located, so as to facilitate the determination of the target panoramic video stream data to be played. Further, the target playback time node corresponding to the panoramic image image of this target track point in the target panoramic video stream data is determined, so that the position where the target panoramic video stream data starts to play meets the panoramic map browsing needs of the target user. Finally, after the target user confirms to play the target panoramic video stream data, the target panoramic video stream data is played from the target playback time node, so that the target user starts browsing from the panoramic image image of the target track point that he wants to browse in the form of video playback, thereby improving the smoothness of roaming of the panoramic street view map and avoiding the jumping feeling of the picture when browsing the panoramic street view map.

[0011] Optionally, before filtering the target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, the method further includes:

[0012] Acquire a driving track of the map collection vehicle and a panoramic image of each collection track point in the driving track;

[0013] Generating overall panoramic video stream data corresponding to the driving trajectory according to a preset frame rate based on the panoramic image of each captured trajectory point;

[0014] The driving trajectory is divided into at least one road section, and the panoramic video stream data corresponding to each road section is filtered from the overall panoramic video stream data.

[0015] By adopting the above technical solution, the panoramic image images of all collected trajectory points are used to generate overall panoramic video stream data that matches the driving trajectory according to the preset frame rate. Then, the panoramic video stream data corresponding to each road section is selected from the overall panoramic video stream data, so as to facilitate the subsequent rapid determination of the panoramic video stream data that needs to be played based on the road section to which the position selected by the target user in the driving trajectory belongs.

[0016] Optionally, the determining of the panoramic image of the target trajectory point further includes, before the target playback time node corresponding to the target panoramic video stream data:

[0017] Selecting the GNSS time of all the collected track points in the track database corresponding to the driving track, and synchronizing the GNSS time of the same collected track point with the corresponding playback time node of the panoramic image in the overall panoramic video stream data;

[0018] The determining of the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data specifically includes:

[0019] Searching the target GNSS time corresponding to the target trajectory point from the trajectory database;

[0020] A playback time node synchronized with the target GNSS time is determined as a target playback time node.

[0021] By adopting the above technical solution, the target GNSS time of the target trajectory point selected by the target user is determined, and based on the synchronization relationship between the GNSS time of the same collected trajectory point and the playback time node of the corresponding panoramic image in the overall panoramic video stream data, the playback time node synchronized with the target GNSS time is determined, that is, the playback time node of the panoramic image corresponding to the target trajectory point, so as to quickly and accurately find the panoramic image corresponding to the target trajectory point in the target panoramic video stream data, and ensure the accuracy of the positioning of the panoramic image that starts to be played in the target panoramic video stream data.

[0022] Optionally, the method further includes:

[0023] Obtaining a panoramic video pause instruction sent by the terminal, and determining a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data when a panoramic image measurement function is enabled;

[0024] Selecting a target image number of a target key frame image from a trajectory database corresponding to the driving trajectory, wherein the trajectory database includes image numbers and corresponding camera pose data of panoramic images captured at different trajectory points in each of the road sections, wherein the camera pose data is pose data of a panoramic camera used to capture the panoramic image, and the target key frame image is a key frame image that is closest to the paused panoramic image in terms of image frame number among the key frame images;

[0025] Based on the target image number, a target depth image associated with the target key frame image is determined, and based on the target depth image and camera pose data corresponding to the paused panoramic image, a distance between points to be measured in the paused panoramic image is determined.

[0026] By adopting the above technical solution, if the target user pauses the playback of the target panoramic video stream data and turns on the panoramic image measurement function, it means that the target user wants to measure the straight-line distance between any two points in the panoramic image corresponding to the current pause in the real three-dimensional space. Then, the target key frame image closest to the paused panoramic image is selected, and finally, the geodetic coordinates of the points to be measured in the paused panoramic image are calculated based on the target depth image associated with the target key frame image and the camera posture data corresponding to the paused panoramic image, and finally the distance between the points to be measured in the real three-dimensional space is determined, thereby realizing the target user's experience of the panoramic image measurement function.

[0027] Optionally, before obtaining the panoramic video pause instruction sent by the terminal, the method further includes:

[0028] Selecting key track points at a preset track distance from each of the collected track points, and determining the panoramic image corresponding to the key track point as a key frame image;

[0029] Converting the point cloud data of the key frame image from the local coordinate system of the laser radar to the global coordinate system using the camera posture data corresponding to the key frame image to obtain converted point cloud data;

[0030] Projecting the converted point cloud data onto the image plane of the panoramic camera to obtain a camera image;

[0031] Calculate the depth value corresponding to each pixel in the camera image, and generate a depth image of the corresponding key frame image based on the depth value.

[0032] By adopting the above technical solution, the panoramic image of some of the collected trajectory points among all the collected trajectory points is determined as the key frame image image, and then the converted point cloud data is determined, so that the point cloud data is consistent with the position and direction of the panoramic camera in the world coordinate system. The converted point cloud data is then projected onto the image platform of the panoramic camera to obtain a camera image in a two-dimensional plane, which facilitates the subsequent generation of a depth image corresponding to the camera image, and further realizes the measurement of the distance between pixel points in the panoramic image.

[0033] Optionally, the method further includes:

[0034] Obtaining a panoramic video pause instruction sent by the terminal, and determining the type of the point of interest to be marked when a panoramic image point of interest marking function is enabled;

[0035] Determining a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data, and marking points of interest corresponding to the point of interest type in the paused panoramic image based on the marking instruction sent by the terminal to obtain marked data;

[0036] In a case where the point of interest type belongs to an urban governance type, the annotation data is set to be exportable.

[0037] By adopting the above technical solution, the marking instructions sent by the terminal are obtained, the points of interest of the target user in the current paused panoramic image are marked, and the corresponding marking data is obtained. If the point of interest type of the marked point of interest belongs to the urban governance type, marking the point of interest in the paused panoramic image can facilitate urban governance-related personnel to mark objects related to urban governance in the paused panoramic image and export the marking data, thereby providing effective data support for refined urban governance.

[0038] Optionally, the method further includes:

[0039] Obtaining a new panoramic video pause instruction sent by other terminals of other users, and determining a new paused panoramic image corresponding to the new panoramic video pause instruction;

[0040] Determine a first similarity between the user portrait of the other user and the target user portrait; if the first similarity exceeds a preset similarity threshold, display the annotation data of the target user in the new paused panoramic image, and send a annotation supplement reminder to the other terminal, wherein the target user portrait is a historical user portrait with the points of interest marked with the urban governance type;

[0041] If the first similarity does not exceed the similarity threshold, then based on the second similarity between the historical user portrait of the marked non-urban governance type point of interest and the user portrait of the other users, and in descending order of the second similarity, the historical annotation data of the corresponding historical user portrait are displayed in the new paused panoramic image. The higher the second similarity, the higher the display priority.

[0042] By adopting the above technical solution, if the first similarity exceeds the preset similarity threshold, it means that the user portraits of other users are closer to the target user portrait and are more concerned about the points of interest related to the urban governance type in the panoramic image. Then the target user's annotation data on this new paused panoramic image will be displayed, and a annotation supplement reminder will be sent to other terminals to remind other users to supplement and improve the annotation data on the new paused panoramic image, so that the annotation data of the points of interest about urban governance will be more complete; if the first similarity does not exceed the similarity threshold, it means that other users are concerned about points of interest other than urban governance. The historical annotation data corresponding to the corresponding historical user portraits will be displayed in this new paused panoramic image in order from high to low according to the second similarity. The higher the second similarity, the closer the other user is to the corresponding historical user portrait, the better the reference effect of the corresponding historical annotation data, and the higher the priority of display, thereby providing a better reference effect for other users.

[0043] In a second aspect of the present application, a device for implementing an interactive panoramic video stream map is provided, specifically comprising:

[0044] The instruction acquisition module is used to obtain the map track point selection instruction sent by the terminal of the target user;

[0045] a video filtering module, configured to determine a target road section where a target track point corresponding to the map track point selection instruction is located, and to filter target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, wherein the panoramic video stream data is a video composed of consecutive panoramic image frames of the corresponding road section;

[0046] A node determination module, configured to determine a target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data;

[0047] The video playback module is used to obtain the panoramic video playback instruction sent by the terminal, and according to the panoramic video playback instruction, play and display the target panoramic video stream data on the terminal starting from the target playback time node.

[0048] By adopting the above technical solution, the instruction acquisition module obtains the map track point selection instruction, the video screening module determines the target road section where the target track point is located, and filters the target panoramic video stream data from the panoramic video stream data of each road section, and then the node determination module determines the target playback time node corresponding to the panoramic image of the target track point in the target panoramic video stream data, and finally the video playback module starts playing and displaying from the target playback time node of the target panoramic video stream data.

[0049] In a third aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps as described in any one of the first aspects are performed.

[0050] In a fourth aspect of the present application, an electronic device is provided, specifically comprising:

[0051] A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the processor is used to load and execute the computer program stored in the memory so that the electronic device performs the method as described in any one of the first aspects.

[0052] In summary, the present application includes at least one of the following beneficial technical effects: after obtaining the map track point selection instruction, the corresponding target track point is determined, thereby determining the location point on the map where the target user wants to view the panoramic image, and then determining the target road section where the target track point is located, thereby facilitating the determination of the target panoramic video stream data to be played. Further, the target playback time node corresponding to the panoramic image of this target track point in the target panoramic video stream data is determined, so that the position where the target panoramic video stream data starts to play meets the target user's panoramic map browsing needs. Finally, after the target user confirms to play the target panoramic video stream data, the target panoramic video stream data is played from the target playback time node, so that the target user starts browsing from the panoramic image of the target track point he wants to browse in the form of video playback, thereby improving the smoothness of roaming in the panoramic street view map and avoiding the jumping feeling of the picture when browsing the panoramic street view map. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of a method for implementing an interactive panoramic video streaming map provided by an embodiment of the present application;

[0054] Figure 2 This is a flowchart of another method for implementing an interactive panoramic video stream map provided by an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of the structure of a device for implementing an interactive panoramic video stream map provided by an embodiment of the present application;

[0056] Figure 4 This is a structural diagram of another device for implementing an interactive panoramic video stream map provided in an embodiment of the present application.

[0057] Explanation of the accompanying drawings: 11. Instruction acquisition module; 12. Video screening module; 13. Node determination module; 14. Video playback module; 15. Road segment division module; 16. Synchronous processing module; 17. Image measurement module; 18. Depth determination module; 19. Interest annotation module; 20. Annotation display module. DETAILED DESCRIPTION

[0058] In order to enable people skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0059] In the description of the embodiments of this application, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0060] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0061] See also Figure 1 The present application discloses a flowchart of a method for implementing an interactive panoramic video stream map. The method can be implemented using a computer program or run on a device for implementing an interactive panoramic video stream map based on a von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application, specifically including:

[0062] S101: Obtaining a map track point selection instruction sent by a terminal of a target user.

[0063] Specifically, the target user is a user who wants to browse the street panorama on the client of the panoramic street view map. The terminal is a personal computer. In other embodiments, it can also be a smart phone. The terminal is installed with the client of the panoramic street view map. The terminal is directly or indirectly connected to the server through a wired or wireless network. The server is the background server of the client, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. In the embodiment of the present application, the server is the executor of the method for realizing the interactive panoramic video streaming map. The map track point selection instruction is the selection instruction corresponding to the map track point where the target user wants to view the street panoramic image. The map track point is the location point in the path track displayed in the map.

[0064] One feasible method for obtaining a map track point selection instruction is: the target user clicks a location in the client's path trajectory on the terminal with the left mouse button, i.e., a map track point, indicating that the target user wants to view the panoramic image of this location. The client sends the generated map track point selection instruction to the server via the Software Development Kit (SDK), and the server ultimately obtains the map track point selection instruction. It should be noted that the map track point selection instruction carries the location information of the map track point.

[0065] S102: Determine the target road section where the target track point corresponding to the map track point selection instruction is located, and filter the target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section.

[0066] Specifically, after receiving a map track point selection instruction, it is necessary to determine the map track point corresponding to the map track point selection instruction, i.e., the target road segment on the map where the target track point is located. The specific process is as follows: A map collection vehicle's driving trajectory, i.e., the route along which panoramic images are collected, is obtained. The map collection vehicle is a vehicle used to collect map information data and is equipped with a panoramic camera, a lidar, and a global navigation satellite system. The panoramic camera is a photographic device capable of capturing a 360-degree view. The panoramic camera simultaneously captures panoramic images of each collected track point on the driving trajectory. A collected track point is a location within the driving trajectory where the panoramic image is collected. Furthermore, urban road network data is obtained from the Digital Earth Open Platform. Based on the road segment information in this urban road network data, the driving trajectory is divided into sections, resulting in at least one road segment. Each section corresponds to a divided track. Furthermore, based on the track where the target track point is located, the target road segment on the map where the target track point is located is determined.

[0067] In addition, a trajectory database corresponding to the driving trajectory is established. The trajectory database includes the section name of each road section, the image number of the panoramic image of each collected trajectory point in each road section, the Global Navigation Satellite System (GNSS) time, location information, and camera attitude data. GNSS time refers to the precise time node when the collection trajectory point is located using the Global Navigation Satellite System. The location information of the collection trajectory point is provided by the Global Navigation Satellite System. The camera attitude data refers to the attitude data of the panoramic camera when collecting the panoramic image of the corresponding collection trajectory point. The camera attitude data can be determined by the panoramic camera's built-in sensors such as the gyroscope and accelerometer.

[0068] Furthermore, the preset video editing software Adobe Premiere Pro generates overall panoramic video stream data corresponding to the driving trajectory based on the panoramic images at each captured trajectory point at a preset frame rate. In other embodiments, the video editing software Final Cut Pro can also be used to generate the overall panoramic video stream data. Furthermore, the panoramic video stream data corresponding to each road segment is filtered out from the overall panoramic video stream data, where the panoramic video stream data is a video composed of consecutive panoramic image frames of the corresponding road segment. In other embodiments, the panoramic video stream data corresponding to each road segment can also be located based on the image number of the panoramic image at each captured trajectory point in the road segment, and the panoramic video stream data for the corresponding road segment is generated separately based on each panoramic image. Finally, the target panoramic video stream data corresponding to the target road segment is filtered out from the panoramic video stream data of each road segment based on the road segment name. It should be noted that the panoramic image is a three-dimensional spatial image captured at the corresponding captured trajectory point using wide-angle technology and displayed in a two-dimensional format.

[0069] S103: Determine the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data.

[0070] Specifically, after the target panoramic video stream data is determined, to avoid a sense of screen jumpiness when the target user browses the panoramic image starting from the target track point and moving forward, the GNSS time of all collected track points in the driving trajectory is first selected from the trajectory database corresponding to the driving trajectory. Then, the panoramic image corresponding to the same collected track point is synchronized with the corresponding GNSS time at the playback time node of the overall panoramic video stream data. Specifically, synchronization can be achieved using the preset video processing software DaVinci Resolve. That is, a correspondence is established between the GNSS time of each collected track point in the trajectory database and the corresponding playback time node in the overall panoramic video stream data.

[0071] Furthermore, the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data is determined. A feasible determination method is: to search for the target GNSS time corresponding to this target trajectory point from all GNSS times in the trajectory database corresponding to the driving trajectory, and finally to determine the playback time node synchronized with the target GNSS time as the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data.

[0072] S104: Obtain the panoramic video play instruction sent by the terminal, and play and display the target panoramic video stream data on the terminal starting from the target play time node according to the panoramic video play instruction.

[0073] Specifically, after the target playback time node corresponding to the target trajectory point selected by the target user is determined, the server sends a reminder pop-up window to the target user's terminal to remind the target user whether to start playing the panoramic image video from the current target trajectory point. If the panoramic video playback instruction sent by the target user's terminal is obtained, it means that the target user wants to browse the panoramic street view from the target trajectory point along the driving trajectory direction, then the server finds the corresponding panoramic image from the target panoramic video stream data according to the target playback time node and renders it, and then starts playing this target panoramic video stream data from the target playback time node, and displays it through the target user's terminal. Specifically, it is displayed on the terminal through streaming media transmission technology and synchronous display mechanism, so that when the target user browses the panoramic street view in the map, the picture is smoother through video playback, avoiding the jumping feeling of the picture.

[0074] In other embodiments, after step S104, the method further includes: obtaining a panoramic video pause instruction sent by the terminal, and determining a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data when the panoramic image measurement function is enabled;

[0075] Selecting a target image number of a target key frame image from a trajectory database corresponding to the driving trajectory;

[0076] Based on the target image number, a target depth image associated with the target keyframe image is determined, and based on the target depth image and the camera posture data corresponding to the paused panoramic image, the distance between the points to be measured in the paused panoramic image is determined.

[0077] Specifically, while browsing the target panoramic video stream, the target user clicks the pause button with their mouse. Accordingly, the server receives the panoramic video pause instruction sent by the terminal. If the target user also clicks the "Panoramic Image Measurement Function" button on the terminal interface, where panoramic image measurement refers to measuring the straight-line distance between any two points in a panoramic image in real three-dimensional space, indicating that the target user wishes to measure the straight-line distance between any two points in the panoramic image corresponding to the current pause, the panoramic image corresponding to the current pause time in the target panoramic video stream is determined as the paused panoramic image. Furthermore, since the trajectory database includes the image number of the panoramic image captured at each trajectory point in each road segment, the target image number of the target keyframe image is selected from the trajectory database. Specifically, based on the current pause time, the image number of the panoramic image corresponding to the captured trajectory point is searched in the trajectory database. Since panoramic images with adjacent image frames also have adjacent image numbers, the target image number is determined. Furthermore, the corresponding target keyframe image is located based on the target image number, and its associated target depth image is determined. The target keyframe image is the keyframe image that is closest to the paused panoramic image in terms of image frame number among the keyframe images. The keyframe image is a panoramic image associated with a depth image. In this embodiment of the application, the depth image is an image that uses the distance from the panoramic camera to each point in the street scene as a pixel value.

[0078] Furthermore, the pixel point that the target user wants to measure in the paused panoramic image, i.e., the point to be measured, is determined, and the image coordinates of the point to be measured are obtained from the target depth image. Then, the geodetic coordinates of the point to be measured are determined based on the image coordinates and the camera posture data corresponding to the paused panoramic image. The specific process is: converting the image coordinates from the image coordinate system to the three-dimensional camera coordinate system, and then converting the image coordinates from the three-dimensional camera coordinate system to the world coordinate system based on the camera posture data corresponding to the paused panoramic image, and finally obtaining the corresponding geodetic coordinates. Among them, the geodetic coordinates are a three-dimensional coordinate system used to describe the position of a point on the surface of the earth, which consists of longitude, latitude and altitude. Furthermore, based on the geodetic coordinates of the point to be measured, the Euclidean distance formula is used to calculate the straight-line distance between the points to be measured in the real three-dimensional space, thereby quickly assisting the target user to know the actual distance of the point to be measured in the actual street scene.

[0079] Before receiving the panoramic video pause command sent by the terminal, the process also includes: selecting key track points from each captured track point in the driving trajectory at intervals of a preset track distance, and determining the panoramic image corresponding to the key track points as key frame images. The preset track distance is set based on human experience. Next, camera pose data corresponding to the key frame images is obtained from the trajectory database, and point cloud data corresponding to the scene in the key frame images is obtained using the laser radar (LiDAR) onboard the map collection vehicle. The point cloud data is primarily used to construct a three-dimensional model of the environment, enabling accurate spatial positioning and environmental perception. The point cloud data is then converted from the local coordinate system of the LiDAR to a global coordinate system based on the camera pose data corresponding to the key frame images, obtaining converted point cloud data. Furthermore, the converted point cloud data is projected onto the image plane of the panoramic camera to obtain a camera image. The specific process involves converting the converted point cloud data to a coordinate system specific to the panoramic camera using the camera's external parameters, and then projecting the converted point cloud data onto the panoramic camera's image plane using the camera's internal parameters to obtain a camera image. Specifically, the three-dimensional coordinates of the point cloud data are converted into two-dimensional pixel coordinates. Finally, the Open Source Computer Vision Library (OpenCV) tool is used to determine the depth value corresponding to each pixel in the camera image. Finally, the perspective projection method is used to project each depth value to the corresponding pixel point in the camera image to obtain the depth image corresponding to the key frame image.

[0080] See also Figure 2 The present application discloses a flowchart of a method for implementing an interactive panoramic video stream map. The method can be implemented using a computer program or run on a device for implementing an interactive panoramic video stream map based on a von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application, specifically including:

[0081] S201: Obtaining a map track point selection instruction sent by a terminal of a target user.

[0082] S202: Determine the target road section where the target track point corresponding to the map track point selection instruction is located, and filter the target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section.

[0083] S203: Determine the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data.

[0084] S204: Obtain the panoramic video play instruction sent by the terminal, and play and display the target panoramic video stream data on the terminal starting from the target play time node according to the panoramic video play instruction.

[0085] For details, please refer to steps S101-S104, which will not be described in detail here.

[0086] S205: Obtain the panoramic video pause instruction sent by the terminal, and determine the type of the interest point to be marked when the panoramic image interest point marking function is enabled.

[0087] S206: Determine the paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data, and mark the points of interest corresponding to the point of interest type in the paused panoramic image based on the marking instruction sent by the terminal to obtain marked data.

[0088] S207: When the point of interest type belongs to the urban governance type, the annotation data is set to be exportable.

[0089] Specifically, when the target user is browsing the target panoramic video stream data being played, he clicks the pause button with his mouse. Accordingly, the server receives the panoramic video pause instruction sent by the terminal, pauses the playback of the target panoramic video stream data, and determines the panoramic image corresponding to the paused playback time node as the paused panoramic image. If the target user also clicks the "Panoramic Image Interest Point Annotation Function" button on the terminal interface with his mouse, where panoramic image interest point annotation refers to marking and noting objects of interest in the panoramic image, then the type of interest point to be annotated sent by the terminal is obtained. For example, the interest point type can be a street style type, and the interest points corresponding to this interest point type can include guardrails, municipal boxes, abandoned utility poles, etc.; the interest point type can also be a roof style type, and the corresponding specific interest points can include solar equipment, photovoltaic panels, etc. Further, the annotation instruction sent by the terminal is obtained, and the interest points corresponding to the interest point type in the current paused panoramic image are annotated to obtain corresponding annotation data. In this embodiment of the application, the annotation data includes the annotation number, geodetic coordinates, etc. of the corresponding interest point.

[0090] Furthermore, it is determined whether this point of interest type belongs to the urban governance type, that is, the type related to urban landscape governance. Query whether the point of interest type exists from the preset urban governance type distribution table. The urban governance type distribution table includes different point of interest types belonging to the urban governance type, which are all set based on human experience. If it exists, it indicates whether this point of interest type belongs to the urban governance type. Then, the annotation data is set to be exportable, so that urban governance-related personnel can mark objects related to urban governance in the paused panoramic image and export the annotation data, thereby providing effective data support for refined urban governance. Among them, in the embodiment of the present application, urban governance mainly refers to the governance of urban landscape.

[0091] In other embodiments, if a new panoramic video pause instruction is received from another terminal of another user, the panoramic image corresponding to the new panoramic video pause instruction in the overall panoramic video stream data is determined as the new paused panoramic image. The other terminal may be a smartphone or a personal computer.

[0092] Furthermore, to determine the user portraits of other users, a feasible way to determine the user portraits is to obtain the historical search terms of other users on the client of this panoramic street view map, and input the historical search terms into the trained portrait determination model, so as to finally determine the user portraits of other users. The portrait determination model can adopt a convolutional neural network model. For example, if there are many search terms about restaurants in the historical search terms of other users, then the corresponding user portrait is likely to be a food lover. Furthermore, the first similarity between the user portraits of other users and the target user portrait, that is, the historical user portraits of the points of interest marked with the urban governance type, is calculated. The specific process is: mapping the user portrait and the target user portrait into vectors under multiple dimensions, calculating the cosine similarity between the two vectors and determining it as the first similarity. The greater the first similarity, the more similar the user portrait is to the target user portrait. If the first similarity exceeds the preset similarity threshold, it means that the user portraits of other users are closer to the target user portrait and they are more concerned about the points of interest related to the urban governance type in the panoramic image. Then the target user's annotation data on this new paused panoramic image will be displayed, and a annotation supplement reminder will be sent to other terminals to remind other users to supplement and improve the annotation data on the new paused panoramic image, so that the annotation data of the points of interest about urban governance will be more complete. Finally, a annotation data completion reminder will be sent to the target user's terminal, so that the target user can export more complete annotation data, thereby strengthening its data support role in providing refined urban governance.

[0093] If the first similarity does not exceed the similarity threshold, indicating that other users are interested in points of interest other than urban governance, then historical user portraits that have been marked with points of interest other than urban governance are selected from the historical user portraits that have been marked with points of interest, and the second similarity between them and the user portraits of other users is calculated. Finally, the historical annotation data corresponding to the corresponding historical user portraits are displayed in this new paused panoramic image in descending order of the second similarity, thereby providing a better reference effect for other users. The higher the second similarity, the closer the other user is to the corresponding historical user portrait, and the better the reference effect of the corresponding historical annotation data, and the higher the priority displayed. For example, if the other user is a food lover, the earlier the historical annotation data of the historical user portrait with a high probability of also being a food lover is displayed (which may include annotation data for restaurants with good reputations), the more reference value it will provide to other users.

[0094] In another embodiment, before playing the target panoramic video stream data, the number of users who have annotated each panoramic image frame in the target panoramic video stream data is counted. That is, users who match the user profile of the target user are selected from each panoramic image frame in descending order of the number of annotated users. Then, the annotated points of interest in each of the selected panoramic image frames that appear more than a threshold number of times are determined and determined as target points of interest, that is, points of interest that are likely to be of interest to the target user. Finally, the playback time node corresponding to each panoramic image frame containing the target point of interest is set as a short pause time node with a short pause duration of 1 second, and the annotated data of the target point of interest is synchronously displayed. This allows the annotated data of the target point of interest that may be of interest to the target user to be presented during the playback of the target panoramic video stream data, thereby enhancing the target user's experience of browsing the panoramic map.

[0095] The implementation principle of the method for realizing an interactive panoramic video stream map in the embodiment of the present application is as follows: after obtaining the map track point selection instruction, determine the corresponding target track point, thereby determining the location point on the map where the target user wants to view the panoramic image image, and then determine the target road section where the target track point is located, so as to facilitate the determination of the target panoramic video stream data to be played. Further, determine the target playback time node corresponding to the panoramic image image of this target track point in the target panoramic video stream data, so that the position where the target panoramic video stream data starts to play meets the panoramic map browsing needs of the target user. Finally, after the target user confirms to play the target panoramic video stream data, start playing this target panoramic video stream data from the target playback time node, so that the target user starts browsing from the panoramic image image of the target track point that he wants to browse in the form of video playback, thereby improving the smoothness of roaming of the panoramic street view map and avoiding the jumping feeling of the picture when browsing the panoramic street view map.

[0096] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0097] See Figure 3 , which is a schematic diagram of the structure of an apparatus for implementing an interactive panoramic video stream map according to an embodiment of the present application. The apparatus for implementing an interactive panoramic video stream map can be implemented as all or part of the apparatus through software, hardware, or a combination of both. The apparatus includes an instruction acquisition module 11, a video screening module 12, a node determination module 13, and a video playback module 14.

[0098] The instruction acquisition module 11 is used to obtain the map track point selection instruction sent by the terminal of the target user;

[0099] The video filtering module 12 is used to determine the target road section where the target track point corresponding to the map track point selection instruction is located, and to filter the target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, where the panoramic video stream data is a video composed of consecutive panoramic image frames of the corresponding road section;

[0100] A node determination module 13 is used to determine a target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data;

[0101] The video playing module 14 is used to obtain the panoramic video playing instruction sent by the terminal, and play and display the target panoramic video stream data on the terminal starting from the target playing time node according to the panoramic video playing instruction.

[0102] Optional, such as Figure 4 As shown, the device further includes a road segment division module 15, which is specifically configured to:

[0103] Acquire a driving track of the map collection vehicle and a panoramic image of each collection track point in the driving track;

[0104] Generate the overall panoramic video stream data corresponding to the driving trajectory according to the preset frame rate based on the panoramic image of each collected trajectory point;

[0105] The driving trajectory is divided into at least one road segment, and the panoramic video stream data corresponding to each road segment is filtered from the overall panoramic video stream data.

[0106] Optionally, the device further includes a synchronization processing module 16, specifically configured to:

[0107] The GNSS time of all collected track points in the track database corresponding to the driving track is selected, and the GNSS time of the same collected track point is synchronized with the playback time node of the corresponding panoramic image in the overall panoramic video stream data.

[0108] Optionally, the node determination module 13 is specifically configured to:

[0109] Find the target GNSS time corresponding to the target trajectory point from the trajectory database;

[0110] A playback time node synchronized with the target GNSS time is determined as the target playback time node.

[0111] Optionally, the device further includes an image measurement module 17, specifically configured to:

[0112] Obtaining a panoramic video pause instruction sent by the terminal, and determining a paused panoramic image corresponding to the panoramic video pause instruction from target panoramic video stream data when a panoramic image measurement function is enabled;

[0113] Selecting a target image number of a target key frame image from a trajectory database corresponding to the driving trajectory, wherein the trajectory database includes image numbers and corresponding camera pose data of panoramic images at different acquisition trajectory points in each road section, wherein the camera pose data is pose data of a panoramic camera that captures the panoramic image, and the target key frame image is a key frame image that is closest to the paused panoramic image in terms of image frame number among the key frame images;

[0114] Based on the target image number, a target depth image associated with the target keyframe image is determined, and based on the target depth image and the camera posture data corresponding to the paused panoramic image, the distance between the points to be measured in the paused panoramic image is determined.

[0115] Optionally, the device further includes a depth determination module 18, specifically configured to:

[0116] Select key track points from each collected track point at a preset track distance, and determine the panoramic image corresponding to the key track point as a key frame image;

[0117] Using the camera pose data corresponding to the key frame image, the point cloud data of the key frame image is converted from the local coordinate system of the lidar to the global coordinate system to obtain the converted point cloud data;

[0118] Projecting the converted point cloud data onto the image plane of the panoramic camera to obtain the camera image;

[0119] Calculate the depth value corresponding to each pixel in the camera image, and generate a depth image of the corresponding key frame image based on the depth value.

[0120] Optionally, the device further includes an interest marking module 19, specifically configured to:

[0121] Obtaining a panoramic video pause instruction sent by the terminal, and determining the type of the points of interest to be marked when the panoramic image point of interest marking function is turned on;

[0122] Determine a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data, and mark points of interest corresponding to the point of interest type in the paused panoramic image based on the marking instruction sent by the terminal to obtain marked data;

[0123] When the point of interest type belongs to urban governance type, the labeled data is set to be exportable.

[0124] Optionally, the device further includes a marking display module 20, specifically configured to:

[0125] Obtaining a new panoramic video pause instruction sent by other terminals of other users, and determining a new paused panoramic image corresponding to the new panoramic video pause instruction;

[0126] Determine a first similarity between the user portraits of other users and the target user portrait. If the first similarity exceeds a preset similarity threshold, display the target user's annotation data in the new paused panoramic image and send a reminder for annotation supplement to other terminals. The target user portrait is a historical user portrait of the points of interest marked with the urban governance type;

[0127] If the first similarity does not exceed the similarity threshold, the historical user portraits of the marked non-urban governance type points of interest and the user portraits of other users will be displayed in the new paused panoramic image in order of the second similarity from high to low. The higher the second similarity, the higher the display priority.

[0128] It should be noted that the above embodiment provides a device for realizing an interactive panoramic video stream map, and when executing the method for realizing an interactive panoramic video stream map, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the above embodiment provides a device for realizing an interactive panoramic video stream map and a method embodiment for realizing an interactive panoramic video stream map, which belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0129] An embodiment of the present application further discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, a method for implementing an interactive panoramic video stream map according to the above embodiment is adopted.

[0130] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that computer-readable medium includes but is not limited to the above-mentioned components.

[0131] Among them, through this computer-readable storage medium, a method for realizing an interactive panoramic video streaming map in the above embodiment is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0132] An embodiment of the present application also discloses an electronic device, in which a computer program is stored in a computer-readable storage medium. When the computer program is loaded and executed by a processor, the above-mentioned method for realizing an interactive panoramic video stream map is adopted.

[0133] Among them, the electronic device can be an electronic device such as a desktop computer, a laptop computer or a cloud server, and the electronic device includes but is not limited to a processor and a memory. For example, the electronic device can also include input and output devices, network access devices and buses, etc.

[0134] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0135] Among them, the memory can be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device, or it can be an external storage device of the electronic device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) or flash memory card (FC) equipped on the electronic device. In addition, the memory can also be a combination of an internal storage unit and an external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or is to be output. This application does not impose any restrictions on this.

[0136] Among them, through this electronic device, a method for realizing an interactive panoramic video stream map in the above embodiment is stored in the memory of the electronic device, and is loaded and executed on the processor of the electronic device for easy use.

[0137] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for realizing an interactive panoramic video stream map, characterized in that: The method comprises: Acquire a driving track of the map collection vehicle and a panoramic image of each collection track point in the driving track; Generating overall panoramic video stream data corresponding to the driving trajectory according to a preset frame rate based on the panoramic image of each captured trajectory point; Dividing the driving trajectory into at least one road segment, and filtering the panoramic video stream data corresponding to each road segment from the overall panoramic video stream data; Obtaining the map track point selection instruction sent by the target user's terminal; Determining a target road section where a target track point corresponding to the map track point selection instruction is located, and selecting target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, wherein the panoramic video stream data is a video composed of consecutive panoramic image frames of the corresponding road section; Determine the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data; Obtaining a panoramic video playback instruction sent by the terminal, and playing and displaying the target panoramic video stream data on the terminal starting from the target playback time node according to the panoramic video playback instruction; Obtaining a panoramic video pause instruction sent by the terminal, and determining a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data when a panoramic image measurement function is enabled; Selecting a target image number of a target key frame image from a trajectory database corresponding to the driving trajectory, wherein the trajectory database includes image numbers and corresponding camera pose data of panoramic images captured at different trajectory points in each of the road sections, wherein the camera pose data is pose data of a panoramic camera used to capture the panoramic image, and the target key frame image is a key frame image that is closest to the paused panoramic image in terms of image frame number among the key frame images; Based on the target image number, a target depth image associated with the target key frame image is determined, and based on the target depth image and camera pose data corresponding to the paused panoramic image, a distance between points to be measured in the paused panoramic image is determined.

2. The method for realizing an interactive panoramic video stream map according to claim 1, characterized in that: The step of determining the panoramic image of the target trajectory point further includes, before the target playback time node corresponding to the target panoramic video stream data: Selecting the GNSS time of all the collected track points in the track database corresponding to the driving track, and synchronizing the GNSS time of the same collected track point with the corresponding playback time node of the panoramic image in the overall panoramic video stream data; The determining of the target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data specifically includes: Searching the target GNSS time corresponding to the target trajectory point from the trajectory database; A playback time node synchronized with the target GNSS time is determined as a target playback time node.

3. The method for realizing an interactive panoramic video stream map according to claim 1, characterized in that: Before obtaining the panoramic video pause instruction sent by the terminal, the method further includes: Selecting key track points at a preset track distance from each of the collected track points, and determining the panoramic image corresponding to the key track point as a key frame image; Converting the point cloud data of the key frame image from the local coordinate system of the laser radar to the global coordinate system using the camera posture data corresponding to the key frame image to obtain converted point cloud data; Projecting the converted point cloud data onto the image plane of the panoramic camera to obtain a camera image; Calculate the depth value corresponding to each pixel in the camera image, and generate a depth image of the corresponding key frame image based on the depth value.

4. The method for realizing an interactive panoramic video stream map according to claim 1, characterized in that: The method further comprises: Obtaining a panoramic video pause instruction sent by the terminal, and determining the type of the point of interest to be marked when a panoramic image point of interest marking function is enabled; Determining a paused panoramic image corresponding to the panoramic video pause instruction from the target panoramic video stream data, and marking points of interest corresponding to the point of interest type in the paused panoramic image based on the marking instruction sent by the terminal to obtain marked data; In a case where the point of interest type belongs to an urban governance type, the annotation data is set to be exportable.

5. The method for realizing an interactive panoramic video stream map according to claim 4, characterized in that: The method further comprises: Obtaining a new panoramic video pause instruction sent by other terminals of other users, and determining a new paused panoramic image corresponding to the new panoramic video pause instruction; Determine a first similarity between the user portrait of the other user and the target user portrait; if the first similarity exceeds a preset similarity threshold, display the annotation data of the target user in the new paused panoramic image, and send a annotation supplement reminder to the other terminal, wherein the target user portrait is a historical user portrait with the points of interest marked with the urban governance type; If the first similarity does not exceed the similarity threshold, then based on the second similarity between the historical user portrait of the marked non-urban governance type point of interest and the user portrait of the other users, and in descending order of the second similarity, the historical annotation data of the corresponding historical user portrait are displayed in the new paused panoramic image. The higher the second similarity, the higher the display priority.

6. A device for implementing an interactive panoramic video stream map, for implementing the method according to any one of claims 1 to 5, characterized in that: include: An instruction acquisition module (11) is used to acquire a map track point selection instruction sent by a terminal of a target user; A video screening module (12) is used to determine the target road section where the target track point corresponding to the map track point selection instruction is located, and to screen target panoramic video stream data corresponding to the target road section from the panoramic video stream data of each road section, wherein the panoramic video stream data is a video composed of continuous panoramic image frames of the corresponding road section; A node determination module (13) is used to determine a target playback time node corresponding to the panoramic image of the target trajectory point in the target panoramic video stream data; The video playing module (14) is used to obtain the panoramic video playing instruction sent by the terminal, and play and display the target panoramic video stream data on the terminal starting from the target playing time node according to the panoramic video playing instruction.

7. A computer-readable storage medium storing a computer program, wherein: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 5 is adopted.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method according to any one of claims 1 to 5 is adopted.

Images