VR panoramic image output method, VR panoramic live broadcast method, and interaction method

Abstract

The present application relates to the technical field of virtual reality. Provided are a VR panoramic image output method, a VR panoramic live broadcast method and an interaction method. The output method comprises: on the basis of geographic space information of an actual object in a shooting range and parameter information of a VR panoramic camera, obtaining a first coordinate by means of a VR panoramic image position mapping model; on the basis of VR panoramic image recognition, obtaining a second coordinate of an object appearing in the VR panoramic image; matching the first coordinate with the second coordinate, and after the matching succeeds, matching the actual object corresponding to the first coordinate with the object appearing in the VR panoramic image and corresponding to the second coordinate; and if the matching succeeds, determining to display the VR image of the actual object in the VR panoramic image. The present application achieves real-time sharing and intercommunication of information of actual objects in a real space and a virtual space, and fuses geographic space information of actual objects and coordinate information in VR panoramic images for secondary calibration, thereby improving the recognition accuracy for VR panoramic images.

Description

VR panoramic image output method, VR panoramic live broadcast method and interactive method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410092796.1 and application date of January 22, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of virtual reality technology, and in particular to a method for outputting VR panoramic images, a method for VR panoramic live broadcasting, and an interactive method. Background Art

[0004] Currently, VR panoramic live streaming, a combination of live video and VR technology, is the primary form of VR content. When users watch through a VR headset, they can freely choose any angle to experience an immersive experience.

[0005] In related technologies, target recognition technology for panoramic images relies on traditional planar video target recognition algorithms. Since panoramic images are spherical in panoramic live broadcasts, they are distorted after being expanded into 2D images. Therefore, the recognition accuracy of existing technologies is low and the reliability is insufficient. Summary of the Invention

[0006] The embodiments of the present application provide a VR panoramic image output method, a VR panoramic live broadcast method, and an interactive method to solve the technical problem of VR panoramic image recognition accuracy.

[0007] In a first aspect, an embodiment of the present application provides a method for outputting a VR panoramic image, comprising:

[0008] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0009] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0010] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0011] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0012] In one embodiment, obtaining at least one first coordinate output by the VR panoramic image position mapping model based on real-time geospatial information of at least one actual object within a shooting range and VR panoramic camera parameter information through the VR panoramic image position mapping model includes:

[0013] Performing feature preprocessing and feature combination on the geospatial information of at least one real object and the VR panoramic camera parameter information, wherein the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, and vectorization;

[0014] The geographic space information and VR panoramic camera parameter information after feature preprocessing and feature combination are input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model.

[0015] In one embodiment, the structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

[0016] In one embodiment, identifying and obtaining, based on the VR panoramic image, at least one second coordinate corresponding to each object appearing in the VR panoramic image, includes:

[0017] Inputting the VR panoramic image into a video object recognition algorithm to identify and obtain relevant information about objects appearing in the VR panoramic image;

[0018] The relevant information includes the second coordinates corresponding to the object and at least one of the following:

[0019] The frame number of the VR panoramic image where the VR image of the object is located;

[0020] the identification of the object;

[0021] the category of the object;

[0022] dimensional data of the object.

[0023] In a second aspect, an embodiment of the present application provides a VR panoramic live broadcast method, the method comprising:

[0024] Continuously executing the first process multiple times to output a VR panoramic live stream in real time, wherein the VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0025] The first process includes:

[0026] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0027] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0028] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0029] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0030] In a third aspect, an embodiment of the present application provides a VR panoramic live broadcast system, the system comprising one or more VR panoramic live broadcast subsystems, each VR panoramic live broadcast subsystem corresponding to its own shooting range;

[0031] The VR panoramic live broadcast subsystem is configured to execute the aforementioned VR panoramic live broadcast method.

[0032] In a fourth aspect, an embodiment of the present application provides an interactive method based on the VR panoramic live broadcast system provided in the third aspect, which is applied to an electronic device that plays VR panoramic live broadcasts, and the method further includes:

[0033] receiving a first input from a user, where the first input is an operation by the user selecting target coordinates, a target object, or a target identifier in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame in a VR panoramic live stream played in real time by the electronic device, and the VR panoramic live stream played in real time by the electronic device is output by a first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0034] In response to the first input, a VR panoramic live broadcast stream output by a second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems is played, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinates or target object or target identifier.

[0035] In one embodiment, the VR panoramic image in the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem includes: coordinates or objects or identifiers corresponding to the shooting range of the first VR panoramic live broadcast subsystem.

[0036] In a fifth aspect, an embodiment of the present application provides an interactive method based on the VR panoramic live broadcast system provided in the third aspect, which is applied to an electronic device that plays VR panoramic live broadcasts, and the method further includes:

[0037] receiving a second input from a user, where the second input is an operation by the user selecting target coordinates, a target store, or a target product in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0038] In response to the second input, a jump link corresponding to the target coordinates or target store or target product is output, and / or a VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinates or target store or target product as the main viewing angle is played.

[0039] In a sixth aspect, an embodiment of the present application provides an interactive method based on the VR panoramic live broadcast system provided in the third aspect, which is applied to an electronic device that plays VR panoramic live broadcasts, and the method further includes:

[0040] receiving a third input from a user, the third input being an operation of the user checking in or leaving a message after selecting target coordinates or a target object in the VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0041] In response to the third input, the user's check-in information or message information is displayed in an area corresponding to the target coordinates or target object in the VR panoramic image currently displayed by the electronic device.

[0042] In one embodiment, the check-in information or message information of the user or other users is displayed in an area corresponding to the coordinates or objects included in the VR panoramic image currently displayed by the electronic device.

[0043] In a seventh aspect, an embodiment of the present application provides a device for outputting a VR panoramic image, comprising:

[0044] a first coordinate acquisition module configured to obtain, based on real-time geospatial information of at least one actual object within a shooting range and parameter information of the VR panoramic camera, at least one first coordinate output by the VR panoramic image position mapping model through a VR panoramic image position mapping model, wherein the at least one first coordinate is a coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0045] A second coordinate acquisition module is configured to identify and obtain at least one second coordinate corresponding to each object appearing in the VR panoramic image based on the VR panoramic image;

[0046] a matching module configured to match the at least one first coordinate with the at least one second coordinate, and to match an actual object corresponding to the successfully matched first coordinate with an object appearing in the VR panoramic image corresponding to the second coordinate;

[0047] The image determination module is configured to determine, when the matching is successful, to display the VR image of the successfully matched actual object in the VR panoramic image.

[0048] In an eighth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory storing a computer program, wherein when the processor executes the program, it implements the steps of the method for outputting VR panoramic images provided in the first aspect, or implements the steps of the method for outputting VR panoramic images provided in the second aspect, or implements the steps of the interactive method provided in the fourth aspect, the fifth aspect, or the sixth aspect.

[0049] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for outputting VR panoramic images provided in the first aspect, or implements the steps of the method for VR panoramic live broadcast provided in the second aspect, or implements the steps of the interactive method provided in the fourth aspect, the fifth aspect, or the sixth aspect.

[0050] The VR panoramic image output method, VR panoramic live broadcast method and interactive method provided in the embodiments of the present application realize real-time sharing and communication of information of actual objects in real space and virtual space by constructing a correlation mapping between the real-time geographic spatial information of actual objects and the VR panoramic image, and integrate the geographic spatial information of actual objects and the coordinate information in the VR panoramic image for secondary calibration, thereby improving the recognition accuracy in the VR panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] FIG1 is a schematic diagram of a flow chart of a method for outputting a VR panoramic image according to an embodiment of the present application;

[0053] FIG2 is a structural diagram of a VR panoramic live broadcast system provided in an embodiment of the present application;

[0054] FIG3 is a flow chart of an interactive method applied to a VR panoramic live broadcast system according to an embodiment of the present application;

[0055] FIG4 is a second flow chart of an interactive method applied to a VR panoramic live broadcast system provided in an embodiment of the present application;

[0056] FIG5 is a third flow chart of an interactive method applied to a VR panoramic live broadcast system provided in an embodiment of the present application;

[0057] FIG6 is a schematic diagram of the structure of a device for outputting a VR panoramic image according to an embodiment of the present application;

[0058] FIG7 illustrates a schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0060] First, let’s introduce the following contents:

[0061] (1) The fifth generation of mobile communication technology (5G) boasts many innovations, including high speed, large data traffic, low latency, and multiple connections. In the 5G era, the high-speed and low-latency transmission characteristics are expected to significantly enhance the user experience of virtual reality (VR) products. 5G+ cloud rendering significantly improves application display effects, reduces hardware costs, and promotes the popularization of VR. Technological innovation continues to achieve breakthroughs, continuously promoting product upgrades, and further clearing obstacles for the popularization of VR.

[0062] The VR content ecosystem is becoming more complete, and a vast application market is opening up: With the gradual implementation of VR applications, the practicality of VR / AR as a tool has gradually become more prominent, and it has shown considerable application potential in the video, marketing, music and game markets, and is expected to promote revolutionary changes in the next generation of interactive entertainment industry.

[0063] (2) VR panoramic live broadcast;

[0064] VR panoramic live streaming is a combination of live video and VR technology, and is the primary form of VR content. When users watch through a VR headset, they can freely choose any angle to experience an immersive experience.

[0065] VR panoramic live broadcast relies on professional equipment for shooting. After capturing images from multiple angles at the same time, it still needs to undergo stitching and other processing to ensure that each frame is a 360-degree panoramic picture.

[0066] When watching, users are placed in the center of the spherical area and can freely rotate the camera angle to watch the dynamic video 360 degrees, without being restricted by time, space, or region, giving them an immersive experience. Panoramic live broadcasts incorporate depth of field, dynamic images, sound, and other factors, with features such as sound and picture alignment and synchronization, creating a strong sense of immersion.

[0067] Panoramic live broadcasts require a dedicated panoramic player, which dynamically adjusts itself based on data from a gesture sensor and is then attached to the surface of a sphere. Using a traditional video player to play panoramic live broadcasts will result in distortion and render the content inaccessible.

[0068] In related technologies, VR panoramic live broadcast mainly focuses on single panoramic live broadcast, and a small number of solutions simply superimpose some interactive operations, such as virtual gifts and barrage, which only superimpose virtual objects in the virtual space. In related technologies, VR panoramic live broadcast solutions have the following shortcomings:

[0069] 1. Single VR panoramic live broadcast is a virtual space created by live streaming images, which lacks interactivity and entertainment, and has low user retention and activity;

[0070] 2. Panoramic live broadcasts with superimposed virtual gifts and bullet comments are similar to traditional 2D videos. They lack entertainment and innovation, fail to make good use of the 3D spatial characteristics of VR panoramic live broadcasts, and lack appeal to users.

[0071] The relevant technologies do not combine the real space geographic information to achieve the fusion and interaction of VR panoramic live broadcast virtual space and real space.

[0072] In related technologies, VR panoramic live broadcast technology mainly builds a single VR panoramic live broadcast. If you need to know the mapping relationship between a certain pixel point in the panoramic live broadcast and the real space position, you can only rely on manual operation. Manual operation has the following disadvantages:

[0073] 1. Large labor input;

[0074] 2. The number of mapping relationships that can be annotated is limited;

[0075] 3. When the VR live broadcast camera position changes or the target object position changes, the mapping relationship cannot be automatically updated.

[0076] In related technologies, target recognition technology for panoramic images relies on traditional planar video target recognition algorithms. Since panoramic images are spherical in panoramic live broadcasts, they are distorted after being expanded into 2D images. Therefore, the recognition accuracy of existing technologies is low and the reliability is insufficient.

[0077] Therefore, the present application provides a VR panoramic image output method, a VR panoramic live broadcast method and an interactive method, which can improve the recognition accuracy of VR panoramic images.

[0078] FIG1 is a flow chart of a method for outputting a VR panoramic image according to an embodiment of the present application. As shown in FIG1 , the method may include the following steps:

[0079] Step 100: Based on real-time geospatial information of at least one actual object within a shooting range and VR panoramic camera parameter information, a VR panoramic image position mapping model is used to obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image. The VR panoramic image is a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time.

[0080] In an embodiment of the present application, a VR panoramic image position mapping model is constructed to establish a mapping relationship between the real-time geographic spatial information of the actual object and the coordinate information of the VR panoramic image, so that the position of the actual object in the VR panoramic live broadcast is obtained by predicting its geographic spatial coordinates.

[0081] The real-time geographic spatial information of the actual object may include: precision coordinates, latitude coordinates, and altitude.

[0082] Specifically, VR panoramic images are spherical images that compress the 3D world. Therefore, there is a certain mapping relationship between the real-time geospatial information of real objects and the coordinate information of the VR panoramic image. This relationship model is affected by internal and external parameters such as the VR camera position, focal length, pixel resolution, and principal distance.

[0083] The input of the VR panoramic image position mapping model can include two parts: one is the real-time geospatial information of at least one actual object, and the other is the VR panoramic camera parameter information. The real-time geospatial information of the actual object can include: precision coordinates, dimensional coordinates, and altitude. The VR panoramic camera parameter information can include: VR camera position, focal length, pixel resolution, main distance and other internal and external parameters;

[0084] Optionally, the real-time geographic spatial information of at least one actual object and the VR panoramic camera parameter information may be subjected to feature preprocessing and then input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model, i.e., the coordinates of the VR image of the at least one actual object in the VR panoramic image predicted by the VR panoramic image position mapping model, such as output obj_geo i The coordinates of the lower left corner in the panoramic image frame (geo_x i , geo_y i ).

[0085] It should be noted that the VR panoramic image position mapping actual object model can be pre-trained.

[0086] It should be noted that the actual object model mapped to the VR panoramic image position can be based on the historical geographic spatial information of at least one actual object and the VR panoramic camera parameter information, and can be obtained through iterative learning and training. All types of neural networks that can predict the coordinates of the VR image of the at least one actual object in the VR panoramic image after training are applicable to the embodiments of the present application and are not limited here.

[0087] Step 110: Based on the VR panoramic image, identify and obtain at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0088] Optionally, a sequence of VR panoramic images can be extracted frame by frame from the VR panoramic live stream generated by the VR panoramic camera, and marked in frame order to obtain {pic k ,k=0,1,2……}.

[0089] The embodiment of the present application can adopt a video target recognition algorithm to identify the category, coordinate position, and size of objects in the VR panoramic image based on the timing information of the previous and next frames of the VR panoramic image, wherein the coordinate position is at least one second coordinate corresponding to each object appearing in the VR panoramic image.

[0090] Step 120: Match the at least one first coordinate with the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0091] Optionally, based on the first coordinate of at least one actual object obtained in step 100 and the at least one second coordinate and related information (such as object category and / or size) corresponding to the objects appearing in the VR panoramic image obtained in step 110, the position of the actual object in the real space in the VR panoramic image is confirmed, and an association mapping of the actual object in the virtual space and the real space is constructed.

[0092] Specifically, the following steps may be included:

[0093] (1) With the VR panoramic camera (such as VR panoramic live broadcast camera) as the center, circle the actual objects within a certain distance range (i.e. shooting range) {obj geoi The selected distance is determined by the range of the environment captured by the VR panoramic camera and the range of the object of interest.

[0094] (2) Based on the map information, obtain the real-time geographic spatial information of the actual objects within the shooting range, and obtain the obj_geo information of the actual objects within the shooting range through the VR panoramic image position mapping model. i Position coordinates in VR panoramic live broadcast image (geo_x i , geo_y i ).

[0095] (3) The first coordinate of at least one actual object obtained in step 100 and the at least one second coordinate and related information (such as object category and / or size) corresponding to the object appearing in the VR panoramic image obtained in step 110 are correlated and matched to obtain a more accurate coordinate of the object obj in the VR panoramic live image. k VR image, and its position coordinates (x k 、y k ).

[0096] Traverse each object obj_det that appears in the VR panoramic image j , in the area with an upper and lower error of 20%, find out whether there is (geo_x i , geo_y i ) falls into it. If it exists, match obj_det j and obj_geo i Is the object category information consistent? If so, confirm that the object obj is present in the VR panoramic live broadcast. k , obj k The information of obj is as follows:k =obj_geo i ; x k =(geo_x i +det_x j ) / 2; y k =(geo_x k +det_x k ) / 2;

[0097] obj k The relevant information is the same as obj_geo i , such as: building name, attraction name, store name, rating, etc.

[0098] If there is a 1-to-many or many-to-1 match, the point with the closer distance will prevail.

[0099] Step 130 : If the matching is successful, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

[0100] The embodiment of the present application is based on the real-time geographic spatial information of actual objects and the associated mapping in the VR panoramic image, so as to realize the real-time sharing and communication of information of actual objects in the real space and the virtual space, and realize the interaction between reality and virtuality.

[0101] The embodiment of the present application proposes a method for constructing a panoramic live broadcast with virtual-real fusion interaction, which associates the target object in the VR live broadcast with the real space mapping to achieve virtual-real space fusion interaction.

[0102] An embodiment of the present application proposes a method for mapping geographic space coordinates to VR panoramic image positions. By constructing a mapping model of geographic coordinates and VR panoramic images, a mapping relationship between the real-space geographic coordinate information of a target object and the VR panoramic image position information is established, thereby obtaining the position of the target object in the VR panoramic image through its geographic space coordinates.

[0103] This embodiment of the present application proposes a method for mapping geospatial coordinates to VR panoramic image positions. Compared to related technologies that rely solely on manual operations, this method has the following advantages:

[0104] (1) Automatically build mapping relationships without additional manual input;

[0105] (2) The mapping relationship that can be annotated can be accurate to the pixel level, and even objects that are very close or overlap in VR panoramic images can be accurately mapped to the real space;

[0106] (3) When the position of the VR panoramic camera changes or the position of the target object changes, the mapping relationship can be automatically updated;

[0107] (4) The mapping relationship is updated and optimized in combination with the target recognition results, making the results more accurate.

[0108] The embodiment of the present application proposes a method for target recognition in VR panoramic images. Compared with the target recognition technology for panoramic images in related technologies (which relies on traditional planar video target recognition algorithms), it has higher accuracy and can be applied to panoramic live broadcasts. Panoramic images are spherical and are distorted when expanded into 2D images. The recognition accuracy of related technologies is low and the reliability is insufficient. The embodiment of the present application incorporates the actual object geospatial information for secondary calibration to ensure improved recognition accuracy.

[0109] The output method of VR panoramic images proposed in the embodiment of the present application realizes real-time sharing and communication of information of actual objects in real space and virtual space by constructing a correlation mapping between the real-time geographic spatial information of actual objects and the VR panoramic image, and integrates the coordinate information of the actual objects in the VR panoramic image into a secondary calibration to improve the recognition accuracy in the VR panoramic image.

[0110] In one embodiment, obtaining at least one first coordinate output by the VR panoramic image position mapping model based on real-time geospatial information of at least one actual object within a shooting range and VR panoramic camera parameter information through the VR panoramic image position mapping model includes:

[0111] Performing feature preprocessing and feature combination on the geospatial information of at least one real object and the VR panoramic camera parameter information, wherein the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, and vectorization;

[0112] The geographic space information and VR panoramic camera parameter information after feature preprocessing and feature combination are input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model.

[0113] Specifically, the input of the VR panoramic image position mapping model can include two parts: one is the real-time geospatial information of at least one actual object, and the other is the VR panoramic camera parameter information. The real-time geospatial information of the actual object can include: precision coordinates, dimensional coordinates, and altitude. The VR panoramic camera parameter information can include: VR camera position, focal length, pixel resolution, main distance and other internal and external parameters;

[0114] Optionally, the real-time geographic spatial information of at least one actual object and the VR panoramic camera parameter information may be subjected to feature preprocessing and then input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model, i.e., the coordinates of the VR image of the at least one actual object in the VR panoramic image predicted by the VR panoramic image position mapping model, such as output obj_geo i The coordinates of the lower left corner in the panoramic image frame (geo_x i , geo_y i ); wherein, feature preprocessing includes: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization, and feature combination based on experience, wherein the feature combination can be constructed according to user needs and experience, and can include: the azimuth angle and distance between the target object and the VR camera, etc., which can reflect the relative position relationship between the target object and the VR camera; for example, it can include the azimuth angle and distance between the actual object and the VR camera; the embodiment of the present application does not limit this.

[0115] Optionally, the VR panoramic image position mapping model can integrate a neural network model and a linear regression model. The linear regression model mainly relies on artificial features and gives full play to the advantages of historical experience; the neural network model has stronger feature reasoning and generalization capabilities, which fully supplements the deficiencies of artificial experience features; the fusion of the two further improves the accuracy of prediction.

[0116] Optionally, the output of the VR panoramic image position mapping model may include: predicting the actual object obj_geo i The coordinates of the lower left corner in the panoramic image frame (geo_x i , geo_y i ).

[0117] In one embodiment, the structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

[0118] It should be noted that the VR panoramic image position mapping model can integrate a neural network model and a linear regression model. The linear regression model mainly relies on artificial features and gives full play to the advantages of historical experience; the neural network model has stronger feature reasoning and generalization capabilities, which fully supplements the deficiencies of artificial experience features; any method of integrating the neural network model and the linear regression model to predict the coordinates of the VR image of the at least one actual object in the VR panoramic image is applicable to the embodiments of this application and is not limited here.

[0119] For example, artificial features are features that are manually designed by domain experts or data scientists based on their understanding and expertise of the dataset. These features are usually created based on insights into the problem domain and knowledge of the data distribution, with the goal of capturing information that may be helpful for prediction tasks.

[0120] In the context of VR panoramic image position mapping models, artificial features may include:

[0121] Geometric features: Geometric properties such as lines, angles, and shapes in panoramic images. These features can help identify structural content in images.

[0122] Statistical features: For example, statistical information such as the color histogram, brightness, and contrast of an image. These features can describe the global properties of the image.

[0123] Transformation features: Features obtained by rotating, scaling, filtering, and other operations on the image. These transformations can enhance the model's adaptability to different perspectives and scales.

[0124] Texture features: The local texture information of an image, such as features extracted by the gray-level co-occurrence matrix (GLCM), can reflect subtle changes in the image surface.

[0125] Contextual features: Features related to the location and relationship of objects in the image, such as the distance and relative position between objects.

[0126] User interaction features: If VR panoramic images contain user interaction data, such as click-through rate, browsing time, etc., these can also serve as useful features.

[0127] Depth information: If available, depth sensors or depth maps generated by stereo vision algorithms can provide additional spatial position information.

[0128] Semantic features: high-level semantic information obtained through image annotation or segmentation, such as scene labels, object categories, etc.

[0129] For example, to fuse a neural network model and a linear regression model to build a VR panoramic image position mapping model, the following steps can be taken:

[0130] Feature extraction: Use neural network models to extract useful features from VR panoramic image samples. These features can be visual information such as image texture, color, edges, etc., or deeper abstract representations.

[0131] Linear regression analysis: After extracting enough features, a linear regression model can be used to analyze the mapping relationship between these features and image locations. Artificial features can also be constructed.

[0132] Model fusion strategy: Features extracted by the neural network model and artificial features are fed into a linear regression model for position mapping. This fusion can be achieved through linear weighting, for example, by taking a weighted sum of the features from different neural networks. Alternatively, more complex fusion methods, such as stacking, can be used to achieve better prediction results by combining multiple sub-learners with their respective strengths.

[0133] Multimodal fusion methods: Considering that VR panoramic images may contain multiple types of data (such as images, depth information, and user interactions), multimodal fusion methods can be used. These include model-independent methods and model-based methods. The former does not directly rely on specific deep learning methods, while the latter explicitly utilizes deep learning models to solve the multimodal fusion problem.

[0134] Training and Optimization: During model training, the parameters of the neural network and linear regression models need to be adjusted to minimize the difference between the predicted and actual locations. This process may require the use of complex optimization algorithms, and because training deep neural networks is typically very time-consuming, it is necessary to rationally allocate computing resources.

[0135] Evaluation and Iteration: Evaluate the performance of the fusion model through experiments and continuously iterate and improve the model based on the evaluation results. This may include adjusting the feature extraction strategy, the selection of fusion methods, and the optimization of model parameters.

[0136] In summary, fusing a neural network model with a linear regression model to construct a VR panoramic image position mapping model is a complex process involving multiple steps, including feature extraction, model fusion, multimodal data processing, and optimization. Through careful design and continuous iteration, an efficient model can be achieved that both captures image details and accurately predicts position.

[0137] In one embodiment, identifying and obtaining, based on the VR panoramic image, at least one second coordinate corresponding to each object appearing in the VR panoramic image, includes:

[0138] Inputting the VR panoramic image into a video object recognition algorithm to identify and obtain relevant information about objects appearing in the VR panoramic image;

[0139] The relevant information includes the second coordinates corresponding to the object and at least one of the following:

[0140] The frame number of the VR panoramic image where the VR image of the object is located;

[0141] the identification of the object;

[0142] the category of the object;

[0143] dimensional data of the object.

[0144] Specifically, the sequence of VR panoramic images can be extracted frame by frame from the VR panoramic live stream generated by the VR panoramic camera, and marked in frame order to obtain {pic k , k=0,1,2……}; then, the key frames in the frame sequence are input into the video target recognition algorithm to identify the category, time, and position of the object appearing in the VR panoramic image; wherein, the video target recognition algorithm may include: Flow-based, YoloV, DFF, FGFA, etc., which can identify the category, time, and position of the object appearing in the VR panoramic image. The embodiment of the present application is not limited to this; taking the Flow-based video target recognition algorithm as an example, it can be recorded in the format shown in the following Table 1:

[0145] Table 1

[0146] Taking the flow-based video object recognition algorithm as an example, the embodiment of the present application adopts a flow-based video object detection algorithm, which has the following advantages over traditional image detection algorithms:

[0147] (1) Continuous VR panoramic images (such as live broadcasts) may contain blurred images due to object motion and camera defocus. There are also cases where the target object is blocked, the appearance changes due to posture changes, and the size changes due to distance. In these scenarios, it is necessary to use the timing information of the previous and next frames in the video to make inferences and predictions;

[0148] (2) The concept of key frames is introduced. Adjacent frames with similar appearance usually lead to similar features. Therefore, it is not necessary to calculate features in all frames, which improves detection efficiency.

[0149] (3) Use the detection features or detection results of the previous and next frames to optimize the detection effect of the current frame and improve the detection accuracy.

[0150] An embodiment of the present application proposes a target recognition method for VR panoramic images, which realizes target recognition in VR panoramic live broadcasts by matching and calibrating the results of a flow-based video target detection algorithm and the results of a VR panoramic live broadcast position mapping model.

[0151] In one embodiment, the present application provides a VR panoramic live broadcast method, the method comprising:

[0152] Continuously executing the first process multiple times to output a VR panoramic live stream in real time, wherein the VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0153] The first process includes:

[0154] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0155] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0156] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0157] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0158] That is, the aforementioned VR panoramic image output method is applied to a live broadcast scene, and the steps in the aforementioned VR panoramic image output method are executed multiple times in succession to form a real-time VR panoramic live broadcast stream.

[0159] The embodiment of the present application proposes a method for constructing a panoramic live broadcast with virtual-real fusion and interaction. By integrating geographic space information, the actual objects in the VR panoramic live broadcast are mapped and associated with the real space. Compared with the existing panoramic live broadcast construction method, it has the advantages of richer panoramic live broadcast information and higher interactivity, breaking the traditional independent virtual space of VR panoramic live broadcast and realizing the fusion and interaction of VR panoramic live broadcast virtual space and real space.

[0160] The embodiment of the present application proposes a method for constructing a panoramic live broadcast that integrates virtual and real interactions. By integrating geographic space coordinates with a VR panoramic live broadcast position mapping model and a panoramic image target recognition model, on the one hand, the accuracy of panoramic live broadcast target recognition is effectively improved, solving the shortcomings of the existing technology; on the other hand, it can automatically generate a mapping relationship between the pixel position of the panoramic live broadcast and the real space coordinates, thereby realizing real-time synchronous sharing of the virtual space information of the panoramic live broadcast and the object information of the real space, creating a panoramic live broadcast that integrates virtual and real interactions.

[0161] The method for constructing a panoramic live broadcast with interactive virtual-real integration proposed in the embodiment of the present application can, on the one hand, effectively improve the target recognition accuracy and solve the shortcomings of the existing technology by integrating geographic space coordinates with a VR panoramic live broadcast position mapping model and a panoramic image target recognition model; on the other hand, it can automatically generate a mapping relationship between the pixel position of the panoramic live broadcast and the real-space coordinates, thereby realizing real-time synchronous sharing of the virtual space information of the panoramic live broadcast and the object information of the real space, and creating a panoramic live broadcast with interactive virtual-real integration.

[0162] FIG2 is a structural diagram of a VR panoramic live broadcast system provided in an embodiment of the present application. As shown in FIG2 , the VR panoramic live broadcast system 200 includes one or more VR panoramic live broadcast subsystems 210 , each of which corresponds to its own shooting range;

[0163] The VR panoramic live broadcast subsystem is configured to execute the aforementioned VR panoramic live broadcast method.

[0164] It should be noted that each VR panoramic live broadcast subsystem 210 in the VR panoramic live broadcast system 200 can realize the VR panoramic live broadcast in the aforementioned VR panoramic live broadcast method embodiment within its own shooting range. For example, the main perspective of the shooting range of VR panoramic live broadcast subsystem A is the playground, with the teaching building and the cafeteria in the background; the main perspective of the shooting range of VR panoramic live broadcast subsystem B is the cafeteria, with the teaching building and the playground in the background. That is, the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem A is the live broadcast with the playground as the main perspective, that is, the live broadcast of objects and events in the playground; the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem B is the live broadcast with the cafeteria as the main perspective, that is, the live broadcast of objects and events in the cafeteria.

[0165] FIG3 is a flow chart of one of the interactive methods applied to a VR panoramic live broadcast system provided in an embodiment of the present application. As shown in FIG3 , an electronic device applied to play VR panoramic live broadcast, the method includes:

[0166] Step 300: Receive a first user input, where the first input is an operation by the user selecting target coordinates, a target object, or a target identifier in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame in a VR panoramic live stream played in real time by the electronic device, and the VR panoramic live stream played in real time by the electronic device is output by a first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0167] Step 310: In response to the first input, play the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinates or target object or target identifier.

[0168] The embodiments of the present application can intelligently construct multi-VR panoramic immersive interactive live broadcasts. For example, in panoramic live broadcast A, you can watch target point B. At the same time, target point B also has a panoramic live broadcast. A jump point to panoramic live broadcast B can be presented at the corresponding position of panoramic live broadcast A. This can automatically associate multiple panoramic live broadcasts and support mutual jumps. Users can select a jump point in multiple panoramic live broadcasts and multiple spaces through the first input, achieving space conversion similar to the real world in VR panoramic live broadcasts, and realizing more immersive interactive live broadcasts.

[0169] For example, the main perspective of the shooting range of VR panoramic live broadcast subsystem A is the playground, with the teaching building and cafeteria in the background; the main perspective of the shooting range of VR panoramic live broadcast subsystem B is the cafeteria, with the teaching building and playground in the background, that is, the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem A is the live broadcast with the playground as the main perspective, that is, the live broadcast of objects and events in the playground; the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem B is the live broadcast with the VR image of the cafeteria as the main perspective, that is, the live broadcast of objects and events in the cafeteria; the electronic device is currently playing the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem A, that is, the foreground content displayed in the display interface is the playground, and the background content includes the teaching building and the cafeteria. The user selects the VR image of the cafeteria through the first input, and the electronic device switches the live broadcast stream to the VR panoramic live broadcast stream output by VR panoramic live broadcast subsystem B, that is, the foreground content displayed in the display interface is switched to the cafeteria, and the background content includes the teaching building and the playground.

[0170] In one embodiment, the VR panoramic image in the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem includes: coordinates or objects or identifiers corresponding to the shooting range of the first VR panoramic live broadcast subsystem.

[0171] FIG4 is a second flow chart of an interactive method applied to a VR panoramic live broadcast system provided in an embodiment of the present application. As shown in FIG4 , an electronic device applied to play VR panoramic live broadcast, the method further includes:

[0172] Step 400: receiving a second input from a user, where the second input is an operation by the user selecting target coordinates, a target store, or a target product in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0173] Step 410: In response to the second input, output a jump link corresponding to the target coordinates or target store or target product, and / or play the VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinates or target store or target product as the main viewing angle.

[0174] The embodiment of the present application can intelligently construct VR panoramic live broadcast immersive cloud shopping. For example: if a store appears in the VR panoramic image of the VR panoramic live broadcast stream, and the user selects the VR image of the store through the second input, a cloud shopping jump link for the store can be provided near the coordinate position corresponding to the VR image of the store, or the live broadcast stream currently displayed by the electronic device is switched to the VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range of the store interior as the main viewing angle, that is, information shopping inside the panoramic live broadcast store. Therefore, users can not only see the real-time situation of the store in the panoramic live broadcast, but also achieve an immersive shopping experience and realize the integration of virtual space and real space.

[0175] FIG5 is a third flow chart of an interactive method applied to a VR panoramic live broadcast system provided in an embodiment of the present application. As shown in FIG5 , an electronic device applied to play VR panoramic live broadcast, the method further includes:

[0176] Step 500: Receive a third input from the user, wherein the third input is an operation of the user checking in or leaving a message after selecting target coordinates or a target object in the VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0177] Step 510: In response to the third input, the user's check-in information or message information is displayed in the area corresponding to the target coordinates or the target object in the VR panoramic image currently displayed by the electronic device.

[0178] The embodiments of the present application can intelligently achieve information sharing and synchronization between virtual and real spaces. For example, when a user is playing a VR panoramic live stream on an electronic device, they can check in or upload messages on the VR image corresponding to different locations on the VR panoramic image in the VR panoramic live stream. At the same time, the check-ins and messages of the user or other users at the same location in the real space will also be displayed at the mapped location on the VR panoramic image in the VR panoramic live stream, enabling user communication and information sharing between virtual and real spaces.

[0179] In one embodiment, the check-in information or message information of the user or other users is displayed in an area corresponding to the coordinates or objects included in the VR panoramic image currently displayed by the electronic device.

[0180] The embodiments of the present application have high application value and broad application prospects, and have rich application scenarios on the CHBN side.

[0181] For example, on the CH side: the embodiments of the present application can enrich the interactive forms of VR panoramic live broadcast, build innovative applications of virtual and real integration such as VR panoramic live broadcast interactive applications, VR immersive online shopping, VR immersive cloud tourism, and VR live broadcast check-in, improve the content richness and entertainment of VR panoramic live broadcast, and improve user stickiness and activity.

[0182] For example, on the B-side: the embodiments of the present application can be applied to many fields such as smart cultural tourism, smart communities, smart villages, smart cities, etc., providing VR panoramic live broadcast with interactive integration of virtual and real, building VR interactive live broadcast in multiple scenes, and creating immersive virtual space shopping, monitoring, roaming, etc.

[0183] For example, at the N end: the embodiment of the present application can realize VR live broadcast content innovation, integrate real scene information into VR live broadcast, and enhance the richness of VR content; through this technical solution, innovative virtual space applications can also be created, integrating and interacting VR panoramic live broadcast with real space.

[0184] The following describes a device for outputting a VR panoramic image provided in an embodiment of the present application. The device for outputting a VR panoramic image described below and the method for outputting a VR panoramic image described above may refer to each other.

[0185] FIG6 is a schematic diagram of the structure of a device for outputting a VR panoramic image according to an embodiment of the present application. As shown in FIG6 , the device 600 includes:

[0186] The first coordinate acquisition module 610 is configured to obtain, based on real-time geospatial information of at least one actual object within a shooting range and parameter information of the VR panoramic camera, at least one first coordinate output by the VR panoramic image position mapping model, wherein the at least one first coordinate is a coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, wherein the VR panoramic image is a VR panoramic image obtained by the VR panoramic camera in real time shooting the shooting range;

[0187] The second coordinate acquisition module 620 is configured to identify and obtain at least one second coordinate corresponding to each object appearing in the VR panoramic image based on the VR panoramic image;

[0188] A matching module 630 is configured to match the at least one first coordinate with the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0189] The image determination module 640 is configured to determine, when the matching is successful, to display the VR image of the successfully matched actual object in the VR panoramic image.

[0190] The output device for VR panoramic images in the embodiment of the present application can implement the various steps of the aforementioned VR panoramic image output method embodiment and achieve the same technical effects, which will not be repeated here.

[0191] The terminal involved in the embodiments of the present application may refer to a device that provides voice and / or data connectivity to a user, a handheld device with wireless connection capabilities, or other processing devices connected to a wireless modem. In different systems, the name of the terminal device may also vary. For example, in a 5G system, the terminal device may be called a user equipment (UE).

[0192] The network devices involved in the embodiments of the present application may be base stations, which may include multiple cells providing services to terminals. Depending on the specific application scenario, a base station may also be called an access point, or may be a device in an access network that communicates with wireless terminal devices over the air interface through one or more sectors, or may be called another name.

[0193] FIG7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG7 , the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. The processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call a computer program in the memory 730 to execute the steps of the aforementioned methods, for example, including:

[0194] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0195] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0196] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0197] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0198] or

[0199] Continuously executing the first process multiple times to output a VR panoramic live stream in real time, wherein the VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0200] The first process includes:

[0201] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0202] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0203] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0204] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0205] or,

[0206] receiving a first input from a user, where the first input is an operation by the user selecting target coordinates, a target object, or a target identifier in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame in a VR panoramic live stream played in real time by the electronic device, and the VR panoramic live stream played in real time by the electronic device is output by a first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0207] In response to the first input, a VR panoramic live broadcast stream output by a second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems is played, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinates or target object or target identifier.

[0208] or,

[0209] receiving a second input from a user, where the second input is an operation by the user selecting target coordinates, a target store, or a target product in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0210] In response to the second input, a jump link corresponding to the target coordinates or target store or target product is output, and / or a VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinates or target store or target product as the main viewing angle is played.

[0211] or,

[0212] receiving a third input from a user, the third input being an operation of the user checking in or leaving a message after selecting target coordinates or a target object in the VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0213] In response to the third input, the user's check-in information or message information is displayed in an area corresponding to the target coordinates or target object in the VR panoramic image currently displayed by the electronic device.

[0214] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0215] On the other hand, embodiments of the present application further provide a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the methods provided in the above embodiments, for example, including:

[0216] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0217] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0218] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0219] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0220] or

[0221] Continuously executing the first process multiple times to output a VR panoramic live stream in real time, wherein the VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0222] The first process includes:

[0223] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0224] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0225] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0226] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0227] or,

[0228] receiving a first input from a user, where the first input is an operation by the user selecting target coordinates, a target object, or a target identifier in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame in a VR panoramic live stream played in real time by the electronic device, and the VR panoramic live stream played in real time by the electronic device is output by a first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0229] In response to the first input, a VR panoramic live broadcast stream output by a second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems is played, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinates or target object or target identifier.

[0230] or,

[0231] receiving a second input from a user, where the second input is an operation by the user selecting target coordinates, a target store, or a target product in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0232] In response to the second input, a jump link corresponding to the target coordinates or target store or target product is output, and / or a VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinates or target store or target product as the main viewing angle is played.

[0233] or,

[0234] receiving a third input from a user, the third input being an operation of the user checking in or leaving a message after selecting target coordinates or a target object in the VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0235] In response to the third input, the user's check-in information or message information is displayed in an area corresponding to the target coordinates or target object in the VR panoramic image currently displayed by the electronic device.

[0236] On the other hand, an embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is configured to cause a processor to execute the steps of the methods provided in the above embodiments, for example, including:

[0237] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0238] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0239] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0240] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0241] or

[0242] Continuously executing the first process multiple times to output a VR panoramic live stream in real time, wherein the VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0243] The first process includes:

[0244] Based on real-time geospatial information of at least one actual object within a shooting range and parameter information of a VR panoramic camera, obtaining, using a VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model, the at least one first coordinate being a coordinate of a VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, the VR panoramic image being a VR panoramic image obtained by the VR panoramic camera capturing the shooting range in real time;

[0245] Based on the VR panoramic image, identifying and obtaining at least one second coordinate corresponding to each object appearing in the VR panoramic image;

[0246] Matching the at least one first coordinate with the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate with the object appearing in the VR panoramic image corresponding to the second coordinate;

[0247] In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

[0248] or,

[0249] receiving a first input from a user, where the first input is an operation by the user selecting target coordinates, a target object, or a target identifier in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame in a VR panoramic live stream played in real time by the electronic device, and the VR panoramic live stream played in real time by the electronic device is output by a first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0250] In response to the first input, a VR panoramic live broadcast stream output by a second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems is played, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinates or target object or target identifier.

[0251] or,

[0252] receiving a second input from a user, where the second input is an operation by the user selecting target coordinates, a target store, or a target product in a VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0253] In response to the second input, a jump link corresponding to the target coordinates or target store or target product is output, and / or a VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinates or target store or target product as the main viewing angle is played.

[0254] or,

[0255] receiving a third input from a user, the third input being an operation of the user checking in or leaving a message after selecting target coordinates or a target object in the VR panoramic image currently displayed by the electronic device, wherein the VR panoramic image currently displayed by the electronic device is a frame of a VR panoramic live stream played in real time by the electronic device;

[0256] In response to the third input, the user's check-in information or message information is displayed in an area corresponding to the target coordinates or target object in the VR panoramic image currently displayed by the electronic device.

[0257] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid-state drives (SSDs)), etc.

[0258] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0259] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for outputting a VR panoramic image, comprising: Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through a VR panoramic image position mapping model, obtaining at least one first coordinate output by the VR panoramic image position mapping model, where the at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively, and the VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time; Based on the VR panoramic image, identifying at least one second coordinate corresponding to the objects appearing in the VR panoramic image respectively; Matching the at least one first coordinate and the at least one second coordinate, and matching the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate; In the case of successful matching, determining to display the VR image of the successfully matched actual object in the VR panoramic image.

2. The output method of the VR panoramic image according to claim 1, wherein, The step of obtaining at least one first coordinate output by the VR panoramic image position mapping model based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information through the VR panoramic image position mapping model includes: Performing feature preprocessing and feature combination on the geospatial information of at least one actual object and the VR panoramic camera parameter information, where the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization; Inputting the geospatial information and VR panoramic camera parameter information after feature preprocessing and feature combination into the VR panoramic image position mapping model, and obtaining at least one first coordinate output by the VR panoramic image position mapping model.

3. The output method of the VR panoramic image according to claim 1 or 2, wherein, The structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

4. The output method of the VR panoramic image according to claim 1, wherein, The step of identifying at least one second coordinate corresponding to the objects appearing in the VR panoramic image respectively based on the VR panoramic image includes: Inputting the VR panoramic image into a video object recognition algorithm, and identifying the relevant information of the objects appearing in the VR panoramic image; Wherein, the relevant information includes the second coordinate corresponding to the object and at least one of the following: The frame number of the VR panoramic image where the VR image of the object is located; The identifier of the object; The category of the object; The size data of the object.

5. A VR panoramic live broadcast method, the method comprising: Continuously executing the first process multiple times to output a VR panoramic live broadcast stream in real time, where the VR panoramic live broadcast stream is composed of the VR panoramic images obtained in the multiple first processes; Wherein, the first process includes: Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model is obtained. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time; Based on the VR panoramic image, at least one second coordinate corresponding to the objects appearing in the VR panoramic image is identified; The at least one first coordinate and the at least one second coordinate are matched, and the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate are matched; In the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

6. A VR panoramic live broadcast system, the system includes one or more VR panoramic live broadcast subsystems, and each VR panoramic live broadcast subsystem corresponds to its own shooting range; The VR panoramic live broadcast subsystem is configured to execute the VR panoramic live broadcast method as described in claim 5.

7. An interaction method based on the VR panoramic live broadcast system described in claim 6, applied to an electronic device for playing VR panoramic live broadcast, the method further includes: Receiving a first input from the user, where the first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream played by the electronic device in real time, and the VR panoramic live broadcast stream played by the electronic device in real time is output by the first VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems; In response to the first input, play the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinate or the target object or the target identifier.

8. The interactive method of the VR panoramic live broadcast system according to claim 7, wherein, The VR panoramic image in the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem includes: the coordinate or object or identifier corresponding to the shooting range of the first VR panoramic live broadcast subsystem.

9. An interaction method based on the VR panoramic live broadcast system described in claim 6, applied to an electronic device for playing VR panoramic live broadcast, the method further includes: Receiving a second input from the user, where the second input is an operation in which the user selects a target coordinate or a target store or a target product in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream played by the electronic device in real time; In response to the second input, output a jump link corresponding to the target coordinate or target store or target product, and / or play a VR panoramic live stream output by a VR panoramic live broadcast subsystem corresponding to a shooting range with the target coordinate or target store or target product as the main perspective.

10. An interaction method for a VR panoramic live broadcast system according to claim 6, applied to an electronic device for playing a VR panoramic live broadcast, the method further comprising: Receiving a third input from a user, where the third input is an operation of the user to punch in or leave a message after selecting a target coordinate or target object in the VR panoramic image currently displayed on the electronic device, and the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream played in real time by the electronic device; In response to the third input, display the user's punch-in information or message information in the area corresponding to the target coordinate or target object in the VR panoramic image currently displayed on the electronic device.

11. The interactive method of the VR panoramic live broadcast system according to claim 10, wherein, Display the punch-in information or message information of the user or other users in the area corresponding to the coordinate or object included in the VR panoramic image currently displayed on the electronic device.

12. An output device for a VR panoramic image, comprising: A first coordinate acquisition module configured to, based on the real-time geospatial information of at least one actual object within a shooting range and VR panoramic camera parameter information, obtain at least one first coordinate output by the VR panoramic image position mapping model through the VR panoramic image position mapping model, where the at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively, and the VR panoramic image is a VR panoramic image obtained by the VR panoramic camera by shooting the shooting range in real time; A second coordinate acquisition module configured to, based on the VR panoramic image, identify at least one second coordinate corresponding to the objects appearing in the VR panoramic image respectively; A matching module configured to match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate; An image determination module configured to, in the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

13. An electronic device, comprising a processor and a memory storing a computer program, where when the processor executes the computer program, it implements the steps of the VR panoramic image output method according to any one of claims 1 to 4, or implements the steps of the VR panoramic live broadcast method according to claim 5, or implements the steps of the interaction method according to any one of claims 7 to 11.

14. A computer program product, comprising a computer program, where when the computer program is executed by a processor, it implements the steps of the VR panoramic image output method according to any one of claims 1 to 4, or implements the steps of the VR panoramic live broadcast method according to claim 5, or implements the steps of the interaction method according to any one of claims 7 to 11.

Images