VR panoramic image output method, VR panoramic live broadcast method and interaction method

Abstract

The invention relates to the technical field of virtual reality, and provides a VR panoramic image output method, a VR panoramic live broadcast method and an interaction method. The output method comprises the steps of obtaining a first coordinate through a VR panoramic image position mapping model based on geographic space information of an actual object in a shooting range and parameter information of a VR panoramic camera; second coordinates of an object appearing in the VR panoramic image are obtained through recognition based on the VR panoramic image; matching the first coordinate with the second coordinate, and matching an actual object corresponding to the successfully matched first coordinate with an object appearing in the VR panoramic image corresponding to the second coordinate; and determining to display the VR image of the actual object in the VR panoramic image under the condition of successful matching. Real-time sharing and intercommunication of information of an actual object in a real space and a virtual space are realized, secondary calibration of geographic space information of the actual object and coordinate information in the VR panoramic image is integrated, and the recognition accuracy in the VR panoramic image is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and particularly to a method for outputting VR panoramic images, a method for VR panoramic live broadcast, and an interaction method. Background Art

[0002] Currently, VR panoramic live broadcast is a combination of video live broadcast and VR technology, and is the main manifestation form of VR content. When a user watches through a VR headset, they can freely view from any angle and experience an immersive sense of presence.

[0003] In related technologies, the target recognition technology for panoramic images relies on traditional planar video target recognition algorithms. Since in panoramic live broadcast, the panoramic image is spherical and there is distortion after being unfolded into a 2D image, the recognition accuracy of the existing technology is relatively low and the reliability is insufficient. Summary of the Invention

[0004] Embodiments of this application provide a method for outputting VR panoramic images, a method for VR panoramic live broadcast, and an interaction method to solve the technical problem of the recognition accuracy of VR panoramic images.

[0005] In a first aspect, embodiments of this application provide a method for outputting a VR panoramic image, including:

[0006] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0007] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0008] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0009] In the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

[0010] In one embodiment, based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, at least one first coordinate output by the VR panoramic image position mapping model is obtained through the VR panoramic image position mapping model, including:

[0011] Perform feature preprocessing and feature combination on the geospatial information of at least one actual object and the VR panoramic camera parameter information, where the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization;

[0012] Input the geospatial information and VR panoramic camera parameter information after feature preprocessing and feature combination into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model.

[0013] In one embodiment, the structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

[0014] In one embodiment, based on the VR panoramic image, identifying at least one second coordinate corresponding to each object that appears in the VR panoramic image includes:

[0015] Input the VR panoramic image into a video object recognition algorithm to identify relevant information about the objects that appear in the VR panoramic image;

[0016] where the relevant information includes the second coordinate corresponding to the object and at least one of the following:

[0017] The frame number of the VR panoramic image in which the VR image of the object is located;

[0018] The identifier of the object;

[0019] The category of the object;

[0020] The size data of the object.

[0021] In a second aspect, an embodiment of the present application provides a VR panoramic live broadcast method, and the method includes:

[0022] Continuously execute the first process multiple times to output a VR panoramic live broadcast stream in real time, where the VR panoramic live broadcast stream is composed of the VR panoramic images obtained in the multiple first processes;

[0023] where the first process includes:

[0024] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model is obtained. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0025] Based on the VR panoramic image, at least one second coordinate corresponding to the objects appearing in the VR panoramic image is identified respectively;

[0026] The at least one first coordinate and the at least one second coordinate are matched, and the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate are matched;

[0027] In the case of successful matching, the VR image of the successfully matched actual object is determined to be displayed in the VR panoramic image.

[0028] In a third aspect, an embodiment of the present application provides a VR panoramic live broadcast system. The system includes one or more VR panoramic live broadcast subsystems, and each VR panoramic live broadcast subsystem corresponds to its own shooting range;

[0029] The VR panoramic live broadcast subsystem is used to execute the foregoing VR panoramic live broadcast method.

[0030] In a fourth aspect, an embodiment of the present application provides an interaction method based on the VR panoramic live broadcast system provided in the third aspect, which is applied to an electronic device for playing VR panoramic live broadcast. The method further includes:

[0031] Receiving a first input from the user. The first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream played by the electronic device in real time, and the VR panoramic live broadcast stream played by the electronic device in real time is output by the first VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems;

[0032] In response to the first input, playing the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems. The shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinate or the target object or the target identifier.

[0033] In one embodiment, the VR panoramic images in the VR panoramic live stream output by the second VR panoramic live subsystem include coordinates, objects, or identifiers corresponding to the shooting range of the first VR panoramic live subsystem.

[0034] In a fifth aspect, an embodiment of the present application provides an interaction method for a VR panoramic live system provided in the third aspect, which is applied to an electronic device for playing a VR panoramic live stream. The method further includes:

[0035] Receiving a second input from the user, where the second input is an operation in which the user selects a target coordinate, a target store, or a target product in the VR panoramic image currently displayed on the electronic device. The VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0036] In response to the second input, outputting a jump link corresponding to the target coordinate, the target store, or the target product, and / or playing the VR panoramic live stream output by the VR panoramic live subsystem corresponding to the shooting range with the target coordinate, the target store, or the target product as the main perspective.

[0037] In a sixth aspect, an embodiment of the present application provides an interaction method for a VR panoramic live system provided in the third aspect, which is applied to an electronic device for playing a VR panoramic live stream. The method further includes:

[0038] Receiving a third input from the user, where the third input is an operation in which the user checks in or leaves a message after selecting a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device. The VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0039] In response to the third input, displaying the user's check-in information or message information in the area corresponding to the target coordinate or the target object in the VR panoramic image currently displayed on the electronic device.

[0040] In one embodiment, the user's or other users' check-in information or message information is displayed in the area corresponding to the coordinates or objects included in the VR panoramic image currently displayed on the electronic device.

[0041] In a seventh aspect, an embodiment of the present application provides an output device for VR panoramic images, including:

[0042] The first coordinate acquisition module is configured to obtain, based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, at least one first coordinate output by the VR panoramic image position mapping model through the VR panoramic image position mapping model. The at least one first coordinate is the coordinates of the VR images of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0043] The second coordinate acquisition module is configured to, based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0044] The matching module is configured to match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0045] The image determination module is configured to, in the case of successful matching, determine the VR image of the successfully matched actual object displayed in the VR panoramic image.

[0046] In a sixth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program. When the processor executes the program, it implements the steps of the VR panoramic image output method provided in the first aspect, or implements the steps of the VR panoramic live broadcast method provided in the second aspect, or implements the steps of the interaction method provided in the fourth aspect or the fifth aspect or the sixth aspect.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the VR panoramic image output method provided in the first aspect, or implements the steps of the VR panoramic live broadcast method provided in the second aspect, or implements the steps of the interaction method provided in the fourth aspect or the fifth aspect or the sixth aspect.

[0048] The VR panoramic image output method, VR panoramic live broadcast method and interaction method provided by the embodiments of the present application realize the real-time sharing and intercommunication of information of actual objects in the real space and the virtual space by constructing the association mapping between the real-time geospatial information of actual objects and the VR panoramic image, and integrate the geospatial information of actual objects and the coordinate information in the VR panoramic image for secondary calibration, so as to improve the recognition accuracy in the VR panoramic image. Description of the Drawings

[0049] To more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0050] Figure 1 It is a schematic flowchart of the method for outputting VR panoramic images provided by an embodiment of the present application;

[0051] Figure 2 It is a schematic structural diagram of the VR panoramic live broadcast system provided by an embodiment of the present application;

[0052] Figure 3 It is one of the schematic flowcharts of the interaction method applied to the VR panoramic live broadcast system provided by an embodiment of the present application;

[0053] Figure 4 It is another schematic flowchart of the interaction method applied to the VR panoramic live broadcast system provided by an embodiment of the present application;

[0054] Figure 5 It is yet another schematic flowchart of the interaction method applied to the VR panoramic live broadcast system provided by an embodiment of the present application;

[0055] Figure 6 It is a schematic structural diagram of the VR panoramic image output device provided by an embodiment of the present application;

[0056] Figure 7 It exemplifies a schematic diagram of the physical structure of an electronic device. Detailed implementation manners

[0057] To make the objectives, technical solutions, and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0058] First, the following content will be introduced:

[0059] (1) The fifth generation of mobile communication technology (5G) has many innovations such as high speed, large traffic, low latency, and multiple connections. In the 5G era, the high-speed and low-latency transmission characteristics are expected to significantly improve the user experience of virtual reality (VR) products; 5G+ cloud rendering greatly improves the display effect of applications, reduces hardware costs, and promotes the popularization of VR. Technological innovation continues to make breakthroughs, continuously promotes product upgrades, and further clears obstacles for the popularization of VR.

[0060] The VR content ecosystem is becoming more complete and a broad application market is opening up: With the gradual implementation of VR applications, the practicality of VR / AR as a tool has gradually become more prominent, and it has shown considerable application potential in the video, marketing, music and game markets, and is expected to promote revolutionary changes in the next generation of interactive entertainment industry.

[0061] (2) VR panoramic live broadcast;

[0062] VR panoramic live broadcast is a combination of live video and VR technology, and is the main form of VR content. When users watch through a VR helmet, they can choose any angle to watch freely and experience an immersive experience.

[0063] VR panoramic live broadcast relies on professional equipment for shooting. After capturing images from multiple angles at the same time, it still needs to go through stitching and other processing to ensure that each frame is a 360-degree panoramic picture.

[0064] When watching, users are placed in the center of the spherical area and can watch dynamic videos 360 degrees around the shooting angle at will, without being restricted by time, space and region, giving them an immersive feeling. Panoramic live broadcast has factors such as depth of field, dynamic images, and sound, and has the characteristics of sound and picture alignment and sound and picture synchronization, giving them a good sense of immersion.

[0065] Panoramic live broadcasts need to be played through a dedicated panoramic player, which needs to be dynamically adjusted based on the data from the gesture sensor and then attached to the surface of a sphere. If you use a traditional video player to play panoramic live broadcasts, they will appear distorted and you will not be able to view the panoramic content normally.

[0066] In the related technologies, VR panoramic live broadcast mainly focuses on single panoramic live broadcast, and a small number of solutions simply superimpose some interactive operations, such as virtual gifts and bullet screens, which only superimpose virtual objects in the virtual space. In the related technologies, VR panoramic live broadcast solutions have the following shortcomings:

[0067] 1. Single VR panoramic live broadcast is a virtual space created by live streaming images, which lacks interactivity and entertainment, and has insufficient user retention and activity;

[0068] 2. The panoramic live broadcast with superimposed virtual gifts and bullet screens is similar to traditional 2D videos, lacking entertainment and innovation. It fails to make good use of the 3D spatial characteristics of VR panoramic live broadcasts and has insufficient attraction to users.

[0069] In the related technologies, the integration and interaction between the virtual space and the real space of VR panoramic live broadcasts are not realized by combining the geographical information of the real space.

[0070] In the related technologies, the VR panoramic live broadcast technology mainly constructs a single VR panoramic live broadcast. If it is necessary to know the mapping correspondence between a certain pixel point in the panoramic live broadcast and the real space position, it can only rely on manual operation. The manual operation has the following disadvantages:

[0071] 1. The labor input is relatively large;

[0072] 2. The number of mappable relationships that can be marked is limited;

[0073] 3. After the position of the VR live camera changes, or the position of the target object changes, the mapping relationship cannot be automatically updated.

[0074] In the related technologies, the target recognition technology for panoramic images relies on traditional planar video target recognition algorithms. Since in panoramic live broadcasts, the panoramic image is spherical and there is distortion after being unfolded into a 2D image, the recognition accuracy of the existing technologies is relatively low and the reliability is insufficient.

[0075] Therefore, the present invention provides an output method for VR panoramic images, a VR panoramic live broadcast method and an interaction method, which can improve the recognition accuracy of VR panoramic images.

[0076] Figure 1 It is a schematic flowchart of the output method for VR panoramic images provided by the embodiments of the present application. As Figure 1 shown, the method may include the following processes:

[0077] Step 100, based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0078] In the embodiments of the present application, by constructing a VR panoramic image position mapping model, a mapping relationship between the real-time geospatial information of the actual object and the coordinate information of the VR panoramic image is established, so as to predict and obtain its position in the VR panoramic live broadcast through the geospatial coordinates of the actual object.

[0079] Among them, the real-time geospatial information of the actual object may include: precision coordinates, dimensional coordinates, and altitude.

[0080] Specifically, the VR panoramic image is a spherical image, which compresses the 3D world. Therefore, there is a certain mapping relationship between the real-time geospatial information of the actual object and the coordinate information of the VR panoramic image. This relationship model is affected by internal and external parameters such as the position of the VR camera, focal length, pixel resolution, and principal distance.

[0081] Among them, the input of the VR panoramic image position mapping model may include two parts. One part is the real-time geospatial information of at least one actual object, and the other part is the VR panoramic camera parameter information. The real-time geospatial information of the actual object may include: precision coordinates, dimensional coordinates, and altitude. The VR panoramic camera parameter information may include internal and external parameters such as the position of the VR camera, focal length, pixel resolution, and principal distance;

[0082] Optionally, the real-time geospatial information of at least one actual object and the VR panoramic camera parameter information can be preprocessed for features and then input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model, that is, the coordinates of the VR images of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. For example, output obj_geo i At the lower left corner coordinates (geo_x i , geo_y i ) in the panoramic image frame.

[0083] It should be noted that the VR panoramic image position mapping actual object model can be pre-trained.

[0084] It should be noted that the VR panoramic image position mapping actual object model can be obtained through iterative learning training based on the historical geospatial information of at least one actual object and the VR panoramic camera parameter information. All types of neural networks that can predict the coordinates of the VR images of the at least one actual object in the VR panoramic image respectively after training are applicable to the embodiments of the present application and are not limited herein.

[0085] Step 110, based on the VR panoramic image, identify at least one second coordinate corresponding to each object that appears in the VR panoramic image;

[0086] Optionally, a sequence of VR panoramic images can be extracted frame by frame from the VR panoramic live stream generated by the VR panoramic camera and marked in the frame order to obtain {pic k , k = 0, 1, 2...}.

[0087] Embodiments of this application can adopt a video object recognition algorithm to recognize the object category, coordinate position, and size in a VR panoramic image based on the temporal information of the front and back frames of the VR panoramic image. Among them, the coordinate position is at least one second coordinate corresponding to the objects appearing in the VR panoramic image respectively.

[0088] Step 120: Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate.

[0089] Optionally, based on the at least one first coordinate of the at least one actual object obtained in step 100, the at least one second coordinate corresponding to the objects appearing in the VR panoramic image obtained in step 110, and related information (such as object category and / or size), confirm the position of the actual object in the real space in the VR panoramic image, and construct an association mapping between the actual object in the virtual space and the real space.

[0090] Specifically, it can include the following steps:

[0091] (1) Centering on a VR panoramic camera (such as a VR panoramic live camera), enclose the actual objects within a certain distance range (i.e., the shooting range). The set selection distance is determined by the environmental range captured by the VR panoramic camera and the object range of concern.

[0092] (2) Based on the map information, obtain the real-time geospatial information of the actual objects within the shooting range, and through the VR panoramic image position mapping model, obtain the position coordinates (geo_x i i i ) of the actual object obj_geo in the VR panoramic live image.

[0093] (3) Perform association matching on the at least one first coordinate of the at least one actual object obtained in step 100, the at least one second coordinate corresponding to the objects appearing in the VR panoramic image obtained in step 110, and related information (such as object category and / or size) to obtain a more accurate VR image of the object obj k k k ) in the VR panoramic live image and its position coordinates (x

[0094] Traverse each object obj_det j appearing in the VR panoramic image, and check whether there is (geo_x i i ​​​​​) fall into it. If so, then match obj_det j and obj_geo i to check if the object category information is the same. If it is, then confirm that the object obj exists in the VR panoramic live broadcast k , obj k 's information is as follows:

[0095] obj k = obj_geo i ;

[0096] x k = (geo_x i + det_x j ) / 2;

[0097] y k = (geo_x k + det_x k ) / 2;

[0098] obj k 's relevant information is the same as obj_geo i , such as: building name, scenic spot name, store name, rating, etc.

[0099] If there is a one-to-many or many-to-one match, the point with the shorter distance shall prevail.

[0100] Step 130, in the case of successful matching, determine the VR image of the successfully matched actual object to be displayed in the VR panoramic image.

[0101] Based on the real-time geospatial information of the actual object and the associated mapping in the VR panoramic image, the embodiments of the present application realize the real-time sharing and intercommunication of information of the actual object between the real space and the virtual space, and realize the interaction between the real and the virtual.

[0102] The embodiments of the present application propose a method for constructing a panoramic live broadcast with virtual-real fusion interaction, which maps and associates the target object in the VR live broadcast with the real space to realize the fusion interaction between the virtual and real spaces.

[0103] The embodiments of the present application propose a method for mapping the geospatial coordinates and the position of the VR panoramic image. By constructing a mapping model between the geospatial coordinates and the VR panoramic image, a mapping relationship between the geospatial coordinate information of the target object in the real space and the position information of the VR panoramic image is established, so as to obtain the position of the target object in the VR panoramic image through its geospatial coordinates.

[0104] The embodiments of the present application propose a method for mapping the geospatial coordinates and the position of the VR panoramic image, which has the following advantages compared with the related art that can only rely on manual operation:

[0105] (1) Automatically construct mapping relationships without additional manual input;

[0106] (2) The mapping relationships that can be marked can be accurate to the pixel level. For objects with extremely close distances and overlapping images in VR panoramic images, accurate mapping to the real space can also be achieved;

[0107] (3) After the position of the VR panoramic camera changes or the position of the target object changes, the mapping relationship can be automatically updated;

[0108] (4) The mapping relationship is updated and optimized in combination with the target recognition result, and the result is more accurate.

[0109] An embodiment of the present application proposes a method for target recognition of VR panoramic images. Compared with the target recognition technology for panoramic images in the related art (relying on traditional planar video target recognition algorithms), the accuracy is higher and it can be applied to panoramic live broadcasts. The panoramic image is spherical, and there is distortion after being unfolded into a 2D image. The recognition accuracy of the related technology is low and the reliability is insufficient. The embodiment of the present application incorporates the geographical space information of the actual object for secondary calibration to ensure the improvement of the recognition accuracy.

[0110] The output method of the VR panoramic image proposed in the embodiment of the present application realizes the real-time sharing and intercommunication of information of the actual object in the real space and the virtual space by constructing the real-time geographical space information of the actual object and the associated mapping in the VR panoramic image, and incorporates the geographical space information of the actual object and the coordinate information in the VR panoramic image for secondary calibration to improve the recognition accuracy in the VR panoramic image.

[0111] In one embodiment, based on the real-time geographical space information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtaining at least one first coordinate output by the VR panoramic image position mapping model includes:

[0112] Performing feature preprocessing and feature combination on the geographical space information of at least one actual object and the VR panoramic camera parameter information, wherein the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization;

[0113] Inputting the geographical space information and VR panoramic camera parameter information after feature preprocessing and feature combination into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model.

[0114] Specifically, the input of the VR panoramic image position mapping model can include two parts. One part is the real-time geospatial information of at least one actual object, and the other part is the VR panoramic camera parameter information. The real-time geospatial information of the actual object can include: precision coordinates, dimensional coordinates, and altitude. The VR panoramic camera parameter information can include internal and external parameters such as the VR camera position, focal length, pixel resolution, and principal distance;

[0115] Optionally, the real-time geospatial information of at least one actual object and the VR panoramic camera parameter information can be subjected to feature preprocessing and then input into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model, that is, the coordinates of the VR images of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image, such as outputting obj_geo i At the lower left corner coordinates (geo_x i , geo_y i ) in the panoramic image frame; among them, the feature preprocessing includes: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization, and feature combinations constructed based on experience. The feature combinations can be constructed according to user needs and experience and can include feature combinations that can reflect the relative position relationship between the target object and the VR camera, such as the azimuth angle and distance between the target object and the VR camera; for example, it can include the feature combination of the azimuth angle and distance between the actual object and the VR camera; the embodiments of the present application do not limit this.

[0116] Optionally, the VR panoramic image position mapping model can fuse a neural network model and a linear regression model. The linear regression model mainly relies on artificial features and gives full play to the advantages of historical experience; the neural network model has stronger feature reasoning and generalization capabilities and fully compensates for the deficiencies of artificial experience features; the two fusions further improve the prediction accuracy.

[0117] Optionally, the output of the VR panoramic image position mapping model can include: predicting the lower left corner coordinates (geo_x i ), geo_y i , geo_y i ) of the actual object obj_geo in the panoramic image frame.

[0118] In one embodiment, the structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

[0119] It should be noted that the VR panoramic image position mapping model can integrate a neural network model and a linear regression model. The linear regression model mainly relies on artificial features and gives full play to the advantages of historical experience. The neural network model has stronger feature inference and generalization capabilities and can fully make up for the deficiencies of artificial experience features. Any method of integrating the neural network model and the linear regression model to predict the coordinates of the VR images of the at least one actual object in the VR panoramic image respectively is applicable to the embodiments of the present application and will not be limited herein.

[0120] For example, artificial features refer to features manually designed by domain experts or data scientists based on their understanding and professional knowledge of the data set. These features are usually created based on insights into the problem domain and an understanding of the data distribution, with the aim of capturing information that may be helpful for the prediction task.

[0121] In the context of the VR panoramic image position mapping model, artificial features may include:

[0122] Geometric features: such as geometric attributes like lines, angles, shapes, etc. in the panoramic image, which can help identify the structural content in the image.

[0123] Statistical features: such as statistical information like the color histogram, brightness, and contrast of the image, which can describe the global attributes of the image.

[0124] Transformation features: features obtained by performing operations such as rotating, scaling, and filtering on the image, which can enhance the model's adaptability to different perspectives and scales.

[0125] Texture features: local texture information of the image, such as features extracted through the gray-level co-occurrence matrix (GLCM), which can reflect the subtle changes on the image surface.

[0126] Context features: features related to the position and relationship of objects in the image, such as the distance between objects, relative positions, etc.

[0127] User interaction features: If the VR panoramic image contains user interaction data, such as click-through rate, browsing time, etc., these can also be used as useful features.

[0128] Depth information: If available, the depth map generated by a depth sensor or a stereo vision algorithm can provide additional spatial position information.

[0129] Semantic features: high-level semantic information obtained through image annotation or segmentation, such as scene labels, object categories, etc.

[0130] For example, to integrate a neural network model and a linear regression model to construct a VR panoramic image position mapping model, the following steps can be taken:

[0131] Feature extraction: Use a neural network model to extract useful features from VR panoramic image samples. These features can be visual information such as the texture, color, and edges of the image, or deeper abstract representations.

[0132] Linear regression analysis: After extracting sufficient features, a linear regression model can be used to analyze the mapping relationship between these features and the image position. Artificial features can also be constructed.

[0133] Model fusion strategy: Take the features extracted by the neural network model and artificial features as inputs and pass them to the linear regression model for position mapping. This fusion can be achieved by linear weighting, for example, by weighted summation of different neural network features. In addition, more complex fusion methods, such as stacking, can also be adopted to achieve better prediction results by combining multiple sub-learners with their own advantages.

[0134] Multimodal fusion method: Considering that VR panoramic images may contain various types of data (such as images, depth information, user interactions, etc.), multimodal fusion methods can be adopted. This includes model-agnostic methods and model-based methods. The former does not directly depend on specific deep learning methods, while the latter uses deep learning models to explicitly solve the multimodal fusion problem.

[0135] Training and optimization: During the model training process, the parameters of the neural network and linear regression model need to be adjusted to minimize the difference between the predicted position and the actual position. This process may require the use of complex optimization algorithms, and since the training of deep neural networks is usually very time-consuming, it is necessary to reasonably arrange computing resources.

[0136] Evaluation and iteration: Evaluate the performance of the fusion model through experiments and continuously iterate and improve the model according to the evaluation results. This may include adjusting the feature extraction strategy, the selection of fusion methods, and the optimization of model parameters.

[0137] In summary, constructing a VR panoramic image position mapping model by fusing a neural network model and a linear regression model is a complex process involving multiple steps such as feature extraction, model fusion, multimodal data processing, and optimization. Through careful design and continuous iteration, an efficient model that can both capture image details and accurately predict positions can be achieved.

[0138] In one embodiment, identifying at least one second coordinate corresponding to each object that appears in the VR panoramic image based on the VR panoramic image includes:

[0139] Input the VR panoramic image into a video object recognition algorithm to identify relevant information about the objects that appear in the VR panoramic image;

[0140] Among them, the relevant information includes the second coordinate corresponding to the object and at least one of the following:

[0141] The frame number of the VR panoramic image where the VR image of the object is located;

[0142] The identifier of the object;

[0143] The category of the object;

[0144] The size data of the object.

[0145] Specifically, from the VR panoramic live stream generated by the VR panoramic camera, a sequence of VR panoramic images can be extracted frame by frame and marked in the order of frames to obtain {pic k , k = 0, 1, 2...}; furthermore, the key frames in the frame sequence are input into the video object recognition algorithm to identify the object categories, time, and positions that appear in the VR panoramic image; among them, the video object recognition algorithm can include: Flow-based, YoloV, DFF, FGFA, etc., which are video object detection algorithms that can identify the object categories, time, and positions that appear in the VR panoramic image. The embodiments of the present application do not limit this; taking the Flow-based video object recognition algorithm as an example, it can be recorded in the format shown in Table 1 below:

[0146] Table 1

[0147]

[0148] Taking the Flow-based video object recognition algorithm as an example, the embodiments of the present application adopt a video object detection algorithm based on the flow-based model, which has the following advantages compared with traditional image detection algorithms:

[0149] (1) In continuous VR panoramic images (such as live broadcasts), there will be blurred images due to object movement, camera defocus, as well as appearance changes caused by the target object being blocked, posture transformation, and size changes caused by distance. In these scenarios, it is necessary to use the temporal information of the front and back frames in the video for inference and prediction;

[0150] (2) Introduce the concept of key frames. Adjacent frames with similar appearances usually result in similar features. Therefore, it is not necessary to calculate features for all frames, improving the detection efficiency;

[0151] (3) Use the detection features or detection results of the front and back frames to optimize the detection effect of the current frame and improve the detection accuracy.

[0152] An embodiment of the present application proposes a method for target recognition of VR panoramic images. Through the matching and calibration of the results of the flow-based video target detection algorithm and the results of the VR panoramic live broadcast position mapping model, the target recognition of VR panoramic live broadcast is realized.

[0153] In one embodiment, an embodiment of the present application provides a VR panoramic live broadcast method, and the method includes:

[0154] Execute the first process multiple times continuously and output the VR panoramic live broadcast stream in real time. The VR panoramic live broadcast stream is composed of VR panoramic images obtained in the multiple first processes;

[0155] Wherein, the first process includes:

[0156] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0157] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0158] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0159] In the case of successful matching, determine the VR image that displays the successfully matched actual object in the VR panoramic image.

[0160] That is, apply the foregoing VR panoramic image output method to the live broadcast scenario, and execute the steps in the foregoing VR panoramic image output method multiple times continuously to form a real-time VR panoramic live broadcast stream.

[0161] An embodiment of the present application proposes a method for constructing a virtual-real fusion interactive panoramic live broadcast. By integrating geospatial information, the actual objects in the VR panoramic live broadcast are mapped and associated with the real space. Compared with the existing panoramic live broadcast construction methods, it has the advantages of richer panoramic live broadcast information and higher interactivity, breaking the traditional VR panoramic live broadcast independent virtual space and realizing the fusion and interaction between the virtual space and the real space of the VR panoramic live broadcast.

[0162] The embodiment of the present application proposes a method for constructing a panoramic live broadcast that integrates virtual and real interactions. By integrating geographic space coordinates with a VR panoramic live broadcast position mapping model and a panoramic image target recognition model, on the one hand, the accuracy of panoramic live broadcast target recognition is effectively improved to address the deficiencies of the prior art; on the other hand, a mapping relationship between the panoramic live broadcast pixel point position and the real space coordinates can be automatically generated, thereby achieving real-time synchronous sharing of the panoramic live broadcast virtual space information and the real space object information, and creating a panoramic live broadcast that integrates virtual and real interactions.

[0163] The method for constructing a panoramic live broadcast with interactive integration of virtual and real proposed in the embodiment of the present application can, by integrating geographic space coordinates with a VR panoramic live broadcast position mapping model and a panoramic image target recognition model, effectively improve the target recognition accuracy and solve the deficiencies of the prior art on the one hand; on the other hand, it can automatically generate a mapping relationship between the pixel position of the panoramic live broadcast and the real space coordinates, thereby achieving real-time synchronous sharing of the virtual space information of the panoramic live broadcast and the object information of the real space, and creating a panoramic live broadcast with interactive integration of virtual and real.

[0164] Figure 2 is a structural diagram of a VR panoramic live broadcast system provided in an embodiment of the present application, such as Figure 2 As shown, the VR panoramic live broadcast system 200 includes one or more VR panoramic live broadcast subsystems 210, each VR panoramic live broadcast subsystem corresponds to its own shooting range;

[0165] The VR panoramic live broadcast subsystem is used to execute the aforementioned VR panoramic live broadcast method.

[0166] It should be noted that each VR panoramic live broadcast subsystem 210 in the VR panoramic live broadcast system 200 can realize the VR panoramic live broadcast in the aforementioned VR panoramic live broadcast method embodiment within its own shooting range. For example, the main perspective of the shooting range of VR panoramic live broadcast subsystem A is the playground, with the teaching building and the cafeteria in the background; the main perspective of the shooting range of VR panoramic live broadcast subsystem B is the cafeteria, with the teaching building and the playground in the background. That is, the VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem A is the live broadcast with the playground as the main perspective, that is, the live broadcast of objects and events in the playground; the VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem B is the live broadcast with the cafeteria as the main perspective, that is, the live broadcast of objects and events in the cafeteria.

[0167] Figure 3 This is one of the flow diagrams of the interactive method applied to the VR panoramic live broadcast system provided in the embodiment of the present application, such as Figure 3 As shown, an electronic device applied to playing VR panoramic live broadcast, the method includes:

[0168] Step 300: Receive the first input from the user. The first input is an operation where the user selects a target coordinate, a target object, or a target identifier in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time, and the VR panoramic live stream that the electronic device is playing in real time is output by the first VR panoramic live subsystem among the multiple VR panoramic live subsystems.

[0169] Step 310: In response to the first input, play the VR panoramic live stream output by the second VR panoramic live subsystem among the multiple VR panoramic live subsystems. The shooting range of the second VR panoramic live subsystem corresponds to the target coordinate, the target object, or the target identifier.

[0170] The embodiments of the present application can intelligently construct a multi-VR panoramic immersive interactive live broadcast. For example: In the panoramic live broadcast of A, the target point B can be viewed. At the same time, there is also a panoramic live broadcast set at the target point B. A jump point to the panoramic live broadcast of B can be presented at the corresponding position in the panoramic live broadcast of A, so that multiple panoramic live broadcasts can be automatically associated and support mutual jumping. Allowing users to select jump points through the first input in multiple panoramic live broadcasts and multiple spaces, realizing a spatial conversion similar to the real world in the VR panoramic live broadcast and achieving a more immersive interactive live broadcast.

[0171] For example, the main perspective of the shooting range of the VR panoramic live subsystem A is the playground, with the teaching building and the cafeteria in the background. The main perspective of the shooting range of the VR panoramic live subsystem B is the cafeteria, with the teaching building and the playground in the background. That is, the VR panoramic live stream output by the VR panoramic live subsystem A is a live broadcast with the playground as the main perspective, that is, a live broadcast of the objects and events in the playground. The VR panoramic live stream output by the VR panoramic live subsystem B is a live broadcast with the VR image of the cafeteria as the main perspective, that is, a live broadcast of the objects and events in the cafeteria. The electronic device is currently playing the VR panoramic live stream output by the VR panoramic live subsystem A, that is, the foreground content displayed on the display interface is the playground, and the background content includes the teaching building and the cafeteria. If the user selects the VR image of the cafeteria through the first input, the electronic device will switch the live stream being played to the VR panoramic live stream output by the VR panoramic live subsystem B, that is, the foreground content displayed on the display interface is switched to the cafeteria, and the background content includes the teaching building and the playground.

[0172] In one embodiment, the VR panoramic image in the VR panoramic live stream output by the second VR panoramic live subsystem includes: the coordinate, object, or identifier corresponding to the shooting range of the first VR panoramic live subsystem.

[0173] Figure 4 It is the second flow diagram of the interactive method applied to the VR panoramic live system provided by the embodiments of the present application. AsFigure 4 As shown, for an electronic device applied to play VR panoramic live broadcast, the method further includes:

[0174] Step 400, receiving a second input from the user, where the second input is an operation in which the user selects a target coordinate or a target store or a target commodity in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream that the electronic device plays in real time;

[0175] Step 410, in response to the second input, outputting a jump link corresponding to the target coordinate or the target store or the target commodity, and / or playing a VR panoramic live broadcast stream output by a VR panoramic live broadcast subsystem corresponding to a shooting range with the target coordinate or the target store or the target commodity as the main perspective.

[0176] Embodiments of the present application can intelligently construct an immersive cloud shopping for VR panoramic live broadcast. For example: when a certain store appears in the VR panoramic image of the VR panoramic live broadcast stream and the user selects the VR image of the store through the second input, a jump link for cloud shopping of the store can be provided near the coordinate position corresponding to the VR image of the store, or the live broadcast stream currently displayed on the electronic device is switched to a VR panoramic live broadcast stream output by a VR panoramic live broadcast subsystem corresponding to a shooting range with the interior of the store as the main perspective, that is, panoramically live broadcast the information shopping inside the store. Therefore, users can not only see the real-time situation of the store in the panoramic live broadcast, but also realize an immersive shopping experience, realizing the integration of the virtual space and the real space.

[0177] Figure 5 It is the third flowchart of the interaction method applied to the VR panoramic live broadcast system provided by the embodiments of the present application. As Figure 5 shown, for an electronic device applied to play VR panoramic live broadcast, the method further includes:

[0178] Step 500, receiving a third input from the user, where the third input is an operation in which the user checks in or leaves a message after selecting a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream that the electronic device plays in real time;

[0179] Step 510, in response to the third input, displaying the user's check-in information or message information in the area corresponding to the target coordinate or the target object in the VR panoramic image currently displayed on the electronic device.

[0180] The embodiments of the present application can intelligently realize the sharing and synchronization of virtual and real space information. For example, when a user plays a VR panoramic live stream on an electronic device, the user can punch in or upload message information on the VR images corresponding to different positions on the VR panoramic image in the VR panoramic live stream. At the same time, the punch-in and message of the user or other users at this position in the real space will also be displayed at the mapped positions on the VR panoramic image in the VR panoramic live stream, realizing user communication and information sharing between the virtual and real spaces.

[0181] In one embodiment, the punch-in information or message information of the user or other users is displayed within the area corresponding to the coordinates or objects included in the currently displayed VR panoramic image of the electronic device.

[0182] The embodiments of the present application have high application value and broad application prospects, and have rich application scenarios in the CHBN segments.

[0183] For example, in the CH segment: The embodiments of the present application can enrich the interactive forms of VR panoramic live broadcasts, build innovative applications integrating virtual and real such as VR panoramic live broadcast interaction applications, VR immersive online shopping, VR immersive cloud tourism, and VR live broadcast punch-in, improve the content richness and entertainment of VR panoramic live broadcasts, and enhance user stickiness and activity.

[0184] For example, in the B segment: The embodiments of the present application can be applied to many fields such as smart culture and tourism, smart communities, smart villages, and smart cities, provide VR panoramic live broadcasts with virtual and real integration interactions, build multi-scenario VR interactive live broadcasts, and create immersive virtual space shopping, monitoring, roaming, etc.

[0185] For example, in the N segment: The embodiments of the present application can realize VR live broadcast content innovation, integrate real-world scene information into VR live broadcasts, and improve the richness of VR content; through this technical solution, innovative applications in the virtual space can also be created to integrate and interact VR panoramic live broadcasts with the real space.

[0186] The output device of the VR panoramic image provided by the embodiments of the present application will be described below. The output device of the VR panoramic image described below can be correspondingly referred to the output method of the VR panoramic image described above.

[0187] Figure 6 is a schematic structural diagram of the output device of the VR panoramic image provided by the embodiments of the present application, as Figure 6 shown, the device 600 includes:

[0188] The first coordinate acquisition module 610 is configured to obtain at least one first coordinate output by the VR panoramic image position mapping model based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0189] The second coordinate acquisition module 620 is configured to identify at least one second coordinate corresponding to the objects appearing in the VR panoramic image based on the VR panoramic image;

[0190] The matching module 630 is configured to match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate;

[0191] The image determination module 640 is configured to determine the VR image of the successfully matched actual object displayed in the VR panoramic image in the case of successful matching.

[0192] The VR panoramic image output device according to the embodiment of the present application can implement the steps of the foregoing VR panoramic image output method embodiment and achieve the same technical effects, which will not be elaborated here.

[0193] The terminal involved in the embodiment of the present application may be a device that provides voice and / or data connectivity to the user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem, etc. In different systems, the name of the terminal device may also be different. For example, in a 5G system, the terminal device may be referred to as a user equipment (UE).

[0194] The network device involved in the embodiment of the present application may be a base station, and the base station may include multiple cells that provide services to terminals. According to different specific application scenarios, the base station may also be referred to as an access point, or may be a device that communicates with wireless terminal devices through one or more sectors on the air interface in the access network, or other names.

[0195] Figure 7 Illustrates a schematic physical structure diagram of an electronic device, such as Figure 7As shown in the figure, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call a computer program in the memory 730 to execute the steps of the foregoing various methods, for example, including:

[0196] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0197] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0198] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0199] In the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

[0200] Or

[0201] Continuously execute the first process multiple times and output a VR panoramic live stream in real time. The VR panoramic live stream is composed of the VR panoramic images obtained in the multiple first processes;

[0202] Wherein, the first process includes:

[0203] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0204] Based on the VR panoramic image, at least one second coordinate corresponding to the objects appearing in the VR panoramic image is identified;

[0205] The at least one first coordinate and the at least one second coordinate are matched, and the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate are matched;

[0206] In the case of successful matching, a VR image for displaying the successfully matched actual object in the VR panoramic image is determined.

[0207] Or,

[0208] Receive a first input from the user, where the first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time, and the VR panoramic live stream that the electronic device is playing in real time is output by the first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0209] In response to the first input, play the VR panoramic live stream output by the second VR panoramic live subsystem among the multiple VR panoramic live subsystems, where the shooting range of the second VR panoramic live subsystem corresponds to the target coordinate or the target object or the target identifier.

[0210] Or,

[0211] Receive a second input from the user, where the second input is an operation in which the user selects a target coordinate or a target store or a target product in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0212] In response to the second input, output a jump link corresponding to the target coordinate or the target store or the target product, and / or play the VR panoramic live stream output by the VR panoramic live subsystem corresponding to the shooting range with the target coordinate or the target store or the target product as the main perspective.

[0213] Or,

[0214] Receive a third input from the user, where the third input is an operation in which the user checks in or leaves a message after selecting a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0215] In response to the third input, display the user's check-in information or message information in the area corresponding to the target coordinate or target object in the VR panoramic image currently displayed on the electronic device.

[0216] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0217] On the other hand, an embodiment of this application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the methods provided in the above-mentioned various embodiments, for example, including:

[0218] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0219] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0220] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0221] In the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

[0222] Or

[0223] Execute the first process multiple times continuously and output the VR panoramic live stream in real time. The VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes;

[0224] Wherein, the first process includes:

[0225] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0226] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively;

[0227] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0228] In the case of successful matching, determine the VR image of the successfully matched actual object to be displayed in the VR panoramic image.

[0229] Or,

[0230] Receive the first input of the user. The first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Wherein, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream played by the electronic device in real time. The VR panoramic live stream played by the electronic device in real time is output by the first VR panoramic live subsystem among the multiple VR panoramic live subsystems;

[0231] In response to the first input, play the VR panoramic live stream output by the second VR panoramic live subsystem among the multiple VR panoramic live subsystems. The shooting range of the second VR panoramic live subsystem corresponds to the target coordinate or the target object or the target identifier.

[0232] Or,

[0233] Receive a second input from the user, where the second input is an operation in which the user selects a target coordinate, a target store, or a target product in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0234] In response to the second input, output a jump link corresponding to the target coordinate, the target store, or the target product, and / or play the VR panoramic live stream output by the VR panoramic live sub-system corresponding to the shooting range with the target coordinate, the target store, or the target product as the main perspective.

[0235] Or,

[0236] Receive a third input from the user, where the third input is an operation in which the user checks in or leaves a message after selecting a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0237] In response to the third input, display the user's check-in information or message information in the area corresponding to the target coordinate or the target object in the VR panoramic image currently displayed on the electronic device.

[0238] On the other hand, an embodiment of the present application further provides a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the methods provided in the above embodiments. For example, it includes:

[0239] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time;

[0240] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image;

[0241] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate;

[0242] In the case of successful matching, determine the VR image of the actually matched object to be displayed in the VR panoramic image.

[0243] Or

[0244] Execute the first process multiple times continuously and output a VR panoramic live stream in real time. The VR panoramic live stream is composed of VR panoramic images obtained in the multiple first processes.

[0245] Wherein, the first process includes:

[0246] Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, obtain at least one first coordinate output by the VR panoramic image position mapping model. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time.

[0247] Based on the VR panoramic image, identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image respectively.

[0248] Match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate.

[0249] In the case of successful matching, determine the VR image of the actually matched object to be displayed in the VR panoramic image.

[0250] Or,

[0251] Receive a first input from the user. The first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Wherein, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream played by the electronic device in real time, and the VR panoramic live stream played by the electronic device in real time is output by the first VR panoramic live subsystem among the multiple VR panoramic live subsystems.

[0252] In response to the first input, play the VR panoramic live stream output by the second VR panoramic live subsystem among the multiple VR panoramic live subsystems. The shooting range of the second VR panoramic live subsystem corresponds to the target coordinate or the target object or the target identifier.

[0253] Or,

[0254] Receive a second input from the user, where the second input is an operation by the user to select a target coordinate, a target store, or a target product in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0255] In response to the second input, output a jump link corresponding to the target coordinate, the target store, or the target product, and / or play the VR panoramic live stream output by the VR panoramic live subsystem corresponding to the shooting range with the target coordinate, the target store, or the target product as the main perspective.

[0256] Or,

[0257] Receive a third input from the user, where the third input is an operation by the user to check in or leave a message after selecting a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device. Here, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time;

[0258] In response to the third input, display the user's check-in information or message information in the area corresponding to the target coordinate or the target object in the VR panoramic image currently displayed on the electronic device.

[0259] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (such as floppy disks, hard disks, magnetic tapes, magneto-optical discs (MO), etc.), optical memory (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drives (SSD)), etc.

[0260] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0261] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for outputting a VR panoramic image, characterized in that, Including: Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model is obtained. The at least one first coordinate is the coordinates of the VR images of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time; Based on the VR panoramic image, at least one second coordinate corresponding to the objects appearing in the VR panoramic image is identified; The at least one first coordinate and the at least one second coordinate are matched, and the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate are matched; In the case of successful matching, determine to display the VR image of the successfully matched actual object in the VR panoramic image.

2. The output method of the VR panoramic image according to claim 1, wherein The step of obtaining at least one first coordinate output by the VR panoramic image position mapping model based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information through the VR panoramic image position mapping model includes: Performing feature preprocessing and feature combination on the geospatial information of at least one actual object and the VR panoramic camera parameter information. Among them, the feature preprocessing includes at least one of the following: discrete feature one-hot encoding, continuous feature discretization, feature smoothing, vectorization; Inputting the geospatial information and VR panoramic camera parameter information after feature preprocessing and feature combination into the VR panoramic image position mapping model to obtain at least one first coordinate output by the VR panoramic image position mapping model.

3. The output method of the VR panoramic image according to claim 1 or 2, characterized in that, The structure of the VR panoramic image position mapping model includes a neural network model and a linear regression model.

4. The output method of the VR panoramic image according to claim 1, characterized in that, The step of identifying at least one second coordinate corresponding to the objects appearing in the VR panoramic image based on the VR panoramic image includes: Inputting the VR panoramic image into a video object recognition algorithm to identify the relevant information of the objects appearing in the VR panoramic image; Among them, the relevant information includes the second coordinate corresponding to the object and at least one of the following: The frame number of the VR panoramic image where the VR image of the object is located; The identifier of the object; The category of the object; The size data of the object.

5. A VR panoramic live broadcast method, characterized in that, The method includes: Continuously executing the first process multiple times and outputting a VR panoramic live stream in real time. The VR panoramic live stream is composed of the VR panoramic images obtained in the multiple first processes; Among them, the first process includes: Based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, through the VR panoramic image position mapping model, at least one first coordinate output by the VR panoramic image position mapping model is obtained. The at least one first coordinate is the coordinate of the VR image of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively. The VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time; Based on the VR panoramic image, at least one second coordinate corresponding to the objects appearing in the VR panoramic image is identified; The at least one first coordinate and the at least one second coordinate are matched, and the actual object corresponding to the successfully matched first coordinate and the object appearing in the VR panoramic image corresponding to the second coordinate are matched; In the case of successful matching, it is determined to display the VR image of the successfully matched actual object in the VR panoramic image.

6. A VR panoramic live broadcast system, characterized in that, The system includes one or more VR panoramic live broadcast subsystems, and each VR panoramic live broadcast subsystem corresponds to its own shooting range; The VR panoramic live broadcast subsystem is used to execute the VR panoramic live broadcast method as claimed in claim 5.

7. An interactive method for a VR panoramic live broadcast system according to claim 6, characterized in that, Applied to an electronic device for playing VR panoramic live broadcast, the method further includes: Receiving a first input from the user, where the first input is an operation in which the user selects a target coordinate or a target object or a target identifier in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream played by the electronic device in real time, and the VR panoramic live broadcast stream played by the electronic device in real time is output by the first VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems; In response to the first input, play the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem among the multiple VR panoramic live broadcast subsystems, and the shooting range of the second VR panoramic live broadcast subsystem corresponds to the target coordinate or the target object or the target identifier.

8. The interactive method of the VR panoramic live broadcast system according to claim 7, wherein The VR panoramic image in the VR panoramic live broadcast stream output by the second VR panoramic live broadcast subsystem includes: the coordinate or object or identifier corresponding to the shooting range of the first VR panoramic live broadcast subsystem.

9. An interactive method for a VR panoramic live broadcast system according to claim 6, characterized in that, Applied to an electronic device for playing VR panoramic live broadcast, the method further includes: Receiving a second input from the user, where the second input is an operation in which the user selects a target coordinate or a target store or a target product in the VR panoramic image currently displayed on the electronic device. Among them, the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live broadcast stream played by the electronic device in real time; In response to the second input, output a jump link corresponding to the target coordinate or the target store or the target product, and / or play the VR panoramic live broadcast stream output by the VR panoramic live broadcast subsystem corresponding to the shooting range with the target coordinate or the target store or the target product as the main perspective.

10. An interactive method for a VR panoramic live broadcast system according to claim 6, characterized in that, Applied to an electronic device for playing VR panoramic live broadcast, the method further includes: Receiving a third input from the user, where the third input is an operation of clocking in or leaving a message after the user selects a target coordinate or a target object in the VR panoramic image currently displayed on the electronic device, and the VR panoramic image currently displayed on the electronic device is a frame in the VR panoramic live stream that the electronic device is playing in real time; In response to the third input, displaying the user's clock-in information or message information in the area corresponding to the target coordinate or the target object in the VR panoramic image currently displayed on the electronic device.

11. The interactive method of the VR panoramic live broadcast system according to claim 10, characterized in that, Displaying the user's or other users' clock-in information or message information in the area corresponding to the coordinates or objects included in the VR panoramic image currently displayed on the electronic device.

12. An output device for VR panoramic images, characterized in that, Comprising: A first coordinate acquisition module, configured to obtain at least one first coordinate output by the VR panoramic image position mapping model based on the real-time geospatial information of at least one actual object within the shooting range and the VR panoramic camera parameter information, where the at least one first coordinate is the coordinates of the VR images of the at least one actual object predicted by the VR panoramic image position mapping model in the VR panoramic image respectively, and the VR panoramic image is the VR panoramic image obtained by the VR panoramic camera shooting the shooting range in real time; A second coordinate acquisition module, configured to identify at least one second coordinate corresponding to the objects that appear in the VR panoramic image based on the VR panoramic image; A matching module, configured to match the at least one first coordinate and the at least one second coordinate, and match the actual object corresponding to the successfully matched first coordinate and the object that appears in the VR panoramic image corresponding to the second coordinate; An image determination module, configured to determine, in the case of successful matching, to display the VR image of the successfully matched actual object in the VR panoramic image.

13. An electronic device, comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, implementing the steps of the VR panoramic image output method according to any one of claims 1 to 4, or implementing the steps of the VR panoramic live broadcast method according to claim 5, or implementing the steps of the interaction method according to any one of claims 7 to 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, implementing the steps of the VR panoramic image output method according to any one of claims 1 to 4, or implementing the steps of the VR panoramic live broadcast method according to claim 5, or implementing the steps of the interaction method according to any one of claims 7 to 11.

Images