Live interaction method and device based on augmented reality, equipment and storage medium

Abstract

The application provides an augmented reality-based live interaction method and device, equipment and a storage medium. The method comprises the following steps: in response to a user access instruction, obtaining live stream data, wherein the live stream data comprises live progress data; calling a virtual object model in an augmented reality database according to the live progress data and audience interaction data; obtaining device sensor data; updating the virtual object model according to the audience interaction data and the device sensor data to obtain a live virtual object; and superimposing the live virtual object at a target position. The live stream data is obtained in response to the user access instruction, and the virtual object is dynamically adjusted according to the live progress data and the audience interaction data, so that the audience can obtain personalized interactive experience during the live broadcast. The augmented reality interaction method greatly improves the audience's participation and interactivity, so that the audience is not only a passive viewer, but also can influence the live broadcast content through interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality technology, and in particular to a live interaction method and device based on augmented reality, equipment and storage medium. BACKGROUND

[0002] Traditional live streaming systems mainly focus on video and audio content recording and playing, and such systems usually lack real-time interaction capabilities with the audience. With the growing demand for remote participation and high-quality interactive experiences in the fields of education, training, and entertainment, traditional live streaming systems are insufficient in terms of interactivity. Simple recording and playing functions cannot meet the pursuit of users for rich and diverse interactive content and personalized experiences.

[0003] In remote education, corporate training, and online meetings, audiences not only want to watch high-quality video and audio content, but also want to participate in real-time through mobile devices, such as interacting with the content, asking questions, participating in voting, etc. The existing technology has obvious shortcomings in providing these interactive functions, resulting in low audience participation and satisfaction.

[0004] In summary, the problems in the prior art need to be solved. SUMMARY

[0005] The present application provides a live interaction method and device based on augmented reality, equipment and storage medium, to solve the defects in the prior art and improve the user's interactive experience.

[0006] The present application provides a live interaction method based on augmented reality, comprising:

[0007] In response to a user access instruction, live stream data is obtained, the live stream data including live progress data;

[0008] According to the live progress data and audience interaction data, a virtual object model is called in an augmented reality database, the augmented reality database being used to save the virtual object model;

[0009] Device sensor data is obtained;

[0010] According to the audience interaction data and the device sensor data, the virtual object model is updated to obtain a live virtual object;

[0011] The live virtual object is superimposed at a target position.

[0012] According to the live interaction method based on augmented reality provided by the present application, after the step of superimposing the live virtual object at the target position, it further comprises:

[0013] acquiring audience preference data;

[0014] generating a customized virtual object according to the audience preference data, the customized virtual object including language subtitles and virtual object annotations.

[0015] According to the live broadcast progress data and the audience interaction data, a virtual object model is called in an augmented reality database, and the step specifically includes:

[0016] According to the live broadcast progress data and the audience interaction data, a model type of the virtual object model is determined.

[0017] According to the model type, a corresponding virtual object model is matched in the augmented reality database.

[0018] According to the live broadcast progress data and the audience interaction data, the live virtual object is updated, and the step specifically includes:

[0019] According to the audience interaction data, attribute information of the live virtual object is determined.

[0020] According to the device sensor data, position information of the live virtual object is determined.

[0021] According to the attribute information and the position information of the live virtual object, the live virtual object is updated.

[0022] According to the live broadcast progress data and the audience interaction data, the device sensor data further includes environmental lighting data, and after the step of determining the attribute information of the live virtual object according to the audience interaction data, the method further includes:

[0023] According to the environmental lighting data, brightness and contrast of the live virtual object are determined.

[0024] According to the environmental lighting data, brightness and contrast of a current device screen are determined.

[0025] According to the live broadcast progress data and the audience interaction data, the device sensor data includes rotation rate data, acceleration data and static direction data, and the step of determining the position information of the live virtual object according to the device sensor data specifically includes:

[0026] According to the rotation rate data, the acceleration data and the static direction data, a posture of a current device is estimated to determine a motion state of the current device.

[0027] According to the motion state, a virtual three-dimensional coordinate of the live virtual object is determined;

[0028] The virtual three-dimensional coordinate is subjected to coordinate transformation to obtain a device two-dimensional coordinate.

[0029] According to the method, the live virtual object is superimposed at the target position.

[0030] The live stream data is input into an environment understanding model to obtain the target position, which is used for superimposing the live virtual object.

[0031] The superimposition order of the live virtual object is determined.

[0032] According to the superimposition order and the device two-dimensional coordinate, the live virtual object is superimposed at the target position.

[0033] The application further provides a live interactive device based on augmented reality, comprising:

[0034] A data acquisition module is configured to acquire live stream data in response to a user access instruction, wherein the live stream data comprises live progress data.

[0035] A model calling module is configured to call a virtual object model from an augmented reality database according to the live progress data and audience interaction data, wherein the augmented reality database is configured to store the virtual object model.

[0036] A sensor module is configured to acquire device sensor data.

[0037] A model updating module is configured to update the virtual object model according to the audience interaction data and the device sensor data to obtain a live virtual object.

[0038] A model superimposition module is configured to superimpose the live virtual object at a target position.

[0039] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the live interactive method based on augmented reality as described above.

[0040] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the live interactive method based on augmented reality as described above.

[0041] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method for live interaction based on augmented reality according to any one of the above.

[0042] The application provides a method and device for live interaction based on augmented reality, and a storage medium. The method comprises the following steps: obtaining live stream data in response to a user access instruction, wherein the live stream data comprises live progress data; calling a virtual object model in an augmented reality database according to the live progress data and audience interaction data, wherein the augmented reality database is used to store the virtual object model; obtaining device sensor data; updating the virtual object model according to the audience interaction data and the device sensor data to obtain a live virtual object; and superimposing the live virtual object at a target position. The live stream data is obtained in response to a user access instruction, and the virtual object is dynamically adjusted according to the live progress data and the audience interaction data, so that the audience can obtain personalized interactive experience during the live broadcast. The augmented reality interaction mode greatly improves the audience's participation and interactivity, so that the audience is not only a passive viewer, but also can influence the live broadcast content through interaction. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 is a flowchart of the method for live interaction based on augmented reality provided by the present application;

[0045] Figure 2 is a structural schematic diagram of the device for live interaction based on augmented reality provided by the present application;

[0046] Figure 3 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] To solve the problems in the prior art, the application provides a live interactive method based on augmented reality to improve the interactive experience of users watching live streaming. The live interactive method based on augmented reality is described below, as shown in Figure 1 including but not limited to the following steps:

[0049] Step 110, in response to the user access instruction, acquiring live streaming data, the live streaming data including live progress data.

[0050] In step 110, when the user accesses the live streaming platform through a mobile device or a computer, the system will first receive the user's access instruction. This instruction is usually triggered by the user clicking the "watch live" button or entering the live room by scanning the two-dimensional code. After successfully processing the user access, the system acquires the current live streaming data from the live server or the content distribution network (CDN). The live streaming data includes video stream, audio stream and real-time data related to live streaming, such as live progress information, interactive data, audience number statistics, etc. After acquiring the live streaming data, the system further analyzes the live progress data. The live progress data includes the time progress of the current live content, the node information of the live activity (such as the ongoing theme discussion or speech session), and the related interactive content (such as the barrage, real-time voting results, etc.).

[0051] By analyzing the live progress data, the system can determine the stage of the current live content and provide appropriate synchronization information for the user, such as displaying the live progress bar, the current discussion theme, or reminding the user whether there is unviewed interactive content.

[0052] Step 120, according to the live progress data and audience interactive data, calling a virtual object model in an augmented reality database, the augmented reality database being used to save the virtual object model.

[0053] In step 120, after obtaining the live streaming data and live progress data, the system parses these data to determine the theme, progress, important nodes (such as key issues discussed, upcoming interactive segments) of the current live content, etc. For example, if the live broadcast is entering an interactive segment, such as a Q&A segment or a voting activity, the system will identify this node and prepare the corresponding virtual object model to enhance the interactive experience of the audience. At the same time, the system will collect real-time audience interaction data. These data include various forms of audience participation in live interactive through mobile devices or computers, such as questions raised, voting choices, sent barrage messages, etc. By analyzing these interactive data, the system can understand the audience's interest points and focus of attention, and thus decide which virtual object models need to be called and displayed to the audience. According to the parsed live progress data and audience interaction data, the system will search and call the corresponding virtual object model in the augmented reality (AR) database. This database stores a plurality of virtual object models related to different scenarios and interactive types, such as 3D charts, virtual characters, prompt boxes, animation effects, etc. For example, in the Q&A segment, the system can call a virtual object model from the AR database to display the questions raised by the audience and their answers, or in the voting activity, a real-time updated voting result chart is displayed. The system will dynamically select the most suitable virtual object model according to the specific circumstances of the current live content and audience interaction.

[0054] Step 130, obtain device sensor data.

[0055] In step 130, the system will obtain data from various sensors in the user device, which may include but not limited to:

[0056] Gyroscope: provides the rotation angle information of the device, used to detect the tilt, rotation and orientation change of the device.

[0057] Accelerometer: measures the linear acceleration of the device, used to detect the moving direction and speed of the device.

[0058] Magnetometer: used to detect the orientation of the device relative to the geomagnetic field, to determine the orientation of the device.

[0059] Camera: used to capture image data of the environment where the device is currently located, providing visual reference for accurate positioning of virtual objects.

[0060] Light sensor: detects the light intensity of the environment, used to adjust the brightness and shadow effect of virtual objects.

[0061] These data will be collected in real time and pre-processed by the internal sensor data processing module of the system, such as data denoising, filtering and calibration. The system will synchronize the data from different sensors in time, ensuring that each data point corresponds to the same moment of the device state. After synchronization, the data is fused to obtain the complete attitude information (position, orientation and motion state) of the device and the environmental feature information (light, distance, etc.).

[0062] Step 140, updating the virtual object model according to the audience interaction data and the device sensor data to obtain a live virtual object.

[0063] In step 140, after receiving the audience's interaction data, the system will analyze these data to determine the content that needs to be updated in the virtual object model. For example, if the audience raises new questions or initiates specific interaction requests (such as clicking on an area to view detailed information) in the live broadcast, the system will mark these data as the basis for updating the virtual object. At the same time, using the data obtained from the device sensors, the system adjusts the display position, angle and effect of the virtual object model in real time. After combining the audience interaction data and sensor data, the system updates the content of the virtual object model. For example, if the audience participates in a vote in the live broadcast, the system will update the display content of the virtual object according to the voting results, such as charts or text information. In addition, the system may trigger specific animation effects or interaction feedback according to the audience's operation. For example, after the audience clicks on a virtual button, the system may display the corresponding expansion information or start a new interaction scene. The updated virtual object model will be rendered as a "live virtual object", which combines real-time audience interaction data and device sensor data.

[0064] Through the above implementation, the system can flexibly update the virtual object model in real time according to the audience's interaction data and device sensor data, ensuring that the audience can have a personalized and highly interactive augmented reality experience during the live broadcast.

[0065] Step 150, superimposing the live virtual object at the target position.

[0066] In step 150, the first step is to determine the target position of the live-stream virtual object in the user's view. Determining the target position depends on several factors: device sensor data, real-world environmental features, live-stream progress data, and audience interaction data. After determining the target position, the virtual object needs to be spatially mapped to ensure accurate display in the user's view. If the target position is a dynamically changing point in the user's mobile device view (e.g., the user's perspective changes as the device moves), the system will adjust the virtual object's mapping in real time to maintain precise alignment with the target position. The processed and rendered live-stream virtual object will then be overlaid on the user's device display. Viewers can see the integration of the virtual object with the real-world environment on their device screen and interact with it or adjust their viewing angle as needed.

[0067] Through the above steps, the system successfully and accurately overlays the live virtual object onto the target location, ensuring that the virtual object's display in the user's view is closely integrated with the real environment, thereby providing the audience with a natural and highly interactive augmented reality experience.

[0068] This invention provides an augmented reality-based live streaming interaction method, apparatus, device, and storage medium. In response to a user access command, it acquires live stream data, including live stream progress data. Based on the live stream progress data and audience interaction data, it retrieves a virtual object model from an augmented reality database, which stores the virtual object model. It also acquires device sensor data; updates the virtual object model based on the audience interaction data and the device sensor data to obtain a live virtual object; and then overlays the live virtual object onto a target location. By acquiring live stream data in response to user access commands and dynamically adjusting the virtual object based on live stream progress data and audience interaction data, viewers can obtain a personalized interactive experience during the live stream. This augmented reality interaction method greatly enhances viewer participation and interactivity, enabling viewers to not only be passive viewers but also influence the live stream content through interaction.

[0069] As a further optional embodiment, after the step of overlaying the live virtual object at the target location, the method further includes:

[0070] Obtain audience preference data;

[0071] Based on the audience preference data, a customized virtual object is generated, which includes language subtitles and virtual object annotations.

[0072] In this embodiment, personalized preference data is obtained from the viewer's device or account. This data includes the viewer's chosen language, viewing habits, subtitle preferences, interest in specific content (such as a preference for technical details or concise summaries), and previous interaction records. Viewer preference data can be actively set by the viewer during the live stream or automatically learned and recorded by the system. For example, the system can derive viewer preferences by analyzing past interaction behavior or questionnaire feedback. Based on the viewer preference data, the system generates customized virtual objects to meet the viewer's specific needs. For example, the system can generate customized virtual objects to meet the viewer's specific needs for specific content or details that the viewer is interested in. The system can generate virtual objects with annotations. These annotations may include textual explanations, background information, relevant links, or multimedia resources to provide viewers with more information while watching the live stream. For example, in educational live streams, viewers may see annotations overlaid next to the content, providing more background knowledge or examples. During the live stream, viewer preferences may change, and the system monitors these changes in real time and adjusts the customized virtual objects accordingly.

[0073] By introducing the acquisition of audience preference data and the generation of customized virtual objects, this invention further enhances the personalized experience of augmented reality interaction, enabling each viewer to obtain live content that highly matches their needs and interests, thereby enhancing the interactivity and user stickiness of the live stream.

[0074] As a further optional embodiment, the step of calling the virtual object model in the augmented reality database based on the live broadcast progress data and audience interaction data specifically includes:

[0075] Based on the live stream progress data and audience interaction data, determine the model type of the virtual object model;

[0076] Based on the model type, a matching process is performed in the augmented reality database to obtain the corresponding virtual object model.

[0077] In this embodiment, the type of virtual object model to be invoked is first determined based on live stream progress data and audience interaction data. This includes the theme of the live stream content, the currently playing content segment, and scene changes. For example, in an educational live stream, the appropriate virtual object model type, such as a chart, 3D model, or text label, might be determined based on the currently explained chapter or topic. It also includes audience interaction behavior during the live stream, such as asking questions, participating in polls, or selecting options. For example, if an audience shows particular interest in a certain issue, a more detailed virtual object model type, such as a model with explanatory text or multimedia, might be selected.

[0078] Based on the determined model type, the system matches it against an augmented reality database to find a corresponding virtual object model. This database pre-stores various types of virtual object models, including but not limited to 3D models, animations, charts, text labels, and video clips. Each model type has multiple instances, which can be selected based on the specific live stream content or interactive needs. The system performs precise matching based on the specific requirements of the model type, such as model size, display effect, and interactivity, ensuring that the selected virtual object model matches the current live stream content and audience interaction needs. For example, if the live stream progress data indicates that the current content is an explanation of a complex mechanical structure, the system will match a corresponding 3D mechanical model.

[0079] As a further optional embodiment, the step of updating the live virtual object based on the audience interaction data and the device sensor data specifically includes:

[0080] Based on the audience interaction data, determine the attribute information of the live virtual object;

[0081] Based on the sensor data from the device, determine the location information of the virtual object being broadcast live;

[0082] The live virtual object is updated based on its attribute and location information.

[0083] In this embodiment, audience interaction data is first analyzed to determine the attribute information of the live-streamed virtual object that needs to be updated. Audience interaction data may include the following:

[0084] Viewer click behavior: If a viewer clicks on a virtual object, the system needs to update the object's display or state, such as displaying additional information or triggering animation effects.

[0085] Audience choices: The preferences expressed by the audience through voting or selection functions may affect the attributes of virtual objects, such as adjusting the level of detail of the content or switching the view mode.

[0086] Interactive requests: Questions or requests from the audience may need to be marked on the virtual object or accompanied by relevant explanations, such as adding text annotations or links to the virtual object.

[0087] Attribute information: Attribute information can include the virtual object's color, size, animation status, displayed text, interactive features, etc. For example, viewers may request to change the virtual object's color or add new annotation information.

[0088] Additionally, the precise position of virtual objects in the user's view is determined using data from the device's sensors (such as gyroscopes, accelerometers, and cameras). The information provided by the device's sensor data includes:

[0089] Device attitude: By acquiring the rotation angle and motion state of the device through gyroscopes and accelerometers, the system can determine the relative position of the virtual object in three-dimensional space.

[0090] Viewpoint and direction: The environmental images captured by the camera and the viewing direction of the device can help the system determine the precise position and angle of virtual objects.

[0091] Environmental changes: The device's sensors can also monitor changes in lighting, spatial transformations, etc., to ensure that the display of virtual objects remains consistent in the real environment.

[0092] Based on the attribute and location information obtained from the above steps, the system updates the live virtual objects in real time to ensure that their display effect matches the audience's interactive needs and the actual situation of the devices.

[0093] As a further optional embodiment, the device sensor data also includes ambient light data, and after the step of determining the attribute information of the live virtual object based on the audience interaction data, the method further includes:

[0094] Based on the ambient lighting data, determine the brightness and contrast of the live-stream virtual object;

[0095] Based on the ambient light data, determine the brightness and contrast of the current device screen.

[0096] In this embodiment, ambient lighting data from the device's sensor data is used to measure the lighting conditions of the audience's environment. Ambient lighting data is typically provided by the device's light sensors and reflects the current light intensity and changes in lighting conditions.

[0097] Based on ambient lighting data, the brightness of virtual objects can be adjusted to ensure their clear visibility under various lighting conditions. If the ambient light intensity is high, the system will increase the brightness of the virtual objects; if the ambient light is low, the brightness will be reduced to maintain the visibility of the virtual objects.

[0098] The brightness of the device screen is adjusted based on ambient lighting data to match the ambient lighting conditions. This ensures that virtual objects displayed on the screen have appropriate brightness and maintain visual comfort under different ambient lighting conditions.

[0099] Understandably, contrast can also be adjusted. After determining the brightness and contrast of the virtual object, as well as the brightness and contrast of the device screen, the system will update the display settings for the virtual object. By incorporating ambient lighting data processing, this embodiment further enhances the visual adaptability and display quality of the virtual object, enabling it to maintain a clear and natural display under various lighting conditions. This method effectively improves the overall quality of the augmented reality experience and enhances the viewer's immersion in different environments.

[0100] As a further optional embodiment, the device sensor data includes rotation rate data, acceleration data, and static orientation data; the step of determining the position information of the live-streamed virtual object based on the device sensor data specifically includes:

[0101] Based on the rotation rate data, the acceleration data, and the static orientation data, the attitude of the current device is estimated to determine the motion state of the current device.

[0102] Based on the motion state, determine the virtual three-dimensional coordinates of the live-streaming virtual object;

[0103] The virtual three-dimensional coordinates are transformed to obtain the device's two-dimensional coordinates.

[0104] In this embodiment, attitude estimation is performed based on rotation rate data, acceleration data, and static orientation data to determine the device's motion state. This includes: calculating the device's rotation angles on each axis using rotation rate data; calculating the device's velocity changes and acceleration state using acceleration data; and determining the device's overall orientation by combining static orientation data. Based on the device's motion state, the system determines the virtual coordinates of the live-streaming virtual object in three-dimensional space. The virtual three-dimensional coordinates represent the virtual object's position in the device view and typically include X, Y, and Z coordinate values. The device attitude estimation results are applied to the virtual object's position calculation to ensure that the virtual object's coordinates are consistent with the device's actual motion state and orientation. For example, when the device moves forward, the virtual object's position is adjusted accordingly.

[0105] Next, the virtual 3D coordinates are converted into 2D coordinates on the device screen. During the coordinate transformation, perspective and the device's viewing angle are considered to calculate the actual position of the virtual object on the screen. The perspective transformation takes into account the device's viewing angle and distance, ensuring that the virtual object's display position on the screen matches its position in 3D space. The resulting 2D device coordinates are then applied to the virtual object's display settings to ensure that the virtual object's position on the device screen accurately reflects its position in 3D space.

[0106] Through these steps, the system can accurately calculate the 3D position of virtual objects based on device sensor data and precisely map it to 2D coordinates on the device screen. This method ensures that the position and display of virtual objects in the augmented reality environment are consistent with the viewer's actual perspective and the device's motion, enhancing the accuracy and immersion of the interactive experience.

[0107] As a further optional embodiment, the step of overlaying the live virtual object onto the target location specifically includes:

[0108] The live stream data is input into the environment understanding model to obtain the target location, which is used to overlay the live virtual object.

[0109] Determine the overlay order of the live-stream virtual objects;

[0110] Based on the superposition order and the device's two-dimensional coordinates, the live virtual object is superimposed onto the target position.

[0111] In this embodiment, data acquired from the live stream is input into an environment understanding model. The environment understanding model can be a deep learning-based image processing model used to analyze environmental information within the live stream content. The model processes the live stream data to identify and determine target locations. A target location refers to the specific location in the live stream frame where virtual objects are overlaid. For example, the model can identify a specific area or object in the live stream content as the target location for overlaying virtual objects. Next, the overlay order of the virtual objects is determined according to preset rules or the requirements of the live stream content. For example, the overlay priority can be arranged based on the importance, interactivity, or logical order of the virtual objects. Each virtual object is assigned an overlay priority to ensure that they are displayed in the correct order during the live stream. This helps optimize the viewer's experience and avoids visual conflicts between virtual objects. Following the determined overlay order, the virtual objects are overlaid one by one onto the target locations. Each virtual object is overlaid at the target location according to its priority order to ensure it is correctly displayed in the live stream content.

[0112] The following describes the augmented reality-based live interactive device provided by the present invention, such as... Figure 2 As shown, the augmented reality-based live interactive device described below and the augmented reality-based live interactive method described above can be referred to in correspondence.

[0113] An augmented reality-based live streaming interactive device includes:

[0114] Data acquisition module 210 is used to acquire live stream data in response to user access instructions, wherein the live stream data includes live stream progress data;

[0115] The model invocation module 220 is used to invoke a virtual object model from the augmented reality database based on the live broadcast progress data and the audience interaction data. The augmented reality database is used to store the virtual object model.

[0116] Sensor module 230 is used to acquire sensor data from the device;

[0117] The model update module 240 is used to update the virtual object model based on the audience interaction data and the device sensor data to obtain a live virtual object;

[0118] The model overlay module 250 is used to overlay the live virtual object onto the target location.

[0119] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an augmented reality-based live interactive method, which includes:

[0120] In response to a user access command, the system acquires live stream data, which includes live stream progress data.

[0121] Based on the live stream progress data and audience interaction data, a virtual object model is retrieved from the augmented reality database, which is used to store the virtual object model.

[0122] Acquire device sensor data;

[0123] The virtual object model is updated based on the audience interaction data and the device sensor data to obtain the live virtual object;

[0124] The live virtual object is overlaid on the target location.

[0125] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the augmented reality-based live interactive method provided by the above methods, the method comprising:

[0127] In response to a user access command, the system acquires live stream data, which includes live stream progress data.

[0128] Based on the live stream progress data and audience interaction data, a virtual object model is retrieved from the augmented reality database, which is used to store the virtual object model.

[0129] Acquire device sensor data;

[0130] The virtual object model is updated based on the audience interaction data and the device sensor data to obtain the live virtual object;

[0131] The live virtual object is overlaid on the target location.

[0132] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the augmented reality-based live interactive method provided by the methods described above, the method comprising:

[0133] In response to a user access command, the system acquires live stream data, which includes live stream progress data.

[0134] Based on the live stream progress data and audience interaction data, a virtual object model is retrieved from the augmented reality database, which is used to store the virtual object model.

[0135] Acquire device sensor data;

[0136] The virtual object model is updated based on the audience interaction data and the device sensor data to obtain the live virtual object;

[0137] The live virtual object is overlaid on the target location.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A live streaming interactive method based on augmented reality, characterized in that, include: In response to a user access command, the system acquires live stream data, which includes live stream progress data. Based on the live stream progress data and audience interaction data, virtual object models are retrieved from the augmented reality database, which is used to store the virtual object models; the virtual object models include 3D models, animations, charts, text labels, and video clips; Acquire device sensor data; The virtual object model is updated based on the audience interaction data and the device sensor data to obtain the live virtual object; The live virtual object is overlaid on the target location; The step of calling the virtual object model from the augmented reality database based on the live stream progress data and audience interaction data specifically includes: Based on the live stream progress data and audience interaction data, the model type of the virtual object model is determined; the live stream progress data includes the theme of the live stream content, the currently playing content segment, and scene changes; the audience interaction data includes the audience's interactive behavior during the live stream, including questions asked, votes participated in, or options selected. Based on the model type, a matching process is performed in the augmented reality database to obtain the corresponding virtual object model; The step of updating the live-stream virtual object based on the audience interaction data and the device sensor data specifically includes: Based on the audience interaction data, determine the attribute information of the live virtual object; Based on the sensor data from the device, determine the location information of the virtual object being broadcast live; The live virtual object is updated based on its attribute information and location information. The step of overlaying the live virtual object onto the target location specifically includes: The live stream data is input into the environment understanding model to obtain the target location, which is used to overlay the live virtual object. Determine the overlay order of the live-stream virtual objects; Based on the superposition order and the device's two-dimensional coordinates, the live virtual object is superimposed onto the target position.

2. The augmented reality-based live streaming interaction method according to claim 1, characterized in that, After the step of overlaying the live virtual object onto the target location, the method further includes: Obtain audience preference data; Based on the audience preference data, a customized virtual object is generated, which includes language subtitles and virtual object annotations.

3. The augmented reality-based live streaming interaction method according to claim 1, characterized in that, The device sensor data also includes ambient light data. Following the step of determining the attribute information of the live-stream virtual object based on the audience interaction data, the method further includes: Based on the ambient lighting data, determine the brightness and contrast of the live-stream virtual object; Based on the ambient light data, determine the brightness and contrast of the current device screen.

4. The augmented reality-based live streaming interaction method according to claim 1, characterized in that, The device sensor data includes rotation rate data, acceleration data, and static orientation data; the step of determining the position information of the live-streaming virtual object based on the device sensor data specifically includes: Based on the rotation rate data, the acceleration data, and the static orientation data, the attitude of the current device is estimated to determine the motion state of the current device. Based on the motion state, determine the virtual three-dimensional coordinates of the live-streaming virtual object; The virtual three-dimensional coordinates are transformed to obtain the device's two-dimensional coordinates.

5. A live interactive device based on augmented reality, characterized in that, include: The data acquisition module is used to acquire live stream data in response to user access commands, the live stream data including live stream progress data; The model invocation module is used to invoke virtual object models from the augmented reality database based on the live broadcast progress data and audience interaction data. The augmented reality database is used to store the virtual object models. The virtual object models include 3D models, animations, charts, text labels, and video clips. The sensor module is used to acquire sensor data from the device. The model update module is used to update the virtual object model based on the audience interaction data and the device sensor data to obtain the live virtual object; The model overlay module is used to overlay the live virtual object onto the target location; The step of calling the virtual object model from the augmented reality database based on the live stream progress data and audience interaction data specifically includes: Based on the live stream progress data and audience interaction data, the model type of the virtual object model is determined; the live stream progress data includes the theme of the live stream content, the currently playing content segment, and scene changes; the audience interaction data includes the audience's interactive behavior during the live stream, including questions asked, votes participated in, or options selected. Based on the model type, a matching process is performed in the augmented reality database to obtain the corresponding virtual object model; The step of updating the live-stream virtual object based on the audience interaction data and the device sensor data specifically includes: Based on the audience interaction data, determine the attribute information of the live virtual object; Based on the sensor data from the device, determine the location information of the virtual object being broadcast live; The live virtual object is updated based on its attribute information and location information. The step of overlaying the live virtual object onto the target location specifically includes: The live stream data is input into the environment understanding model to obtain the target location, which is used to overlay the live virtual object. Determine the overlay order of the live-stream virtual objects; Based on the superposition order and the device's two-dimensional coordinates, the live virtual object is superimposed onto the target position.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the augmented reality-based live interactive method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the augmented reality-based live interactive method as described in any one of claims 1 to 4.

Images